A method and system for student facial expression recognition in a classroom environment

Abstract

The present application relates to a method and system for student facial expression recognition in a classroom environment, and relates to the field of expression recognition technology. The method comprises: obtaining a student facial image, classifying the student facial image into a corresponding expression library; performing face recognition on the student facial image to obtain a face recognition image; calibrating feature points on the face recognition image, calculating corresponding feature values ​​based on the feature points, and obtaining a face image data set; training an SVM training model based on the face image data set to obtain an expression recognition model; obtaining a student facial image in a classroom environment, performing face recognition, feature point calibration, and feature value calculation on the student facial image in the classroom environment; and determining the expression category of the student facial image in the classroom environment based on the expression recognition model. In this way, the listening status and comprehension ability of the students in class can be analyzed, and the teaching level of the teacher can also be reflected from the quality of the students' classes.

Description

Technical Field

[0001] The present application relates to the field of expression recognition technology, and more specifically to a method and system for student facial expression recognition in a classroom environment. Background Art

[0002] With the rapid development of computer vision technology, facial expression recognition of students in classroom environment has gradually attracted attention. Expression recognition refers to separating specific expression states from a given static image or dynamic video sequence, so as to determine the listening state of the identified object. Student expression recognition in classroom environment is the most objective way for teachers and parents to understand the status and process of students in class. For teachers, it is impossible to take care of every student even the best teachers. Especially for those average students, it is more difficult to attract the attention of teachers, which is undoubtedly very detrimental to their development. Understanding the status of students in class through computers can enable teachers to understand the process of each student in class, so that teachers can impart knowledge to each student in a more targeted and efficient manner. For parents, the only way to understand their children's learning is from test scores and communication with teachers, which is very subjective and not specific to some extent.

[0003] Facial expression recognition can not only tell the degree of students' understanding of different knowledge, but also find out the degree of students' interest in different knowledge. If they encounter a subject that they are interested in, they will show an excited and positive facial state, such as a smile. If they find it difficult to understand or dislike a subject, they will show a negative expression, such as a frown. This will more objectively discover the strengths and weaknesses, interests and talents of students. However, the recognition of learning expressions is more difficult than that of basic expressions. The main reason is that the expressions produced during the learning process are mostly subtle and have little distinction. Summary of the invention

[0004] In order to facilitate the recognition of students' facial expressions in a classroom environment, this application proposes a method for recognizing students' facial expressions in a classroom environment, which solves the above problem by collecting students' facial information. It should be noted that the actions involving facial information collection are applied in legal scenarios such as classrooms, and the user's explicit authorization is obtained before actual execution.

[0005] In the first aspect, the present application provides a method for recognizing student facial expressions in a classroom environment, which adopts the following technical solutions:

[0006] A method for recognizing student facial expressions in a classroom environment, comprising:

[0007] Acquire student facial images, establish several expression libraries according to expression categories, classify the acquired student facial images by expression, and put them into corresponding expression libraries;

[0008] Performing face recognition on the student facial images in the expression library to obtain a face recognition image;

[0009] Calibrate feature points of the face recognition image, calculate corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in a corresponding expression library, and obtain a face image data set;

[0010] Establishing an SVM training model, and training the SVM training model according to the face image data set to obtain an expression recognition model;

[0011] Acquire facial images of students in a classroom environment, perform face recognition on the facial images of students in the classroom environment, calibrate feature points, and calculate feature values ​​based on the feature points;

[0012] Based on the expression recognition model, the expression category is determined based on the feature values ​​of the student's facial images in the classroom environment.

[0013] By adopting the above technical solution, firstly, multiple images are taken for each student, and expression classification, face recognition, and feature point calibration are performed to obtain a face image data set. The SVM training model is trained with the face image data set to obtain an expression recognition model. The facial images of students in the classroom environment are taken, and the images are analyzed. The student expressions are judged according to the expression recognition model, thereby analyzing the students' listening status and comprehension ability in class. In addition, the quality of students' classes can also reflect the teacher's teaching level.

[0014] Optionally, before performing the face recognition step on the student facial images, pixel normalization processing is performed on the student facial images in all expression libraries.

[0015] By adopting the above technical solution, the pixels of the student's facial image are normalized so that each pixel value of the student's facial image is between 0 and 1, which helps to facilitate subsequent face recognition.

[0016] Optionally, the MB-LBP feature algorithm is used to perform face recognition on the student's facial image, and the specific recognition process includes:

[0017] Divide the student's facial image into a plurality of sub-regions, calculate the average gray value of each sub-region, and obtain a gray value matrix;

[0018] Calculate the MB-LBP eigenvalue of the corresponding area according to the gray value matrix;

[0019] The MB-LBP feature values ​​are classified using a classifier to obtain a face recognition image.

[0020] By adopting the above technical solution, MB-LBP is a rectangular area encoded by a local binary pattern operator, and the correlation between pixels is reflected by regionalizing the neighborhood pixels. MB-LBP features can capture large-scale structures, which may be the main features in the image structure, and can perform face recognition through a smaller feature set, and obtain a face recognition image through the feature + classifier mode.

[0021] Optionally, the classifier is a cascade classifier, including several cascaded strong classifiers, each level of strong classifier is composed of several weak classifiers; the weak classifier is a multi-branch regression tree.

[0022] By adopting the above technical solution, a cascade classifier is used to classify MB-LBP features. Each layer of the cascade classifier is a strong classifier. These strong classifiers form a multi-layer classification structure, which allows each layer to pass almost all face samples and reject a large number of non-face samples, thereby spending more time in areas where faces may exist and reducing the overall calculation time.

[0023] Optionally, feature point calibration is performed on the face recognition image, and corresponding feature values ​​are calculated according to the calibrated feature points, specifically:

[0024] The positions of eyebrows, eyes, nose, mouth and chin in the face recognition image are calibrated using a DLIB-based machine learning method; corresponding feature values ​​are calculated according to the positions and angles between the eyebrows, eyes, nose, mouth and chin.

[0025] By adopting the above technical solutions, DLIB has good support for facial feature extraction, and has a trained facial feature point extraction model for developers to use. The positions of eyebrows, eyes, nose, mouth and chin are calibrated according to actual needs, and the required feature values ​​are calculated based on the positions and angles between these parts, so as to compare facial features later.

[0026] Optionally, face recognition is performed on the facial images of students in the classroom environment, facial feature points are calibrated, and facial feature values ​​are calculated based on the facial feature points, specifically:

[0027] The MB-LBP feature algorithm is used to perform face recognition on student facial images in a classroom environment to obtain a face recognition image in the classroom environment; the positions of eyebrows, eyes, nose, mouth and chin in the face recognition image in the classroom environment are calibrated; and corresponding feature values ​​are calculated according to the positions and angles between the eyebrows, eyes, nose, mouth and chin.

[0028] By adopting the above technical solution, when analyzing students' expressions in a classroom environment, the MB-LBP feature algorithm is also used for face recognition, and the feature points and feature values ​​corresponding to the face image data set are calibrated, so as to facilitate the determination of students' status according to the established expression recognition model.

[0029] In the second aspect, the present application provides a student facial expression recognition system for use in a classroom environment, which adopts the following technical solutions:

[0030] A student facial expression recognition system for use in a classroom environment, comprising

[0031] An image acquisition module, used to capture facial images of students;

[0032] The expression library module is used to establish a number of expression libraries according to expression categories, classify the student facial images collected by the image collection module, and put them into the corresponding expression library;

[0033] A face detection and extraction module is used to perform face recognition on the student facial images in the expression library module to obtain a face recognition image;

[0034] A feature point calibration module is used to calibrate the feature points of the face recognition image, calculate the corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in the corresponding expression library, and obtain a face image data set;

[0035] A model training module is used to establish an SVM training model, and train the SVM training model according to the face image data set to obtain an expression recognition model;

[0036] The expression determination module is used to determine the expression category of the feature values ​​of the student's facial images in the classroom environment based on the expression recognition model.

[0037] By adopting the above technical solution, multiple images of each student are taken through the image acquisition module, and the taken images are classified into expression categories and stored in the corresponding expression library in the expression library module. Face recognition and feature point calibration are performed through the face detection and extraction module and the feature point calibration module to obtain a face image data set. The SVM training model is trained with the face image data set to obtain an expression recognition model. The facial images of students in the classroom environment are taken, the images are analyzed, and the expressions of the students are judged according to the expression recognition model, thereby analyzing the students' listening status and comprehension ability in class.

[0038] Optionally, the system further includes: an image preprocessing module, used for performing pixel normalization processing on the student facial images in all expression libraries before performing face recognition on the student facial images.

[0039] By adopting the above technical solution, the pixels of the student's facial image are normalized through the image preprocessing module, so that each pixel value of the student's facial image is between 0 and 1, which helps to facilitate subsequent face recognition.

[0040] In a third aspect, the present application provides a device for student facial expression recognition in a classroom environment, which adopts the following technical solution:

[0041] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program:

[0042] Acquire student facial images, establish several expression libraries according to expression categories, classify the acquired student facial images by expression, and put them into corresponding expression libraries;

[0043] Performing face recognition on the student facial images in the expression library to obtain a face recognition image;

[0044] Calibrate feature points of the face recognition image, calculate corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in a corresponding expression library, and obtain a face image data set;

[0045] Establishing an SVM training model, and training the SVM training model according to the face image data set to obtain an expression recognition model;

[0046] Acquire facial images of students in a classroom environment, perform face recognition on the facial images of students in the classroom environment, calibrate feature points, and calculate feature values ​​based on the feature points;

[0047] The step of determining the expression category of the feature values ​​of the facial images of students in a classroom environment based on the expression recognition model.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0049] A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor when it is run:

[0050] Acquire student facial images, establish several expression libraries according to expression categories, classify the acquired student facial images by expression, and put them into corresponding expression libraries;

[0051] Performing face recognition on the student facial images in the expression library to obtain a face recognition image;

[0052] Calibrate feature points of the face recognition image, calculate corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in a corresponding expression library, and obtain a face image data set;

[0053] Establishing an SVM training model, and training the SVM training model according to the face image data set to obtain an expression recognition model;

[0054] Acquire facial images of students in a classroom environment, perform face recognition on the facial images of students in the classroom environment, calibrate feature points, and calculate feature values ​​based on the feature points;

[0055] The step of determining the expression category of the feature values ​​of the facial images of students in a classroom environment based on the expression recognition model.

[0056] In summary, this application includes the following beneficial technical effects:

[0057] 1. By classifying students' facial expressions, performing face recognition, and calibrating feature points, and using the established expression recognition model to judge students' expressions, we can analyze students' listening status and comprehension ability in class.

[0058] 2. Using the MB-LBP feature algorithm to perform face recognition on student facial images can capture more information about the image structure. In addition, face recognition can be performed using a smaller feature set, which reduces training time.

[0059] 3. Since DLIB has a trained facial feature point extraction model for developers to use, it is more convenient to calibrate facial feature points using the DLIB method. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of a method for recognizing student facial expressions in a classroom environment according to one embodiment of the present application.

[0061] Figure 2 This is a specific flow chart of a method for recognizing student facial expressions in a classroom environment according to one embodiment of the present application.

[0062] Figure 3 This is a schematic diagram of 23 key point calibration based on DLIB in one embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0064] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make modifications to the present embodiment without any creative contribution as needed. However, as long as it is within the scope of the claims of the present application, it shall be protected by the patent law.

[0065] like Figure 1 and Figure 2 As shown, an embodiment of the present application discloses a method for recognizing student facial expressions in a classroom environment, comprising the following steps.

[0066] Step 1: obtain student facial images, establish several expression libraries according to expression categories, classify the expressions of the obtained student facial images, and put them into the corresponding expression libraries.

[0067] Collect multiple facial images of each student, where multiple images are taken for each expression of each student. Divide the required expressions into several expression categories, establish an expression library for each expression category, and store all the collected student facial images in the corresponding expression library according to the expression category, and label them in order.

[0068] In this embodiment, students' classroom expressions can be roughly divided into three types: happy, normal and confused, and three corresponding expression libraries are established. In a classroom environment, a micro camera needs to be set up in front of each student to collect video images. Each expression library of these three types of expressions requires at least 100 students of all ages to be entered, and at least 50 sets of similar expression data for each person. Obviously, the larger the data sample size of the expression library, the more accurate the analysis result.

[0069] Step 2, performing face recognition on the student facial images in the expression library to obtain a face recognition image, and saving it as a picture through OpenCV.

[0070] Step 21, before performing face recognition, first perform pixel normalization processing on the student facial images in all expression libraries.

[0071] Normalization is to make subsequent data processing more convenient, and limit the data to be processed to a certain range after processing. The pixels of the student's facial image are normalized so that each pixel value of the student's facial image is between 0 and 1. When the pixels of the image are in the range of 0-1, since they are still between 0-255, the image is still valid and can be viewed normally.

[0072] Step 22, performing face recognition on the student's facial image.

[0073] In this embodiment, the MB-LBP feature algorithm is used to perform face recognition on the student's facial image. MB-LBP (Multi-block Local Binary Patterns) encodes the intensity of a rectangular area through a local binary pattern operator, and the resulting binary pattern can describe different local structures of the image. The image is divided into small blocks (Blocks), and each small block is further divided into rectangular areas (similar to cells in HOG). The average grayscale value in the rectangular area is used as the grayscale value of the current rectangular area, which is compared with the grayscale of the surrounding rectangular areas to form an LBP feature. The generated feature is called MB-LBP.

[0074] The specific recognition process of the MB-LBP feature algorithm includes:

[0075] 1) First, the student's facial image is divided into several sub-regions of size 3s×3t. Any domain window of a sub-region of size 3s×3t is divided into 9 sub-windows of size s×t. The average grayscale value of each sub-region is calculated to obtain a 3×3 grayscale value matrix.

[0076] 2) calculating the MB-LBP eigenvalues ​​of the corresponding area according to the gray value matrix;

[0077] The gray value matrix is ​​calculated to obtain the MB-LBP eigenvalue of the region. The calculation method is as follows:

[0078] Remember any point g c The 3×3 window whose field pixels are defined in clockwise direction is g i (0≤ i ≤7), g i The size of represents the gray value of the pixel. Use the central pixel gc to threshold the neighboring pixels g i (0≤ i ≤7), and a binary string is obtained, which becomes the MB-LBP code of point gc. The calculation formula is as follows:

[0079] (1)

[0080] Will S ( g i , g c ) is weighted and summed, and the resulting value is called the center pixel g c MB-LBP eigenvalues.

[0081] (2)

[0082] The MB-LBP value is between 0 and 255, that is, an image contains a total of 256 MB-LBP modes.

[0083] The generated binary patterns are used as the feature values ​​of MB-LBP features. These binary patterns can show image edges, lines, points, flat areas, corners, etc. at different scales and positions.

[0084] 3) Using a classifier to classify the MB-LBP feature values ​​to obtain a face recognition image.

[0085] The MB-LBP feature values ​​obtained above are classified according to a classifier, wherein the classifier is a cascade classifier, including a plurality of cascaded strong classifiers, and each level of the strong classifier is composed of a plurality of weak classifiers.

[0086] The classifier in this embodiment can be an AdaBoost classifier, and the weak classifier can adopt a multi-branch regression tree. AdaBoost is a strong classifier obtained by training a group of weak classifiers to obtain a superposition of these weak classifiers. The AdaBoost algorithm is an adaptive Boosting algorithm. The adaptiveness lies in that the samples that were misclassified by the previous classifier will be used to train the next classifier. This method is an iterative algorithm, and a new weak classifier is added in each round until a predetermined sufficiently small error rate is reached. Each training sample is assigned a weight, indicating the probability of it being selected into the training set by a certain classifier. If a certain sample point has been accurately classified, then in the construction of the next training set, its probability of being selected is reduced; on the contrary, if a certain sample point is not accurately classified, then its weight is improved.

[0087] The principle of the Adaboost algorithm is to reasonably combine multiple weak classifiers to make them a strong classifier. The iterative idea is: only one weak classifier is trained in each iteration, and the trained weak classifier will be used in the next iteration. That is to say, in the Nth iteration, there are a total of N weak classifiers, of which N-1 are previously trained, and their various parameters will no longer change. This time, the Nth classifier is trained. The relationship between the weak classifiers is that the Nth weak classifier is more likely to classify the data that the first N-1 weak classifiers did not classify correctly, and the final classification output depends on the comprehensive effect of these N classifiers. During the training process, each training sample (i.e., MB-LBP feature value) is assigned a weight, indicating the probability of it being selected into the training set by a weak classifier. If a sample has been accurately classified, then the probability of it being selected in the construction of the next training set is reduced; on the contrary, if a sample is not correctly classified, then its weight is increased.

[0088] Step 3, calibrate the feature points of the face recognition image, calculate the corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in the corresponding expression library, and obtain the face image data set.

[0089] Step 31, calibrating feature points of the face recognition image.

[0090] Student facial feature extraction is the most important task in facial recognition extracted by convolutional neural methods and is the difference between the pros and cons of deep learning. Deep learning neural networks are the best solution for object detection, pattern recognition, and face recognition.

[0091] In this embodiment, the positions of eyebrows, eyes, nose, mouth and chin in the face recognition image are calibrated using a DLIB-based machine learning method. DLIB is a powerful convolutional neural network and one of the widely used deep neural networks because it is well adapted to the translation invariance of images. Convolutional neural networks use three main basic ideas for facial feature extraction: local receptive field, shared weights and sets.

[0092] a. Local Acceptable Field

[0093] In a convolutional neural network, the layers of hidden neurons have small connections, unlike other neural networks where local regions of input image pixels have connections for each input pixel. To be more precise, each neuron in the first hidden layer will be connected to a small region of the input neurons.

[0094] This area in the input image is called the local receptive field of the hidden neuron, which is a small window on the input pixel. As each connection learns, the hidden neuron learns too. The receptive field slides by each pixel to connect with each hidden layer neuron.

[0095] b. Shared weights and biases

[0096] The local receptive fields connected to the hidden layers with the same step length will share the same weights and biases throughout the hidden layer. For the jth, kth hidden neuron, the output is as follows:

[0097] (3)

[0098] in σ is the neural activation function or sigmoid function, b is the shared bias, w 1,m is the shared weight, a j+1,k+m is j+1 , k+m Input at.

[0099] The neurons in the first hidden layer detect exactly the same features but at different locations in the input image. Therefore, the map from the input image to the hidden layer is called a feature map and the bias used for the feature map is called a shared bias. The shared weights and biases are called kernels or filters. There are many feature maps for face recognition, but kernels effectively reduce the number of parameters.

[0100] c. Collection layer

[0101] The convolutional layer is followed by a pooling layer, which is used to simplify the information in the output of the feature map from the convolutional layer into a condensed feature map. Many pooling techniques can be used in convolutional neural networks, but in this system, maximum pooling is used for the facial recognition system. The output feature map returns the output with the maximum activation in the 2 x 2 input regions.

[0102] In this embodiment, the 23 feature points at the eyebrows, eyes, nose, mouth and chin in each face recognition image can be calibrated. Figure 3 As shown, the 23 feature points specifically include:

[0103] brow:

[0104] LBl: left end of left eyebrow,

[0105] LBr: left eyebrow right end,

[0106] RBl: left end of right eyebrow,

[0107] RBr: right end of right eyebrow,

[0108] Bc: between the eyebrows.

[0109] Eye:

[0110] LE1: Left eye left end,

[0111] LEr: Left eye right end,

[0112] LEu: upper left eye,

[0113] LEd: lower left eye,

[0114] LEc: left eye midpoint (midpoint between LEl and LEr),

[0115] REl: left end of right eye,

[0116] REr: Right end of right eye,

[0117] REu: upper right eye,

[0118] REd: lower end of right eye,

[0119] REc: midpoint of right eye (midpoint between Rel and REr).

[0120] Nose:

[0121] Nl: left end of left nostril,

[0122] Nr: right end of right nostril,

[0123] Nc: The center of the lower end of the nose.

[0124] Mouth:

[0125] Ml: left corner of mouth,

[0126] Mr: Right corner of mouth,

[0127] Mu: upper lip,

[0128] Md: lower end of lower lip.

[0129] Chin:

[0130] C: Lower part of the chin.

[0131] Step 32: Calculate corresponding feature values ​​according to the calibrated feature points.

[0132] Facial expressions are the result of facial muscle activity, so the classification of expressions cannot be simply determined by the position change of key points. When making facial expressions, the tension and relaxation of muscles will cause obvious wrinkles in certain key facial positions. After video acquisition, these key parts will appear light and dark on the image. After grayscale processing by computer vision, the image intensity of the area can be obtained. In other words, the position and angle between feature points and the image intensity of key facial positions are used as key data to determine the expression state of the person being tested.

[0133] In this embodiment, 24 feature values, such as angles, distances, and image intensity between feature points, can be calculated based on the 23 feature points calibrated in each face, and the results are stored in a database. The 24 feature values ​​are specifically shown in Table 1.

[0134] Table 1 Calculated 24 eigenvalues

[0135] Step 33, through feature point calibration and feature value calculation, the same type of expressions can be quantified and subjective expressions can be converted into objective quantitative data. The 23 feature points corresponding to each face in the expression library and the 24 feature values ​​calculated for each face are saved as expression data in several TXT documents respectively, and the TXT documents containing expression data are stored in the corresponding expression library according to the expression type. The expression library finally obtained is the face image data set.

[0136] Step 4, establishing an SVM training model, training the SVM training model according to the face image data set to obtain an expression recognition model.

[0137] The SVM method is to apply statistical learning theory to the hyperplane that separates the two-level classification problem. The core idea of ​​SVM is to adjust the discriminant function to maximize the separability information of the boundary cases. In the field of machine learning, it is a supervised learning model that is usually used for pattern recognition, classification, and regression analysis, and is suitable for most classification needs. Given a set of situations belonging to one of the two categories of student expressions, training involves searching for a hyperplane that leaves the maximum number of situations of the same class on the same side, while maximizing the distance between the two classes and the hyperplane.

[0138] According to the expression category, the expression data is extracted from the corresponding expression library (the expression data in each face image includes 23 feature points and corresponding 24 feature values), and the extracted expression data is input into the SVM training model. The model uses SVM (support vector machine) to classify and train several feature value data, and saves the results in an XML file to obtain an expression recognition model. The macro expression after manual classification is input into the computer to classify the expression data.

[0139] Step 5, obtaining facial images of students in a classroom environment, performing face recognition on the facial images of students in the classroom environment, calibrating facial feature points, and calculating facial feature values ​​based on the facial feature points.

[0140] In this embodiment, the MB-LBP feature algorithm can also be used to perform face recognition on the student facial image in the classroom environment to obtain a face recognition image in the classroom environment. The 23 facial feature points in the face recognition image in the classroom environment are calibrated, and the corresponding 24 facial feature values ​​are calculated.

[0141] Step 6: Based on the expression recognition model, perform expression determination on the facial feature values.

[0142] The 23 facial feature points and 24 facial feature values ​​of the students in the classroom environment calibrated in step 5 are matched with the trained expression recognition model to automatically determine the expression category.

[0143] After facial expressions are determined, the classroom facial expression data of each student can be integrated and analyzed to give a statistical chart of facial expression distribution time. Teachers can understand the students' classroom participation level and the students' knowledge acceptance level in this class by checking the distribution ratio of their facial expressions. For example, if the confused expression lasts too long, it means that the students are struggling to absorb knowledge in class, and if the normal expression lasts for a long time, it means that they are not in the state of listening to the class. Testing the students' class status can show the students' learning situation to parents relatively objectively, and they can understand how long the students are focused in this class and how long they are distracted, so that parents can better supervise and urge them.

[0144] At the same time, through the data analysis results of students, we can calculate the statistical distribution of students' participation in class and give an evaluation analysis of a teacher's teaching situation. We can also analyze the students' hobbies and personalities based on the analysis results of each student. Based on all information data such as student information, expression judgment results, time data, etc., we can perform various data analyses and output the corresponding analysis results.

[0145] The present application also discloses a student facial expression recognition system for use in a classroom environment, comprising:

[0146] The image acquisition module is used to capture the facial images of students; the image acquisition module can be a miniature camera or a network hard disk recorder, which is assumed to be in front of each student.

[0147] The expression library module is used to establish a number of expression libraries according to expression categories, classify the student facial images collected by the image collection module, and put them into corresponding expression libraries.

[0148] The face detection and extraction module is used to perform face recognition on the student facial images in the expression library module to obtain a face recognition image.

[0149] The feature point calibration module is used to calibrate the feature points of the face recognition image, calculate the corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in the corresponding expression library, and obtain the face image data set.

[0150] The model training module is used to establish an SVM training model, and train the SVM training model according to the face image data set to obtain an expression recognition model.

[0151] The expression determination module is used to determine the expression category of the feature values ​​of the student's facial images in the classroom environment based on the expression recognition model.

[0152] In this embodiment, the system may further include an image preprocessing module for performing pixel normalization processing on the student facial images in all expression libraries before performing face recognition on the student facial images.

[0153] The present application also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0154] Acquire student facial images, establish several expression libraries according to expression categories, classify the acquired student facial images by expression, and put them into corresponding expression libraries;

[0155] Performing face recognition on the student facial images in the expression library to obtain a face recognition image;

[0156] Calibrate feature points of the face recognition image, calculate corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in a corresponding expression library, and obtain a face image data set;

[0157] Establishing an SVM training model, and training the SVM training model according to the face image data set to obtain an expression recognition model;

[0158] Acquire facial images of students in a classroom environment, perform face recognition on the facial images of students in the classroom environment, calibrate feature points, and calculate feature values ​​based on the feature points;

[0159] The step of determining the expression category of the feature values ​​of the facial images of students in a classroom environment based on the expression recognition model.

[0160] The present application also discloses a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is executed when a processor is executed:

[0161] Acquire student facial images, establish several expression libraries according to expression categories, classify the acquired student facial images by expression, and put them into corresponding expression libraries;

[0162] Performing face recognition on the student facial images in the expression library to obtain a face recognition image;

[0163] Calibrate feature points of the face recognition image, calculate corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in a corresponding expression library, and obtain a face image data set;

[0164] Establishing an SVM training model, and training the SVM training model according to the face image data set to obtain an expression recognition model;

[0165] Acquire facial images of students in a classroom environment, perform face recognition on the facial images of students in the classroom environment, calibrate feature points, and calculate feature values ​​based on the feature points;

[0166] The step of determining the expression category of the feature values ​​of the facial images of students in a classroom environment based on the expression recognition model.

[0167] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0168] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. A method for recognizing student facial expressions in a classroom environment, characterized in that: include: Acquire student facial images, establish several expression libraries according to expression categories, classify the acquired student facial images by expression, and put them into corresponding expression libraries; Performing face recognition on the student facial images in the expression library to obtain a face recognition image; The step of performing face recognition on the student facial images in the expression library to obtain a face recognition image includes: The MB-LBP feature algorithm is used to perform face recognition on the student's facial image. The specific recognition process includes: Divide the student's facial image into a plurality of sub-regions, calculate the average gray value of each sub-region, and obtain a gray value matrix; The student's facial image is divided into a plurality of sub-regions, and the average gray value of each sub-region is calculated to obtain a gray value matrix, including: The student's facial image is divided into several sub-regions of size 3s×3t. Any domain window of a sub-region of size 3s×3t is divided into 9 sub-windows of size s×t. The average gray value of each sub-region is calculated to obtain a 3×3 gray value matrix. Calculate the MB-LBP eigenvalue of the corresponding area according to the gray value matrix; The calculating the MB-LBP eigenvalue of the corresponding area according to the gray value matrix includes: Remember any point g c The 3×3 window whose field pixels are defined in clockwise direction is g i (0≤ i ≤7), g i The size represents the gray value of the pixel, and the neighborhood pixels are thresholded using the center pixel gc. g i (0≤ i ≤7), and a binary string is obtained, which becomes the MB-LBP code of point gc. The calculation formula is as follows: （1） ； Will S ( g i , g c ) is weighted and summed, and the resulting value is called the center pixel g c MB-LBP eigenvalues; （2）； Using a classifier to classify the MB-LBP feature values ​​to obtain a face recognition image; The classifier is a cascade classifier, including several cascaded strong classifiers, each level of strong classifier is composed of several weak classifiers; the weak classifier is a multi-branch regression tree; The classifier includes an AdaBoost classifier, wherein the AdaBoost classifier is obtained by training a group of weak classifiers to obtain a strong classifier obtained by superimposing these weak classifiers; Calibrate feature points of the face recognition image, calculate corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in a corresponding expression library, and obtain a face image data set; The feature point calibration of the face recognition image and the calculation of the corresponding feature value according to the calibrated feature points are specifically as follows: The positions of eyebrows, eyes, nose, mouth and chin in the face recognition image are calibrated by using a DLIB-based machine learning method; corresponding feature values ​​are calculated according to the positions and angles between the eyebrows, eyes, nose, mouth and chin; Establishing an SVM training model, and training the SVM training model according to the face image data set to obtain an expression recognition model; Acquire facial images of students in a classroom environment, perform face recognition on the facial images of students in the classroom environment, calibrate feature points, and calculate feature values ​​based on the feature points; Based on the expression recognition model, the expression category is determined based on the feature values ​​of the student's facial images in the classroom environment.

2. The method for recognizing student facial expressions in a classroom environment according to claim 1, characterized in that: Before performing the face recognition step on the student facial images, pixel normalization processing is performed on the student facial images in all expression libraries.

3. The method for recognizing student facial expressions in a classroom environment according to claim 1, characterized in that: Perform face recognition on the student's facial image in the classroom environment, calibrate facial feature points, and calculate facial feature values ​​based on the facial feature points, specifically: The MB-LBP feature algorithm is used to perform face recognition on student facial images in a classroom environment to obtain a face recognition image in the classroom environment; the positions of eyebrows, eyes, nose, mouth and chin in the face recognition image in the classroom environment are calibrated; and corresponding feature values ​​are calculated according to the positions and angles between the eyebrows, eyes, nose, mouth and chin.

4. A student facial expression recognition system for use in a classroom environment, characterized in that: include: An image acquisition module, used to capture facial images of students; The expression library module is used to establish a number of expression libraries according to expression categories, classify the student facial images collected by the image collection module, and put them into the corresponding expression library; A face detection and extraction module is used to perform face recognition on the student facial images in the expression library module to obtain a face recognition image; The step of performing face recognition on the student facial images in the expression library to obtain a face recognition image includes: The MB-LBP feature algorithm is used to perform face recognition on the student's facial image. The specific recognition process includes: Divide the student's facial image into a plurality of sub-regions, calculate the average gray value of each sub-region, and obtain a gray value matrix; The student's facial image is divided into a plurality of sub-regions, and the average gray value of each sub-region is calculated to obtain a gray value matrix, including: The student's facial image is divided into several sub-regions of size 3s×3t. Any domain window of a sub-region of size 3s×3t is divided into 9 sub-windows of size s×t. The average gray value of each sub-region is calculated to obtain a 3×3 gray value matrix; Calculate the MB-LBP eigenvalue of the corresponding area according to the gray value matrix; The calculating the MB-LBP eigenvalue of the corresponding area according to the gray value matrix includes: Remember any point g c The 3×3 window whose field pixels are defined in clockwise direction is g i (0≤ i ≤7), g i The size of represents the gray value of the pixel. Use the central pixel gc to threshold the neighboring pixels g i (0≤ i ≤7), and a binary string is obtained, which becomes the MB-LBP code of point gc. The calculation formula is as follows: （1） ； Will S ( g i , g c ) is weighted and summed, and the resulting value is called the center pixel g c MB-LBP eigenvalues; （2）； Using a classifier to classify the MB-LBP feature values ​​to obtain a face recognition image; The classifier is a cascade classifier, including several cascaded strong classifiers, each level of strong classifier is composed of several weak classifiers; the weak classifier is a multi-branch regression tree; The classifier includes an AdaBoost classifier, wherein the AdaBoost classifier is obtained by training a group of weak classifiers to obtain a strong classifier obtained by superimposing these weak classifiers; A feature point calibration module is used to calibrate the feature points of the face recognition image, calculate the corresponding feature values ​​according to the calibrated feature points, store the feature points and feature values ​​in the corresponding expression library, and obtain a face image data set; The feature point calibration of the face recognition image and the calculation of the corresponding feature value according to the calibrated feature points are specifically as follows: The positions of eyebrows, eyes, nose, mouth and chin in the face recognition image are calibrated by using a DLIB-based machine learning method; corresponding feature values ​​are calculated according to the positions and angles between the eyebrows, eyes, nose, mouth and chin; A model training module is used to establish an SVM training model, and train the SVM training model according to the face image data set to obtain an expression recognition model; Acquire facial images of students in a classroom environment, perform face recognition on the facial images of students in the classroom environment, calibrate feature points, and calculate feature values ​​based on the feature points; The expression determination module is used to determine the expression category of the feature values ​​of the student's facial images in the classroom environment based on the expression recognition model.

5. The student facial expression recognition system for use in a classroom environment according to claim 4, characterized in that: The system also includes: an image preprocessing module, which is used to perform pixel normalization processing on the student facial images in all expression libraries before performing face recognition on the student facial images.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 3 are executed.

Images