A student facial expression recognition method and system

Abstract

The application provides a student facial expression recognition method and system, comprising: acquiring an image containing all student faces to be recognized; using a face recognition algorithm to recognize the faces of all students to be recognized in the image, and cutting to obtain a face picture of each student to be recognized; inputting the face picture of each student to be recognized into a trained convolutional neural network to obtain the probability of each facial expression category corresponding to each student to be recognized, and taking the facial expression category corresponding to the maximum probability as the facial expression recognition result of the student to be recognized; wherein the convolutional neural network comprises a plurality of convolutional layers connected in sequence, a fully connected layer and a softmax layer, and each convolutional layer is composed of a BN layer, an activation layer and an average pooling layer. The accuracy of facial expression recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a student facial expression recognition method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] At present, some students change from offline learning to online learning, and in the process of student classroom learning, the problem to be solved is how to evaluate the psychological state of each student in real time and accurately, so that the teacher can better master the learning state of the student and timely adjust the teaching strategy to improve the teaching efficiency.

[0004] For students, when students feel uncomfortable, they may have facial expressions such as frowning and tightly closed lips, and timely identification of these information of students helps to understand the psychological state of students in time.

[0005] However, the student facial expression recognition has the following problems: real-time collection of facial expressions of each student in the classroom; students are dynamic, if a separate camera is directly installed in front of each student, not only will the students feel pressure and nervousness, but also will affect the collection of facial expressions of the students, and further affect the prediction result and interfere with the judgment made by the teacher teaching; low facial expression recognition accuracy. SUMMARY

[0006] In order to solve the technical problems existing in the background art, the present application provides a student facial expression recognition method and system, which directly divides into several facial expression categories after a full connection layer, thereby improving the accuracy of facial expression recognition.

[0007] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0008] The first aspect of the present application provides a student facial expression recognition method, which comprises:

[0009] obtaining an image containing all the faces of students to be identified;

[0010] using a face recognition algorithm to identify the faces of all the students to be identified in the image, and cutting to obtain a face picture of each student to be identified;

[0011] inputting the face picture of each student to be identified into a trained convolutional neural network to obtain the probability of each facial expression category corresponding to each student to be identified, and taking the facial expression category corresponding to the maximum probability as the facial expression recognition result of the student to be identified;

[0012] The convolutional neural network comprises a plurality of convolutional layers, a full connection layer and a softmax layer connected in sequence, each convolutional layer is composed of a BN layer, an activation layer and an average pooling layer.

[0013] Further, the face picture clipping method is:

[0014] The position information of the face of each student to be identified in the image is obtained, and the face picture of each student to be identified is clipped based on the position information of the face of each student to be identified.

[0015] Further, the training process of the convolutional neural network is:

[0016] A training set is generated.

[0017] The training set is input into the convolutional neural network, and iterative training is performed until the loss function converges, so that the trained convolutional neural network is obtained.

[0018] Further, the loss function adopts a cross-entropy loss function.

[0019] Further, the specific method for generating the training set is: obtaining an original training set, performing data enhancement on each face picture in the original training set, combining the face pictures after data enhancement with the original training set, and obtaining the training set.

[0020] Further, the specific method for generating the training set is that the data enhancement method is flipping, rotation or cutting.

[0021] The second aspect of the present application provides a student facial expression recognition system, which comprises:

[0022] An image acquisition module configured to acquire an image containing the faces of all students to be identified;

[0023] A face recognition module configured to identify the faces of all students to be identified in the image by using a face recognition algorithm, and to clip the face pictures of each student to be identified;

[0024] A facial expression recognition module configured to input the face picture of each student to be identified into the trained convolutional neural network to obtain the probability of each facial expression category corresponding to each student to be identified, and to take the facial expression category corresponding to the maximum probability as the facial expression recognition result of the student to be identified.

[0025] The convolutional neural network comprises a plurality of convolutional layers, a full connection layer and a softmax layer connected in sequence, each convolutional layer is composed of a BN layer, an activation layer and an average pooling layer.

[0026] Further, a network training module is further included, which is configured to:

[0027] generating a training set;

[0028] inputting the training set into the convolutional neural network, performing iterative training until the loss function converges, and obtaining the trained convolutional neural network.

[0029] A third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the student facial expression recognition method.

[0030] A fourth aspect of the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the student facial expression recognition method when executing the program.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The present application provides a student facial expression recognition method, which integrates facial expression feature extraction and expression classification into an end-to-end network, uses VGG19 to complete expression recognition and classification, removes multiple fully connected layers in the traditional VGG19, and directly divides into 7 categories after a fully connected layer to identify, thereby improving the accuracy of facial expression recognition. BRIEF DESCRIPTION OF DRAWINGS

[0033] The drawings constituting a part of the specification of the present application are used to provide further understanding of the present application, and the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitations on the present application.

[0034] Figure 1 is a student facial expression recognition method flowchart of the first embodiment of the present application;

[0035] Figure 2 is a training flowchart of the convolutional neural network of the first embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with the drawings and embodiments.

[0037] It should be pointed out that the following detailed description is all exemplary, and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0038] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0039] Embodiment one

[0040] The embodiment provides a student facial expression recognition method, as shown in the figure, which specifically comprises the following steps: Figure 1

[0041] Step 1, obtaining an image containing all the faces of students to be recognized;

[0042] The image containing all the faces of students to be recognized can be a screenshot containing all the faces of students in online teaching, or a picture containing all the faces of students collected by a camera in a classroom.

[0043] Step 2, face detection: using a face recognition algorithm to recognize the faces of all students to be recognized in the image obtained in step 1, obtaining the position information of the face of each student to be recognized in the image obtained in step 1, and cutting to obtain the face picture of each student to be recognized based on the position information of the face of each student to be recognized.

[0044] Step 3, inputting the face picture of each student to be recognized into a trained convolutional neural network to obtain the probability of each facial expression category corresponding to each student to be recognized, and taking the facial expression category corresponding to the maximum probability as the facial expression recognition result of the student to be recognized. Teachers can predict the psychological state of students according to the recognized facial expression classification of students, so as to better grasp the learning state of students.

[0045] The convolutional neural network is based on VGG19, removes multiple fully connected layers in the traditional VGG19, and directly divides the last fully connected layer into 7 categories (i.e. 7 facial expression categories) to identify and obtain the output probability of each category. At this time, the probability is not normalized, and the probability normalization processing is performed through the softmax layer. Therefore, the convolutional neural network of the present application comprises a plurality of convolutional layers (16 convolutional layers), a fully connected layer and a softmax layer connected in sequence.

[0046] ​Each convolutional layer is composed of a BatchNorm layer (BN layer), an activation layer (the activation layer adopts a relu layer) and an average pooling layer. The convolutional layer is used for extracting facial expression features of the face picture. The facial expression features refer to the related feature data of facial muscle and texture changes, which are accurately extracted and analyzed through the presentation of data.

[0047] As shown in Figure 2 The training process of the convolutional neural network is as follows:

[0048] (1) Generating a training set. Obtain the original training set (CK+ dataset). In order to prevent the convolutional neural network from overfitting too fast, data augmentation is performed on each face picture in the original training set, that is, some image transformations are artificially made, such as flipping, rotating and cutting, etc. The face pictures after data augmentation are combined with the original training set to obtain the training set.

[0049] (2) Input the training set into the convolutional neural network and perform iterative training until the loss function converges, thereby obtaining the trained convolutional neural network.

[0050] The loss function adopts a cross-entropy loss function.

[0051] The present application integrates facial expression feature extraction and facial expression classification into an end-to-end network, and uses VGG19 to complete expression recognition and classification. The multiple fully connected layers in the traditional VGG19 are removed, and the network is directly divided into 7 categories after a fully connected layer to identify, thereby improving the accuracy of facial expression recognition. By inferring the learning state of students through facial expressions, it is beneficial for teachers to master the learning state of each student in class, realizes the detection and evaluation of each student in class, so that the teacher can better master the learning state of the student, adjust the teaching strategy, and improve the teaching efficiency.

[0052] Embodiment two

[0053] The present embodiment provides a student facial expression recognition system, which specifically comprises the following modules:

[0054] An image acquisition module configured to acquire images containing all faces of students to be identified;

[0055] A face recognition module configured to recognize the faces of all students to be identified in the images by using a face recognition algorithm, and to crop the face pictures of each student to be identified;

[0056] The facial expression recognition module is configured to: input the face picture of each student to be recognized into the trained convolutional neural network, obtain the probability of each facial expression category corresponding to each student to be recognized, and take the facial expression category corresponding to the maximum probability as the facial expression recognition result of the student to be recognized.

[0057] The convolutional neural network comprises a plurality of convolutional layers, a full connection layer and a softmax layer connected in sequence, each convolutional layer is composed of a BN layer, an activation layer and an average pooling layer.

[0058] The network training module is configured to:

[0059] generate a training set;

[0060] input the training set into the convolutional neural network, and perform iterative training until the loss function converges, so as to obtain the trained convolutional neural network.

[0061] It should be noted that each module in the embodiment corresponds to each step in the first embodiment one by one, and the specific implementation process is the same, which will not be repeated here.

[0062] Embodiment three

[0063] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the student facial expression recognition method according to the first embodiment.

[0064] Embodiment four

[0065] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor realizes the steps in the student facial expression recognition method according to the first embodiment when executing the program.

[0066] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer usable program code.

[0067] The embodiments of methods, apparatuses (systems) and computer program products according to the present application can be described in the general context of method steps and processes, which can be implemented in one embodiment by a program of instructions on a computer-readable storage medium executed by a computer or other programmable apparatus. The apparatuses can be specially constructed for executing the embodiments of methods, apparatuses (systems) and computer program products according to the present application or can include a computer or other programmable apparatus. Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram Figure 1 The apparatus can be specially constructed for performing the functions described in a flow or multiple flows and / or blocks in a flowchart and / or a block diagram.

[0068] The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in a flow or multiple flows and / or blocks in a flowchart and / or a block diagram. Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram Figure 1 The apparatus can be specially constructed for performing the functions described in a flow or multiple flows and / or blocks in a flowchart and / or a block diagram.

[0069] The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in a flow or multiple flows and / or blocks in a flowchart and / or a block diagram. Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram Figure 1 The apparatus can be specially constructed for performing the functions described in a flow or multiple flows and / or blocks in a flowchart and / or a block diagram.

[0070] It is understood by those skilled in the art that all or part of the above-mentioned embodiment methods can be implemented by computer program instructions to instruct related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the flow of each method embodiment as described above. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0071] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A student facial expression recognition method, characterized in that, The method comprises the following steps: obtaining an image containing all faces of students to be identified; using a face recognition algorithm to identify the faces of all students to be identified in the image, and cutting out a face picture of each student to be identified; the cutting method of the face picture is as follows: obtaining position information of the face of each student to be identified in the image, and cutting out the face picture of each student to be identified based on the position information of the face of each student to be identified; inputting the face picture of each student to be identified into a trained convolutional neural network to obtain the probability of each facial expression category corresponding to each student to be identified, performing probability normalization processing through a softmax layer, and taking the facial expression category corresponding to the maximum probability as the facial expression recognition result of the student to be identified; wherein the convolutional neural network comprises a plurality of convolutional layers, a fully connected layer and a softmax layer connected in sequence, and each convolutional layer is composed of a BN layer, an activation layer and an average pooling layer.

2. The student facial expression recognition method of claim 1, wherein, The training process of the convolutional neural network is as follows: generating a training set; inputting the training set into the convolutional neural network for iterative training until the loss function converges, thereby obtaining the trained convolutional neural network.

3. The method of claim 2, wherein the student facial expression recognition method is characterized by, The loss function adopts a cross-entropy loss function.

4. The method of claim 2, wherein the step of identifying the student facial expression is further defined as: identifying the student facial expression based on the student facial expression data. The specific method for generating the training set is to obtain an original training set, perform data enhancement on each face picture in the original training set, combine the face pictures after data enhancement with the original training set, and obtain the training set.

5. The method of claim 4, wherein the student facial expression recognition method is characterized by, The specific method for generating the training set is that the data enhancement method is flipping, rotation or cutting.

6. A student facial expression recognition system characterized by, The method comprises the following steps: an image acquisition module configured to obtain an image containing all faces of students to be identified; a face recognition module configured to use a face recognition algorithm to identify the faces of all students to be identified in the image, and cut out a face picture of each student to be identified; the cutting method of the face picture is as follows: obtaining position information of the face of each student to be identified in the image, and cutting out the face picture of each student to be identified based on the position information of the face of each student to be identified; a facial expression recognition module configured to input the face picture of each student to be identified into a trained convolutional neural network to obtain the probability of each facial expression category corresponding to each student to be identified, perform probability normalization processing through a softmax layer, and take the facial expression category corresponding to the maximum probability as the facial expression recognition result of the student to be identified; wherein the convolutional neural network comprises a plurality of convolutional layers, a fully connected layer and a softmax layer connected in sequence, and each convolutional layer is composed of a BN layer, an activation layer and an average pooling layer.

7. A student facial expression recognition system as claimed in claim 6, wherein, The method further comprises a network training module configured to: generate a training set; input the training set into the convolutional neural network for iterative training until the loss function converges, thereby obtaining the trained convolutional neural network.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps in the student facial expression recognition method according to any one of claims 1-5. The program is executed by the processor to realize the steps in the student facial expression recognition method according to any one of claims 1-5.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the student facial expression recognition method in any one of claims 1-5 when executing the program.

Images