A face expression recognition method fusing parallel network features

Abstract

The application discloses a face expression recognition method fusing parallel network features, comprising the following steps: firstly, improving a traditional VGG13 network, and replacing the last group of ordinary convolution groups in the traditional VGG13 network with a group of empty convolution groups; secondly, adding a weighted processing layer between the fourth layer and the fifth layer of the convolution layer of an Alexnet network, and using the improved Alexnet network and the improved VGG13 network to simultaneously extract features from a data set, fusing the extracted features, and then classifying and recognizing the features. The purpose of the innovation is to obtain a larger receptive field by using empty convolution under the condition that time and parameters are hardly increased, and to convert single feature information into more abundant feature information by means of feature fusion. The application can be used in online classes, improves the control of teachers on the whole teaching quality, adjusts the teaching content in time according to the reactions of students, improves the classroom efficiency, and optimizes the classroom rhythm.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression recognition, and in particular to a facial expression recognition method integrating parallel network features. Background Art

[0002] In today's society, intelligence and wisdom are at the forefront, guiding the development of our times and urban progress. Intelligence and wisdom inevitably involve the communication and collaboration between humans and computers. For example, in 1) technology, with the emergence and continuous improvement of robots, robots can now communicate face-to-face with humans, deriving information not only from language and emotions but also from facial expressions, enabling smoother communication and discussion. 2) In the medical field, facial expressions are particularly important for patients and newborns in pain who are unable to express themselves through words or sounds. 3) In the field of public security, long-term national stability is essential, and this extends to every city and region. Through the research and popularization of expression classification systems, surveys of happiness and pain indexes, and even monitoring the furtive expressions of strangers, can contribute to regional prosperity and development.

[0003] While facial expression recognition technology is currently advanced, most methods extract features from a single network model and utilize standard convolutions throughout the entire model. For example, the traditional VGG19 network has 16 standard convolutional layers, each with the same convolution kernel. These kernels are of a single type and have a voiding rate of 1, resulting in moderate accuracy and high recognition error rates. Summary of the Invention

[0004] Purpose of the invention: In response to the problems pointed out in the background technology, the present invention provides a facial expression recognition method that integrates parallel network features. Without increasing parameters and data sets, two networks are used to extract features, and hollow convolution is used to increase the receptive field and extract clearer features, so that the network model obtains multi-scale features, improves the accuracy of the training model and makes the model more robust.

[0005] Technical solution: The present invention discloses a facial expression recognition method integrating parallel network features, comprising the following steps:

[0006] Step 1: Obtain facial expression images and perform image preprocessing;

[0007] Step 2: Construct an improved VGG13 network model to obtain the training feature parameter values ​​and model weights, thereby obtaining the attribute characteristics of each facial expression. The improved VGG13 network model replaces the last group of 3×3 ordinary convolution groups with a dilated convolution group with a dilation rate of 2;

[0008] Step 3: Add a weighted processing layer between the fourth and fifth convolutional layers of the Alexnet network, input the dataset into the improved VGG13 network and the improved Alexnet network model, and fuse the features of the two after the output of the first fully connected layer;

[0009] Step 4: Use the trained improved VGG13 network model weights and the obtained attribute features to recognize facial expressions.

[0010] Furthermore, the image preprocessing in step 1 includes: the data set contains data pictures and classification categories, which are also called labels. The data set contains seven categories: anger, disgust, fear, happiness, sadness, surprise, and normal, that is, the labels are marked as anger, disgust, fear, happiness, sadness, surprise, and normal; the data pictures are cropped and normalized, the data images are labeled and categorized, and the faces are located.

[0011] Furthermore, the VGG13 network model is divided into two sections, one for feature extraction and the other for classification and recognition;

[0012] The feature extraction part of the VGG13 network model is as follows: first, an image of size 224×224×3 is input, and after two convolutional layers with a filter of 3×3, a 224×224×1 matrix is ​​obtained. Since the first layer has 64 convolution kernels, the original 224×224×1 becomes 224×224×64. The convolution kernel used in the pooling layer is 2×2×64 with a stride of 2. After pooling, the dimension of the resulting matrix is ​​half of the original, 112×112×128. Similarly, after a series of convolution + pooling, two convolutions with a channel number of 512 and a filter of 3×3 and a maximum pooling operation are performed.

[0013] The classification and recognition parts of the VGG13 network model are as follows: the classification part consists of three fully connected layers with the number of nodes being [4096, 4096, 1000]. The last softmax layer is used to predict the image, and the prediction result is converted to a value between (0, 1), that is, a probability, and the label corresponding to the highest probability is output.

[0014] Furthermore, the improved VGG13 network model includes: replacing the last group of 3×3 ordinary convolution groups with a dilated convolution group with a dilation rate of 2, wherein the dilated convolution group with a dilation rate of 2 fills a row or a column of 0s between two adjacent values ​​in the convolution kernel;

[0015] Assuming that the convolution kernel size of the dilated convolution is k and the number of holes is d, the equivalent convolution kernel size is k′, and for a 3×3 convolution kernel, k=3. The equivalent convolution kernel formula is as follows:

[0016] k'=k+(k-1)×(d-1)(1)

[0017] RF i+1 =RF i +(k'-1)×S i (2)

[0018]

[0019] The calculation formula of the receptive field of the current layer is as shown in formula (2) and (3), where RF i+1 Represents the receptive field of the current layer, RF i represents the receptive field of the previous layer, k′ represents the size of the convolution kernel, S i Represents the product of the strides of all previous layers, excluding this layer.

[0020] Furthermore, the improved Alexnet network model consists of an input layer, 5 convolutional layers, 1 weighted processing layer, and 3 fully connected layers. The input size of the Alexnet network model is 227×227×3, and stacked convolutional layers, i.e., convolutional layer + convolutional layer + pooling layer, are used to extract image features. The convolutional layer extracts feature information, and the pooling layer compresses information. After the image passes through the fourth convolutional layer of the Alexnet network, the size becomes 13*13, and the weight of each position in the sub-area is attenuated by 0.1% of the distance of the position, that is, for every increase of 1 pixel in the distance, the weight decreases by 0.013; the relationship between A and d is: A=1-0.013d, where A is the weight and d is the Manhattan distance from the position to the feature center.

[0021] Furthermore, the feature fusion in step 3 is specifically as follows: the one-dimensional feature vectors D1 and D2 finally obtained by improving the VGG13 network and the Alexnet network model are fused, and the obtained D is passed through the fully connected layer and the softmax layer to obtain the category prediction, and the fusion method is: D=D1+D2.

[0022] Furthermore, step 4 specifically includes: saving the trained model with the highest accuracy, then inputting the facial expression test data to be recognized into the trained network to obtain a new feature vector, and performing classification prediction on the new feature vector through the softmax method, and the prediction result is its predicted label.

[0023] Beneficial effects:

[0024] The present invention changes the last convolution group of the VGG13 network model into a dilated convolution with a dilated rate of 2. Under the premise of ensuring that the width, height and number of channels of the first four groups of feature maps of the entire network model are not changed, the last group is improved, and a multi-feature fusion image is formed while almost no parameter amount is increased during the calculation process. The Alexnet network model is added to the VGG13 network structure to obtain deeper features and realize the complementary fusion of shallow semantics and high-level semantics. The fused image is used to improve the accuracy of the training model without increasing the training samples and the network, and the model has better robustness. A weighted processing layer is added between the fourth and fifth convolutional layers of the Alexnet network to effectively improve the accuracy.

[0025] Typically, each convolution kernel in VGG13 uses a 3×3 kernel size. However, by varying the dilation rate at each layer, the receptive field varies, capturing multi-scale information. This, of course, does not affect the size of the feature map, thus avoiding information loss caused by downsampling while also utilizing multi-scale information. The presence or absence of dilated convolution does not affect the size of the output feature map. In other words, the size of the output feature map is unrelated to the number of dilations. Therefore, dilated convolution can be used to increase the receptive field while maintaining the same size. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of the present invention;

[0027] Figure 2 This is a diagram of the neural network structure of the present invention;

[0028] Figure 3 It is the ordinary convolution graph in the original VGG13 network;

[0029] Figure 4 To improve the dilated convolution graph in the VGG13 network;

[0030] Figure 5 A facial feature image of a person who is sad according to an embodiment of the present invention;

[0031] Figure 6 This is a result diagram of the facial expression recognition method integrating parallel network features of the present invention;

[0032] Figure 7 This is a comparison chart of benefits before and after using the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0034] like Figure 1As shown, the present invention discloses a facial expression recognition method integrating parallel network features, which is applied to the recognition of facial expressions in online classes, comprising the following steps:

[0035] Step 1: Obtain facial expression images and perform image preprocessing. The dataset contains both data images and classification categories, also known as labels. This dataset contains seven categories: anger, disgust, fear, happiness, sadness, surprise, and normal, with the labels marked as anger, disgust, fear, happiness, sadness, surprise, and normal. The data images are cropped and normalized, the labels are correctly labeled, the faces are accurately located, and the data samples are free of interference such as watermarks and cartoons.

[0036] Step 2: Construct an improved VGG13 network model to obtain the training feature parameter values ​​and model weights, so as to obtain the attribute characteristics of each facial expression. The improved VGG13 network model replaces the last group of 3×3 ordinary convolution groups with a dilated convolution group with a dilation rate of 2.

[0037] like Figure 2 As shown, the VGG13 network model is roughly divided into two parts: feature extraction and classification and recognition. The feature extraction part of the VGG13 network model works as follows: First, an image of size 224×224×3 is input. This image passes through two convolutional layers with a 3×3 filter, resulting in a 224×224×1 matrix. Because the first layer has 64 convolution kernels, the original 224×224×1 matrix becomes 224×224×64. The pooling layer uses a 2×2×64 kernel with a stride of 2. After pooling, the resulting matrix is ​​halved in size, becoming 112×112×128. Similarly, after a series of convolution + pooling, two convolution and maximum pooling operations with a channel number of 512 and a filter of 3×3 are finally performed; the classification and recognition parts of the VGG13 network model are specifically as follows: the classification part consists of three layers of fully connected layers with the number of nodes [4096, 4096, 1000]. The last layer, the softmax layer, is used to predict the image. The predicted result is converted to a value between (0, 1), that is, a probability, and the label corresponding to the largest probability is output.

[0038] like Figure 3 and Figure 4As shown, the last group of 3×3 ordinary convolution groups is replaced by a dilated convolution group with a dilation rate of 2. The dilated convolution group with a dilation rate of 2 fills a row or column of 0s between two adjacent values ​​in the convolution kernel. When the dilation rate is 2, the dilated convolution can be regarded as an ordinary convolution, that is, the ordinary convolution is a special case of dilated convolution. It can be easily seen from the figure that the receptive field after ordinary convolution is 3, and the receptive field after dilated convolution with a dilation rate of 2 is 5. The same 3×3 convolution can have the effect of a 5×5 convolution. The dilated convolution can increase the receptive field without increasing the number of parameters (parameters = convolution kernel size + bias). Assuming that the convolution kernel size of the dilated convolution is k and the number of holes is d, its equivalent convolution kernel size is k′. For example, for a 3×3 convolution kernel, k = 3. The equivalent convolution kernel formula is as follows:

[0039] k'=k+(k-1)×(d-1)(1)

[0040] RF i+1 =RF i +(k'-1)×S i (2)

[0041]

[0042] The calculation formula of the receptive field of the current layer is as shown in formula (2) and (3), where RF i+1 Represents the receptive field of the current layer, RF i represents the receptive field of the previous layer, k′ represents the size of the convolution kernel, S i It represents the product of the strides of all previous layers, excluding the current layer. Calculation shows that for the same 3×3 convolution, the receptive field of a normal convolution is 3, while the receptive field of a dilated convolution with a dilation rate of 2 is 7.

[0043] Step 3: Add a weighted processing layer between the fourth and fifth convolutional layers of the Alexnet network, input the dataset into the improved VGG13 network and the improved Alexnet network model, and fuse the features of the two after the output of the first fully connected layer.

[0044] The improved Alexnet network model consists of an input layer, five convolutional layers, one weighted processing layer, and three fully connected layers. It uses a stacked convolutional layer (convolutional layer + convolutional layer + pooling layer) to extract image features. The Alexnet network model has an input size of 227×227×3. After passing through the fourth convolutional layer of the Alexnet network, the image size is reduced to 13*13. The weight of each position within a subregion is decayed by 0.1% of the distance to that position, meaning that for every pixel increase in distance, the weight decreases by 0.013. The relationship between A and d is: A = 1 - 0.013d. Where A is the weight and d is the Manhattan distance from the position to the feature center.

[0045] After a series of operations such as continuous convolution and pooling, the two networks finally obtain a one-dimensional feature vector D1 and D2. The D obtained by fusing D1 and D2 is passed through the fully connected layer and the softmax layer to obtain the category prediction. The fusion method is: D = D1 + D2.

[0046] Step 4: Use the trained improved VGG13 network model weights and the obtained attribute features to recognize facial expressions in online classes.

[0047] Save the model with the highest accuracy, then input the test data into the trained network to obtain a new feature vector, and perform classification prediction on the new feature vector using the softmax method. The prediction result is its predicted label.

[0048] Depend on Figure 5 、 Figure 6 As shown, Figure 5 The facial features of sadness. When a student has a sad expression, it means that the student needs further explanation of this part of the class content, or is not paying attention. The teacher needs to explain it to the student again and provide further ideological education. Figure 6 This figure shows the results of a facial expression recognition method that integrates parallel network features, with a test accuracy of 93.506%. This shows that replacing the last set of normal convolutions in the traditional VGG13 network with a dilated convolution group, adding a weighted processing layer between the fourth and fifth convolutional layers of the Alexnet network, and using both the improved Alexnet network and the improved VGG13 network to extract features from the dataset simultaneously can achieve high accuracy.

[0049] Online classes make classes more efficient and time-saving. Teachers and students can access the classroom at any time due to special circumstances, eliminating unnecessary complex procedures. The present invention is applied in online classes to increase meaningful interactions. Teachers can manage the class more intuitively and judge students' classroom status and acceptance level through their expressions, achieving twice the result with half the effort.

[0050] like Figure 7 As shown in the figure, the monthly test scores of students before and after use are recorded, with a full score of 130 points, and the student scores before and after use are clearly compared. Therefore, it can be seen that the present invention can be used in online classes to improve teachers' control over the overall teaching quality, make timely adjustments to teaching content based on students' responses, improve classroom efficiency, and optimize classroom rhythm.

[0051] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made in accordance with the spirit of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A facial expression recognition method integrating parallel network features, characterized in that: The following steps are involved: Step 1: Obtain facial expression images and perform image preprocessing; Step 2: Construct an improved VGG13 network model to obtain the training feature parameter values ​​and model weights, thereby obtaining the attribute characteristics of each facial expression. The improved VGG13 network model replaces the last group of 3×3 ordinary convolution groups with a dilated convolution group with a dilation rate of 2; Step 3: Add a weighted processing layer between the fourth and fifth convolutional layers of the Alexnet network, input the dataset into the improved VGG13 network and the improved Alexnet network model, and fuse the features of the two after the output of the first fully connected layer; Step 4: Use the trained improved VGG13 network model weights and the obtained attribute features to recognize facial expressions.

2. The facial expression recognition method integrating parallel network features according to claim 1, characterized in that: The image preprocessing in step 1 includes: the data set contains data pictures and classification categories, which are also called labels. The data set contains seven categories: anger, disgust, fear, happiness, sadness, surprise, and normal, that is, the labels are marked as anger, disgust, fear, happiness, sadness, surprise, and normal; the data pictures are cropped and normalized, the data images are labeled and categorized, and the faces are located.

3. The facial expression recognition method integrating parallel network features according to claim 1, characterized in that: The VGG13 network model is divided into two sections, one for feature extraction and the other for classification and recognition; The feature extraction part of the VGG13 network model is as follows: first, an image of size 224×224×3 is input, and after two convolutional layers with a filter of 3×3, a 224×224×1 matrix is ​​obtained. Since the first layer has 64 convolution kernels, the original 224×224×1 becomes 224×224×64. The convolution kernel used in the pooling layer is 2×2×64 with a stride of 2. After pooling, the dimension of the resulting matrix is ​​half of the original, 112×112×128. Similarly, after a series of convolution + pooling, two convolutions with a channel number of 512 and a filter of 3×3 and a maximum pooling operation are performed. The classification and recognition parts of the VGG13 network model are as follows: the classification part consists of three fully connected layers with the number of nodes being [4096, 4096, 1000]. The last softmax layer is used to predict the image, and the prediction result is converted to a value between (0, 1), that is, a probability, and the label corresponding to the highest probability is output.

4. The facial expression recognition method integrating parallel network features according to claim 3 is characterized in that: The improved VGG13 network model includes: replacing the last group of 3×3 ordinary convolution groups with a dilated convolution group with a dilation rate of 2, wherein the dilated convolution group with a dilation rate of 2 fills a row or a column of 0s between two adjacent values ​​in the convolution kernel; Assuming that the convolution kernel size of the dilated convolution is k and the number of holes is d, the equivalent convolution kernel size is k′, and for a 3×3 convolution kernel, k=3. The equivalent convolution kernel formula is as follows: k'=k+(k-1)×(d-1)(1) RF i+1 =RF i +(k'-1)×S i (2) The calculation formula of the receptive field of the current layer is as shown in formula (2) and (3), where RF i+1 Represents the receptive field of the current layer, RF i represents the receptive field of the previous layer, k′ represents the size of the convolution kernel, S i Represents the product of the strides of all previous layers, excluding this layer.

5. The facial expression recognition method integrating parallel network features according to claim 3 is characterized in that: The improved Alexnet network model consists of an input layer, 5 convolutional layers, 1 weighted processing layer, and 3 fully connected layers. The input size of the Alexnet network model is 227×227×3. Stacked convolutional layers, i.e., convolutional layer+convolutional layer+pooling layer, are used to extract image features. The convolutional layer extracts feature information, and the pooling layer compresses information. After the image passes through the fourth convolutional layer of the Alexnet network, the size becomes 13*13. The weight of each position in the sub-area is attenuated by 0.1% of the current position distance, that is, the weight decreases by 0.013 for every pixel increase in distance. The relationship between A and d is: A=1-0.013d, where A is the weight and d is the Manhattan distance from the current position to the feature center.

6. The facial expression recognition method integrating parallel network features according to claim 1, characterized in that: The feature fusion in step 3 is specifically as follows: the one-dimensional feature vectors D1 and D2 finally obtained by improving the VGG13 network and the Alexnet network model are fused, and the obtained D is passed through the fully connected layer and the softmax layer to obtain the category prediction, and the fusion method is: D=D1+D2.

7. The facial expression recognition method integrating parallel network features according to claim 1, characterized in that: Step 4 specifically includes: saving the trained model with the highest accuracy, then inputting the facial expression test data to be recognized into the trained network to obtain a new feature vector, and performing classification prediction on the new feature vector through the softmax method. The prediction result is its predicted label.

Images