Expression recognition method based on feature fusion

Abstract

The present invention discloses a method for facial expression recognition based on feature fusion. First, data augmentation operation is performed on a data set. Then, the semantic level features of different layers of a VGGNet‑19 network are extracted by using a feature fusion sub-network and input into a fully connected layer. A splicing layer is used to splice different fully connected layers in parallel. Finally, facial expression classification is performed through the fully connected layer to achieve the facial expression recognition task. The characteristic of this network structure is that the network can be trained from scratch without the need for a pre-trained model. The performance of the present invention on the FER2013 facial expression data set reaches a relatively high level.

Description

Technical Field

[0001] The invention relates to an expression recognition method based on feature fusion, which is applicable to the field of multimedia signal processing. Background Art

[0002] In modern society, communication and exchange are indispensable. People are communicating in different ways and channels all the time. In general, the way humans communicate and convey information can usually be divided into two parts: language and non-language. Through language, people can directly obtain relevant information in the words, which is a more direct way. However, in the actual communication process, people can always obtain more information through other non-language methods besides language. Among them, facial expressions play a very important role. Facial expressions change with people's emotions all the time, contain rich information, and are a natural medium for transmitting information.

[0003] Expression recognition is an important area of ​​computer vision. It classifies expressions from static images or dynamic video sequences, simulates the way humans recognize expressions, and determines human emotional states, so that computers can accurately judge emotional states through expressions like humans. Therefore, expression recognition can be widely used in human-computer interaction applications such as driver fatigue monitoring, lie monitoring, medical treatment, security, education, entertainment, virtual reality, and augmented reality.

[0004] Compared with traditional expression recognition methods, expression recognition based on deep learning methods can greatly reduce the reliance on separate modules such as image preprocessing and feature extraction. By constructing a deep neural network structure, the computer can extract multi-layer features of the image, learn more abstract expression features, and integrate them into an end-to-end model, which can improve classification accuracy and have stronger robustness. The deep learning method is similar to the way the human brain recognizes things. It has shown far better accuracy than traditional methods for abstract pictures, text, voice, etc. In addition, it also demonstrates the ability to process more complex data sets. Therefore, in the field of expression recognition, methods based on deep learning are also increasingly used by researchers.

[0005] However, in expression recognition based on deep neural networks, the neural network learns different features of the input image through different convolutional layers, and each convolutional layer outputs feature maps of different sizes. As the depth of the network increases, the feature maps output by the convolutional layer also differ in the learned features. For shallow convolutional layers, the output feature maps contain more low-level features, such as texture, details, and other features. For deep convolutional layers, as the size of the feature map becomes smaller and smaller, the feature map can extract more high-level semantic features and some of the strongest features. At this time, after the feature map is visualized, it is difficult for the human eye to distinguish the relationship with the original image. This is because in the process of the network layer from shallow to deep, the feature map has to undergo a series of downsampling operations, and downsampling will inevitably lose many detailed features extracted by the shallow network in order to reduce the size of the feature map and reduce the amount of calculation. Summary of the invention

[0006] Purpose of the invention: In view of the above-mentioned prior art, a facial expression recognition method based on feature fusion is proposed to solve the problem that traditional convolutional neural networks are insufficient in extracting facial expression features.

[0007] Technical solution: The expression recognition method based on feature fusion includes the following steps:

[0008] Step 1: Augment the data set;

[0009] Step 2: Input the augmented image obtained in step 1 into the VGGNet-19 network for training, and extract the main network features after passing through the fully connected layer;

[0010] Step 3: After the third and fourth pooling operations, the feature maps are respectively input into the feature fusion sub-network for training, and finally fused with the main network for classification.

[0011] Furthermore, the step 1 includes the following sub-steps:

[0012] Step 1.1: Randomly crop and horizontally flip the images in the data set, specifically: randomly crop the grayscale images with a size of 48×48 in the data set, and the random cropping is to select a 40×40 window to crop the input image. The window position is random, and the size of the cropped image is 40×40; then the randomly cropped data set is randomly horizontally flipped with a probability of 0.5, that is, processed according to the following formula:

[0013] I preprocess =Flip(Crop(I input ))

[0014] Among them, the probability of performing a random horizontal flip of Flip is P 1 (Flip) = 0.5, Crop (Iinput ) is to perform cropping operation on the input image; I input is the input original grayscale image, I preprocess Output image after random cropping and horizontal flipping;

[0015] Step 1.2: Use the noise adding method to continue data enhancement on the preliminary processing obtained in step 1.1, and add Gaussian white noise to a small number of images after cropping and flipping with a probability of 0.2; that is, process according to the following formula:

[0016] I noise =Gaussian(I preprocess )

[0017] Among them, I noise is the output augmented image, and Gaussian represents the Gaussian function.

[0018] Furthermore, the step 2 includes the following sub-steps:

[0019] Step 2.1: Input the data-augmented image into the VGGNet-19 main network;

[0020] Step 2.2: After inputting the network, it passes through 2 layers of convolution with 64 3×3 convolution kernels each, 2 layers of convolution with 128 3×3 convolution kernels each, 4 layers of convolution with 256 3×3 convolution kernels each, 4 layers of convolution with 512 3×3 convolution kernels each, 4 layers of convolution with 512 3×3 convolution kernels each and a fully connected layer to output the global feature expression.

[0021] Furthermore, the step 3 includes the following sub-steps:

[0022] Step 3.1: Connect the third pooling layer and the fourth pooling layer to two feature fusion sub-networks respectively, and then output feature expressions of different levels through the fully connected layer respectively; the feature fusion sub-network includes a convolution layer with a convolution kernel of size 1×1, a batch normalization layer and a ReLU nonlinear activation layer;

[0023] Step 3.2: The feature expressions of different levels output by the feature fusion sub-network are parallel-joined with the global feature expressions of the main network;

[0024] Step 3.3: Input the fused feature expression into the random dropout layer. The strategy of the random dropout layer is to invalidate part of the input data with a probability of 0.5.

[0025] Step 3.4: Send the features to the fully connected layer and use the Softmax layer to normalize the output probability to get the final classification result.

[0026] Beneficial effects: In order to solve the problem that traditional convolutional neural networks are insufficient in extracting facial expression features, the method of the present invention introduces a feature fusion branch based on the VGGNet-19 network to fuse the facial expression features extracted from feature maps of different layers, so as to achieve the purpose of strengthening the mid- and low-level features, so that the recognition effect of the model on the data set collected in an uncontrolled environment is better and more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the expression recognition method based on feature fusion of the present invention;

[0028] Figure 2 A schematic diagram of the network structure of the expression recognition method based on feature fusion of the present invention;

[0029] Figure 3 It is a schematic diagram of the feature fusion module of the present invention;

[0030] Figure 4 It is the VGGNet network structure;

[0031] Figure 5 It is the accuracy curve of the training process;

[0032] Figure 6 This is the confusion matrix of the present invention on the FER2013 private test set. DETAILED DESCRIPTION

[0033] The present invention will be further explained below in conjunction with the accompanying drawings.

[0034] like Figure 1 As shown in the figure, an expression recognition method based on feature fusion integrates semantic signs at different levels to solve the problem of single feature extraction of traditional deep neural network, including the following steps:

[0035] Step 1: Perform data augmentation on the FER2013 dataset, specifically:

[0036] Step 1.1: Randomly crop and horizontally flip the images in the FER2013 dataset. Specifically, randomly crop the grayscale images of size 48×48 in the FER2013 dataset. Random cropping means selecting a 40×40 window to crop the input image. The window position is random, and the size of the cropped image is 40×40. Then, the randomly cropped dataset is randomly horizontally flipped with a probability of 0.5, that is, processed according to the following formula:

[0037] I preprocess =Flip(Crop(I input ))

[0038] Among them, the probability of performing a random horizontal flip of Flip is P1 (Flip) = 0.5, Crop (I input ) is to perform cropping operation on the input image; I input is the input original grayscale image, I preprocess Output image after random cropping and horizontal flipping.

[0039] Step 1.2: Use the noise adding method to continue data enhancement on the preliminary processing obtained in step 1.1, and add Gaussian white noise to a small number of images after cropping and flipping with a probability of 0.2; that is, process according to the following formula:

[0040] I noise =Gaussian(I preprocess )

[0041] Among them, I noise is the output augmented image, and Gaussian represents the Gaussian function.

[0042] Step 2: Input the augmented image obtained in step 1 into the VGGNet-19 network for training, and extract the main network features after passing through the fully connected layer, specifically:

[0043] Step 2.1: Input the data-augmented image into the VGGNet-19 main network. VGGNet-19 is Figure 4 Middle E structure;

[0044] Step 2.2: If Figure 2 As shown in the figure, after the input network, it passes through the C1 layer (2 layers of convolution, each with 64 3×3 convolution kernels), C2 layer (2 layers of convolution, each with 128 3×3 convolution kernels), C3 layer (4 layers of convolution, each with 256 3×3 convolution kernels), C4 layer (4 layers of convolution, each with 512 3×3 convolution kernels), C5 (4 layers of convolution, each with 512 3×3 convolution kernels) and the fully connected layer F1 to output the global feature expression. Among them, each layer of convolution is connected to a pooling layer.

[0045] Step 3: After the third and fourth pooling operations, the feature maps are respectively input into the feature fusion sub-network for training, and finally fused with the main network for classification, specifically:

[0046] Step 3.1: Connect the third pooling layer P3 and the fourth pooling layer P4 to two feature fusion subnetworks FFM respectively, and then output feature expressions of different levels through the fully connected layers F2 and F3 respectively. Figure 3 As shown in the figure, the feature fusion sub-network contains a convolution layer with a convolution kernel of size 1×1, a batch normalization layer and a ReLU nonlinear activation layer. The feature fusion sub-network extracts features of different levels from the middle layer of the main network, so that the semantic features of different levels of the main network can be further fused.

[0047] Step 3.2: The feature expressions of different levels output by the feature fusion subnetwork are spliced ​​in parallel with the global feature expressions of the main network; that is, the fully connected layers F1-F3 are spliced ​​in parallel through the splicing layer CO, and the features of different levels are fused and input together to the next layer of the network. Splicing the channels increases the width of the network and the adaptability of the network to scale.

[0048] Step 3.3: Input the fused feature expression into the random dropout layer DR. The strategy of the random dropout layer is to invalidate part of the input data with a probability of 0.5, which can effectively prevent the model from overfitting and increase robustness.

[0049] Step 3.4: Send the features to the fully connected layer FC and use the Softmax layer to normalize the output probability to get the final classification result.

[0050] like Figure 5 As shown in the figure, the entire iteration process is 250 iterations. The model accuracy starts to stabilize around the 150th iteration. Since the FER2013 dataset itself has watermarks and occlusions, a small number of images are incorrectly labeled or even have no face images, the accuracy of both the public test set and the private test set finally stabilizes at a level slightly above 70%. Figure 6 This is the confusion matrix on the FER2013 private test set. In the confusion matrix, happiness and surprise have the highest prediction accuracy. Happiness and surprise are significantly different from other expressions in the direction of facial muscle movement and are not easily mispredicted by the network. The feature fusion module effectively improves the accuracy of the expression recognition task. Overall, the recognition rate of the expression recognition method based on feature fusion in the present invention on the FER2013 data set reaches 72.95%, and the performance reaches a relatively high level.

[0051] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. The expression recognition method based on feature fusion is characterized by: The steps include: Step 1: Augment the data set; Step 2: Input the augmented image obtained in step 1 into the VGGNet-19 network for training, and extract the main network features after passing through the fully connected layer; Step 3: After the third and fourth pooling operations, the feature maps are respectively input into the feature fusion sub-network for training, and finally fused with the main network for classification; The step 1 includes the following sub-steps: Step 1.1: Randomly crop and horizontally flip the images in the data set, specifically: randomly crop the grayscale images with a size of 48×48 in the data set, and the random cropping is to select a 40×40 window to crop the input image. The window position is random, and the size of the cropped image is 40×40; then the randomly cropped data set is randomly horizontally flipped with a probability of 0.5, that is, processed according to the following formula: I preprocess =Flip(Crop(I input )) The probability of performing a random horizontal flip of Flip is P1(Flip)=0.5, and Crop(I input ) is to perform cropping operation on the input image; I input is the input original grayscale image, I preprocess Output image after random cropping and horizontal flipping; Step 1.2: Use the noise adding method to continue data enhancement on the preliminary processing obtained in step 1.1, and add Gaussian white noise to a small number of images after cropping and flipping with a probability of 0.2; that is, process according to the following formula: I noise =Gaussian(I preprocess ) Among them, I noise is the output augmented image, Gaussian represents the Gaussian function; The step 2 includes the following sub-steps: Step 2.1: Input the data-augmented image into the VGGNet-19 main network; Step 2.2: After inputting the network, it passes through 2 layers of convolution with 64 3×3 convolution kernels each, 2 layers of convolution with 128 3×3 convolution kernels each, 4 layers of convolution with 256 3×3 convolution kernels each, 4 layers of convolution with 512 3×3 convolution kernels each, 4 layers of convolution with 512 3×3 convolution kernels each and a fully connected layer to output the global feature expression; The step 3 includes the following sub-steps: Step 3.1: Connect the third pooling layer and the fourth pooling layer to two feature fusion sub-networks respectively, and then output feature expressions of different levels through the fully connected layer respectively; the feature fusion sub-network includes a convolution layer with a convolution kernel of size 1×1, a batch normalization layer and a ReLU nonlinear activation layer; Step 3.2: The feature expressions of different levels output by the feature fusion sub-network are parallel-joined with the global feature expressions of the main network; Step 3.3: Input the fused feature expression into the random dropout layer. The strategy of the random dropout layer is to invalidate part of the input data with a probability of 0.

5. Step 3.4: Send the features to the fully connected layer and use the Softmax layer to normalize the output probability to get the final classification result.

Images