A multi-feature fusion expression recognition method

Abstract

The present invention proposes an expression recognition method based on self-correction and fusion of multi-scale features to improve VGGNet. The method first adds a multi-scale feature extraction network structure to the front end of the VGGNet16 network, and fuses the branch features of multiple features extracted by networks of different depths to retain more complete feature information; then, by using batch normalization after each convolutional layer and Dropout after the fully connected layer, the convergence rate of the network model is accelerated and the probability of gradient vanishing and overfitting in the network model is reduced; finally, a self-correction network is added to the back end of the network to suppress the impact of sample annotation uncertainty on network model training. This method can recognize seven basic expressions with higher accuracy on annotated uncertain data sets.

Description

Technical Field

[0001] The technology of the present invention belongs to the fields of image processing, pattern recognition, and artificial intelligence. Specifically, it relates to an expression recognition method of improving VGGNet by self-correction and fusion of multi-scale features. Background Art

[0002] Facial expressions are an intuitive reflection of human psychological states, containing rich emotional information, and can intuitively reflect a person's true thoughts. In daily human communication, we can not only convey information through language and text, but also through some body movements and facial expressions to complete communication between people. Studies have shown that facial expressions and body movements can often convey key information better than language. Facial expressions are a common form of non-verbal expression that can effectively convey personal emotions and intentions. We can observe other people's facial expression information through our eyes, and then the brain will analyze the information obtained to obtain other people's psychological state, completing the expression and communication of emotions between people. In the process of communication, facial expressions can give language emotional colors, and facial expressions can clearly show a person's true emotions, which is more accurate than language.

[0003] Facial expression recognition research has accumulated a wealth of theories so far, and the research objects have gradually shifted from laboratory scenes to real scenes. Although many research results have been achieved in facial expression research, the uncertainty of data set annotation has always been a major challenge. Deep learning network training requires a large amount of standard data drive, but for the expression data sets publicly available by domestic and foreign research institutions, because everyone often has their own subjective impressions of expression judgment, and everyone's subjective impression may be affected by their culture, growth environment, and personality, the labels of facial expression data may be wrong. Therefore, the quality of existing facial expression recognition data sets is uneven, and the amount of data in existing data sets is relatively scarce. These problems restrict the improvement of expression recognition accuracy. When using incorrectly labeled expression data samples to train network models, the following situations may occur: First, the network may over-learn the features of the wrong sample data, thereby interfering with the expression feature learning of the correct sample data; second, the network may over-learn the features on the wrong sample data and overfit. If there are too many wrong samples, it will also make it difficult for the network to learn the real expression features and make it difficult for the network to converge. Summary of the invention

[0004] The purpose of the present invention is to solve the problems that the existing manually annotated facial expression data sets for expression recognition still have annotation uncertainty and the convolutional neural network extraction features are imperfect, so as to achieve more accurate facial expression recognition.

[0005] In order to achieve the above-mentioned purpose, the present invention provides an expression recognition method for improving VGGNet based on self-correction and fusion of multi-scale features. The method mainly includes five parts: the first part is to preprocess the data set; the second part is to extract and fuse multi-scale features of the VGGNet16 network; the third part is to add batch normalization and Dropout modules; the fourth part is to add a self-correction network at the back end of the network; and the fifth part is network training and testing.

[0006] The first part consists of two steps:

[0007] Step 1: Download the facial expression dataset in natural scenes (RAF-DB), and then select 12,271 single-label training set images and 3,068 test set images in the dataset as model training and test samples;

[0008] Step 2: Determine whether the image is a grayscale image. If not, convert it into a grayscale image, then transform its size to 96×96, and perform basic operations such as rotation and translation on the image to enhance the training sample set to form the final training set sample;

[0009] The second part consists of two steps:

[0010] Step 3: Input the training samples in step 2 into the VGGNet16 network, add a multi-scale feature extraction network structure to the front end of the VGGNet16 network as the first layer of the network, use three convolution kernels of different sizes to extract features from the original image, and perform feature fusion on the multiple features extracted by the three convolution kernels of different sizes. Compared with using only a 3×3 convolution kernel, this structure loses less information and can retain more complete feature information.

[0011] Step 4: Add a shortcut connection between the second convolution group and the third convolution group of the improved VGGNet16 network in step 3 before the fully connected layer to form a branch feature fusion module, which fuses the features of different stages together and enables the network to learn more features;

[0012] The third part consists of two steps:

[0013] Step 5: Use batch normalization after each convolutional layer of the improved VGGNet16 network in steps 3 and 4 to speed up the convergence rate of the network model and increase the generalization of the model;

[0014] Step 6: Use Dropout after each fully connected layer of the improved VGGNet16 network in steps 3, 4, and 5 to discard some neurons and add some interference to increase the robustness of the model and reduce the number of model parameters;

[0015] Part 4 consists of five steps:

[0016] Step 7: The features extracted by the improved VGGNet16 network in steps 3, 4, 5, and 6 are weighted by the self-attention module, and the key samples and non-key samples are distinguished by the weight value, and the key samples are given priority to increase the network's weight value for key samples.

[0017] Step 8: The criticality weights obtained in step 7 are weighted by the cross entropy loss function used by the original network model using the logit model to avoid misleading network learning by non-critical samples.

[0018] Step 9, compare the criticality weights obtained in step 7 through the sorting regularization module, then sort all the weights from high to low in order of size, and divide the criticality weight values ​​into critical parts and non-critical parts according to a certain ratio.

[0019] In step 10, the weight values ​​of the key part and the non-key part obtained in step 9 are averaged, and the mean of the key part is constrained to be higher than the mean of the non-key part by a threshold through sorting regularization. The two loss functions of the self-attention module and the sorting regularization module are combined to obtain the final loss function of the self-correction network.

[0020] Step 11, the non-critical part samples obtained in step 9 are subtracted from the predicted probability of the original label of the sample by the re-labeling module. If the difference between the two is greater than the given range parameter value, the sample is re-labeled and its maximum predicted probability is used as the correct label of the re-labeled sample.

[0021] Part 5 consists of two steps:

[0022] Step 12, debug the network structure hyperparameters from step 3 to step 11, and obtain the final training model;

[0023] Step 13, input the training set in step 1 into the training model in step 12 to obtain the expression recognition image result.

[0024] The present invention proposes an expression recognition method based on self-correction and fusion of multi-scale features to improve VGGNet. The method first adds a multi-scale feature extraction network structure to the front end of the VGGNet16 network, and fuses the branch features of multiple features extracted by networks of different depths to retain more complete feature information. Then, by using batch normalization after each convolutional layer and using Dropout after the fully connected layer, the convergence rate of the network model is accelerated and the probability of gradient vanishing and overfitting in the network model is reduced. Finally, a self-correction network is added to the back end of the network to suppress the impact of sample annotation uncertainty on network model training, and seven basic expressions are recognized. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an overall block diagram of a specific implementation of the present invention;

[0026] Figure 2 A multi-scale feature extraction structure diagram of the present invention;

[0027] Figure 3 This is an example diagram of batch normalization of the present invention;

[0028] Figure 4 It is the Dropout structure diagram of the present invention;

[0029] Figure 5 This is a diagram of the self-correcting network structure of the present invention. DETAILED DESCRIPTION

[0030] In order to better understand the present invention, the expression recognition method based on self-correction and fusion of multi-scale features to improve VGGNet of the present invention is described in more detail below in conjunction with the accompanying drawings. In the following description, the detailed description of the current prior art may dilute the subject content of the present invention, and these descriptions will be ignored here.

[0031] Figure 1 It is a general block diagram of a specific implementation of the present invention, which specifically includes the following 13 steps, which are performed in the following steps:

[0032] Step 1, download the facial expression dataset in natural scenes (RAF-DB), which contains pictures of different ages, races and skin colors. It contains various postures, lighting conditions, interference factors such as glasses, hair and self-occlusion, and is committed to solving the problem of facial expression recognition in real environments. Then select 12271 single-label training set pictures and 3068 test set pictures in the data set as model training test samples.

[0033] Step 2: Determine whether the image is a grayscale image. If not, convert it to a grayscale image to minimize the impact of lighting on facial expression recognition. We first perform a series of random transformations, including flipping, rotating, and cutting. Then, we transform the data image size to 104×104, and then randomly cut it into 96×96 images. Then, we randomly rotate the image between 0 and 15 degrees and perform a horizontal mirror operation to form the final training set samples.

[0034] Step 3: Input the training samples in step 2 into the VGGNet16 network, and add a multi-scale feature extraction module structure at the front end of the VGGNet16 network as the first layer of the network. This module can more completely retain the features of the expression image at the beginning of the network. Figure 2As shown in the figure, three convolution modules are connected in parallel, including 1×1 and 3×3 convolutions in series, 1×1 and 5×5 convolutions in series, and 1×1 and 7×7 convolutions in series. The three parallel modules are fused to extract features. Since the receptive fields obtained when convolution kernels of different scales are convolved are different, the extracted image information will also be different. Therefore, the image information extracted by combining all the receptive fields of convolution kernels of different sizes in parallel can extract more complete image feature information. A 1×1 convolution structure is used in each convolution kernel branch to reduce the dimension of the feature map of each parallel branch network, and then 3×3, 5×5 and 7×7 convolutions are performed in low dimensions to extract different feature information. At this time, the number of parameters is greatly reduced, and it avoids the situation where only a single convolution kernel is used in the initial part of the network, resulting in incomplete image feature extraction and loss of some key features.

[0035] Step 4: Add a shortcut connection between the second convolution group and the third convolution group of the improved VGGNet16 network in step 3 before the fully connected layer to form a branch feature fusion module, which fuses the features of different stages together and enables the network to learn more features.

[0036] Step 5: Use batch normalization after each convolutional layer of the improved VGGNet16 network in steps 3 and 4 to speed up the convergence rate of the network model and increase the generalization of the model. Batch normalization performs batch normalization on the network input data according to each batch, normalizing each batch of sample data to a standard normal distribution with a mean of 0 and a variance of 1. The following is the calculation method and specific steps for a single batch of the batch normalization layer:

[0037] Calculate the mean of each batch of input data:

[0038]

[0039] Calculate the variance of each batch of input data:

[0040]

[0041] Normalize the input data to constrain the input data to follow a normal distribution:

[0042]

[0043] Translate and scale the input data using a linear transformation:

[0044]

[0045] in, m refers to the number of samples in a single batch, x i is a single batch of input sample data. is the sample mean of a single batch, is the sample variance of a single batch, y i is a single batch output sample data, and are two trainable linear transformation parameters, which make the new data distribution closer to the real distribution of samples and improve the network's expressiveness. Figure 3 Shown is a data comparison chart after batch normalization of the input data.

[0046] Step 6: Use Dropout after each fully connected layer on the improved VGGNet16 network of steps 3, 4, and 5. During the training process, a part of neurons are probabilistically discarded. They are usually added between fully connected layers. This changes the connection mode of the network to a certain extent and destroys the interaction between feature information. This is equivalent to adding some noise to enhance the generalization of the model. Figure 4 This is the Dropout structure diagram. The specific operation is to randomly discard some neurons in each iteration of the training process. These neurons no longer participate in the forward propagation process and the backward propagation process, that is, the weights and bias parameters of these neurons are in a forced dormant state. This prevents the network from over-learning some neurons with interference information, effectively improves the generalization of the network model, and greatly reduces the number of network parameters.

[0047] Step 7: The features extracted by the improved VGGNet16 network in steps 3, 4, 5, and 6 are passed through a linear fully connected layer to obtain the criticality weight, and then the attention weight is mapped to between 0 and 1 through the Sigmoid activation function; F =[ x 1, x 2, . . . , x n ]for n The face input feature map of the picture, the module formula is:

[0048]

[0049] in It is i The criticality weight of each sample, represents the fully connected layer parameters of the self-attention module, is the Sigmoid activation function.

[0050] Step 8: The criticality weights obtained in step 7 are weighted by the cross entropy loss function used by the original network model using the logit model to avoid misleading the network learning by non-critical samples. V jFor the j classifiers, is the weighted loss function, the formula is as follows:

[0051] ;

[0052] Step 9, compare the criticality weights obtained in step 7, then arrange all weights from high to low, and divide the criticality weight values ​​into critical parts and non-critical parts according to a certain ratio.

[0053] Step 10, take the average of the weight values ​​of the key part and the non-key part obtained in step 9, and constrain the mean of the key part to be higher than the mean of the non-key part by a threshold through sorting regularization. The loss function formula of this module is:

[0054]

[0055] in D Indicates a certain threshold range. This parameter can be adaptively determined by the network or specified manually. is the sample mean of the key part, is the mean of the non-critical part samples, N is the number of samples in a batch, M is the number of samples in the key part. Finally, the two loss functions of the self-attention module and the sorting regularization module are combined to obtain the final loss function of the self-correction network:

[0056]

[0057] Step 11: Subtract the maximum predicted probability from the predicted probability of the original label of the sample through the re-labeling module. If the difference between the two is greater than the given range parameter value, the sample will be re-labeled and its maximum predicted probability will be used as the correct label of the re-labeling. The formula of this module is as follows:

[0058]

[0059] in, represents the modified pseudo-label, D 2 is the given range parameter, P max represents the maximum predicted probability of the sample, P gt is the original label prediction probability of the sample, y max and y t is the index of the maximum predicted probability and the original label.

[0060] Step 12, debug the network structure hyperparameters from step 3 to step 11, use the stochastic gradient descent optimizer, the training is divided into 200 batches, the initial learning rate is 0.01, starting from the 15th iteration, and decrease every ten rounds. During the training process, the self-correction network module divides the training images into key and non-key samples in a ratio of 7:3. And set the threshold of the mean of the key and non-key samples to 0.15. The ratio between the cross entropy weighted loss function and the sorting regularization loss function in the overall loss function is 1:1. After training to the 80th batch, the re-labeling module is added. The default difference range parameter between the maximum predicted probability and the original label probability in the re-labeling module is 0.2, and the final training model is obtained;

[0061] Step 13, input the training set in step 1 into the training model in step 12 to obtain the expression recognition image result.

[0062] Although the above describes an illustrative specific embodiment of the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiment. For ordinary technicians in this technical field, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A multi-feature fusion expression recognition method, characterized in that It includes five parts: data set preprocessing, multi-scale feature extraction and fusion of VGGNet16 network, adding batch normalization and Dropout modules, adding a self-correction network to the back end of the network, and network training and testing: The first part consists of two steps: Step 1: Download the RAF-DB dataset of facial expressions in natural scenes, and then select 12,271 single-label training set images and 3,068 test set images from the dataset as model training and test samples; Step 2: determine whether the image is a grayscale image. If not, convert it into a grayscale image to minimize the impact of lighting on facial expression recognition. Then transform its size to 96×96, and perform basic operations such as rotation and translation on the image to enhance the training sample set and form the final training set sample. The second part consists of two steps: Step 3: Input the training samples in step 2 into the VGGNet16 network, add a multi-scale feature extraction network structure to the front end of the VGGNet16 network as the first layer of the network, use three convolution kernels of different sizes to extract features from the original image, and perform feature fusion on the multiple features extracted by the three convolution kernels of different sizes. Compared with using only a 3×3 convolution kernel, this structure loses less information and can retain more complete feature information. Step 4: Add a shortcut connection between the second convolution group and the third convolution group of the improved VGGNet16 network in step 3 before the fully connected layer to form a branch feature fusion module, which fuses the features of different stages together and enables the network to learn more features; The third part consists of two steps: Step 5: Use batch normalization after each convolutional layer of the improved VGGNet16 network in steps 3 and 4 to speed up the convergence rate of the network model and increase the generalization of the model; Step 6: Use Dropout after each fully connected layer of the improved VGGNet16 network in steps 3, 4, and 5 to discard some neurons and add some interference to increase the robustness of the model and reduce the number of model parameters; Part 4 consists of five steps: Step 7: The features extracted by the improved VGGNet16 network in steps 3, 4, 5, and 6 are weighted by the self-attention module, and the key samples and non-key samples are distinguished by the weight value, and the key samples are given priority, so as to increase the weight value of the network for the key samples; Step 8: The criticality weight obtained in step 7 is weighted by the cross entropy loss function used by the original network model using the logit model to avoid misleading the network learning by non-critical samples; Step 9, compare the criticality weights obtained in step 7 through the sorting regularization module, then sort all the weights from high to low, and divide the criticality weight values ​​into critical parts and non-critical parts according to a certain ratio; Step 10, average the weight values ​​of each part in step 9, and constrain the mean of the key part to be higher than the mean of the non-key part by a threshold through sorting regularization. The two loss functions of the self-attention module and the sorting regularization module are combined to obtain the final loss function of the self-correction network; Step 11: Subtract the maximum predicted probability from the predicted probability of the original label of the sample through the re-labeling module for the non-critical part samples obtained in step 9. If the difference between the two is greater than the given range parameter value, the sample will be re-labeled, and its maximum predicted probability will be used as the correct label of the re-labeled sample; Part 5 consists of two steps: Step 12, debug the network structure hyperparameters from step 3 to step 11, and obtain the final training model; Step 13, input the training set in step 1 into the training model in step 12 to obtain the expression recognition image result.

2. The method for facial expression recognition based on multi-feature fusion according to claim 1, characterized in that: The multi-scale feature extraction network structure of step 3 is used to retain more complete feature information.

3. The expression recognition method of multi-feature fusion according to claim 1, characterized in that: In step 5, batch normalization is used after each convolutional layer to speed up the convergence rate of the network model and increase the generalization of the model.

4. The method for facial expression recognition based on multi-feature fusion according to claim 1, characterized in that: Using step 6, use Dropout after each fully connected layer to discard some neurons and add some interference to increase the robustness of the model and reduce the number of model parameters.

5. The method for facial expression recognition based on multi-feature fusion according to claim 1, characterized in that In steps 7, 8, 9, 10, and 11, a self-correcting network is used to suppress the impact of sample labeling uncertainty on network model training.

Images