A facial expression recognition method based on expression transfer

Abstract

This invention discloses a facial expression recognition method based on expression transfer, belonging to the field of deep learning and image processing technology. This invention improves the accuracy of expression recognition tasks by extracting high-purity expression-related features. The method involves: extracting global feature vectors from expression images using existing mature networks, and using a decomposer to decompose the global feature vectors into feature vector A and feature vector B; performing expression transfer training, transforming feature vector A into expression-related features and feature vector B into expression-independent features; and using the expression-related features for expression recognition, outputting the expression category. This invention leverages the auxiliary task of expression transfer to decompose the original expression feature vector containing interference information into expression-related feature vectors and expression-independent feature vectors. The expression-related feature vectors are of high quality, which is more conducive to expression recognition tasks.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and image processing, specifically to a facial expression recognition method based on expression transfer. Background Technology

[0002] Common facial expressions include calmness, happiness, anger, disgust, sadness, fear, and surprise. These expressions effectively reflect a person's psychological and physiological state, leading to the widespread application of facial expression recognition in fields such as psychology, video games, intelligent surveillance, and vehicle security. With the development of deep learning, facial expression recognition methods based on convolutional neural networks have gradually emerged. The traditional approach involves using mature networks (such as ResNet and GoogleNet) to extract features from expression images and then using these features for expression recognition. Therefore, the quality of the features determines the accuracy of the recognition.

[0003] However, facial expression images contain a large amount of interfering information unrelated to the expression itself, making it difficult to extract high-quality features. This interfering information can confuse the network during training; for example, facial features such as wrinkles are often mistaken for certain facial expressions, causing the network to misclassify the target expression as a similar expression from another category. Interfering information also reduces the weight of micro-expression features, making micro-facial movements harder for the network to perceive. For instance, during convolution, subtle micro-facial movements are erased, resulting in them having a very small weight in the features extracted by the network. Summary of the Invention

[0004] This invention provides a facial expression recognition method based on expression transfer, which improves the accuracy of expression recognition by extracting high-purity, high-quality features from expression images.

[0005] The technical solution adopted in this invention is as follows:

[0006] A facial expression recognition method based on expression transfer, the method includes the following steps:

[0007] Step 1: Construct a facial expression recognition image dataset, preprocess the images in the image dataset to obtain sample images of a specified size (e.g., 224*224*3), set corresponding expression category labels for each sample image, and use a specified number of sample images as the training set;

[0008] Step 2, Construct the network model:

[0009] A decomposer is connected after the selected first feature extraction network for image classification to decompose the image features extracted by the feature extraction network into two sub-feature vectors;

[0010] Construct a network with the same structure as the first feature extraction network as the second feature extraction network, and connect a fully connected layer after the second feature extraction network as a discriminator to output the probability of the image being real or fake;

[0011] A decoding network is selected as the generator to generate new facial expression images;

[0012] Step 3: Train the network model built in Step 2:

[0013] All sample images in the training set are input into the feature extraction network, and the decomposer obtains two sub-feature vectors for each sample image: feature vectors A and B.

[0014] Input all sample images in the training set into the discriminator, set the discrimination label of all sample images to true, and fine-tune the network parameters of the discriminator based on the binary cross-entropy loss. Stop when the binary cross-entropy loss meets the specified condition.

[0015] For each sample image in the training set, a sample image is randomly selected from the training set as its reference image. The feature vector A of the sample image in the training set is added to the feature vector B of the reference image and then input into the generator to output a new sample image.

[0016] The new sample image is then input into the optimized discriminator, and the discrimination label of the new sample image is set to pseudo. The network parameters of the discriminator, generator, and first and second feature extraction networks are optimized based on the binary cross-entropy loss. When the expression transfer is detected to be successful, the optimization is stopped and step 4 is executed.

[0017] Among them, successful expression transfer means that the expression in the new sample image is consistent with that in the original sample image, and the facial features are consistent with those in the reference image.

[0018] Step 4: Add a fully connected layer with a dimension equal to the number of expression categories after the output feature vector A of the decomposer. The expression recognition model is composed of the first feature extraction network, the branch of the decomposer output feature vector A, and a fully connected layer.

[0019] Input all sample images in the training set into the facial expression recognition model, and fine-tune the network parameters of the facial expression recognition model based on multi-class cross-entropy loss. Stop when the classification accuracy meets the specified conditions to obtain the trained facial expression recognition model.

[0020] Step 5: Perform image preprocessing on the image to be recognized so that the preprocessed image to be recognized matches the input of the facial expression recognition model;

[0021] The preprocessed image to be recognized is then input into the trained facial expression recognition model, and the facial expression recognition result is obtained based on its output.

[0022] Furthermore, in step 1, the preprocessing is either upsampling or downsampling. Upsampling uses cubic interpolation, while downsampling uses alternating rows and columns based on the target size and the original size.

[0023] Furthermore, in step 2, the first feature extraction network is the first to second-to-last layers of ResNet18, and the initialization parameters are selected from AlexNet parameters, and the parameters are not frozen.

[0024] Furthermore, in step 2, the decomposer comprises one output layer, one hidden layer, and two output layers. The input layer is the output layer of the first feature extraction network with a dimension of 512. The hidden layer is a fully connected layer with a dimension of 256. Both output layers are fully connected to the hidden layer and have a dimension of 128. That is, the dimension of the hidden layer is half that of the input layer, and the dimensions of the two output layers are half that of the hidden layer.

[0025] Furthermore, the initialization parameters of the second feature extraction network are the same as those of the first feature extraction network.

[0026] Furthermore, in step 2, the generator adopts the generator structure of AdaGAN.

[0027] Furthermore, in step 3, when optimizing the network parameters of the discriminator, generator, and first and second feature extraction networks based on binary cross-entropy loss (expression transfer training), only the network parameters of the discriminator are updated first, and then the network parameters of the feature extraction network, discriminator, and generator are updated together.

[0028] Furthermore, in step 4, the network parameter tuning of the facial expression recognition model is stopped when the classification accuracy is greater than or equal to a specified threshold (preferably 0.95).

[0029] The technical solution provided by this invention brings at least the following beneficial effects:

[0030] This invention utilizes the auxiliary task of facial expression transfer to decompose the original facial expression feature vector containing interference information into facial expression-related feature vectors and facial expression-independent feature vectors. The facial expression-related feature vectors are of high quality and are more conducive to facial expression recognition tasks. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of a facial expression recognition method based on expression transfer provided by an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] Facial expressions effectively reflect a person's psychological and physiological state, leading to the widespread application of facial expression recognition in fields such as psychology, intelligent monitoring, and vehicle safety. However, facial expression images contain a large amount of interfering information, making high-precision facial expression recognition difficult to achieve. To address this, this invention proposes a facial expression recognition method based on expression transfer. By extracting high-purity expression-related features, the accuracy of facial expression recognition is improved. This method comprises three steps: 1) Extracting global feature vectors from facial expression images using existing mature networks, and then using a decomposer to decompose the global feature vectors into feature vector A and feature vector B; 2) Performing expression transfer training, transforming feature vector A into expression-related features and feature vector B into expression-independent features; 3) Using expression-related features to perform facial expression recognition and output the expression category.

[0035] like Figure 1 As shown in the figure, the facial expression recognition method based on expression transfer provided by this invention specifically includes the following steps:

[0036] Step 1: Construct the experimental dataset. Divide the experimental dataset into training and validation sets according to the principle of 90% training set and 10% validation set. For example, select the Extended Cohn-Kanade dataset or the Oulu-CASIA dataset.

[0037] Step 2: Resize all samples to 224*224*3 pixels to fit the input standard of the ResNet18 network. Upsampling uses cubic interpolation, and downsampling is performed by sampling every other row and column based on the target size and the original size.

[0038] Step 3: Build a ResNet18 network model for image feature extraction, using AlexNet weights and the feature extraction part of the original model, without freezing the network parameters;

[0039] Step 4: Add a decomposer after the network model. The input layer of the decomposer is the output of the feature extraction network model, with a dimension of 512. The hidden layer is a fully connected layer with a dimension of 256. The output layer consists of two fully connected layers with a dimension of 128. Each layer is activated using the RuLU function.

[0040] Step 5: Establish a discriminator to distinguish between genuine and fake images. The ResNet18 network structure is still used, with AlexNet weights and the feature extraction part of the original model. The network parameters are not frozen, and a fully connected layer with dimension 1 is added after the network.

[0041] Step 6: Build an image generator, using the generator structure in AdaGAN, with parameters initialized randomly and network parameter settings not frozen;

[0042] Step 7: Input all training set samples into the feature extraction network, and output feature vector A and feature vector B for each sample;

[0043] Step 8: Input all training set samples into the discriminator, set all sample labels to true, calculate the binary classification cross-entropy loss, and use the Adam optimizer to update the discriminator network parameters;

[0044] Step 9: For each sample image in the training set, randomly select one image from the training set as its reference image;

[0045] Step 10: Add the feature vector A of all training set sample images to the feature vector B of its reference image, and then input the sum into the generator to output a new image;

[0046] Step 11: Input the new image into the discriminator, set the label to pseudo, calculate the binary cross-entropy loss, and use the Adam optimizer to update the parameters of the discriminator, generator, and feature extraction network;

[0047] Step 12: Observe the new image. If the expression in the input image is successfully transferred to its reference image, it means that feature vector A has evolved into an expression-related vector, while feature vector B has evolved into an expression-independent vector. The expression transfer training is over, and proceed to step 13. Otherwise, proceed to step 7.

[0048] Step 13: Add a fully connected layer with a dimension equal to the number of expression categories after the output feature vector A of the feature extraction network and the decomposer. In this embodiment of the invention, the number of expression categories is 7.

[0049] Step 14: Input all training samples into the feature extraction network and output the expression category of each sample;

[0050] Step 15: Calculate the multi-class cross-entropy loss and update the feature extraction network parameters using the Adam optimizer;

[0051] Step 16: Calculate the classification accuracy. If the accuracy is higher than 0.95, end the training and go to step 17; otherwise, go to step 14.

[0052] Step 17: Use the trained feature extraction network to validate the validation set samples.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0054] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on expression transfer, characterized in that, Includes the following steps: Step 1: Construct a facial expression recognition image dataset, preprocess the images in the image dataset to obtain sample images of a specified size, set corresponding expression category labels for each sample image, and use a specified number of sample images as the training set; Step 2, Construct the network model: A decomposer is connected after the selected first feature extraction network for image classification to decompose the image features extracted by the feature extraction network into two sub-feature vectors; Construct a network with the same structure as the first feature extraction network as the second feature extraction network, and connect a fully connected layer after the second feature extraction network as a discriminator to output the probability of the image being real or fake; A decoding network is selected as the generator to generate new facial expression images; Step 3: Train the network model built in Step 2: All sample images in the training set are input into the feature extraction network, and the decomposer obtains two sub-feature vectors for each sample image: feature vectors A and B. Input all sample images in the training set into the discriminator, set the discrimination label of all sample images to true, and fine-tune the network parameters of the discriminator based on the binary cross-entropy loss. Stop when the binary cross-entropy loss meets the specified condition. For each sample image in the training set, a sample image is randomly selected from the training set as its reference image. The feature vector A of the sample image in the training set is added to the feature vector B of its reference image and then input into the generator to output a new sample image. The new sample image is then input into the optimized discriminator, and the discrimination label of the new sample image is set to pseudo. The network parameters of the discriminator, generator, and first and second feature extraction networks are optimized based on the binary cross-entropy loss. When the expression transfer is detected to be successful, the optimization is stopped and step 4 is executed. Among them, successful expression transfer means that the expression in the new sample image is consistent with that in the original sample image, and the facial features in the new sample image are consistent with those in the reference image. Step 4: Add a fully connected layer with a dimension equal to the number of expression categories after the output feature vector A of the decomposer. The expression recognition model is composed of the first feature extraction network, the branch of the decomposer output feature vector A, and a fully connected layer. Input all sample images in the training set into the facial expression recognition model, and fine-tune the network parameters of the facial expression recognition model based on multi-class cross-entropy loss. Stop when the classification accuracy meets the specified conditions to obtain the trained facial expression recognition model. Step 5: Perform image preprocessing on the image to be recognized so that the preprocessed image to be recognized matches the input of the facial expression recognition model; The preprocessed image to be recognized is then input into the trained facial expression recognition model, and the facial expression recognition result is obtained based on its output.

2. The method as described in claim 1, characterized in that, In step 1, the preprocessing is either upsampling or downsampling. Upsampling uses cubic interpolation, while downsampling uses alternating rows and columns based on the target size and the original size.

3. The method as described in claim 1, characterized in that, In step 2, the first feature extraction network is the first to second-to-last layers of ResNet18, and the initialization parameters are selected from AlexNet parameters, and the parameters are not frozen.

4. The method as described in claim 1, characterized in that, In step 2, the decomposer includes an output layer, a hidden layer, and two output layers. The input layer is the output layer of the first feature extraction network, and the dimension of the hidden layer is half that of the input layer. The dimensions of the two output layers are half that of the hidden layer.

5. The method as described in claim 1, characterized in that, The initialization parameters of the second feature extraction network are the same as those of the first feature extraction network.

6. The method as described in claim 1, characterized in that, In step 2, the generator adopts the generator structure of AdaGAN.

7. The method as described in claim 1, characterized in that, In step 4, network parameter tuning of the facial expression recognition model is stopped when the classification accuracy is greater than or equal to the specified threshold.

Images