Face Expression Recognition Method Based on Attention Images

Abstract

The present invention discloses a method for facial expression recognition based on attention images, which mainly solves the problems that the key facial regions mined in the prior art are inaccurate and relatively rough. The present invention generates attention images with the same resolution and the same labels as each facial expression image for mining the key regions of the face. The present invention jointly trains a convolutional neural network using the facial expression image and its corresponding label, the attention image and its corresponding label. The trained network of the present invention pays more attention to the key region pixels in the facial expression image, can achieve automatic positioning of the key facial regions at the pixel level, and improves the recognition accuracy of the facial expression image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and further relates to a facial expression recognition method based on attention images in the field of image recognition technology. The present invention can be applied to recognize the facial expression categories of human faces in many complex scenarios such as intelligent education, assisted medical treatment, and vehicle safety. Background Art

[0002] Facial expression recognition refers to using a computer to extract the features of facial expression images, and combining the existing prior knowledge of humans to perform feature modeling, mining the relationship between facial expression images and emotions, so as to recognize the categories of facial expressions. Facial expressions can effectively express personal emotions and are an intuitive reflection of human emotions. Therefore, facial expression recognition technology has a wide range of applications in artificial intelligence fields such as intelligent education, assisted medical treatment, and vehicle safety. Due to the characteristics of large intra-class differences and small inter-class differences in the facial expression recognition task, the performance of the facial expression recognition task is not very ideal, which requires the network to be able to well mine the key regions of the human face.

[0003] Yong Li et al. proposed a convolutional neural network model based on an attention mechanism in their published paper "Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism" (IEEE Transactions on Image Processing: 2439–2450, 2019) to perceive the occluded regions of the human face and focus on the most discriminative unoccluded regions. The implementation steps of this method are: collecting facial expression images and performing facial key point detection on them; using the facial expression images as input data and using a convolutional neural network to extract features from them to generate corresponding feature maps; performing global feature encoding on the feature maps and learning the importance weights through an attention network; decomposing the feature maps into 24 local blocks according to the facial key points, performing local feature encoding on each local block and learning the importance weights through an attention network; fusing the obtained local features and global features according to their corresponding importance weights and inputting them into a classifier to recognize the categories of facial expressions. The disadvantages of this method are: the mining of the key regions of the human face depends on the detection of facial key points. If the detection effect of facial key points is not good, it will lead to inaccurate mining of the key regions of the human face and is prone to misjudging the facial expression categories of facial images.

[0004] Harbin University of Science and Technology discloses a face expression recognition method based on an attention mechanism in its patent document "A Face Expression Recognition Method Based on an Attention Mechanism" (Application No.: 202110663990.7, Publication No.: CN 113392766 A). The implementation steps of this method are as follows: collect a data set and preprocess the data set; label the key points of facial features for each face expression image; crop the image at key positions and scale the cropped image; input the obtained image and the whole face expression image into a neural network for the recognition and acquisition of local features and global features; fuse the obtained features and input them into a classifier for the recognition of face expression categories. The deficiencies of this method are as follows: this method mines the key areas of the face by cropping the face image into blocks, and the mining of key areas is based on image blocks rather than individual pixel points, resulting in relatively rough mined key areas and affecting the recognition accuracy of face expression images. Summary of the Invention

[0005] The object of the present invention is to propose a face expression recognition method based on an attention image for the deficiencies of the above-mentioned prior art, so as to solve the problems that the mined key areas of the face are not accurate enough and the mined key areas of the face are relatively rough in the prior art when mining the key areas of the face.

[0006] To achieve the above object, the idea of the present invention is to generate an attention image with the same resolution and the same label as each face expression image. The attention image only contains the pixel points of the key areas of the face expression image. Using the attention image to mine the key areas of the face can overcome the problem that the mined key areas of the face are not accurate enough due to the dependence on the face key point detection technology in the prior art when mining the key areas of the face, and improve the recognition accuracy of face expression images. The present invention jointly trains a convolutional neural network with the face expression image and its corresponding label, the attention image of the image and its corresponding label, so that the network pays more attention to the pixel points of the key areas in the face expression image, realizes the automatic positioning of the key areas of the face at the pixel level, and solves the problem that the mined key areas of the face are relatively rough in the prior art.

[0007] The specific steps to achieve the object of the present invention are as follows:

[0008] Step 1, generate a training set: [[ID=!6]]

[0009] Step 1.1, collect at least 10,000 face expression images to form a sample set. The sample set includes at least 3 emotional categories of different face expressions, and at least 280 images are collected for each emotional category;

[0010] Step 1.2, perform bilinear sampling on each image in the sample set with a sampling resolution of 224×224, normalize the sampled image, and form a training set by combining all the normalized facial expression images and their corresponding labels;

[0011] Step 2, build a convolutional neural network and set the parameters of the convolutional neural network;

[0012] Step 3, use the convolutional neural network to generate corresponding attention images for each facial expression image in the training set:

[0013] Step 3.1, input a randomly selected facial expression image from the training set into the convolutional neural network, and output the probability value that the selected image belongs to the true label and the feature map of the selected image;

[0014] Step 3.2, calculate the weight of each channel in the feature map of the selected image according to the following formula:

[0015]

[0016] where a k represents the weight of the k-th channel A k in the feature map of the selected image, m and n respectively represent the total number of elements of channel A k in the vertical height and horizontal width directions, i and j respectively represent the element numbers of channel A k in the vertical height and horizontal width directions, and h represents the probability value that belongs to the true label after inputting the selected image into the convolutional neural network;

[0017] Step 3.3, perform weighted summation on all channels in the feature map to obtain the weighted feature map of the selected image;

[0018] Step 3.4, calculate the mean value of all elements in the weighted feature map, and set the elements in the weighted feature map that are less than or equal to the mean value to 0 to obtain the importance matrix of the selected image;

[0019] Step 3.5, normalize the importance matrix, and scale the normalized importance matrix to the same size as the selected image to obtain the attention matrix of the selected image;

[0020] Step 3.6, multiply the selected image by its attention matrix to obtain the attention image of the selected image, and use the label of the selected image as the label of the attention image;

[0021] Step 3.7, determine whether all facial expression images in the training set have been selected. If so, execute Step 4; otherwise, execute Step 3.1;

[0022] Step 4, train the convolutional neural network:

[0023] Input the images in the training set, their corresponding labels, the attention images of the images, and their corresponding labels into the convolutional neural network in batches. Use the Adam optimizer to optimize the training process. Through the gradient descent algorithm, iteratively update the parameters of each layer in the convolutional neural network until the cross-entropy loss function of the convolutional neural network converges, and obtain the trained convolutional neural network.

[0024] Step 5, identify the expression category in the face image:

[0025] Adopt a sampling resolution of 224×224, perform bilinear sampling on each face image to be recognized, perform normalization processing on the sampled image, and input the normalized image into the trained convolutional neural network to output the expression category of the face image.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] First, the present invention generates an attention image with the same resolution and the same label as each face expression image to mine the key areas of the face, overcomes the deficiency of the prior art in that the mined key areas of the face are not accurate enough, and enables the present invention to improve the recognition accuracy of face expression images.

[0028] Second, the present invention jointly trains the convolutional neural network using the face expression image and its corresponding label, the attention image and its corresponding label, solves the defect that the mined key areas of the face in the prior art are relatively rough, enables the trained network of the present invention to pay more attention to the key area pixels in the face expression image, and realizes the automatic positioning of the key areas of the face at the pixel level. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart for implementing the present invention;

[0030] Figure 2 is an effect diagram of the attention image generated by the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] The following further describes the present invention in detail with reference to the drawings and embodiments.

[0032] Refer to Figure 1 and the embodiments to further describe the implementation steps of the present invention in detail.

[0033] Step 1, generate a training set and a test set.

[0034] Step 1.1, in the embodiment of the present invention, seven types of facial expression images labeled as angry, disgusted, fearful, happy, sad, surprised, and neutral are collected from the outdoor facial expression dataset RAF-DB. At least 350 images are collected for each emotion category, and a total of 15,539 facial expression images are collected to form a sample set.

[0035] Step 1.2, using a sampling resolution of 224×224, perform bilinear sampling on each image in the sample set, and perform normalization processing on the sampled images to obtain a normalized sample set.

[0036] Step 1.3, randomly select 12,271 facial expression images from the normalized sample set and their corresponding labels to form a training set, with at least 280 images for each emotion category; the remaining 3,068 facial expression images and their corresponding labels form a test set, with at least 70 images for each emotion category.

[0037] Step 2, build a convolutional neural network and set the parameters of the convolutional neural network.

[0038] Step 2.1, build a convolutional neural network, whose structure is in turn: input layer, first convolutional layer, second convolutional layer, first pooling layer, third convolutional layer, fourth convolutional layer, second pooling layer, fifth convolutional layer, sixth convolutional layer, seventh convolutional layer, third pooling layer, eighth convolutional layer, ninth convolutional layer, tenth convolutional layer, fourth pooling layer, eleventh convolutional layer, twelfth convolutional layer, thirteenth convolutional layer, fifth pooling layer, flattening layer, first fully connected layer, dropout layer, second fully connected layer, output layer.

[0039] Step 2.2, set the parameters of the convolutional neural network as follows:

[0040] Set the dimension size of the input layer to b×224×224×3, where b represents the number of samples selected for one input of the convolutional neural network. In the embodiment of the present invention, b is set to 128 during the training phase;

[0041] Set the number of convolutional kernels of the first to thirteenth convolutional layers to 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512 in turn, the convolutional kernel size is set to 3×3, the stride is set to 1, and the activation function uses the rectified linear unit;

[0042] Set the pooling window of the first to fifth pooling layers to 2×2, and the stride to 2;

[0043] The flattening layer uses the flatten function to stretch the input matrix into a vector;

[0044] Set the number of nodes in the first fully connected layer to 512, and use the leaky rectified linear unit as the activation function; set the number of nodes in the second fully connected layer to 7;

[0045] The dropout layer uses the dropout function to set each neuron to 0 with probability p. In the embodiment of the present invention, p = 0.3;

[0046] In the training stage, the output result of the output layer consists of the probability value that the input image belongs to the true label and the feature map of the input image. In the test stage, the output layer only outputs the predicted result of the expression category of the input image.

[0047] Step 3, use the convolutional neural network to generate corresponding attention images for each face expression image in the training set.

[0048] Step 3.1, input a randomly selected face expression image from the training set into the convolutional neural network, and output the probability value that the selected image belongs to the true label and the feature map of the selected image.

[0049] Step 3.2, calculate the weight of each channel in the feature map of the selected image according to the following formula:

[0050]

[0051] where ak represents the weight of the k-th channel Ak in the feature map of the selected image, m and n respectively represent the total number of elements in the vertical height and horizontal width directions of the channel Ak, i and j respectively represent the element serial numbers in the vertical height and horizontal width directions of the channel Ak, and k represents the probability value that the selected image belongs to the true label after being input into the convolutional neural network. In the embodiment of the present invention, m = n = 7.

[0052] Step 3.3, perform weighted summation on all channels in the feature map according to the following formula to obtain the weighted feature map of the selected image:

[0053]

[0054] where Map represents the weighted feature map of the selected image, D represents the total number of channels of the feature map, k represents the channel serial number of the feature map, ak represents the weight of the k-th channel Ak in the feature map. In the embodiment of the present invention, D = 512.

[0055] Step 3.4, calculate the mean value of all elements in the weighted feature map, and set the elements in the weighted feature map that are less than or equal to the mean value to 0 to obtain the importance matrix of the selected image.

[0056] Step 3.5, perform normalization processing on the importance matrix, and scale the normalized importance matrix to the same size as the selected image to obtain the attention matrix of the selected image.

[0057] Step 3.6: Multiply the selected image by its attention matrix to obtain the attention image of the selected image, and set the label of the attention image to be the same as that of the selected image.

[0058] Step 3.7: Determine whether all face expression images in the training set have been selected. If so, execute Step 4; otherwise, execute Step 3.1.

[0059] Step 4: Train the convolutional neural network.

[0060] Input the images in the training set, their corresponding labels, the attention images of the images, and their corresponding labels into the convolutional neural network batch by batch. Use the Adam optimizer to optimize the training process. Through the gradient descent algorithm, iteratively update the parameters of each layer in the convolutional neural network until the cross-entropy loss function of the convolutional neural network converges, and obtain the trained convolutional neural network.

[0061] In the embodiment of the present invention, the training set is divided into batches of 64 images each.

[0062] The cross-entropy loss function is as follows:

[0063]

[0064] where log represents the logarithm operation with base 10, h i represents the probability value belonging to the true label output by the i-th image in the training set after passing through the convolutional neural network, represents the probability value belonging to the true label output by the convolutional neural network for the attention image corresponding to the i-th image in the training set.

[0065] Step 5: Identify the expression category in the face image.

[0066] Divide the test set into batches of 64 images each, and input them into the trained convolutional neural network batch by batch to obtain the expression category corresponding to each face image.

[0067] The effect of the present invention can be further demonstrated by the following simulation.

[0068] 1. Simulation experiment conditions.

[0069] The hardware platform for the simulation experiment of the present invention is: the graphics processor is GeForce GTX 2080Ti GPU with a video memory of 11G.

[0070] The software platform for the simulation experiment of the present invention is: Windows 10 operating system and python 3.6, TensorFlow deep learning development framework.

[0071] The data for the simulation experiment of the present invention is collected from two outdoor face expression datasets, RAF-DB and AffectNet.

[0072] Collect seven types of face expression images labeled as angry, disgusted, fearful, happy, sad, surprised, and neutral from the outdoor face expression dataset RAF-DB. At least 70 images are collected for each emotion category, and a total of 3068 face expression images are collected to form a sample set. Using a sampling resolution of 224×224, perform bilinear sampling on each image in the sample set, and perform normalization processing on the sampled images. Combine all the normalized face expression images and their corresponding labels to form the test set of the RAF-DB dataset in the simulation experiment of the present invention.

[0073] Collect seven types of face expression images labeled as angry, disgusted, fearful, happy, sad, surprised, and neutral from the outdoor face expression dataset AffectNet. 500 images are collected for each emotion category, and a total of 3500 face expression images are collected to form a sample set. Using a sampling resolution of 224×224, perform bilinear sampling on each image in the sample set, and perform normalization processing on the sampled images. Combine all the normalized face expression images and their corresponding labels to form the test set of the AffectNet dataset in the simulation experiment of the present invention.

[0074] 2. Simulation content and result analysis:

[0075] Simulation experiment 1 of the present invention is to use the network trained by the present invention and the network trained by the prior art to classify the face images in the two test sets in the simulation conditions respectively, and obtain two classification results for each method.

[0076] The prior art refers to a convolutional neural network model based on an attention mechanism proposed by Yong Li et al. in their published paper "Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism" (IEEE Transactions on Image Processing: 2439–2450, 2019).

[0077] In order to evaluate the effect of the simulation of the present invention, use the following classification accuracy formula to evaluate the two classification results of each method in the simulation experiment of the present invention respectively. The evaluation results are shown in Table 1.

[0078]

[0079] Table 1. Precision comparison table of classification results between the present invention and the prior art

[0080] Data set Prior art (%) The present invention (%) 1. RAF-DB 85.07 86.68 2. AffectNet 58.78 59.08

[0081] As can be seen from Table 1, the classification accuracies of the present invention in the RAF-DB and AffectNet test sets are 86.68% and 59.08% respectively, both higher than those of the prior art, proving that the present invention can obtain better classification accuracy for facial expression images.

[0082] The following will further describe the attention images generated by the convolutional neural network before training and the trained convolutional neural network respectively for a training picture in the embodiments of the present invention with reference to Figure 2 .

[0083] Figure 2 (a) is a training picture in the embodiments of the present invention, Figure 2 (b) is an attention image with the same resolution as the training picture generated by the convolutional neural network before training, Figure 2 (c) is an attention image with the same resolution as the training picture generated by the trained convolutional neural network.

[0084] Figure 2 (b) and Figure 2 In (c), the change of pixel point values between 0 and 255 is expressed by the color from black to white. Visually observing the pixel points in Figure 2 (b) and Figure 2 (c), if the color of the pixel point is closer to black, it means that the pixel point corresponding to the pixel point at the corresponding position in Figure 2 (a) is a non-critical pixel point; if the color of the pixel point is closer to white, it means that the pixel point corresponding to the pixel point at the corresponding position in Figure 2 (a) is a critical pixel point.

[0085] Combined with Figure 2 (b) and Figure 2 (c), it can be seen that Figure 2 In (b), the key areas such as the forehead, the corners of the eyes and the corners of the mouth with large facial movement changes in Figure 2 (a) are all close to black, indicating that Figure 2 (b) fails to detect the key areas in Figure 2 (a). While Figure 2 In (c), the key areas such as the forehead, the corners of the eyes and the corners of the mouth with large facial movement changes in Figure 2 (a) are all close to white, indicating that Figure 2 (c) better detects the key areas in Figure 2 (a), proving that the method for facial expression recognition based on attention images proposed by the present invention can accurately detect the key areas of the human face.

Claims

1. A face expression recognition method based on attention images, characterized in that The specific steps of this recognition method are as follows: Step 1, generate a training set; Step 2, build a convolutional neural network and set the parameters of the convolutional neural network; Step 3, use the convolutional neural network to generate corresponding attention images for each facial expression image in the training set: Step 3.1, input a randomly selected facial expression image from the training set into the convolutional neural network, and output the probability value that the selected image belongs to the true label and the feature map of the selected image; Step 3.2, calculate the weights of each channel in the feature map of the selected image according to the following formula: ； Among them, represents the weight of the th channel in the feature map of the selected image, and respectively represent the total number of elements of the channel in the vertical height and horizontal width directions, and respectively represent the element sequence numbers of the channel in the vertical height and horizontal width directions, represents the probability value belonging to the true label output after inputting the selected image into the convolutional neural network; Step 3.3, perform weighted summation on all channels in the feature map to obtain the weighted feature map of the selected image; Step 3.4, calculate the mean value of all elements in the weighted feature map, set the elements in the weighted feature map that are less than or equal to the mean value to 0, and obtain the importance matrix of the selected image; Step 3.5, perform normalization processing on the importance matrix, and scale the normalized importance matrix to the same size as the selected image to obtain the attention matrix of the selected image; Step 3.6, multiply the selected image by its attention matrix point by point to obtain the attention image of the selected image, and use the label of the selected image as the label of the attention image; Step 3.7, determine whether all facial expression images in the training set have been selected. If so, execute Step 4; otherwise, execute Step 3.1; Step 4, train the convolutional neural network; Step 5, use the trained convolutional neural network to identify the expression categories in facial images.

2. The method for facial expression recognition based on attention images according to claim 1, wherein The generation of the training set described in Step 1 means collecting at least 10,000 facial expression images to form a sample set. The sample set includes at least 3 emotional categories of different facial expressions, and at least 280 images are collected for each emotional category; using a sampling resolution of 224×224, perform bilinear sampling on each image in the sample set, perform normalization processing on the sampled images, and form a training set with all the normalized facial expression images and their corresponding labels.

3. The method for facial expression recognition based on attention images according to claim 1, wherein The structure of the convolutional neural network described in Step 2 is as follows: input layer, first convolutional layer, second convolutional layer, first pooling layer, third convolutional layer, fourth convolutional layer, second pooling layer, fifth convolutional layer, sixth convolutional layer, seventh convolutional layer, third pooling layer, eighth convolutional layer, ninth convolutional layer, tenth convolutional layer, fourth pooling layer, eleventh convolutional layer, twelfth convolutional layer, thirteenth convolutional layer, fifth pooling layer, flattening layer, first fully connected layer, dropout layer, second fully connected layer, output layer.

4. The method for facial expression recognition based on attention images according to claim 1, characterized in that, The parameters of the convolutional neural network described in Step 2 are set as follows: Set the dimension size of the input layer to ×224×224×3, where represents the number of samples selected for a single input of the convolutional neural network, 32; The number of convolutional kernels of the first to thirteenth convolutional layers is set to 64, 64, 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512 in sequence. The size of the convolutional kernels is set to 3×3, the stride is set to 1, and the rectified linear unit is used as the activation function for all; The pooling windows of the first to fifth pooling layers are all set to 2×2, and the stride is set to 2; The flattening layer uses the flatten function to stretch the input matrix into a vector; Set the number of nodes in the first fully connected layer to 512, and use the leaky rectified linear unit as the activation function; Set the number of nodes in the second fully connected layer to , 3 and equal to the number of label categories in the training set; The random inactivation layer uses the dropout function to set each neuron to 0 with probability 0, 0 1.

5. The method for facial expression recognition based on attention images according to claim 1, wherein The weighted sum of all channels in the feature map described in step 3.3 is obtained by the following formula: ； Among them, represents the weighted feature map of the selected image, D represents the total number of channels of the feature map, represents the channel number of the feature map, represents the th channel in the feature map.

6. The method for facial expression recognition based on attention images according to claim 1, wherein Training the convolutional neural network described in step 4 means that the images in the training set, their corresponding labels, the attention images of the images, and their corresponding labels are input into the convolutional neural network in batches. The Adam optimizer is used to optimize the training process. Through the gradient descent algorithm, the parameters of each layer in the convolutional neural network are iteratively updated until the cross-entropy loss function of the convolutional neural network converges, and a trained convolutional neural network is obtained.

7. The method for facial expression recognition based on attention images according to claim 6, characterized in that The cross-entropy loss function is as follows: ； Among them, represents the logarithmic operation with base 10, represents the probability value belonging to the true label output by the convolutional neural network for the th image in the training set, represents the probability value belonging to the true label output by the convolutional neural network for the attention image corresponding to the th image in the training set.

8. The method for facial expression recognition based on attention images according to claim 1, characterized in that Identifying the expression category in the face image described in step 5 means that with a sampling resolution of 224×224, bilinear sampling is performed on each face image to be identified, the sampled image is normalized, and the normalized image is input into the trained convolutional neural network to output the expression category of the face image.

Images