Facial expression recognition method based on face sub-blocks

Abstract

A face part block-based expression recognition method comprises the following steps: 1) constructing a full convolution expression recognition network using face part blocks: a face feature extraction stage adopts two complementary convolution networks, one is an expression recognition network based on picture time information, used for extracting overall features of the whole face, and the other is an expression recognition network based on picture geometric information, used for extracting local features of face part blocks; 2) a training stage: training the network using an expression recognition data set, using a cross-entropy function as a loss function, and adopting an online difficult example sample retraining strategy in the process; 3) a testing stage: inputting a test image into the network to obtain an expression recognition result. In the feature extraction stage, the application fully utilizes picture information and face key point information, constructs two complementary small networks, extracts overall features of the face and local features of face part blocks, and improves the robustness of face expression recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and relates to image recognition in multimedia technology, and is an expression recognition method based on face partitioning. BACKGROUND

[0002] Expression recognition is to determine the facial expression of a person or multiple persons in a given still face image or dynamic face image sequence. Therefore, expression recognition systems can be generally divided into two categories, one is a static picture expression recognition system, and the other is a dynamic sequence expression recognition system. In the static picture expression recognition system, the network only considers the geometric features of a single picture, while the dynamic sequence expression recognition system also considers the time features of related frames. The specific research contents of the facial expression recognition system include face detection, facial expression feature extraction and facial expression recognition.

[0003] Face detection: that is, to detect the presence of a face and determine its position from various different scenes. This part of the system is generally completed in the data preprocessing stage. For the current expression dataset, it is a single face, and there is no multi-person task. The face detection currently has very mature algorithms, and the more detailed face key point detection is also included in the expression recognition task.

[0004] Facial expression feature extraction: that is, to determine the description method of the detected facial expression and the existing facial expression in the database. Similar to other fields of computer vision, the expression recognition system based on traditional methods uses manual features. The representative methods include local binary pattern (LBP) and non-negative matrix factorization (NMP). After 2013, large data sets such as FER2013 and EmotiW appeared, providing sufficient data volume for deep network training. At the same time, the outstanding performance of convolutional neural network in image recognition tasks makes more and more people use deep network in the expression recognition task. The expression recognition system based on deep network has achieved excellent results in the accuracy of recognition compared with the traditional method system.

[0005] Facial expression recognition: compare the facial expression to be recognized with the known facial expression in the database, and obtain related information. The deep network combines the facial expression feature extraction and the facial expression recognition task together, and is realized end to end. The network inputs a face image, and the network outputs the expression label of the picture.

[0006] Although the deep network has achieved excellent results in the expression recognition field compared with the traditional method, there are still many things to improve.

[0007] Firstly, deep network needs a large amount of data set for training to solve the problem of overfitting, but the existing expression recognition data set does not support the training of general network structure of deep network, and these networks often perform well on one data set, but the performance is greatly reduced when changing to another data set. For example, the network cannot perfectly balance the performance on the laboratory data set and the field data set. Usually, the performance of the network is excellent on the laboratory data set because of small size and clean data, but the overall performance of the network is poor on the challenging field data set.

[0008] Secondly, the existing expression recognition data set has obvious individual differences, such as age, personality, race and expression level. Especially for field data set, in order to solve the problems of large cost, small size and difficult annotation in the laboratory data set, the method of video frame interception from film and television is adopted, and the data label is obtained by plot reasoning. These differences have a significant impact on the training of expression recognition network.

[0009] The appearance of these problems presupposes that the feature extraction part of the expression recognition network needs to be carefully designed, so as to enhance the generalization ability of the network and make the expression recognition system applicable to reality. SUMMARY

[0010] The problem to be solved by the present application is that the existing expression recognition method is mostly based on the front face image, only considers the overall features of the face picture, ignores the connection between the facial movement muscle blocks closely related to the expression, and thus the recognition result is not accurate enough.

[0011] The technical scheme of the present application is: an expression recognition method based on face sub-blocks, two deep networks of main network and branch network are constructed for face feature extraction, the main network is an expression recognition network based on picture time information, which extracts the overall features of the face, the branch network is an expression recognition network based on picture geometric information, which captures the connection between the face sub-blocks, finds out the area with the largest contribution to network recognition, and extracts the local features of the face sub-blocks, before entering the classifier for recognition, the two part features are fused to improve the recognition accuracy of the expression, and finally the expression recognition result is given by the classifier; wherein the face sub-blocks refer to the division of the face into N movement blocks according to the expression action, the division method is from point to surface, first the core point of each sub-block is determined, and then the sub-block area is calculated according to the distance from the core point, and the coordinates of the core point are obtained according to the face key points.

[0012] Further, the present application comprises the following steps:

[0013] 1) Construct a full convolutional facial expression recognition network: the facial expression recognition network includes data preprocessing, facial feature extraction and expression classification three stages, data preprocessing is used for face alignment and face cutting, get face normalization data, facial feature extraction includes face feature extraction and face patch feature extraction, get face feature representation, expression classification stage according to face feature classification prediction, the convolution layer in the network is processed by batch normalization, and PReLU is used as the activation function;

[0014] 2) Training stage: training the full convolutional neural network constructed by using facial expression recognition data set, using cross entropy function as loss function;

[0015] 3) Test stage: input test image into network, get expression classification result.

[0016] Further, in step 1), data preprocessing is divided into two stages, first, 68 key points of face detection, and then face cutting, face cutting is to get face contour according to face key point coordinates, and cut out face image.

[0017] As a preferred mode, in step 1), the facial feature extraction stage includes two small networks, namely the main network and the branch network, the main network includes two convolution layers, two maximum pooling layers, and two fully connected layers, the convolution layer and the maximum pooling layer are all adopted 5*5 convolution kernel, PReLU is used as the activation function, and the fully connected layer adopts dropout to prevent overfitting; The input of branch network is the key point coordinates of face image, the branch network first selects the center point of face patch, then extracts the features of the patch, and calculates the attention of the patch to get the heatmap of the patch, the convolution kernel of branch network adopts 3*3, and the step is 1.

[0018] As a preferred mode, the main network uses 3D convolution operation to capture features from time and space dimensions at the same time, uses 3 image sequences as input, the 3D filter of the main network does not share weight on the time axis, so that different filters produce adaptive changes, the activation function of the first layer of the two convolution layers of the main network is Where f x,y,i is the i-th feature activation map of position (x, y), R and S are the number of rows and columns of the filter respectively, T a is the total number of input frames, is the value of position (x+r, y+s) at time t, is the i-th filter coefficient of (r, s) in t frame, b iis the bias coefficient of the i-th filter, and is the activation function, ReLU is used, the output of the convolutional layer of the main network is reduced by half in the pooling layer, and the final output value is input into the softmax classifier through two fully connected layers for classification.

[0019] As a preferred mode, the branch network divides the face into 12 motion blocks according to the expression, the branch network divides the face into blocks for each picture, a convolutional network is used to calculate the hotspot map of the input face block image, a hotspot matrix is obtained, a high weight is set for the area close to the center point, and a low weight is set for the area far from the center point, then the hotspot matrix is integrated into the main network, the hotspot matrix and the feature matrix are summed, and another feature dimension is brought into calculation.

[0020] In the expression classification stage of step 1), the features of the heatmap of the branch network are added to the second layer of the main network, the hotspot map heatmap of the face block extracted by the branch network is multiplied with the feature map featuremap generated by the convolution of the main network, pixel-by-pixel summation is performed, the fused features are obtained, and finally the main network is output to the softmax classifier for classification.

[0021] Compared with the prior art, the present application has the following advantages:

[0022] 1) The present application uses two small complementary networks to extract features at different levels of face images, improves the robustness of the expression recognition network, and has good performance on both field data sets and laboratory data sets, and the embodiments also show the accuracy of the present application.

[0023] 2) The present application adds a face block strategy to the expression recognition task, and performs attention calculation on the block, so that the network can learn the area that contributes greatly to the expression recognition result, thereby improving the accuracy of expression recognition. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the overall network architecture diagram of the expression recognition based on the face block of the present application.

[0025] Figure 2 is the structure diagram of the two complementary networks in the face feature extraction stage of the present application.

[0026] Figure 3 is the processing diagram of the face feature extraction stage and the classification stage of the present application.

[0027] Figure 4 is the visualization of the expression recognition network based on the face block of the present application, figure (a) is a schematic diagram of the face block, and figure (b) is a hotspot map generated after the face block passes through the convolutional network. DETAILED DESCRIPTION

[0028] The application provides a face expression recognition method based on face patches, fuses face features at different levels by constructing two small complementary networks, and obtains a result of classifying six basic expressions.

[0029] The application comprises the following steps: 1) constructing a full convolutional expression recognition network by using face patches: the network comprises three stages of data preprocessing, face feature extraction and comprehensive prediction, wherein the face feature extraction stage adopts two complementary convolutions, one of which is an expression recognition network based on picture time information and is used for extracting overall features of the whole face, and the other is an expression recognition network based on picture geometric information and is used for extracting local features of the face patches; 2) a training stage: the network is trained by using an expression recognition dataset, a cross-entropy function is used as a loss function, and an online difficult example sample retraining strategy is adopted in the process; 3) a testing stage: the test image is input into the network to obtain an expression recognition result. In the feature extraction stage, the application fully utilizes picture information and face key point information, constructs two complementary small networks, extracts overall features of the face and local features of the face patches, and improves the robustness of face expression recognition.

[0030] The full convolutional expression recognition network constructed by the application adds the features of the face motion unit to the identification task of the auxiliary network, compared with other face expression recognition networks. The full convolutional expression recognition network of the application is divided into two parts, one is a main network, which is an expression recognition network based on time information, this part of network processes the continuous pictures, extracts the overall features of the face pictures, and performs expression recognition. Considering that the completion of an expression is a dynamic action, the comparison of continuous pictures produces the change of the expression, therefore, the main network of the application utilizes the time information, and effectively captures the change of the expression along with time. The main network of the application adds the part of the 3D convolutional network on the time axis. The second part of the full convolutional expression recognition network of the application is a branch network, which is based on the geometric information of the picture to complete the expression recognition. First, the input face picture is marked with 68 face key points, then the face is blocked according to the distance rule according to the key points, that is, the division of the face motion unit, and finally the branch network performs attention calculation on each face motion unit block. Overall, the main network focuses on the overall features of the high-level semantics, and the branch network is used to capture the relationship of the face motion unit ignored by the main network.

[0031] The specific implementation steps of the application are as follows:

[0032] 1) constructing a full convolutional face expression recognition network: comprising three stages of data preprocessing, face feature extraction and expression classification.

[0033] Data preprocessing includes face key point detection and data cropping. First, the whole picture sequence is horizontally flipped. For picture cropping, the boundary is calculated according to the face key point, and the picture size input into the network is 96*96.

[0034] Face feature extraction is realized by establishing two small complementary networks, one is the main network, an expression recognition network based on picture time information, and the other is the branch network, an expression recognition network based on picture geometric information. The CNN network in this paper includes two convolutional layers, two pooling layers, two fully connected layers, and the activation function uses ReLU, and the dropout parameter is 0.4. The specific architecture of the expression recognition network based on face patches is as shown in Figure 1 ,2.

[0035] The main network, i.e. the convolutional neural network based on time information, uses 3D convolution operation in the CNN convolution operation stage to capture features from time and space dimensions at the same time. 3D convolution is performed by stacking multiple consecutive frames to form a cube and performing convolution calculation with a 3D kernel, so that the feature map on the convolution layer is connected with the consecutive frames of the previous layer to obtain time information. In the prior art, C3D shares weights along the time axis to capture spatio-temporal features. The present application uses three image sequences as input, but the 3D filter does not share weights along the time axis, which is to make different filters produce adaptive changes. The activation function of the first layer of the convolution layer of the main network is where f x,y,i is the i-th feature activation map of position (x, y), R and S are the number of rows and columns of the filter respectively. T a is the total number of input frames, is the value of position (x+r, y+s) at time t. is the i-th filter coefficient of (r, s) in t frame, b i is the bias coefficient of the i-th filter. δ(·) is the activation function, and ReLU is used. In order to perform effective calculation, the output of the convolution layer is reduced by half in the pooling layer. The final output value is classified by two fully connected layers and softmax. In the process of training the network, the method of stochastic gradient descent is used for optimization, and the methods of dropout and weight decay are used for regularization.

[0036] The branch network, i.e. the expression recognition network based on picture geometric information, is used to capture the connection between the facial movement muscle blocks, and the input of the branch network is a single picture. The rule of facial muscle movement corresponds to the basic expression of human face, but the effect of the facial expression recognition through the facial action coding is not satisfactory due to the difficulty in accurately positioning the facial movement unit. Since it is very difficult to accurately extract the facial movement unit, the application proposes a local block feature extraction method for a human face to assist the human face expression recognition. Firstly, the application performs normalization processing on the human face picture, and detects the key points of the human face. Then, the center point is found through the key point position, and the human face is blocked according to the distance method. Finally, the network gives different parameters to different movement blocks, and performs attention calculation.

[0037] According to the facial anatomy, some facial movement units appear in pairs, and others appear independently. The division of the facial unit of the human face is generated by the facial muscle movement, so there must be a connection between the facial movement units generated by the same muscle movement. Psychological research shows that there is a correlation between the facial movement units, and there is a correlation between the facial movement units and the expression categories. The relationship between the facial movement units can be divided into positive correlation and no correlation. The positive correlation represents that these facial units may appear at the same time, and according to the anatomical finding, the facial movement of the positive correlation is generally controlled by the same muscle or adjacent muscle group. For example, AU1 (inner eyebrow up) and AU2 (outer eyebrow up) are both controlled by the occipitofrontal muscle. The lateral contraction of the occipitofrontal muscle will appear AU2, and the central contraction will produce AU1. Because the lateral and central parts of the occipitofrontal muscle generally contract at the same time, AU2 and AU1 generally appear together, so the two movement units are positively correlated. On the other hand, the negative correlation represents that some facial movement units rarely or even never appear at the same time. According to the facial anatomy, some muscles of the face cannot be activated at the same time. That is to say, the appearance of some facial units means that the probability of the appearance of other facial movement units will be greatly reduced, or even none. The relationship between AU and the six basic expressions is shown in Table 1.

[0038] Table 1: Correlation probability of six basic expressions and specific facial units

[0039] Expression Facial Part Anger AU4, AU7, AU24 (>= 0.7), AU10 (0.26), AU17 (0.52) Disgust AU9, AU10, AU17 (>= 0.7), AU2 (0.57) AU26 (0.33) Fear AU1, AU4, AU20, AU25 (>= 0.7) Happiness AU12, AU25 (>= 0.7) Sadness AU4, AU15, AU17 (>= 0.7) Surprise AU4, AU2, AU25, AU26 (>= 0.7)

[0040] If the face patches in FACS are directly used, the calculation complexity is too large. According to psychological research, it is found that the parts related to expression include eyebrow, lip and other parts. Therefore, the number of face patches is reduced in the present application, and coarse-grained division is performed. The reason for this is that on the one hand, the calculation complexity can be reduced, and on the other hand, the detection difficulty of the motion block can be reduced. At present, the detection accuracy of the facial motion unit is not satisfactory, and there is a large error in directly detecting 68 AUs. And the network will have a negative effect on the network based on the above consideration. The network of the present application divides the face into 12 motion blocks, including the muscle regions of the eyebrow, eye, mouth and nose regions. The specific implementation is that the picture is first cropped to a size of 120*120. The patching mode is from point to surface, and the core point of each patch is first determined, and then the patch region is obtained according to the distance calculation from the core point. At present, the accuracy of face key point detection is quite high, and the coordinates of the 68 key points of the face detected and calibrated in the data preprocessing stage are directly or indirectly obtained, and the core point of each patch is determined according to the anatomy of the face. For some patches, the core point coincides with a certain face key point, and the key point coordinates can be directly used, and for the patches whose core point does not coincide with the face key point, the position of the core point in the face is determined according to the distance between the two eyes. Since different parts of the face have different contribution levels to each expression, expression recognition needs to find the area with high contribution level to the expression, and give high weight and high response, and discard the irrelevant area. The branch network of the present application is designed to complete this task, and the branch network generates different heat maps for each patch of the picture, and then inputs the heat map into the main network as another feature dimension for calculation. The branch network calculates the heat map of the input face patch image to obtain a heat matrix, and sets a high weight for the position close to the core point and a low weight for the position far away from the core point. When the branch network and the main network feature are fused, the feature of the heatmap of the branch network is added to the second layer of the main network, and the feature map generated by the main network is summed pixel by pixel, so that the main network pays more attention to valuable information, and at the same time, the feature information is not lost.

[0041] Since the face is a symmetrical structure, the core points also have a pairing phenomenon. And for the adjacent muscle group controlled face patches, there may also be a phenomenon that two patches share a core point. The specific selection standard of the core point in the embodiment of the present application is shown in Table 2.

[0042] Table 2 Selection of core points of face patches

[0043] Patch Number Patch Name Patch Center Point 1 Inner Brow Up Upper 1 / 2 of inner eyebrow 2 Outer Brow Up Upper 1 / 3 of outer eyebrow 3 Brow Lowerer Lower 1 / 3 of middle of eyebrow 4 Cheek Raiser Below one eye lid 5 Eye Wide Open Middle of eye 6 Upper Lip Raiser Middle of upper lip 7 Corner Lip Depressor Corner of mouth 8 Dimpler Corner of mouth 9 Corner Lip Down Corner of mouth 10 Chin Raiser Lower 1 / 2 of chin 11 Lip Pucker Middle of lip 12 Lip Sucker Middle of lip

[0044] The visualization result of the expression recognition network based on face patches is shown in Figure 4 .

[0045] In the expression classification stage, a 1*1 convolution is used to realize communication between 256 channels in the output features of the feature processing stage, and then a 3*3 convolution layer is used to obtain a small size classification result, wherein the number of channels n is equal to the number of classification labels of the data set, and the size is 1 / 8 of the original picture size. Finally, an up-sampling layer is used to expand the size to the original picture size through linear interpolation.

[0046] 2) Training stage: the full convolutional neural network constructed is trained using a facial expression recognition data set, and a cross-entropy function is used as a loss function; wherein, a stochastic gradient descent method is used for optimization, and a dropout and weight decay method is used for regularization, and the full connection layer of the main network uses dropout of 0.4.

[0047] 3) Test stage: the test image is input into the network after zero mean processing, and the recognition result is obtained.

[0048] The following uses the network trained to perform performance testing to illustrate the effect of the application.

[0049] CK+ is an expansion of Cohn-Kanda data set, and is a commonly used data set for expression recognition tasks. CK only includes static pictures, and CK+ includes dynamic videos. CK+ includes 123 people and 593 video sequences. Among them, 118 objects have 327 video sequences containing 7 basic expressions. The expression labels are anger, fear, happiness, sadness, surprise, contempt, and a neutral expression. In CK+, only the last frame of the video sequence is generally marked. The label includes 7 kinds of facial expressions and facial codes. The pictures in the CK+ data set are forward-facing human face pictures, and there is no problem of occlusion and illumination. The expression recognition network based on the whole picture alone has a recognition rate of 97.3, and the expression recognition network based on the face part block is added. The performance of the network is improved by 0.5 percentage points. The results of the latest research are shown in Table 3, so it can be seen that the results of the application are very excellent on the CK data set.

[0050] Table 3: Results of expression recognition network of CK data set

[0051] Name Input Type Accuracy STM-ExpLte (Dynamic Expression Recognition Network Based on Spatio-Temporal Information) Video Sequences 94 DTGAN (Jointly Fine-Tuned Expression Recognition Network) Video Sequences 97.25 DTAN+ (A Small and Robust Expression Recognition Network) Static Images 97.37 Ours-C3D (Invention) Video Sequences 98 Ours (Invention) Static Images 97.3

[0052] The AFEW dataset has short video clips of facial expressions in near-real environments, with subjects ranging in age from 1-70 years, including a large number of video frames of children and adolescents. The care conditions in the dataset also have differences, including indoor, outdoor and night conditions, compared with the experimental dataset, this dataset is closer to the natural scene. SFEW is selected from AFEW more difficult video frames, these factors also make SFEW dataset the most difficult in the expression recognition dataset. SFEW contains 95 experimental objects, including 700 images taken from movies. The difficulty of these images includes unconstrained head posture, changing lighting conditions, large age span, different image resolution, etc. The label of SFEW dataset is divided into 6 basic expressions and one neutral (NE) expression, of which the 6 basic expressions include anger (AN), disgust (DI), fear (FE), happiness (HA), sadness (SA), and surprise (SU). The accuracy of the local network of the present application, i.e., the expression recognition network based on image time information, on the SFEW dataset is 52.6, while the result on the overall network of the present application is 53.4.

Claims

1. A face-based partial patch based expression recognition method, characterized in that The two deep networks of a main network and a branch network are constructed for facial feature extraction, the main network is an expression recognition network based on picture time information, the overall features of the face are extracted, the branch network is an expression recognition network based on picture geometric information, the connection between the face patches is captured, the area with the largest contribution to the network recognition is found out, the local features of the face patches are extracted, the two partial features are fused before entering the classifier for recognition, the expression recognition accuracy is improved, and finally the expression recognition result is given by the classifier; wherein the face patches refer to the face being divided into N motion blocks according to the expression action, the block division mode is from point to surface, the center point of each block is determined first, the center point of each block is determined according to the facial anatomy, and then the block area is calculated according to the distance from the center point, and the coordinates of the center point are obtained according to the face key points; comprising the following steps: 1) Constructing a full convolutional facial expression recognition network: the facial expression recognition network includes three stages of data preprocessing, facial feature extraction and expression classification, the data preprocessing is used for face alignment and face cropping to obtain normalized face data, the facial feature extraction includes overall facial feature extraction and facial patch feature extraction to obtain face feature representation, and the expression classification stage classifies and predicts according to the face feature, and the convolutional layers in the network are all subjected to batch normalization processing, and ReLU is used as the activation function; in step 1), the facial feature extraction stage includes two small networks, i.e. the main network and the branch network, the main network includes two convolutional layers, two maximum pooling layers and two fully connected layers, the convolutional layers and the maximum pooling layers are both 5*5 convolutional kernels, ReLU is used as the activation function, and the fully connected layers adopt dropout to prevent overfitting; the branch network input is the key point coordinates of the face image, the branch network first selects the center point of the face patch, then extracts the features of the block, and calculates the attention of the block to obtain the heatmap of the block, and the convolutional kernel of the branch network adopts 3*3 with a step of 1; Wherein, the branch network divides the face into 12 motion blocks according to the expression, including the muscle regions of the eyebrow, eye, mouth and nose regions, the branch network calculates the heatmap of the input face patch image by using the convolutional network for each face patch of the image, sets a high weight for the area close to the center point and a low weight for the area far away from the center point, then the heatmap is fused into the main network, and the heatmap and the feature matrix are summed as another feature dimension for calculation; 2) Training stage: the full convolutional neural network constructed is trained by using a facial expression recognition dataset, and a cross-entropy function is used as a loss function; 3) Test stage: input the test image into the network to obtain the expression classification result.

2. The method of claim 1, wherein the method is characterized by In step 1), the data preprocessing is divided into two stages, first, the 68 key points of the face are detected, and then the face is cropped, the face contour is obtained according to the face key point coordinates, and the face image is cropped.

3. The method of claim 1, wherein the method is characterized by The main network uses 3D convolution operation to capture features from time and space dimensions simultaneously, uses three image sequences as input, the 3D filter of the main network does not share weights on the time axis, so that different filters produce adaptive changes, the activation function of the first layer of the two convolution layers of the main network is where f x,y,i is the i-th feature activation map of position (x, y), R and S are the number of rows and columns of the filter respectively, T a is the total number of input frames, is the value of position (x+r, y+s) at time t, is the i-th filter coefficient of (r, s) in t frame, b i is the bias coefficient of the i-th filter, δ(·) is the activation function, ReLU is used, the output of the convolution layer of the main network is reduced by half in the pooling layer, and the final output value is classified by two fully connected layers and a softmax classifier.

4. The method of claim 1, wherein the method is characterized by In the expression classification stage of step 1), the features of the heatmap of the branch network are added to the second layer of the main network, the heatmap of the face part block extracted by the branch network is multiplied with the feature map generated by the convolution of the main network, pixel-by-pixel summation is performed, and finally the fused features are output to the softmax classifier by the main network for classification.

5. The method of claim 1, wherein the method further comprises: In the training stage, the method of stochastic gradient descent is used for optimization, and the methods of dropout and weight decay are used for regularization, and the dropout of the fully connected layer of the main network is 0.4.

Images