Network model and method for improving facial expression recognition accuracy in video

Abstract

The application provides a network model and method for improving the facial expression recognition accuracy in a video, the method comprising: inputting an initial video image, and performing feature extraction through a 3D convolution network; fusing an AU perception attention module in the 3D convolution network, paying attention to the key area of facial emotional expression through the AU perception attention module, and learning the features of the key parts of the face; encoding the obtained feature map through a capsule network, and encoding the enhanced features through the dynamic routing between the capsules; decoding through three fully connected layers, and realizing the final expression classification through a nonlinear squeeze function. The application realizes higher-precision facial expression recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of facial expression recognition, in particular to a network model and method for improving the accuracy of facial expression recognition in a video. BACKGROUND

[0002] With the development of computer science, facial expression recognition technology (FER) has a more and more extensive application in the real society, such as human-computer interaction, video emotion analysis, safe driving, etc. Therefore, facial expression recognition technology has always been one of the hot research fields in computer vision. For the problem of facial expression recognition in a video, many solutions have been proposed by researchers, but because of the bias in the expression of different people and the complex spatiotemporal features of the video, these algorithms have certain limitations, such as the method based on traditional computer vision algorithm to extract hand-designed features, which is complex to operate and has poor robustness, and the method based on classic CNN cannot effectively process the feature relationship between facial structures, such as relative position relationship, relative size relationship, feature direction, etc., so as to not effectively focus on the key area of facial emotional expression, resulting in poor recognition effect and other problems. Therefore, in the field of computer vision, facial expression recognition is still a challenging and highly applicable problem.

[0003] In the research of expression recognition, the deep learning method is usually better than the hand-crafted feature extraction algorithm based on traditional machine learning, and shows good performance: not only has stronger feature extraction ability, but also has better expression recognition rate under interference scenes such as light, posture and occlusion. However, most of the current facial expression recognition methods use traditional convolutional neural networks to work, and only focus on the feature representation of the face, rarely considering the relative relationship between the facial motion units (AU). Although convolutional neural networks are good at capturing features, they are not good at exploring the relationship between features, such as relative position relationship, relative size relationship, feature direction, etc. In addition, for facial expression recognition in a video, the ordinary convolutional neural network does not consider the time dimension features of the video. Due to these problems, the existing facial expression recognition method has poor recognition effect and low robustness. SUMMARY

[0004] In view of the defects in the prior art, the purpose of the present application is to provide a network model and method for improving the accuracy of facial expression recognition in a video, which uses a combination of 3D deep convolutional network and capsule network instead of ordinary convolutional neural network, solving the problem that the traditional convolutional neural network cannot well process the spatiotemporal features of the video and ignores the relative position relationship between facial features.

[0005] To solve the above problems, the technical scheme of the present application is:

[0006] A network model for improving the accuracy of facial expression recognition in a video, comprising a 3D convolution network module based on a capsule network and an AU perception attention module, the 3D convolution network module based on the capsule network is used to extract features of video time and spatial sequence information, the AU perception attention module is fused in the 3D convolution network module, the key area of facial emotional expression is focused on through the AU perception attention module, and the features of key parts of the face are learned.

[0007] Optionally, the network model further comprises encoding the enhanced features through dynamic routing between capsules, decoding through three fully connected layers, and finally realizing expression classification results through a nonlinear squeeze function.

[0008] Optionally, the three fully connected layers are two fully connected layers with ReLU activation function and one fully connected layer with Sigmoid activation function, and the nonlinear squeeze function is defined as: Where v j is the output vector of capsule j, b j is the input vector of the capsule.

[0009] Further, the present application also provides a method for improving the accuracy of facial expression recognition in a video, comprising the following steps:

[0010] Inputting an initial video image, and extracting features through a 3D convolution network;

[0011] Fusing an AU perception attention module in the 3D convolution network, focusing on the key area of facial emotional expression through the AU perception attention module, and learning the features of key parts of the face;

[0012] Encoding the obtained feature map through a capsule network, and encoding the enhanced features through dynamic routing between capsules;

[0013] Decoding through three fully connected layers, and finally realizing expression classification through a nonlinear squeeze function.

[0014] Optionally, the step of inputting an initial video image and extracting features through a 3D convolution network specifically comprises extracting spatio-temporal features from a video track through an improved 3DResNet convolutional neural network, the video track is divided into continuous non-overlapping small segments, and each small segment contains N frames.

[0015] Optionally, the step of inputting the initial video image and extracting features through the 3D convolutional network further comprises: when extracting features, in order to obtain feature dependency in the spatial dimension and the time dimension, a non-local attention block is used to directly calculate the relationship between the time position and the spatial position.

[0016] Optionally, the step of fusing the AU-aware attention module in the 3D convolutional network, through the AU-aware attention module, focuses on the key area of facial emotional expression, and learns the features of the key parts of the face, specifically comprises: when constructing the attention map, first find the key facial landmarks, define the AU center through the key facial landmarks, obtain the AU center, and construct the attention map based on the center position.

[0017] Optionally, the step of encoding the obtained feature map through the capsule network and mapping the enhanced features through dynamic routing between capsules comprises: the features obtained through the AU attention constraint are sent to the PrimaryCaps layer of the capsule network, for the PrimaryCaps layer capsule, the activated capsule is encoded with position information, the DigitCaps layer adopts the length of each capsule activity vector to represent the probability of each class appearing, and is used to calculate the classification loss, each capsule receives the input from the previous layer of capsules between the PrimaryCaps layer and the DigitCaps layer, and the network performs a dynamic routing consistency algorithm.

[0018] Optionally, the step of decoding through three fully connected layers and realizing the final expression classification through a nonlinear squeeze function comprises: connecting three fully connected layers after the DigitCaps layer for image reconstruction, and the three fully connected layers are two fully connected layers with ReLU activation function and one fully connected layer with Sigmoid activation function.

[0019] Optionally, in the step of decoding through three fully connected layers and realizing the final expression classification through a nonlinear squeeze function, the nonlinear squeeze function is defined as: where v j is the output vector of the capsule j, b j is the input vector of the capsule, and the prediction probability of the final expression category is obtained through the nonlinear squeeze function.

[0020] Compared with the prior art, the 3D convolutional network based on the capsule network can better extract the features of the video time and spatial sequence information, and enhance the feature representation capability, while the face AU-aware attention module is added for mining the potential information in the key area of the face, and higher-precision facial expression recognition is realized, and the generalization is good. BRIEF DESCRIPTION OF DRAWINGS

[0021] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in conjunction with the accompanying drawings:

[0022] Figure 1 A network model for improving the facial expression recognition accuracy in a video provided for an embodiment of the application;

[0023] Figure 2 A method flowchart for improving the facial expression recognition accuracy in a video provided for an embodiment of the application. DETAILED DESCRIPTION

[0024] The application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the application, a number of changes and improvements can be made. These are within the scope of the application.

[0025] Specifically, Figure 1 A network model for improving the facial expression recognition accuracy in a video provided for an embodiment of the application, as shown in Figure 1 The network model for improving the facial expression recognition accuracy in a video mainly includes two modules: (1) a 3D convolution network module based on a capsule network: an improved 3D convolution network module is used as a backbone network for feature extraction, so as to better extract the features of video time and spatial sequence information. (2) Action Unit (AU) perception attention module: an AU perception attention module is fused in the 3D convolution network to focus on the key areas of emotional expression, which is used to mine the potential information (position, size, direction) in the key areas of the face. Then, the enhanced features are mapped through the dynamic routing between the capsules, decoded through three fully connected layers, and finally the expression classification results are realized by using a nonlinear squeezing function.

[0026] Figure 2 A method flowchart for improving the facial expression recognition accuracy in a video provided for an embodiment of the application, as shown in Figure 2 The method includes the following steps:

[0027] S1: input the initial video image, and extract features through a 3D convolution network;

[0028] Specifically, as shown in Figure 1The illustrated 3D deep convolutional network and capsule network-based facial expression recognition model in video extracts spatio-temporal features from a video track, i.e., a video segment, through an improved 3D ResNet convolutional neural network. The model takes T consecutive frames, i.e., a video track, as input. The video track is first divided into consecutive non-overlapping small segments, and each small segment contains N frames. When extracting features, in order to obtain the feature dependency in the spatial and temporal dimensions, a non-local attention block is used in the network to solve this problem. The non-local operation can directly calculate the relationship between two positions (which can be time position, spatial position and space-time position) to quickly capture long-range dependencies, and can ensure that the input scale and output scale are invariant. This design can be well embedded into the 3D network architecture.

[0029] S2: Fuse the AU-aware attention module in the 3D convolutional network. The AU-aware attention module focuses on the key areas of facial emotional expression, and learns the features of the key parts of the face.

[0030] Specifically, when constructing the attention map, first find the key facial landmarks. The key facial landmarks can be used to define the AU center. Specifically, the AU center is obtained by using the scaling distance or directly using the existing facial landmarks. Then the scaling distance is used as a reference to measure the displacement of the facial pixels. The scaling distance is the distance between the outer corners of the two eyes, which helps to find the AU center. Due to the symmetry of the face, each AU considers a pair of points. After obtaining the AU center, the attention map can be constructed based on the center position. The generated attention map is embedded in the convolution operation of the first 3D residual block. After the convolution in the 3D residual block is completed, the attention features and the convolution results are added element by element for fusion, and then used as the input of the next stage.

[0031] S3: Encode the obtained feature map through the capsule network, and encode the enhanced feature map through the dynamic routing between the capsules.

[0032] Specifically, the features obtained through the AU attention constraint are sent to the PrimaryCaps layer of the capsule network. For the PrimaryCaps layer capsules, the activated capsules perform position information encoding. The DigitCaps layer uses the length of each capsule activity vector to represent the probability of each class appearing, and is used to calculate the classification loss. Each capsule receives input from the previous layer of capsules between the PrimaryCaps layer and the DigitCaps layer, and the network performs a dynamic routing consistency algorithm. As the capsule level rises, more and more position information encoding is converted into probability encoding by the real value part of the output vector of the capsule.

[0033] S4: Decode through three fully connected layers, and realize the final expression classification through a nonlinear squeezing function.

[0034] The three fully connected layers behind the DigitCaps layer are connected for image reconstruction, which are two fully connected layers with ReLU activation function and one fully connected layer with Sigmoid activation function, which is the decoding stage of the capsule network. The capsule structure in the embodiment uses an additional reconstruction loss to promote the DigitCaps layer to encode the input video frame, and the reconstruction loss is established by the Euclidean distance between the pixels of the output video frame of the last layer of the fully connected layer with Sigmoid activation function and the pixels of the original video frame. The capsule can represent the instantiation parameters of a specific type of entity by an activity vector, the direction of the activity vector represents some self information of the entity, such as relative position information, spatial direction, etc., and the length of the vector can be used to represent the probability of the occurrence of a specific entity, therefore, the length of the vector must be between 0 and 1, in order to ensure that the length of the output vector is within this range, a squashing function is used, and the function is nonlinear, and is defined as:

[0035]

[0036] Where v j is the output vector of the capsule j, b j is the input vector of the capsule. The squashing function is divided into two parts, the scaling ratio of the input vector b j , and the unit vector of the input vector b j , and the final prediction probability of the expression category is obtained through the nonlinear squashing function.

[0037] Compared with the prior art, the 3D convolutional network based on the capsule network used in the application can better extract the features of the video time and spatial sequence information, and enhance the feature representation capability, and at the same time, the face AU perception attention module is added, which is used for mining the potential information in the key area of the face, and higher precision face expression recognition is realized, and the application has good generalization.

[0038] The specific embodiments of the application are described above. It should be understood that the application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the application. In the case of no conflict, the embodiments of the application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for improving the accuracy of facial expression recognition in a video, characterized in that, The method comprises the following steps: input initial video images, and perform feature extraction through a 3D convolution network; AU perception attention modules are fused in the 3D convolution network, the AU perception attention modules pay attention to key areas of facial emotional expression, and the features of key parts of the face are learned, specifically including the following steps: when constructing an attention map, first, key facial landmarks are found, the AU center is defined through the key facial landmarks, after the AU center is obtained, the attention map is constructed based on the center position; the obtained feature map is encoded through a capsule network, and the enhanced features are encoded through dynamic routing between capsules, specifically including the following steps: the features obtained through AU attention constraints are sent to the PrimaryCaps layer of the capsule network, for the PrimaryCaps layer capsules, the activated capsules are subjected to position information encoding, the DigitCaps layer adopts the length of each capsule activity vector to represent the probability of each class appearing, and is used to calculate a classification loss, each capsule between the PrimaryCaps layer and the DigitCaps layer receives input from the capsules of the previous layer, and the network performs a dynamic routing consistency algorithm; decoding is performed through three fully connected layers, and final expression classification is realized through a nonlinear squeeze function.

2. The method for improving the accuracy of facial expression recognition in a video according to claim 1, wherein, The step of inputting initial video images and performing feature extraction through a 3D convolution network specifically comprises the following steps: spatial and temporal features are extracted from a video track through an improved 3DResNet convolutional neural network, the video track is divided into continuous non-overlapping small segments, and each small segment contains N frames.

3. The method for improving the accuracy of facial expression recognition in a video according to claim 2, wherein, The step of inputting initial video images and performing feature extraction through a 3D convolution network specifically further comprises the following steps: when extracting features, in order to obtain feature dependency in the spatial dimension and the temporal dimension, a non-local attention block is used to directly calculate the relationship between the temporal position and the spatial position.

4. The method for improving the accuracy of facial expression recognition in a video according to claim 1, wherein, The step of decoding through three fully connected layers and realizing final expression classification through a nonlinear squeeze function specifically comprises the following steps: three fully connected layers are connected behind the DigitCaps layer for image reconstruction, the three fully connected layers are two fully connected layers with ReLU activation functions and one fully connected layer with a Sigmoid activation function, respectively.

5. The method for improving the accuracy of facial expression recognition in a video according to claim 4, wherein, The step of decoding through three fully connected layers and realizing the final expression classification through a nonlinear squeeze function is defined as: where v j is the output vector of the capsule j, b j is the input vector of the capsule, and the prediction probability of the final expression category is obtained through the nonlinear squeeze function.

Images