Facial expression recognition method based on spatiotemporal fusion network

Abstract

The present invention discloses a facial expression recognition method based on a spatiotemporal fusion network, comprising the following steps: first, preprocessing the input image; then, constructing a spatiotemporal fusion network that can learn facial expression features from two different dimensions, time domain and space domain, respectively, using training samples to train the two modules of the spatiotemporal fusion network respectively, and finally using an optimal fusion method based on a fine-tuning strategy to effectively fuse the spatiotemporal features of expressions learned in different dimensions; during testing, inputting a sequence of facial expression images to be detected into the trained spatiotemporal fusion network to obtain the classification category of the expression. The present invention has the following beneficial effects: the present invention uses the optimal fusion method based on the fine-tuning strategy, which can more effectively fuse the learned spatiotemporal features of expressions, improve the learning ability of expression features, and can improve the accuracy of facial expression recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of facial expression recognition, and in particular to a facial expression recognition method based on a spatiotemporal fusion network, which can improve the accuracy of facial expression recognition. Background Art

[0002] Facial expressions are one of the most natural and universal signals that humans use to convey their emotional states. Facial expression recognition has broad application prospects in fields such as commerce, security, and medicine. The ability to quickly and accurately identify facial expressions is of great significance for its research and application. Current research on facial expression recognition can be roughly divided into two categories: static image-based methods and dynamic video sequence-based methods. Facial expression recognition based on static images only requires the use of peak images of the expression. However, the generation of an expression itself corresponds to a continuous process of facial muscle movement. Using only static peak expression images will lose some important information, such as the motion information of the facial expression. Therefore, in order to further integrate the temporal characteristics of expressions, we need to study facial expression recognition methods based on dynamic video sequences to improve the algorithm's expression recognition performance.

[0003] In recent years, researchers have proposed a series of facial expression recognition methods based on video sequences. However, as a specialized facial analysis task, facial expression recognition has its own unique characteristics. In particular, facial expressions can be considered as the dynamic changes of key facial components (such as eyebrows, eyes, nose, and mouth). The changes in these key components, combined together, form the changes in the overall facial expression. To address the dynamic changes of key facial components, existing methods have attempted to extract dynamic expression features in the temporal domain from consecutive frames through manual design or deep learning methods. However, on the one hand, manually designed features are often difficult to correctly extract the temporal features contained in facial image sequences; on the other hand, directly inputting image sequences into deep neural networks cannot effectively utilize prior knowledge of key facial components, which is not conducive to learning temporal expression features. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art in which expression classification based on video features is very difficult, the motion features of facial expressions in the time domain cannot be accurately extracted, and the learned spatiotemporal features of expressions cannot be effectively integrated, which will lead to insufficient learning ability of the network and low recognition accuracy, the present invention provides a facial expression recognition method based on a spatiotemporal fusion network that can improve the accuracy of facial expression recognition.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A facial expression recognition method based on a spatiotemporal fusion network comprises the following steps:

[0007] (1-1) Preprocessing of input video sequence;

[0008] (1-2) Construct a dual-stream spatiotemporal fusion network;

[0009] (1-2-1) Use the LTCNN module to extract the temporal features of facial expressions;

[0010] (1-2-2) Use the DMF module to extract the spatial features of facial expressions;

[0011] (1-2-3) Use the fine-tuning fusion module to effectively fuse the spatiotemporal features of expressions in two different dimensions;

[0012] (1-3) Classify facial expression image sequences.

[0013] The present invention inputs a face video sequence with facial expression information. First, the input video sequence is preprocessed; then, a spatiotemporal fusion network is constructed that can learn facial expression features from two different dimensions, namely, the temporal domain and the spatial domain, and the two modules of the spatiotemporal fusion network are trained separately using training samples; finally, an optimal fusion method based on a fine-tuning strategy is used to effectively fuse the expression features learned from different dimensions; during testing, the facial expression video sequence to be detected is input into the trained spatiotemporal fusion network to obtain the classification category of the expression.

[0014] Preferably, the specific steps of step (1-2-1) are as follows:

[0015] (2-1) Initialize the LTCNN module;

[0016] (2-2) The LTCNN module consists of four convolutional layers, two fully connected layers, and a Softmax classification layer. The convolutional layers and fully connected layers further extract high-level semantic features from the input class feature map to learn more discriminative expression temporal features contained in the key point trajectories in the class feature map.

[0017] (2-3) Finally, the network is trained using cross entropy as the loss function, and the expression category is estimated through the Softmax layer.

[0018] Preferably, the specific steps of step (1-2-2) are as follows:

[0019] (3-1) Initialize the DCNN submodule in the DMF module;

[0020] (3-2) Using the softmax loss function on the facial expression database, the DCNN submodule is adjusted to obtain the pre-trained DCNN submodule;

[0021] (3-3) The DCNN submodule uses multiple fully connected branch layers, each of which has a fixed-size fully connected layer and an associated threshold π i ,i∈{1, 2, ..., N};

[0022] (3-4) Embed the image x into the Euclidean space to obtain the embedded feature f i (x), During training, each branch is constrained by a symmetric triplet loss, denoted as loss i ,i∈{1,2,...,N}。

[0023] (3-5) In each batch of training, each training sample is used as a query image, and image samples with the same expression as the query sample and image samples with different expressions are selected to construct a triplet; the query image is used as an anchor point Image samples with the same expression as the query sample are regarded as positive samples Image samples with different expressions as negative samples The form of symmetric triplet loss is as follows

[0024]

[0025] (3-6) Fuse the expression spatial features learned by all branches in the N-metric submodule;

[0026] (3-7) Use two fully connected layers and the softmax loss function to classify facial expressions.

[0027] Preferably, the specific steps of step (1-2-3) are as follows:

[0028] (4-1) A fine-tuning-based fusion strategy is used to better complement the spatiotemporal features of facial expressions learned by the two modules.

[0029] Preferably, the specific steps of the fine-tuning fusion strategy in step (4-1) are as follows:

[0030] (5-1) First, different learning rates and learning strategies are used to train the LTCNN and DMF modules respectively;

[0031] (5-2) Remove the softmax layer of the network in the two trained modules, and cascade the last fully connected layer of the two networks together as the input of the newly added fully connected layer to form a spatiotemporal fusion network, thereby achieving effective fusion of the output features of the two networks;

[0032] (5-3) Finally, we further fine-tune the spatiotemporal fusion network composed of the pre-trained DMF and LTCNN modules and perform expression classification through a softmax layer. Four different fine-tuning schemes are used to adjust the network with a very small learning rate.

[0033] Preferably, the specific steps of the four different fine-tuning schemes in step (5-3) are as follows:

[0034] (6-1) Local fine-tuning: fix the parameters of the two modules and only fine-tune the newly added fully connected fusion layer and softmax classification layer at the end;

[0035] (6-2) Fine-tune the fixed DMF module, fix the parameters in the DMF module, and jointly fine-tune the LTCNN module and the newly added fully connected fusion layer and softmax classification layer;

[0036] (6-3) Fine-tuning of the fixed LTCNN module: Fix the parameters in the LTCNN module and jointly fine-tune the DMF module and the newly added fully connected fusion layer and softmax classification layer.

[0037] (6-4) Overall fine-tuning: All modules in the spatiotemporal fusion network are fine-tuned as a whole.

[0038] Preferably, the specific steps of steps (1-3) are as follows:

[0039] (7-1) During training, the DMF and LTCNN sub-networks in the spatiotemporal fusion network are first trained separately. The total loss of the DMF module is defined as follows: Among them, loss0 is the loss of the softmax loss function; the loss of the LTCNN module is the softmax loss function.

[0040] (7-2) Fine-tuning is performed based on different fine-tuning fusion strategies, and the final classification is performed by the softmax classification layer in the fine-tuning fusion module.

[0041] Preferably, the specific steps of step (1-1) are as follows:

[0042] (8-1) All sequences are unified into a fixed sequence of 11 frames using the sequence normalization method, and then the facial key points are used to crop the facial region of interest in each frame and the cropped image is scaled to obtain a fixed facial image sequence;

[0043] (8-2) For a fixed facial image sequence, a key point extraction algorithm is used to propose 51 key point coordinates for each frame of the image to form a class feature map. The class feature map is then enhanced using an online data enhancement method. The enhanced class feature map is used as the input data for the LTCNN module.

[0044] (8-3) The last expression peak frame image of each sequence is extracted, and the peak expression image is enhanced using offline and online data enhancement methods. The enhanced image is used as the input data of the DMF module.

[0045] Therefore, the present invention has the following beneficial effects: the present invention uses a facial key point trajectory convolutional neural network to accurately extract the motion features of facial expressions in the time domain, and uses a deep metric fusion network to extract the spatial domain features of facial expressions, which makes the learned expression features more diverse and robust; the present invention uses an optimal fusion method based on a fine-tuning strategy, which can more effectively fuse the learned expression temporal and spatial features, thereby improving the ability to learn expression features; and can improve the accuracy of facial expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the present invention;

[0047] Figure 2 Schematic diagram of the result of data preprocessing of the present invention;

[0048] Figure 3 It is a structural diagram of the spatiotemporal fusion network of the present invention;

[0049] Figure 4 This is a structural diagram of the time domain feature extraction module of the present invention - the convolutional neural network of facial key point trajectory;

[0050] Figure 5 It is a structural diagram of the spatial feature extraction module of the present invention - the deep metric fusion network;

[0051] Figure 6 It is a structural diagram of the fine-tuning fusion module of the present invention; DETAILED DESCRIPTION

[0052] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0053] like Figure 1 The embodiment shown is a facial expression recognition method based on a spatiotemporal fusion network, comprising the following steps:

[0054] Step 100: Preprocessing of input video sequence

[0055] 101 Sequence Normalization

[0056] The original video sequence is Figure 2As shown in (a), the starting frame of the sequence is taken as the first frame, the expression peak frame is taken as the 11th frame, and then the other 9 frames are selected from the sequence at equal intervals. Finally, all sequences are unified into a fixed sequence of 11 frames. Then, for each frame in the fixed sequence, the facial key points are used to crop the facial region of interest, and the cropped image is scaled to a size of 236*236 to obtain a facial image sequence after frame number normalization, as shown in Figure 1. Figure 2 As shown in (b).

[0057] 102 Key Point Detection and Standardization

[0058] Facial key point detection is used to propose 51 facial key points in the face image, which correspond to the four expression-focused areas of eyebrows, eyes, nose and mouth. The facial key point sequence is as follows: Figure 2 As shown in (c). Let P (t) is the key point of the tth frame, then P (t) for:

[0059] P (t) =[x1 (t) ,y1 (t) ,x2 (t) ,y2 (t) ,...,x 51 (t) ,y 51 (t) ]

[0060] Among them, (x i (t) ,y i (t) ) is the coordinate of the i-th key point in the t-th frame. The coordinate of each point is subtracted from the coordinate of the nose center point, and then each coordinate is divided by the standard deviation of the key point coordinates in all frames of the current sequence to perform key point normalization:

[0061]

[0062] Among them, (x c (t) ,y c (t) ) is the coordinate of the nose center point in the t-th frame, (σ x ,σ y ) is the standard deviation of the key point coordinates in all frames of the current sequence.

[0063] 103 Input Data Generation

[0064] The X / Y coordinates of the key points are stored in two channels separately to obtain a 11*51*2 dual-channel class feature map as the input data of the LTCNN module; the expression peak image of the last frame of the image sequence is used as the input data of the DMF module, such as Figure 2 As shown in (d).

[0065] 104 Data Augmentation

[0066] Class feature map enhancement: Using online data enhancement, key points are randomly flipped horizontally and Gaussian noise is randomly added to the key point coordinates. The specific form is as follows:

[0067]

[0068] in, Indicates the noise added to the x-coordinate of the i-th key point in the t-th frame, setting Similarly, noise can be added to the y coordinates of the key points as input data to the LTCNN module.

[0069] Expression peak image enhancement: The image is enhanced using offline and online data enhancement methods. Offline enhancement is to rotate the input image accordingly to {-10°, -5°, 0, 5°, 10°} to obtain the enhanced image data; online enhancement is to randomly crop the image from four directions (center, upper left, lower left, upper right, lower right) to obtain a 224*224 size image during network training, and randomly flip it horizontally as the input data of the DMF module.

[0070] Step 200: Build a spatiotemporal fusion network

[0071] like Figure 3 As shown in the figure, the spatiotemporal fusion network mainly consists of three modules: the LTCNN module for extracting temporal features of facial expressions, the DMF module for extracting spatial features of facial expressions, and the fine-tuning fusion module for fusing spatiotemporal expression features.

[0072] Step 201: Extract facial expression temporal features using the LTCNN module

[0073] like Figure 4 As shown in Figure 2, the LTCNN module consists of 4 convolutional layers and 2 fully connected layers. The size and number of convolution kernels in each convolutional layer are as follows: Figure 3 As shown in Figure 2, the two fully connected layers contain 1024 and 256 neurons respectively. During training, the module is initialized using Xavier and constrained by the softmax loss function.

[0074] Step 202: Extract facial expression spatial features using the DMF module

[0075] like Figure 5 As shown in the figure, the DCNN submodule is initialized with the VGG16-Face model; on the FER2013 facial expression database, the DCNN submodule is fine-tuned using the softmax loss function to obtain the pre-trained DCNN submodule;

[0076] The N-metric submodule uses multiple fully connected branch layers, each containing 512 neurons. The last fully connected layer of the DCNN submodule serves as the input of each branch of the N-metric submodule. Each branch has a fixed-size fully connected layer and an associated threshold π. i ,i∈{1,2,...,N}; embed the image x into the Euclidean space to obtain the embedded feature f i (x), During training, each branch is constrained by a symmetric triplet loss, denoted as loss i ,i∈{1, 2, ..., N};

[0077] In each batch of training, each training sample is used as a query image, and image samples with the same expression as the query sample and image samples with different expressions are selected to construct a triplet; the query image is used as an anchor point. Image samples with the same expression as the query sample are regarded as positive samples Image samples with different expressions as negative samples The form of symmetric triplet loss is as follows

[0078]

[0079]

[0080] Step 203: Use the fine-tuning fusion module to fuse the learned spatiotemporal features of facial expressions.

[0081] like Figure 6 As shown in the figure, the softmax layers of the two trained modules are removed, and the last fully connected layers of the two networks are cascaded together as the input of a newly added fully connected layer with a size of 512 neurons. This fusion of the output features of the two networks is then input to the final softmax layer for expression classification. Four different fine-tuning strategies are used:

[0082] (1) Local fine-tuning: fix the parameters of the two modules and only fine-tune the newly added fully connected fusion layer and softmax classification layer at the end;

[0083] (2) Fine-tune the fixed DMF module, fix the parameters in the DMF module, and jointly fine-tune the LTCNN module and the newly added fully connected fusion layer and softmax classification layer;

[0084] (3) Fine-tuning of the fixed LTCNN module: Fix the parameters in the LTCNN module and jointly fine-tune the DMF module and the newly added fully connected fusion layer and softmax classification layer.

[0085] (4) Overall fine-tuning: All modules in the spatiotemporal fusion network are fine-tuned as a whole.

[0086] Step 300: Classify facial expression image sequences

[0087] During training, the DMF and LTCNN sub-networks in the spatiotemporal fusion network are first trained separately. The total loss of the DMF module is defined as follows: Where loss0 is the loss of the softmax loss function; the loss of the LTCNN module is the softmax loss function. Fine-tuning is then performed based on different fine-tuning fusion strategies, and the final classification is performed by the softmax classification layer in the fine-tuning fusion module.

[0088] It should be understood that the present embodiment is only used to illustrate the present invention and is not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the appended claims of the application.

Claims

1. A facial expression recognition method based on spatiotemporal fusion network, characterized in that: The steps include: (1-1) Preprocessing of input video sequence; (1-2) Construct a dual-stream spatiotemporal fusion network; (1-2-1) Use the LTCNN module to extract the temporal features of facial expressions; (1-2-2) Use the DMF module to extract the spatial features of facial expressions; (1-2-3) Use the fine-tuning fusion module to effectively fuse the expression spatiotemporal features of two different dimensions; (1-3) Classify facial expression image sequences; The specific steps of step (1-2-1) are as follows: (2-1) Initialize the LTCNN module; (2-2) The LTCNN module consists of four convolutional layers, two fully connected layers, and a Softmax classification layer. The convolutional layers and fully connected layers are used to further extract high-level semantic features from the input class feature map to learn more discriminative expression temporal features contained in the key point trajectories in the class feature map. (2-3) Finally, the network is trained using cross entropy as the loss function, and the expression category is estimated through the Softmax layer; The specific steps of step (1-2-2) are as follows: (3-1) Initialize the DCNN submodule in the DMF module; (3-2) On the facial expression database, the DCNN submodule is adjusted using the softmax loss function to obtain a pre-trained DCNN submodule; (3-3) The DCNN submodule uses multiple fully connected branch layers, each of which has a fixed-size fully connected layer and an associated threshold π i ,i∈{1,2,...,N}; (3-4) Embed the image x into the Euclidean space to obtain the embedded feature fi(x). During training, each branch is constrained by a symmetric triplet loss, denoted as loss i ,i∈{1,2,...,N}; (3-5) In each batch of training, each training sample is used as a query image, and image samples with the same expression as the query sample and image samples with different expressions are selected to construct a triplet; the query image is used as an anchor point Image samples with the same expression as the query sample are regarded as positive samples Image samples with different expressions as negative samples The form of the symmetric triplet loss is as follows (3-6) Fuse the expression spatial features learned by all branches in the N-metric submodule; (3-7) Use two fully connected layers and the softmax loss function to classify facial expressions; The specific steps of step (1-2-3) are as follows: (4-1) A fine-tuning-based fusion strategy is used to better complement the spatiotemporal features of facial expressions learned by the two modules; The specific steps of the fine-tuning fusion strategy in step (4-1) are as follows: (5-1) First, different learning rates and learning strategies are used to train the LTCNN and DMF modules respectively; (5-2) Remove the softmax layer of the network in the two trained modules, and cascade the last fully connected layer of the two networks together as the input of the newly added fully connected layer to form a spatiotemporal fusion network, thereby achieving effective fusion of the output features of the two networks; (5-3) Finally, we further fine-tune the spatiotemporal fusion network composed of the pre-trained DMF and LTCNN modules, and perform expression classification through the softmax layer. Four different fine-tuning schemes are used to adjust the network; The specific steps of the four different fine-tuning schemes in step (5-3) are as follows: (6-1) Local fine-tuning: fix the parameters of the two modules and only fine-tune the newly added fully connected fusion layer and softmax classification layer at the end; (6-2) Fine-tune the fixed DMF module, fix the parameters in the DMF module, and jointly fine-tune the LTCNN module and the newly added fully connected fusion layer and softmax classification layer; (6-3) Fine-tuning of the fixed LTCNN module: fix the parameters in the LTCNN module, and jointly fine-tune the DMF module and the newly added fully connected fusion layer and softmax classification layer; (6-4) Overall fine-tuning: All modules in the spatiotemporal fusion network are fine-tuned as a whole; The specific steps of steps (1-3) are as follows: (7-1) During training, the DMF and LTCNN sub-networks in the spatiotemporal fusion network are first trained separately. The total loss of the DMF module is defined as follows: Among them, loss0 is the loss of the softmax loss function; The loss of the LTCNN module is the softmax loss function; (7-2) Fine-tune based on different fine-tuning fusion strategies, and finally perform final classification by the softmax classification layer in the fine-tuning fusion module; The specific steps of step (1-1) are as follows: (8-1) Using the sequence normalization method, all sequences are unified into a fixed sequence of 11 frames, and then the facial key points are used to crop the facial region of interest in each frame and the cropped image is scaled to obtain a fixed facial image sequence; (8-2) For a fixed facial image sequence, a key point extraction algorithm is used to propose 51 key point coordinates for each frame of the image to form a class feature map, and an online data enhancement method is used to enhance the class feature map. The enhanced class feature map is used as the input data of the LTCNN module; (8-3) The last expression peak frame image of each sequence is extracted, and the peak expression image is enhanced using offline and online data enhancement methods. The enhanced image is used as the input data of the DMF module.

Images