A method for extracting and recognizing micro-expression features based on deep learning

Abstract

The present invention provides a method for extracting and identifying micro-expression features based on deep learning, comprising the following steps: constructing a four-layer pyramid optical flow model based on key frames, inputting micro-expression video key frames, and obtaining optical flow features; extracting LBP features from three orthogonal planes of a micro-expression image sequence, and cascading the normalized feature gradient histograms obtained in three dimensions into LBP-TOP histogram vectors; converting the optical flow features into a histogram, and cascading and fusing them with the LBP-TOP histogram vectors of the image sequence to obtain a histogram representation of the fused features; constructing a shallow CNN model based on a residual module, introducing a macro expression data set and a micro expression data set, and constructing a cross-dataset micro expression recognition model based on a CNN-GCN transfer learning network; inputting the fused features into a trained cross-dataset micro expression recognition model based on a CNN-GCN transfer learning network for classification, and obtaining micro expression types. This method effectively improves the recognition accuracy of micro expressions.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method for extracting and recognizing micro-expression features based on deep learning. Background Art

[0002] Micro-expressions are spontaneous, relatively subtle facial expression signals. Unlike macro-expression signals, micro-expressions do not change with subjective will, last for about 1 / 25 to 1 / 5 seconds, and have a small amplitude. Studies have shown that micro-expressions can reflect people's potential true emotions when they are under repression. Since the form of micro-expressions is highly correlated with the psychological state when lying, micro-expression recognition has good application prospects in national security, clinical medicine, case detection, judicial interrogation and other fields.

[0003] In terms of micro-expression feature extraction, the existing optical flow method is mainly a method to calculate the image motion information in the optical flow field by using the correlation of pixels in the spatiotemporal domain under the assumption of constant brightness and small amplitude motion. Since the optical flow method combined with the region of interest (ROI) can detect the changes in facial action corresponding to the Facial Action Coding System (FACS), Liu et al. proposed the main directional mean optical-flow (MDMO) method for micro-expression feature extraction, normalized the average current of different ROI directions, and classified and identified the obtained MDMO feature vector through the support vector machine (SVM), which can effectively reduce the feature dimension. Lu et al. proposed a feature extraction method based on motion boundary histogram fusion, which calculated its differential optical flow vector in the horizontal and vertical directions respectively, and verified and evaluated it through the LOSO protocol after feature fusion. The above methods have strict requirements on the application scenarios and need to be modified accordingly to improve their generalization ability; on the other hand, the traditional method directly calculates the optical flow of the image sequence, which has a high computational cost.

[0004] In terms of classification models, convolutional neural networks can mine deep features of visual images and have excellent performance on a variety of image classification datasets. Liong et al. proposed a dual-weighted directional optical flow method (Bi-WOOF method), which only uses the peak frame and the starting frame in the video for feature extraction, and then inputs it into a two-dimensional CNN for classification, achieving good results. However, due to factors such as the insufficient number of samples in the micro-expression database, the above model has an overfitting problem and the model accuracy is low.

[0005] To this end, the present invention proposes a new micro-expression feature extraction and recognition method based on deep learning. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a micro-expression feature extraction and recognition method based on deep learning.

[0007] The present invention provides the following technical solutions.

[0008] A method for extracting and recognizing micro-expression features based on deep learning, comprising the following steps:

[0009] Construct a four-layer pyramid optical flow model based on key frames, input micro-expression video key frames, and obtain optical flow features;

[0010] LBP features are extracted from three orthogonal planes of the micro-expression image sequence, and the normalized feature gradient histograms obtained in three dimensions are cascaded into an LBP-TOP histogram vector;

[0011] The optical flow features are converted into histograms and cascaded with the LBP-TOP histogram vectors of the image sequence to obtain the histogram representation of the fused features;

[0012] Build a shallow CNN model based on residual modules, introduce macro expression datasets and micro expression datasets, and build a cross-dataset micro expression recognition model based on CNN-GCN transfer learning network;

[0013] The fused features are input into the trained cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network for classification to obtain the micro-expression type.

[0014] Preferably, the acquisition of the optical flow feature comprises the following steps:

[0015] For the micro-expression video key frame I with a size of m*n, a downsampling operation is performed, represented by d, and the video key frame I is transformed into a new image of m / 2*n / 2;

[0016] Use the warp operator to distort the image, perform bilinear interpolation, and train the convolutional network in an independent and serialized manner;

[0017] Input a new image, start from the top layer of the pyramid, initialize the residual of the previous layer of optical flow field to 0, add it to the upsampling result of the previous layer of optical flow field, pass the calculated optical flow field to the attention network model, repeat the operation to build a four-layer residual network, calculate the residual, and finally calculate the optical flow field, that is, obtain the optical flow feature;

[0018] Among them, the residual optical flow of each pyramid level is expressed as:

[0019]

[0020] In the formula, {G0,L,Gk} represents a set of trained convolutional neural network models, v k Represents the k-layer optical flow residual, which is determined by the model based on the video frame at the k-layer and And the result u(V k-1 ) is calculated; w and u represent the distortion and upsampling operations on the image respectively. Represents a distorted new image, which is the result of upsampling the optical flow field of the previous layer u(V k-1 ) distorts the second image of layer k; V k Represents the optical flow field result of the kth layer, and the upsampling result u(V k-1 ) and the current layer optical flow residual v k After adding, we get:

[0021] V k =u(V k-1 )+v k .

[0022] Preferably, the step of cascading the normalized feature gradient histograms obtained in three dimensions into an LBP-TOP histogram vector specifically includes:

[0023] The uniform coding method is used to divide the micro-expression image sequence into several non-overlapping block structures in three dimensions: horizontal direction x, vertical direction y and time axis t.

[0024] LBP features are extracted from three orthogonal planes of the micro-expression image sequence respectively;

[0025] Among them, the XY plane represents the spatial information of the video frame, the XT and YT planes represent the motion texture information in the horizontal and vertical directions obtained by scanning the video sequence along the time axis, respectively, and the normalized feature gradient histograms obtained in three dimensions are cascaded into an LBP-TOP histogram vector.

[0026] Preferably, the construction of the shallow CNN model based on the residual module includes the following steps:

[0027] The feature data set obtained by feature extraction is divided into training set, test set and validation set, and a shallow CNN model is built with convolutional layer, pooling layer, fully connected layer and dropout layer as the core.

[0028] Preferably, it also includes:

[0029] Using the skip shortcut connection module, the shallow feature x is input to the output of the training module as the initial residual;

[0030] The deep features in this training module are represented as:

[0031] H(x)=F(x)+x

[0032] Let F(x) infinitely approach 0 to obtain an approximate identity mapping. By learning the residual function F(x), the deep feature training network is decomposed into multiple shallow multi-scale training modules based on the residual structure:

[0033] F(x)→0,H(x)≈x.

[0034] Preferably, the construction of the cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network specifically includes:

[0035] The CASME II feature dataset is used as the training set, SAMM as the test set, and pre-training is performed on the macro-expression dataset FER2013. The quantitative advantages of macro-expressions are used to assist micro-expression recognition, and a cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network is constructed.

[0036] Preferably, the network framework of the cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network includes a CNN-based image feature training module and a GCN-based emotion classification module.

[0037] Preferably, the step of inputting the fusion features into a trained cross-dataset micro-expression recognition model based on a CNN-GCN transfer learning network for classification comprises the following steps:

[0038] The fused features are input into the trained cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network, and the image features learned by CNN are max-pooled to obtain the new global features x:

[0039] x=G[f MP (F CNN (I c ,w CNN ))]

[0040] Among them, I c Represents the fused feature representation of the input image, w CNN Represents the weight parameter in the CNN network model, F CNN is the iterative operation of the hidden layer of the CNN model, f MP () represents the maximum pooling function, G() is the global calculation function of the pooling feature;

[0041] By associating key facial action units and corresponding emotion categories in different micro-expression image sequences, GCN is used to train the target classifier to obtain the hidden association information of different category labels;

[0042] The node features of the previous layer of the current network are repeatedly iterated as the input of this layer, and the node features are updated, so that the target classifier models the strategy of the label vector and classifies the micro-expression images.

[0043] Beneficial effects of the present invention:

[0044] This method first uses a deep learning network model to improve the optical flow extraction effect, and combines the idea of ​​residual network to convert the approximate estimation of the optical flow velocity vector into a strategic solution problem of the optical flow residual. Without minimizing the loss function, the optical flow vector can be automatically updated to improve the generalization ability of the model; secondly, the attention mechanism is used to construct a pyramid optical flow model based on key frames, giving peak frames higher weights, while improving the calculation accuracy and reducing the spatial complexity; further, since the constructed optical flow model only calculates the image velocity vector and ignores the inter-frame motion information of the image sequence, this method fuses the local binary pattern (Local Binary Patterns from ThreeOrthogonalPlanes, LBP-TOP) features of the three orthogonal planes with the optical flow features to obtain the final feature representation, extract the spatiotemporal texture features and optical strain information of the video sequence, to make up for the lack of sequence information description in the optical flow method, while reducing the feature dimension and reducing the computational cost.

[0045] This method improves the CNN network architecture and constructs a micro-expression cross-dataset recognition model based on the CNN-GCN transfer learning network. The unique advantages of the graph convolutional network GCN are used to analyze the hidden connections between image fusion features and label vectors. According to the idea of ​​weight-based transfer learning network, the model is pre-trained on the macro-expression dataset, and the advantages of macro-expression data are used to alleviate the overfitting problem of the network model. At the same time, the SAMM micro-expression dataset is introduced to enhance the diversity of data samples and further improve the network recognition performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a method flow chart of an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of a key-frame-based pyramid residual optical flow feature calculation process according to an embodiment of the present invention;

[0048] Figure 3 It is a block feature extraction and three orthogonal plane schematic diagram of LBP-TOP in an embodiment of the present invention;

[0049] Figure 4 It is a micro-expression recognition model diagram based on optical flow motion features and LBP-TOP features according to an embodiment of the present invention;

[0050] Figure 5is a CNN model diagram with a residual module added in an embodiment of the present invention;

[0051] Figure 6 Schematic diagram of a micro-expression recognition network framework based on CNN-GCN according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] Example 1

[0054] A method for extracting and recognizing micro-expression features based on deep learning of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0055] S1. Construct a four-layer pyramid optical flow model based on key frames, input micro-expression video key frames, and obtain optical flow features. Specifically:

[0056] This method improves the optical flow method, trains CNN at each pyramid level, automatically optimizes and updates the optical flow vector, and gradually refines the image motion boundaries and details without minimizing the loss function. It improves the calculation accuracy in real time while reducing the spatial complexity and saving the space occupied by the model. At the same time, considering that the peak frame of micro-expression contains most of the optical strain information, this method combines the attention mechanism on the basis of the constructed optical flow model, assigns more weight to the peak frame, and obtains a more accurate optical flow vector expression.

[0057] The algorithm is described as follows:

[0058] For the micro-expression video key frame I with a size of m*n, a downsampling operation is performed, represented by d, and the video key frame I is transformed into a new image of m / 2*n / 2;

[0059] Use the warp operator to distort the image, perform bilinear interpolation, and train the convolutional network in an independent and serialized manner;

[0060] Input a new image, start from the top layer of the pyramid, initialize the residual of the previous layer of optical flow field to 0, add it to the upsampling result of the previous layer of optical flow field, pass the calculated optical flow field to the attention network model, repeat the operation to build a four-layer residual network, calculate the residual, and finally calculate the optical flow field, that is, obtain the optical flow feature;

[0061] Among them, the residual optical flow of each pyramid level is expressed as:

[0062]

[0063] In the formula, {G0,L,G k} represents a set of trained convolutional neural network models, v k Represents the k-th layer optical flow residual, which is determined by the model based on the video frame at the k-th layer and And the result u(V k-1 ) is calculated; w and u represent the distortion and upsampling operations on the image respectively. Represents a distorted new image, which is the result of upsampling the optical flow field of the previous layer u(V k-1 ) distorts the second image of layer k; V k Represents the optical flow field result of the kth layer, and the upsampling result u(V k-1 ) and the current layer optical flow residual v k After adding, we get:

[0064] V k =u(V k-1 )+v k .

[0065] The calculation process of pyramid residual optical flow features based on key frames is as follows Figure 2 shown.

[0066] S2. Extract LBP features from three orthogonal planes of the micro-expression image sequence, and concatenate the normalized feature gradient histograms obtained in three dimensions into an LBP-TOP histogram vector. Specifically:

[0067] The algorithm feature extraction process is as follows Figure 3 As shown, Figure 3 (a) is the block feature extraction of LBP-TOP. Figure 3 (b) Three orthogonal planes of the micro-expression sequence.

[0068] In the process of extracting LBP-TOP features, directly operating on the image sequence will lead to the loss of pixel information and cannot process motion information and image details well. Therefore, it is necessary to divide the micro-expression sequence into blocks. Using uniform coding, the image sequence is divided into several non-overlapping block structures in three dimensions: horizontal direction x, vertical direction y and time axis t. LBP features are extracted from three orthogonal planes of the micro-expression image sequence, where the XY plane represents the spatial information of the video frame, and the XT and YT planes represent the horizontal and vertical motion texture information obtained by scanning the video sequence along the time axis. The normalized feature gradient histograms obtained in the three dimensions are cascaded into the LBP-TOP histogram vector, which can effectively describe the spatiotemporal texture information of the video frame.

[0069] S3, convert the optical flow features into histograms, and perform cascade fusion with the LBP-TOP histogram vector of the image sequence to obtain the histogram representation of the fused features. The algorithm model is as follows Figure 4 shown.

[0070] S4. Build a shallow CNN model based on the residual module. Based on this, introduce the macro expression dataset and micro expression dataset, and build a cross-dataset micro expression recognition model based on the CNN-GCN transfer learning network. Specifically:

[0071] S4.1. The construction of a shallow CNN model based on a residual module includes the following steps:

[0072] The feature data set obtained by feature extraction is divided into training set, test set and validation set, and a shallow CNN model is built with convolutional layer, pooling layer, fully connected layer and dropout layer as the core.

[0073] Considering that it will be combined with GCN to build a deep network model in the future, network training will become more and more difficult as the number of layers increases. To address this problem, this method further improves the shallow CNN and divides the overall CNN network architecture into several training modules according to different feature scales.

[0074] Using the skip shortcut connection module, the shallow feature x is input to the output of the training module as the initial residual;

[0075] The deep features in this training module are represented as:

[0076] H(x)=F(x)+x

[0077] Let F(x) infinitely approach 0 to obtain an approximate identity mapping. By learning the residual function F(x), the deep feature training network is decomposed into multiple shallow multi-scale training modules based on the residual structure:

[0078] F(x)→0,H(x)≈x.

[0079] The improved model is Figure 5 shown.

[0080] S4.2. Construction of a cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network, specifically including:

[0081] The CASME II feature dataset is used as the training set, SAMM as the test set, and pre-training is performed on the macro-expression dataset FER2013. The quantitative advantages of macro-expressions are used to assist micro-expression recognition, and a cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network is constructed.

[0082] Among them, the network framework of the cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network includes a CNN-based image feature training module and a GCN-based emotion classification module. The principle diagram is as follows: Figure 6 shown.

[0083] Figure 6 In the above example, C represents the number of emotion categories in the dataset, and F CNN It means that CNN is used to extract features from the image. The feature dimension extracted is D×h×w. After global maximum pooling, the dimension is reduced to D. In the GCN-based sentiment classification module, the word embedding vectors of different categories of labels are first obtained according to the combination of face action units AUs in each key area. GC represents the graph convolution operation on each label vector. The word embedding vector dimension of the original label is d. After feature mapping, the label vector dimension is transformed to D, which is determined by the dimension of the image feature representation obtained in CNN.

[0084] S5. Input the fused features into the trained cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network for classification to obtain the micro-expression type. Specifically:

[0085] The fused features are input into the trained cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network, and the image features learned by CNN are max-pooled to obtain the new global features x:

[0086] x=G[f MP (F CNN (I c ,w CNN ))]

[0087] Among them, I c Represents the fused feature representation of the input image, w CNN Represents the weight parameter in the CNN network model, F CNN is the iterative operation of the hidden layer of the CNN model, f MP () represents the maximum pooling function, G() is the global calculation function of the pooling feature;

[0088] By associating key facial action units and corresponding emotion categories in different micro-expression image sequences, GCN is used to train the target classifier to obtain the hidden association information of different category labels;

[0089] The node features of the previous layer of the current network are repeatedly iterated as the input of this layer, and the node features are updated, so that the target classifier models the strategy of the label vector and classifies the micro-expression images.

[0090] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting and recognizing micro-expression features based on deep learning, characterized in that: The following steps are involved: Construct a four-layer pyramid optical flow model based on key frames, input micro-expression video key frames, and obtain optical flow features; LBP features are extracted from three orthogonal planes of the micro-expression image sequence, and the normalized feature gradient histograms obtained in three dimensions are cascaded into an LBP-TOP histogram vector; The optical flow features are converted into histograms and cascaded with the LBP-TOP histogram vectors of the image sequence to obtain the histogram representation of the fused features; Build a shallow CNN model based on residual modules, introduce macro expression datasets and micro expression datasets, and build a cross-dataset micro expression recognition model based on CNN-GCN transfer learning network; Input the fused features into the trained cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network for classification to obtain the micro-expression type; The construction of the shallow CNN model based on the residual module includes the following steps: The feature dataset obtained by feature extraction is divided into training set, test set and validation set, and a shallow CNN model is built with convolution layer, pooling layer, fully connected layer and dropout layer as the core; Also includes: Using the skip shortcut connection module, the shallow feature x is input to the output of the training module as the initial residual; The deep features in this training module are represented as: H(x)=F(x)+x Let F(x) infinitely approach 0 to obtain an approximate identity mapping. By learning the residual function F(x), the deep feature training network is decomposed into multiple shallow multi-scale training modules based on the residual structure: F(x)→0,H(x)≈x The construction of the cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network specifically includes: The network framework of the cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network includes a CNN-based image feature training module and a GCN-based emotion classification module; the CASME II feature dataset is used as a training set, the SAMM is used as a test set, and pre-training is performed on the macro expression dataset FER2013, the quantitative advantage of macro expressions is used to assist micro-expression recognition, and a cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network is trained and constructed; The step of inputting the fusion features into the trained cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network for classification includes the following steps: The fused features are input into the trained cross-dataset micro-expression recognition model based on the CNN-GCN transfer learning network, and the image features learned by CNN are max-pooled to obtain the new global features x: x=G[f MP (F CNN (I c ,w CNN ))] Among them, I c represents the fused feature representation of the input image, w CNN Represents the weight parameter in the CNN network model, F CNN is the iterative operation of the hidden layer of the CNN model, f MP () represents the maximum pooling function, G() is the global calculation function of the pooling feature; By associating key facial action units and corresponding emotion categories in different micro-expression image sequences, GCN is used to train the target classifier to obtain the hidden association information of different category labels; The node features of the previous layer of the current network are repeatedly iterated as the input of this layer, and the node features are updated, so that the target classifier models the strategy of the label vector and classifies the micro-expression images.

2. The method for extracting and identifying micro-expression features based on deep learning according to claim 1, characterized in that: The acquisition of the optical flow feature includes the following steps: For the micro-expression video key frame I with a size of m*n, a downsampling operation is performed, represented by d, and the video key frame I is transformed into a new image of m / 2*n / 2; Use the warp operator to distort the image, perform bilinear interpolation, and train the convolutional network in an independent and serialized manner; Input a new image, start from the top layer of the pyramid, initialize the residual of the previous layer of optical flow field to 0, add it to the upsampling result of the previous layer of optical flow field, pass the calculated optical flow field to the attention network model, repeat the operation to build a four-layer residual network, calculate the residual, and finally calculate the optical flow field, that is, obtain the optical flow feature; Among them, the residual optical flow of each pyramid level is expressed as: In the formula, {G0,…,G k } represents a set of trained convolutional neural network models, v k Represents the k-th layer optical flow residual, which is determined by the model based on the video frame at the k-th layer and And the result u(V k-1 ) is calculated; w and u represent the distortion and upsampling operations on the image respectively. Represents a distorted new image, which is the result of upsampling the optical flow field of the previous layer u(V k-1 ) distorts the second image of layer k; V k Represents the optical flow field result of the kth layer, and the upsampling result u(V k-1 ) and the current layer optical flow residual v k After adding, we get: V k =u(V k-1 )+v k 。 3. The method for extracting and identifying micro-expression features based on deep learning according to claim 1, characterized in that: The process of cascading the normalized feature gradient histograms obtained in three dimensions into an LBP-TOP histogram vector specifically includes: The uniform coding method is used to divide the micro-expression image sequence into several non-overlapping block structures in three dimensions: horizontal direction x, vertical direction y and time axis t. LBP features are extracted from three orthogonal planes of the micro-expression image sequence respectively; Among them, the XY plane represents the spatial information of the video frame, the XT and YT planes represent the motion texture information in the horizontal and vertical directions obtained by scanning the video sequence along the time axis, respectively, and the normalized feature gradient histograms obtained in three dimensions are cascaded into an LBP-TOP histogram vector.

Images