A micro-expression recognition method based on multi-dimensional feature fusion

Abstract

The present invention discloses a micro-expression recognition method based on multi-dimensional feature fusion, comprising four steps: image preprocessing, extraction of optical flow feature components, construction of a multi-dimensional feature fusion network, and micro-expression classification using the trained model. In order to compensate for the facial detail information lost by the model, the present invention constructs a feature fusion module that fuses the shallow features extracted by the feature fusion module with the abstract features extracted by the dual-stream convolutional network, and uses both for model classification. The extracted high-dimensional fused features are assigned different weights by a channel attention module, allowing the model to pay more attention to channels with high contribution, further improving the accuracy of micro-expression recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of expression recognition, and in particular to a micro-expression recognition method based on multi-dimensional feature fusion. Background Art

[0002] Microexpressions are short-lived, low-intensity expressions that often appear involuntarily when people attempt to conceal their true inner thoughts. Because microexpressions typically linger on the face for only 1 / 25 to 1 / 3 of a second, most people are unaware of their presence. These unique expressions are believed to be linked to a person's self-defense mechanism, reflecting their true inner thoughts. Accurately identifying microexpressions helps people make appropriate judgments and decisions, making microexpression recognition extremely important. Microexpression recognition is essentially an image classification problem. In recent years, with the advancement of deep learning, computers have begun to be used for microexpression analysis and recognition. The advantage is that by training accurate and stable models, large-scale microexpression recognition tasks can be automatically and efficiently processed at low cost, saving manpower and resources.

[0003] Reference 1 [Gan YS, Liong ST, Yau WC, et al. Off-apexnet on micro-expression recognition system [J]. Signal Processing: Image Communication, 2019, 74: 129-139.] designed a dual-branch convolutional neural network model, extracting two components of the optical flow features of the starting frame and the vertex frame as the input of the two branches of the model. On this basis, Reference 2 [Jin Qiushi, Xu Huangchao, Liu Kunhong, et al. Ga-apexnet: Genetic algorithm in apex frame network for micro-expression recognition system [C] / / Proceedings of the Journal of Physics: Conference Series. Suzhou: IOP Press, 2020, 1544 (1): 012149.] introduced a genetic algorithm (GA) to reduce the dimensionality of the features learned by the model, retaining only the features that are beneficial to classification. Finally, after testing, the accuracy of the model has been further improved. Reference 3 [Nie Xuan, Takalkar MA, Duan Mengyang, et al. GEME: Dual-stream multi-task gender-based micro-expression recognition [J]. Neurocomputing, 2021, 427: 13-28.], Reference 4 [[Zhou Ling, Mao Qirong, Xue Luoyang. Dual-inception network for cross-database micro-expression recognition [C] / / Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition. Lille: IEEE Press, 2019: 1-5.] also uses a dual-stream network model to extract different features. The difference is that reference [3] introduces a multi-task learning method, using a network branch to detect gender to assist model classification.The overall idea of ​​reference [4] is the same as that of reference [1], which is to extract the optical flow feature components of the starting frame and the vertex frame as input. However, reference [4] uses the inception block module when constructing the model, that is, it uses convolution kernels of multiple sizes in parallel at the same level.

[0004] The above existing models do not pay enough attention to facial details, and the micro-expression recognition effect needs to be improved. Summary of the Invention

[0005] To address the problem that existing models lack attention to detailed facial information, the present invention discloses a micro-expression recognition method based on multidimensional feature fusion and designs a two-stream convolutional neural network based on multidimensional feature fusion. To compensate for the lost facial details, a feature fusion module is constructed to fuse the shallow features extracted by the feature fusion module with the abstract features extracted by the two-stream convolutional network, which are then used together for model classification. The extracted high-dimensional fused features are assigned different weights by a channel attention module, allowing the model to pay more attention to channels with high contribution, further improving the accuracy of micro-expression recognition.

[0006] In view of this, the technical solution adopted by the present invention is: a micro-expression recognition method based on multi-dimensional feature fusion, characterized in that it includes the following steps:

[0007] Step 1: Image preprocessing, including image grayscale conversion, facial key point detection, facial alignment and cropping, and scale normalization.

[0008] Step 2: Use the iterative Lucas-Kanade algorithm to extract the horizontal component u and the vertical component v of the optical flow between the starting frame and the vertex frame.

[0009] Step 3: Construct a multi-dimensional feature fusion network, specifically: use the horizontal component u and the vertical component v of the optical flow as the input of the two-stream convolutional neural network. The backbone network adopts a symmetrical structure. The first, third and fourth layers use convolution layers to extract the texture information and edge features of the optical flow features. The second and fifth layers use maximum pooling layers to downsample the input feature dimensions. The output of the first convolution layer in the backbone network is fused through the feature fusion module, and then fused again with the output features of the maximum pooling layer of the fifth layer of the two branches to obtain an output containing facial detail information and abstract features. After the multi-dimensional feature fusion, the channel attention module is introduced to assign different importance to different channels, highlighting the features that are useful for model classification and discrimination. Then two fully connected layers are introduced, and the features output from the fully connected layers are passed to the output layer for classification through the softmax function.

[0010] Step 4: Use the trained model to classify micro-expressions.

[0011] The present invention has the following beneficial technical effects:

[0012] The present invention uses the horizontal and vertical components of optical flow as inputs to the model, constructs a multi-dimensional feature fusion module, fuses the shallow features of facial details extracted by it with the abstract features extracted by the two-stream convolutional network, introduces a channel attention module, and achieves good recognition results. Specifically:

[0013] (1) The fast and robust iterative Lucas-Kanade (iLK) algorithm is used to obtain the optical flow feature component, which can better highlight the subtle movements of the face.

[0014] (2) In the dual-stream convolutional neural network model, the horizontal component of the optical flow and the vertical component of the optical flow are learned and fused separately, and the constructed network model has high recognition accuracy.

[0015] (3) The shallow features extracted by the feature fusion module are fused with the abstract features extracted by the two-stream convolutional network and used together for model classification to enrich the features.

[0016] (4) Introduce the channel attention mechanism at the appropriate position, assign different importance to different channels, and highlight the features that are useful for model classification and discrimination. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a multi-dimensional feature fusion model;

[0018] Figure 2 It is the feature fusion module FFM; DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clear, the specific embodiments of the present invention will be further described in detail below.

[0020] A micro-expression recognition method based on multi-dimensional feature fusion is divided into four steps: image preprocessing, extracting optical flow feature components, building a multi-dimensional feature fusion network, and using the trained model to classify micro-expressions.

[0021] Step 1: Image preprocessing

[0022] Compared to regular expressions, microexpressions have three distinct characteristics: short duration, low intensity, and localized motion. Given these characteristics, image preprocessing is crucial for microexpression recognition. This includes image grayscale conversion, facial landmark detection, facial alignment and cropping, and scale normalization.

[0023] Since the number of micro-expression samples is small, if color images are used in the experiment, it will cause a series of problems such as difficulty in training. Therefore, the present invention uses formula (1) to grayscale the original color image.

[0024] Gray(x,y)=0.299×R(x,y)+0.587×G(x,y)+0.114×B(x,y) (1)

[0025] Among them, Gray is the pixel grayscale value, x, y represent the spatial position index of the target pixel, and R, G, B are the values ​​of the three channels of the RGB image.

[0026] The original images in the micro-expression dataset contain a large amount of background image noise, including the laboratory background wall and the participants' headphones. Only the facial area of ​​the subject is helpful for the model to recognize micro-expressions. Therefore, the present invention aligns and crops the facial area.

[0027] Before cropping the facial region, the present invention uses a cascaded shape regression method to detect facial key points in the micro-expression dataset. This cascaded shape regression method adjusts the initial shape, allowing for convenient data augmentation, high computational efficiency, and strong versatility.

[0028] Based on the facial key points extracted by positioning, the present invention uses formula (2) and formula (3) to calculate the distance d between the two eyes and the angle θ between the line connecting the center coordinates of the left and right eyes and the horizontal direction.

[0029]

[0030]

[0031] Where x1, y1 are the coordinates of the inner corner of the left eye, x2, y2 are the coordinates of the inner corner of the right eye, d is the distance between the two eyes, and θ is the angle between the line connecting the left and right eyes and the horizontal direction.

[0032] Then, the midpoint of the eye distance between the two eyes (x0, y0) is calculated using the coordinates of the two eyes using formula (4).

[0033]

[0034] The center of the eye distance is used as the rotation center. The position of the key points of the face after rotation is calculated as shown in formula (5):

[0035]

[0036] Where x,y are the spatial position indices of the target pixel before adjustment, and x′,y′ are the spatial position indices of the target pixel after adjustment. After face alignment, a face cropping operation is performed, usually cropping the participant's facial area into a rectangle.

[0037] Finally, scale normalization is performed. In the image preprocessing stage of the present invention, all micro-expression images are uniformly adjusted to a size of 112×112 using formula (6).

[0038]

[0039] Where g′ is the pixel value of the image after adjustment, g is the pixel value of the image before adjustment, w and h are the width and height of the image before adjustment, and w′ and h′ are the width and height of the image after adjustment.

[0040] Step 2: Extract optical flow feature components

[0041] Optical flow can extract representative motion features. After completing image preprocessing in step 1, the present invention uses the iterative Lucas-Kanade (iLK) algorithm to extract the horizontal component u and vertical component v of the optical flow between the start frame and the vertex frame. The start frame is the moment when the micro-expression appears, and the vertex frame is the moment when the micro-expression has the largest amplitude and is the frame richest in facial information.

[0042] The key idea behind the optical flow method is to calculate the temporal changes in the same pixel between two adjacent frames, thereby determining the motion changes of the same pixel between the two frames. Therefore, optical flow is often used in image processing to describe moving objects. The iterative Lucas-Kanade (iLK) solver is applied to each level of the image pyramid. iLK is a fast and robust alternative to the TVL1 algorithm.

[0043] Step 3: Build a multi-dimensional feature fusion model

[0044] The multi-dimensional feature fusion model of the present invention is as follows Figure 1 As shown, the horizontal component u and vertical component v of the optical flow are used as inputs to a two-stream convolutional neural network. The backbone network adopts a symmetrical structure. The first, third, and fourth layers use convolutional layers to extract texture information and edge features from the optical flow features. All convolution kernels in the model are set to 5×5 with a stride of 1. The number of kernels in the convolutional layers is 64, 128, and 128, respectively, and the ReLU function is used as the activation function. The second and fifth layers of the backbone network use max pooling layers to downsample the input feature dimensions, reducing the number of parameters by half. The pooling layer window size is 2×2 with a stride of 1.

[0045] The output of the first convolutional layer in the backbone network is fused through the Feature Fusion Module (FFM) and then fused again with the output features of the fifth layer of the maximum pooling layer of the two branches to obtain an output containing facial detail information and abstract features for the final classification and judgment of the model.

[0046] After multi-dimensional feature fusion, the output feature dimension is relatively high. Feature maps that do not contribute much to classification will interfere with the model's judgment and affect the final classification accuracy. The present invention introduces a channel attention module (Squeeze-and-Excitation Networks, SENet) to solve this problem, assigning different importance to different channels and highlighting features that are useful for model classification and discrimination. Then, two fully connected layers are introduced to reduce the number of parameters in the model and prevent overfitting. The number of nodes in each layer is 256. The features output from the fully connected layer are passed to the output layer and classified by the softmax function. Detailed model parameters are shown in Table 1.

[0047] Specifically, the probability of being classified as the lth type of expression is:

[0048]

[0049] Where p, x, l, L, and y are the model prediction label, input image, predicted emotion label, total number of emotion categories, and true label, respectively. j , x n Represent the output values ​​of the jth and nth nodes respectively.

[0050] The present invention uses the Adam optimizer to update the model parameters and perform optimization. The loss function uses the classic cross entropy loss function (CrossEntropy Loss, CELoss), and the calculation formula is as follows:

[0051]

[0052] where y i is a label and y i ∈{0,1},p i is the predicted value output by the model.

[0053] Table 1 Model parameters

[0054]

[0055] The FFM module is introduced in detail below.

[0056] The detailed structure of the FFM designed by the present invention is as follows Figure 2 shown.

[0057] Figure 2 In the above example, Conv represents the convolution layer and M represents the maximum pooling layer. Assume that the input feature is x m , that is, the first m The layer output contains the features of c channels, so the input features can be expressed as Before fusion, a one-dimensional convolution operation is performed on the two features from the shallow layer to unify their dimensions. The transformed features can be expressed as:

[0058]

[0059] Where, χ m ,χ m′ are the input features and output features, c is the channel of the input features and output features, w is the convolution kernel weight, and b is the bias term. Represents the input features of a certain channel.

[0060] After the one-dimensional convolution operation, the two feature maps are superimposed based on the channel, that is,

[0061]

[0062] χ n′ Indicates the output of the FFM lower branch after the convolution layer (see Figure 2 ).

[0063] Then, the Relu function is used to activate the fused high-dimensional features and input them into the maximum pooling layer to reduce the feature dimension and extract more representative features.

[0064] Step 4: Use the trained model to classify micro-expressions

[0065] The Leave-One-Subject-Out (LOSO) method is used on the CASME II dataset. That is, there are samples of K participants in the dataset. The image data of one person is selected as the test set, and the data of the remaining K-1 people are used as the training set. The sample of each person will be used as a test set.

[0066] The deep learning frameworks used are TensorFlow 2.4.0 and Keras 2.4.3, with a learning rate of 0.0001, a batch size of 32, and 100 training epochs.

[0067] The unweighted F1-score (UF1) value and unweighted average recall (UAR) of the test results are calculated using formulas (11)-(14):

[0068]

[0069] in:

[0070]

[0071]

[0072] in:

[0073]

[0074] Where C is the total number of emotion categories, TP i TN i FP i 、FN i They represent the number of true positives, true negatives, false positives, and false negatives in the i-th category, respectively.

[0075] Compared with the current methods, the leave-one-out cross-validation (LOSO) method is used on the CASME II micro-expression dataset, and the recognition effect is better. The comparison between UF1 and UAR is shown in the following table.

[0076] method UF1 UAR <![CDATA[LBP-TOP[ 1 ]]]> 0.7026 0.7429 <![CDATA[Bi-WOOF[ 2 ]]]> 0.7805 0.8026 <![CDATA[OFF-Apex [3] ]]> 0.8764 0.8681 <![CDATA[Ga-apexnet [4] ]]> 0.8634 0.9232 <![CDATA[MERFGR[ 5 ]]]> 0.871 0.8798 <![CDATA[MJFN[ 6 ]]]> 0.9151 0.8871 <![CDATA[MERASTC[ 7 ]]]> 0.854 0.862 <![CDATA[MERSiamC3D[ 8 ]]]> 0.8818 0.8763 This embodiment 0.9406 0.9502

[0077] The references mentioned in the above table are as follows:

[0078] [1]Zhao Guoying,Pietikainen M.Dynamic texture recognition using localbinary patterns with an application to facial expressions[J].IEEETransactions on Pattern Analysis and Machine Intelligence,2007,29(6):915-928.

[0079] [2]Liong ST,SeeJ.,Wong KS,et al.Less is more:Micro-expressionrecognition from video using apex frame[J].Signal Processing:ImageCommunication,2018,62:82-92.

[0080] [3]Gan YS, Liong ST, Yau WC, et al. Off-apexnet on micro-expressionrecognition system[J]. Signal Processing:Image Communication, 2019,74:129-139.

[0081] [4]Jin Qiushi,Xu Huangchao,Liu Kunhong,et al.Ga-apexnet:Geneticalgorithm in apex frame network for micro-expression recognition system[C] / / Proceedings of the Journal of Physics:Conference Series.Suzhou:IOP Press,2020,1544(1):012149.

[0082] [5]Lei Ling,Chen Tong,Li Shigang,et al.Micro-expression recognitionbased on facial graph representation learning and facial action unit fusion[OL]URL:[2021-6-25]. https: / / openaccess.thecvf.com / content / CVPR2021W / AUVi / papers / Lei_Micro-Expression_Rec ognition_Based_on_Facial_Graph_Representation_Learning_and_Facial_CVPRW_2021_paper.pdf.

[0083] [6]Li Xinyu,Wei Guangshun,Wang Jie,et al.Multi-scale joint featurenetwork for micro-expression recognition[J].Computational Visual Media,2021,7(3):407-417.

[0084] [7]Gupta P.MERASTC:Micro-expression recognition using effectivefeature encodings and 2D convolutional neural network[J].IEEE Transactions onAffective Computing,doi:10.1109 / TAFFC.2021.3061967,2021.

[0085] [8]Zhao Sirui,Tao Hanqing,Zhang Yangsong,et al.A two-stage 3D CNNbased learning method for spontaneous micro-expression recognition[J].Neurocomputing,2021,448:276-289。

Claims

1. A micro-expression recognition method based on multi-dimensional feature fusion, characterized in that: The following steps are involved: Step 1: Image preprocessing, including image grayscale conversion, facial key point detection, facial alignment and cropping, and scale normalization; Step 2: Use the iterative Lucas-Kanade algorithm to extract the horizontal component u and the vertical component v of the optical flow between the starting frame and the vertex frame; Step 3: Build a multi-dimensional feature fusion network, specifically: The horizontal component u and the vertical component v of the optical flow are used as the input of the two-stream convolutional neural network. The backbone network adopts a symmetrical structure. The first, third, and fourth layers use convolutional layers to extract texture information and edge features of the optical flow features. The second and fifth layers use maximum pooling layers to downsample the input feature dimensions. The output of the first convolutional layer in the backbone network is fused through the feature fusion module and then fused again with the output features of the maximum pooling layer of the fifth layer of the two branches to obtain an output containing facial detail information and abstract features. After multi-dimensional feature fusion, a channel attention module is introduced to assign different importance to different channels, highlighting features that are useful for model classification and discrimination. Then, two fully connected layers are introduced. The features output from the fully connected layers are passed to the output layer and classified by the softmax function. The feature fusion module is specifically: Let the input feature be x m , that is, the output of the mth layer of the network, contains the features of c channels, then the input features can be expressed as Before fusion, a one-dimensional convolution operation is performed on the two features from the shallow layer to unify their dimensions. The transformed features are represented as follows: Where, χ m ,χ m′ are input features and output features, c is the channel of input features and output features, w is the convolution kernel weight, b is the bias term, Represents the input features of a certain channel, After the one-dimensional convolution operation, the two feature maps are superimposed based on the channel, that is, χ n′ Represents the output of the lower branch of the FFM module after the convolution layer; Then use the Relu function to activate the fused high-dimensional features and input them into the maximum pooling layer; Step 4: Use the trained model to classify micro-expressions.

2. The micro-expression recognition method based on multi-dimensional feature fusion according to claim 1, characterized in that: The facial key point detection described in step 1 uses a cascade shape regression-based method to detect facial key points in the micro-expression dataset.

3. The micro-expression recognition method based on multi-dimensional feature fusion according to claim 2, characterized in that: The face alignment and cropping in step 1 includes: Use formula (1) and formula (2) to calculate the distance d between the two eyes and the angle θ between the line connecting the center coordinates of the left and right eyes and the horizontal direction; Where x1, y1 are the coordinates of the inner corner of the left eye, x2, y2 are the coordinates of the inner corner of the right eye, d is the distance between the two eyes, and θ is the angle between the line connecting the left and right eyes and the horizontal direction; Then, the midpoint of the eye distance between the two eyes (x0, y0) is calculated using the coordinates of the two eyes using formula (3). The center of the eye distance is used as the rotation center. The position of the key points of the face after rotation is calculated as shown in formula (4): Among them, x, y are the spatial position indexes of the target pixel before adjustment, and x′, y′ are the spatial position indexes of the target pixel after adjustment. After face alignment, face cropping operation is performed.

4. The micro-expression recognition method based on multi-dimensional feature fusion according to claim 1, characterized in that: The starting frame in step 2 is the moment when the micro-expression appears, and the apex frame is the moment when the micro-expression has the largest amplitude.

5. A micro-expression recognition method based on multi-dimensional feature fusion according to any one of claims 1 to 4, characterized in that: The probability of the micro-expression being classified as the first type of expression is: Where p, x, l, L, and y are the model prediction label, input image, predicted emotion label, total number of emotion categories, and true label, respectively. j , x n Represent the output values ​​of the jth and nth nodes respectively.

Images