A facial micro-expression recognition method based on local diversity driven deep network

Abstract

This invention provides a facial micro-expression recognition method based on a local diversity-driven deep network, specifically comprising: Step 1: Preprocessing the sample data, the preprocessing including: calculating the original expression stream image of the original images in the dataset, performing motion detail enhancement on the calculated original expression stream image, and finally expanding the motion detail-enhanced expression stream image; Step 2: Establishing a local diversity facial micro-expression recognition network, and extracting micro-expression-related features from the expanded expression stream image; Step 3: Using a softmax classifier to classify the features from Step 2. This invention improves the accuracy of micro-expression recognition.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology. Background Technology

[0002] Unlike macrofacial expressions, microfacial expressions are spontaneous facial movements characterized by small amplitude, short duration (generally less than 500ms), and localized muscle movements. Microfacial expressions are the result of conscious or unconscious suppression of facial expressions and reveal people's true emotions. When people try to hide their emotions, uncontrollable microfacial expressions reveal their true feelings. Due to the objectivity of microfacial expressions, their recognition has wide applications in fields such as psychology and clinical diagnosis, interrogation, and public safety.

[0003] Similar to macro-expression recognition, micro-expression recognition can be divided into image preprocessing, feature extraction, and expression classification. The most significant factor affecting the effectiveness of micro-expression recognition is the extraction of relevant features from image sequences. Traditional methods rely on handcrafted features such as optical flow and use conventional classification models like support vector machines and random forests to recognize facial micro-expressions based on extracted motion attributes.

[0004] Recently, many studies have proposed custom deep neural networks (DNNs) to learn and extract micro-expression-related features. Some researchers extract features from facial images of subjects, especially peak frames. However, even peak frames in micro-expression sequences suffer from low motion intensity. Therefore, some researchers input computed motion attributes into DNNs to mine micro-expression-related features. Compared to peak frames, these motion attributes are more sensitive to subtle muscle movements. Therefore, features automatically extracted from motion attributes are more suitable for recognizing micro-expressions. Although researchers have successfully inputted motion attributes into custom DNNs, several problems remain. First, because the process of collecting micro-expression data is very expensive, all micro-expression databases are small. Therefore, even with data augmentation using random cropping or flipping of input images, DNNs are prone to overfitting. Furthermore, strictly speaking, optical flow is calculated from two adjacent frames, while micro-expression flow is calculated from the subject's starting and peak frames. Therefore, motion attributes calculated from expression flow may encounter some interfering factors, such as slight facial tremors and changes in lighting. Second, even from augmented motion attributes, it is difficult to extract motion-related features. Previous studies have mostly used networks employed for macro-expression recognition, neglecting the differences between macro-expression recognition and micro-expression recognition. Specifically, facial micro-expressions are often caused by low-intensity muscle movements and exist in localized but diverse facial areas. Summary of the Invention

[0005] Purpose of the invention: In order to solve the problems existing in the prior art, the present invention provides a facial micro-expression recognition method based on local diversity-driven deep networks.

[0006] Technical Solution: This invention provides a facial micro-expression recognition method based on local diversity-driven deep networks, which specifically includes the following steps:

[0007] Step 1: Preprocess the sample data. The preprocessing includes: calculating the original expression stream image of the original image in the dataset, enhancing the motion details of the calculated original expression stream image, and finally expanding the expression stream image after motion detail enhancement.

[0008] Step 2: Establish a local diversity facial micro-expression recognition network to extract micro-expression-related features from the augmented expression stream images;

[0009] Step 3: Use a softmax classifier to classify the features from Step 2.

[0010] Furthermore, the preprocessing in step 1 specifically includes:

[0011] Step 1.1: Resize the original images in the dataset, then extract the facial muscle motion attributes between the start and peak frames of micro-expressions to obtain the expression flow between the start and peak frames of micro-expressions:

[0012] I t (x, y) = I t+a (x+u t (x, y)δt, y+v t (x, y)δt)

[0013] Where t represents the start frame, I t (x, y) represents the light intensity of the pixel at coordinates (x, y) in the starting frame, a represents the time interval between the starting frame and the peak frame, and u t (x, y) and v t (x, y) represent the horizontal and vertical components of the facial expression flow, respectively. t+a (x+u t (x, y)δt, y+v t (x, y)δt) represents the coordinates in the peak frame as (x+u) t (x, y)δt, y+v t The light intensity of pixel (x, y)δt);

[0014] Step 1.2: Based on u t (x, y) and v t Calculate the strain component s at (x, y): where u=[u(x,y),v(x,y)] T T represents transpose. To express differentiation; according to u t (x, y), v t (x, y) and s yield the original facial expression stream image;

[0015] Step 1.3: Downsample the original start frame and peak frame in the dataset, and calculate the expression flow image between the downsampled start frame and peak frame according to Step 1.1 and Step 1.2; then upsample the expression flow image to the size of the original expression flow image to obtain a blurred expression flow image with motion unrelated to micro-expressions;

[0016] Step 1.4: Subtract the blurred expression stream image from the original expression stream image by pixels to obtain a detailed expression stream image related to micro-expressions; then add this detailed expression stream image to the original expression stream image by pixels to obtain a motion-detail-enhanced expression stream image;

[0017] Step 1.5: Adjust the weights of the horizontal and vertical components of the motion detail-enhanced facial expression stream image and calculate the corresponding strain components to expand the number of motion detail-enhanced facial expression stream images.

[0018] Furthermore, the local diversity facial micro-expression recognition network includes first to fifth feature downsampling modules, first to ninth feature enhancement modules, a first adaptive pooling layer, and a fully connected layer; the first feature downsampling module, the first feature enhancement module, the second feature downsampling module, the second feature enhancement module, the third feature enhancement module, the third feature downsampling module, the fourth to seventh feature enhancement modules, the fourth feature downsampling module, the eighth feature enhancement module, the fifth feature downsampling module, the ninth feature enhancement module, the first adaptive pooling layer, and the fully connected layer are connected sequentially.

[0019] Furthermore, the first to fifth feature downsampling modules have the same structure, each including a first convolutional layer, a first batch processing normalization layer, a max pooling layer, and a first P-ReLU activation function layer connected in sequence.

[0020] Furthermore, the feature enhancement module is used to enhance micro-expression related features. The first to ninth feature enhancement modules have the same structure, each including a second convolutional layer, a second batch normalization layer, a third convolutional layer, a third batch normalization layer, a second P-ReLU activation function layer, a spatial channel attention module, and a local diversity feature mining module connected in sequence.

[0021] Furthermore, for the input feature X, the spatial channel attention module outputs the corresponding feature. Sa(·) represents the spatial attention module, and Ca(·) represents the channel attention module;

[0022] The spatial attention module includes a fourth and a fifth convolutional layer and a sigmoid function activation. The fourth and fifth convolutional layers transform the input features of size C×H×W into a spatial attention map of size 1×H×W, where C represents the number of channels, H is the height, and W is the width. The sigmoid function activation multiplies the spatial attention map element-wise with the input features to obtain an output feature of size C×H×W.

[0023] The channel attention module includes a second adaptive pooling layer, a multilayer perceptron, and a softmax activation function. The second adaptive pooling layer transforms the input features of size C×H×W into a C-dimensional vector, and inputs the C-dimensional vector into the multilayer perceptron to obtain a channel attention map. The softmax activation function layer multiplies the channel attention map with the input features element by element to obtain an output feature of size C×H×W.

[0024] Furthermore, the local diversity feature mining module includes a sixth convolutional layer, used to transform input features into pattern features, and sets a local diversity loss function L in the local diversity feature mining module. l-d To achieve the mining of local diversity features, L l-d The expression is:

[0025]

[0026] Where, σ 2 Let f be the variance among all channels, λ be a constant, N be the number of channels for the mode feature, and f be the variance among all channels. i For the i-th channel of the pattern feature, L lc The expression is: Where τ is a constant, θ(f) i ) represents the activation region of the i-th channel.

[0027] Furthermore, when training the facial micro-expression recognition network and softmax classifier for local diversity, the loss function is:

[0028]

[0029] Where ω(epoch) is the weight that changes with each epoch. For all local diversity loss L l-d The mean, L CE Let cross-entropy be the loss function. Where K is the number of samples, p k Let q represent the true probability distribution. k This represents the probability distribution of the prediction.

[0030] Beneficial effects:

[0031] (1) Considering complex factors such as slight facial tremors and changes in lighting, this invention preprocesses the acquired micro-expression images to remove dynamics unrelated to micro-expressions and enhance the motion details related to micro-expressions. To address the limited size of the micro-expression database, a data augmentation strategy based on u / v / s images of faces is proposed. This improves the accuracy of micro-expression motion feature extraction.

[0032] (2) Considering the low intensity of micro-expression movements and the small and diverse regions where expressions occur, this invention proposes a Locally Diverse Facial Micro-expression Recognition Network (LD-FMERN). A spatial channel attention module is introduced to refine the extracted features. A local diversity feature mining strategy is proposed to enable the network to focus on small and diverse facial regions to find clues related to micro-expressions.

[0033] (3) Softmax is used to classify different micro-expressions, and an adaptive loss function of cross-entropy plus local diversity loss is used to constrain LD-FMERN to concentrate on different facial regions at different stages, thereby improving the accuracy of micro-expression recognition. Attached Figure Description

[0034] Figure 1 This is a flowchart of the method of the present invention;

[0035] Figure 2 This is a flowchart of the preprocessing process of the present invention;

[0036] Figure 3 Figure 1 shows the structure diagram of the LD-FMERN network, where Figure (a) is the overall structure diagram of the LD-FMERN network, Figure (b) is the structure diagram of the FDM module, and Figure (c) is the structure diagram of the FEM module.

[0037] Figure 4 This is a structural diagram of the spatial channel attention module;

[0038] Figure 5 This is a structural diagram of the local diversity mining module. Detailed Implementation

[0039] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0040] like Figure 1As shown, the facial micro-expression recognition method proposed in this invention includes an image preprocessing module and a local-diverse facial-micro-expression recognition network (LD-FMERN). The latter includes feature down-sampling modules (FDMs) and feature enhancement modules (FEMs). This invention can simultaneously perform feature extraction and classification of micro-expression images. The invention preprocesses the input micro-expression images, including expression flow computation, motion detail enhancement, and data sample expansion. In the feature extraction stage, channel space attention and local diversity mining strategies are used to obtain more accurate micro-expression features. During the Softmax solution, an adaptive loss function constraint of cross-entropy plus local diversity loss is adopted to improve the accuracy of micro-expression recognition.

[0041] The specific operating steps of this invention are as follows:

[0042] 1) Perform data preprocessing, calculate facial expression flow, enhance motion details, and expand the sample size.

[0043] (1): Expression stream u / v / s calculation

[0044] The original images in the dataset were resized to 148*148 pixels. The TV-L1 algorithm was used to calculate the expression flow, thereby extracting facial muscle motion attributes between the micro-expression initiation frame and the peak frame. Unlike the optical flow defined between consecutive frames, considering the short duration of micro-expressions, the expression flow defined between the initiation frame and the peak frame is as follows: I t (x, y) = I t+a (x+u t (x, y)δt, y+v t (x, y)δt), where I t Let I be the light intensity of the starting frame, (x, y) represent the pixel position, and I... t (x, y) represents the light intensity of the pixel at coordinates (x, y) in the starting frame, t represents the starting frame, a represents the interval between the starting frame and the peak frame, and I t+a For peak frame light intensity, u t (x, y) and v t (x, y) represent the horizontal and vertical components of the facial expression flow (i.e., the u / v image), respectively. Based on the first-order u and v, the second-order strain component s (i.e., the s image) is calculated, defined as follows: In the formula u=[u(x, y), v(x, y)] T , denoted by , and T denotes transpose.

[0045] The TV-L1 algorithm (a mature optical flow calculation method) is used to solve the equations to calculate u / v, and then s is obtained;

[0046] According to u t (x, y), v t (x, y) and s yield the original facial expression stream image (i.e., the u / v / s image).

[0047] (2): Enhanced motion details

[0048] Figure 2 A schematic diagram of the motion detail enhancement algorithm is provided.

[0049] Generate the original u / v / s image from the initial start frame and peak frame according to step (1). Downsample the original start frame and peak frame to 64*64 (unit: pixels), calculate the corresponding expression stream u / v / s image according to step (1), and upsample the obtained expression stream image to the original size to obtain a blurred expression stream image that may only contain motion unrelated to micro-expressions. Subtract the blurred expression stream image from the original expression stream image by pixels to obtain the detailed expression stream image related to micro-expressions. Add the calculated detailed expression stream image to the original expression stream image by pixels to generate a motion-detail-enhanced expression stream image.

[0050] (3): Data sample expansion

[0051] Since the sample size in current micro-expression benchmark databases is insufficient and the cost of collecting micro-expression samples is very high, this invention augments the motion detail enhancement of facial expression stream images. The weights of the horizontal component u and the vertical component v of the facial expression stream are adjusted, and the corresponding strain component s is calculated. The weight of the u component increases from 0.1 to 1.9 in steps of 0.1, while the weight of the v component decreases from 1.9 to 0.1 in steps of 0.1. Using this method, the number of u / v / s images can be expanded by 19 times. The generated u / v / s images still reflect the motion associated with micro-expressions, but the degree of attention paid to the u / v components differs.

[0052] 2) Construct a locally diverse facial micro-expression recognition network to automatically extract effective features.

[0053] like Figure 3As described in (a) of this embodiment, the structure diagram of the local diversity facial micro-expression recognition network is given, including expression stream image input, five feature down-sampling modules (FDMs) and nine feature enhancement modules (FEMs). The last FEM module is followed by an adaptive pooling layer to convert the 2D feature map into a 1D feature vector, and the output is fed into the fully connected layer fc.

[0054] like Figure 3 As shown in (b), the Feature Downsampling Module (FDM) downsamples the feature map to avoid overfitting and reduce computational complexity. It consists of a convolutional layer (Conv), a batch normalization layer to accelerate convergence, a max-pooling layer for downsampling the input feature map, and a P-ReLU layer as the activation function. Initially, the FDM has 3 input channels, 16 output channels, and a convolutional kernel size of 3.

[0055] like Figure 3 As shown in (c), the Feature Enhancement Module (FEM) enhances micro-expression-related features. It consists of two convolutional layers (Conv), each followed by a batch normalization layer. The output of the second batch normalization layer is fed into a P-ReLU activation layer. The activated output is then sequentially fed into a spatial-channel modulator and a local-diverse feature mining module. Initially, the Feature Enhancement Module has 16 input channels, 16 output channels, and a kernel size of 3.

[0056] like Figure 4 The diagram shown is a structural diagram of the spatial channel attention module in the feature enhancement module.

[0057] The spatial channel attention module in the feature enhancement module adopts a parallel computing structure. For the input feature X, the output feature is... Where Sa(·) represents the spatial attention module and Ca(·) represents the channel attention module. The spatial attention module contains two convolutional layers, which transform the input features of size C×H×W (C is the number of channels, H is the feature map height, and W is the feature map width) into a 1×H×W spatial attention map. After activation by the sigmoid function, the resulting spatial attention map is multiplied element-wise with the input features to obtain an output feature of size C×H×W. The channel attention module contains an adaptive pooling layer, which transforms the input features of size C×H×W into a C-dimensional vector. This vector is then passed through a multi-layer perceptron (MLP). In this implementation, a three-layer perceptron is used, with C, C / 4, and C neurons in each layer, respectively, to obtain a channel attention map of size C. After passing through the softmax function, the resulting channel attention map is multiplied element-wise with the input features to obtain an output feature of size C×H×W.

[0058] like Figure 5 The diagram shown is a structural diagram of the local diversity feature mining module in the feature enhancement module.

[0059] The local diversity mining module within the feature enhancement module includes a convolutional layer that transforms input features of size C×H×W into pattern features of size C / 4×H×W. This is achieved by introducing an additional loss function L. lc Local constraints are applied to each channel of the pattern features in a self-supervised manner. Where N is the number of channels, τ is the number of attempts (in this implementation, τ is 0.15), used to control the ratio between the activated region and the entire feature map. θσf i ) represents the activation region of the i-th channel, f i Let be the i-th channel of the pattern feature. The variance σ among all channels is calculated. 2 To introduce diversity constraints, we optimize the local diversity loss L. l-d To achieve local diversity feature mining, λ is a constant, and in this implementation, λ is set to 0.1.

[0060] 3) Use softmax for facial expression classification

[0061] The softmax function represents the classification results of a multi-class classification problem in probabilistic form, for a given input x: Where both the input x and the output y are N-dimensional vectors, Each value in the output vector is in the range [0,1].

[0062] In deep networks, the values ​​in the softmax function can be represented by N neurons, and the probability y of classifying each input x can be expressed as: Where P(y=N|x) represents the probability that input x belongs to the Nth class.

[0063] For solving the softmax function, this invention employs an adaptive loss function L, which combines cross-entropy and local diversity loss. adap As a loss function, it is defined as follows: Where L CE Let ω be the cross-entropy loss function, and ω(epoch) be the weights that vary with each epoch. L represents the local diversity loss calculated by all feature enhancement modules. l-d The mean of the cross-entropy loss function. Where K is the number of samples, p k Let q represent the true probability distribution. k This represents the predicted probability distribution. ω(epoch) is initially set to 0.1. This embodiment uses a gradient descent-based backpropagation algorithm to minimize this distribution, thereby obtaining the network output.

[0064] The above description provides a brief overview of the present invention and is not limited to the scope of work described above. Any simple modifications made to the ideas and methods of the present invention for use in other devices, or any improvements and refinements made without changing the main conceptual principles of the present invention, are all within the protection scope of the present invention.

Claims

1. A facial micro-expression recognition method based on local diversity-driven deep networks, characterized in that, The method specifically includes the following steps: Step 1: Preprocess the sample data. The preprocessing includes: calculating the original expression stream image of the original images in the dataset; performing motion detail enhancement on the calculated original expression stream image; and finally, augmenting the motion detail-enhanced expression stream image. The TV-L1 algorithm is used to calculate the expression stream, and the horizontal and vertical components of the expression stream are obtained. and and according to and Calculate the strain component s; adjust the weights of the horizontal and vertical components of the motion-detail-enhanced facial expression stream image, and calculate the corresponding strain components. The weight of the u component increases from 0.1 to 1.9 in steps of 0.1, while the weight of the v component decreases from 1.9 to 0.1 in steps of 0.1, thereby expanding the number of motion-detail-enhanced facial expression stream images. Step 2: Establish a locally diverse facial micro-expression recognition network to extract micro-expression-related features from the expanded expression stream image; the locally diverse facial micro-expression recognition network includes first to fifth feature downsampling modules, first to ninth feature enhancement modules, a first adaptive pooling layer, and a fully connected layer; the first feature downsampling module, the first feature enhancement module, the second feature downsampling module, the second feature enhancement module, the third feature enhancement module, the third feature downsampling module, the fourth to seventh feature enhancement modules, the fourth feature downsampling module, the eighth feature enhancement module, the fifth feature downsampling module, the ninth feature enhancement module, the first adaptive pooling layer, and the fully connected layer are connected sequentially; The feature enhancement module is used to enhance micro-expression related features. The first to ninth feature enhancement modules have the same structure, including a local diversity feature mining module and a second convolutional layer, a second batch processing normalization layer, a third convolutional layer, a third batch processing normalization layer, a second P-Relu activation function layer, and a spatial channel attention module connected in sequence. The output of the spatial channel attention module is connected to the local diversity feature mining module. The local diversity feature mining module includes a sixth convolutional layer for transforming input features into pattern features. A local diversity loss function is set within the local diversity feature mining module. To achieve the mining of local diversity features, The expression is: ； in, The variance across all channels, Where N is a constant, and N is the number of channels for the pattern features. For the i-th channel of the pattern feature, The expression is: ,in It is a constant. H represents the activation region of the i-th channel, where H is the height and W is the width. Step 3: Use a softmax classifier to classify the features from Step 2.

2. The facial micro-expression recognition method based on local diversity-driven deep networks according to claim 1, characterized in that, The preprocessing in step 1 specifically includes: Step 1.1: Adjusting the size of the original images in the dataset, and then extracting the facial muscle motion attributes between the start frame and the peak frame of the micro-expression to obtain the expression flow between the start frame and the peak frame of the micro-expression. ； Where t represents the start frame. Indicates the coordinates in the starting frame are The light intensity of the pixels, This indicates the time interval between the start frame and the peak frame. and These represent the horizontal and vertical components of the facial expression flow, respectively. Indicates the coordinates in the peak frame as The light intensity of a pixel; Step 1.2: Based on and Calculate the strain component s: ,in T represents transpose. To express differentiation; according to , And s obtain the original facial expression stream image; Step 1.3: Downsample the original start frame and peak frame in the dataset, and calculate the expression flow image between the downsampled start frame and peak frame according to Step 1.1 and Step 1.2; then upsample the expression flow image to the size of the original expression flow image to obtain a blurred expression flow image with motion unrelated to micro-expressions; Step 1.4: Subtract the blurred expression stream image from the original expression stream image by pixels to obtain a detailed expression stream image related to micro-expressions; then add the detailed expression stream image to the original expression stream image by pixels to obtain a motion-enhanced expression stream image.

3. The facial micro-expression recognition method based on local diversity-driven deep networks according to claim 1, characterized in that, The first to fifth feature downsampling modules have the same structure, each including a first convolutional layer, a first batch processing normalization layer, a max pooling layer, and a first P-ReLU activation function layer connected in sequence.

4. The facial micro-expression recognition method based on local diversity-driven deep networks according to claim 1, characterized in that, For input feature X, the spatial channel attention module outputs the corresponding feature. , This indicates a spatial attention module. This indicates the channel attention module; The spatial attention module includes a fourth and a fifth convolutional layer and a sigmoid function activation. The fourth and fifth convolutional layers transform the input features of size C×H×W into a spatial attention map of size 1×H×W, where C represents the number of channels, H is the height, and W is the width. The sigmoid function activation multiplies the spatial attention map element-wise with the input features to obtain an output feature of size C×H×W. The channel attention module includes a second adaptive pooling layer, a multilayer perceptron, and a softmax activation function. The second adaptive pooling layer transforms the input features of size C×H×W into a C-dimensional vector and inputs the C-dimensional vector into the multilayer perceptron to obtain a channel attention map. The softmax activation function layer multiplies the channel attention map element-wise with the input features to obtain an output feature of size C×H×W.

5. The facial micro-expression recognition method based on local diversity-driven deep networks according to claim 1, characterized in that, When training a facial micro-expression recognition network with local diversity and a softmax classifier, the loss function is: epoch ; in, As the weights change with epoch, For all local diversity loss The mean, Let cross-entropy be the loss function. Where K is the number of samples, Represents the true probability distribution. This represents the probability distribution of the prediction.

Images