A micro-expression recognition method based on convolutional neural network and optical flow features

Abstract

This invention belongs to the field of facial expression recognition and provides a micro-expression recognition method based on convolutional neural networks and optical flow features, which is used to improve the adaptability and accuracy of micro-expression recognition. The method first constructs a face detection module to perform face detection on each image frame in a video segment, outputs facial key point information, and extracts facial regions of interest (ROIs) based on the facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow. Then, optical flow information and optical strain information are introduced to characterize the spatiotemporal information of facial movement and the intensity information of facial deformation. The optical flow eigenvalue of each image frame is calculated, and a binary search algorithm is used to search for the image frame with the maximum optical flow eigenvalue, which is used as the vertex frame for micro-expression recognition. Finally, a micro-expression recognition model based on a convolutional neural network is constructed, which uses the horizontal and vertical components of the optical flow and optical strain of the vertex frame as input. The micro-expression recognition model outputs a micro-expression category prediction result.

Description

Technical Field

[0001] The present invention belongs to the field of expression recognition, relates to micro-expression recognition technology, and specifically provides a micro-expression recognition method based on convolutional neural networks and optical flow features. Background Art

[0002] Facial expressions can directly convey emotional information. Compared to macro-expressions, which are subject to human control, micro-expressions, which are fleeting, convey more accurate information. Since their discovery in 1966, research on micro-expressions has continued to develop, and the demand for their application has also been growing.

[0003] Micro-expression recognition can capture subtle facial movements and provide authentic emotional feedback, making it widely used in fields such as psychotherapy, national security, and justice. With the continuous development of artificial intelligence (AI), image processing techniques based on deep learning are becoming increasingly mature, and facial detection technology is becoming more accurate. Compared with traditional machine learning techniques, convolutional neural networks (CNNs) are more powerful feature extractors and classifiers. Based on this, the present invention provides a micro-expression recognition method based on CNNs and optical flow features. Summary of the Invention

[0004] The purpose of the present invention is to provide a micro-expression recognition method based on convolutional neural networks and optical flow features to improve the adaptability and accuracy of micro-expression recognition.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A micro-expression recognition method based on convolutional neural network and optical flow features includes the following steps:

[0007] Step 1: Use the face detection module to perform face detection on each image frame in the video segment and output the facial key point information;

[0008] Step 2: Extract four facial regions of interest based on facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow;

[0009] Step 3: Use the TV_L1 optical flow calculation algorithm to calculate the optical strain of all image frames and the first image frame in the video segment, and calculate the optical strain value of each region of interest in the image frame, and add them as the optical flow feature value of the image frame;

[0010] Step 4: Using the first frame of the video segment as the micro-expression start frame and the last frame as the micro-expression end frame, a binary search algorithm is used to search for the image frame with the largest optical flow eigenvalue in the image frame sequence of the video segment, and the image frame is used as the micro-expression vertex frame;

[0011] Step 5: The optical strain, horizontal component of optical flow, and vertical component of optical flow of the vertex frame are used as input, and the micro-expression recognition model based on the convolutional neural network outputs the micro-expression category prediction results. The micro-expression categories include positive, negative, surprise, and others.

[0012] Furthermore, in step 1, the face detection module includes an MTCNN face detection model and an LBF face key point detection model; the MTCNN face detection model performs face detection on each image frame of the video segment, and outputs a stable face bounding box in combination with the C-OF tracking algorithm; the LBF face key point detection model extracts 68 facial key points based on the face bounding box, and outputs them as face key point information.

[0013] Furthermore, in step 2, the facial region of interest extraction process is to determine the upper left vertex and the lower right vertex of the rectangular box corresponding to the nose, mouth, left eye and eyebrow, right eye and eyebrow through the facial key point information, and extract the corresponding facial region of interest from the rectangular box.

[0014] Furthermore, in step 3, the optical flow O of the current image frame relative to the first image frame is:

[0015] O={(u(x,y),v(x,y))|x=1,2,...,X,y=1,2,...,Y},

[0016] Where u represents the horizontal component of the optical flow, v represents the vertical component of the optical flow, (x, y) represents the pixel coordinates; X and Y represent the width and height of the image frame respectively;

[0017] The optical strain ε is calculated based on the horizontal component u and the vertical component v of the optical flow:

[0018]

[0019] Among them, (ε xx , ε yy ) is the normal optical strain component, (ε xy , ε xy ) is the tangential optical strain component;

[0020] Calculate the optical strain value ε(x,y) of each pixel in the current image frame:

[0021]

[0022] The optical strain value of each region of interest is obtained by cumulative calculation, and then the optical strain values of the four regions of interest are added together to obtain the optical flow feature value of the current image frame.

[0023] Furthermore, the micro-expression recognition model based on convolutional neural networks includes: 3 feature extraction modules, 2 fully connected layers and a softmax layer; among them, the 3 feature extraction modules use the same network structure, and respectively extract features of the optical strain, horizontal component of optical flow and vertical component of optical flow of the vertex frame, and obtain corresponding one-dimensional feature vectors; the 3 one-dimensional feature vectors are spliced to achieve feature fusion, and the fused feature vectors are converted in dimension through 2 fully connected layers, and finally the softmax layer performs micro-expression category prediction and outputs the category prediction result.

[0024] Furthermore, the feature extraction module includes: 2 convolutional models, a maximum pooling layer and a fully connected layer; the two convolutional models use the same network structure, consisting of two 3×3 convolutional layers connected together, with the stride of the first convolutional layer being 1 and the stride of the second convolutional layer being 2; the input passes through the two convolutional modules and the maximum pooling layer in sequence to obtain a three-dimensional feature map, which is then flattened and converted into a one-dimensional feature map, and then output after passing through the fully connected layer.

[0025] Based on the above technical solution, the beneficial effects of the present invention are:

[0026] The present invention provides a micro-expression recognition method based on a convolutional neural network and optical flow features, wherein the categories of micro-expressions include positive, negative, surprise, and others. First, a face detection module is constructed, and face detection is performed on each image frame in a video segment by the face detection module to output facial key point information. Four facial regions of interest are extracted based on the facial key point information: nose, mouth, left eye and eyebrow, and right eye and eyebrow, as core areas for subsequent micro-expression recognition. Then, optical flow information (horizontal component of optical flow and vertical component of optical flow) and optical strain information are introduced to characterize the spatiotemporal information of facial movement and the intensity information of facial deformation. The optical flow feature value of each image frame is calculated, and a binary search algorithm is used to search for the image frame with the largest optical flow feature value as the vertex frame in micro-expression recognition. Finally, a micro-expression recognition model based on the convolutional neural network is constructed, and the optical flow horizontal component, optical flow vertical component, and optical strain of the vertex frame are used as input. The micro-expression recognition model outputs a micro-expression category prediction result.

[0027] In summary, the present invention can realize the recognition of positive, negative, surprising and other micro-expressions with good adaptability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the process of micro-expression recognition method based on convolutional neural network and optical flow features in the present invention.

[0029] Figure 2 This is a network structure diagram of the micro-expression recognition model based on convolutional neural network in the present invention.

[0030] Figure 3 This is a network structure diagram of the feature extraction module in the micro-expression recognition model of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0032] This embodiment provides a micro-expression recognition method based on convolutional neural network and optical flow features, the process of which is as follows: Figure 1 As shown in the figure, first, a sliding time window of 0.5 seconds is set to divide the full long video to be recognized into several non-overlapping video segments, and then micro-expression recognition is performed on each video segment; specifically, the following steps are included:

[0033] Step 1: Use the face detection module to perform face detection on each frame of the video segment and output the facial key point information;

[0034] Specifically, the face detection module includes the MTCNN face detection model and the LBF face key point detection model. The MTCNN face detection model performs face detection on each image frame of the video segment and outputs a stable face bounding box in combination with the C-OF tracking algorithm. The LBF face key point detection model extracts 68 facial key points from the face bounding box and outputs them as facial key point information.

[0035] Step 2: Extract four facial regions of interest (ROIs) based on the facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow;

[0036] Specifically, corresponding to the nose, mouth, left eye and eyebrow, right eye and eyebrow, the upper left vertex and the lower right vertex of the rectangular box are determined by the facial key point information, and the corresponding facial region of interest is extracted from the rectangular box;

[0037] Step 3: Use the TV_L1 optical flow calculation algorithm to calculate the optical strain of the face area in all image frames in the video segment and the first image frame, and calculate the optical strain value of each region of interest in the image frame, and add them as the optical flow feature value of the image frame;

[0038] Specifically, the optical flow consists of a vertical component and a horizontal component, which is represented as a tuple:

[0039] O={(u(x,y),v(x,y))|x=1,2,...,X,y=1,2,...,Y},

[0040] Where O represents the optical flow of the image frame, u represents the horizontal component of the optical flow, v represents the vertical component of the optical flow, (x, y) represents the pixel coordinates; X and Y represent the width and height of the image frame respectively;

[0041] The optical strain ε is calculated based on the horizontal component u and the vertical component v. The optical strain ε can provide the intensity information of the facial deformation. The optical strain ε is expressed as:

[0042]

[0043] Among them, (ε xx , ε yy ) is the normal optical strain component, (ε xy , ε xy ) is the tangential optical strain component;

[0044] The optical strain value ε(x,y) of each pixel in the image frame is calculated:

[0045]

[0046] The optical strain value of each region of interest is obtained by cumulative calculation, and then the optical flow feature value of the image frame is obtained;

[0047] Step 4: Using the first frame of the video segment as the micro-expression start frame and the last frame as the micro-expression end frame, a binary search algorithm is used to search for the image frame with the largest optical flow eigenvalue in the image frame sequence of the video segment, and the image frame is used as the micro-expression vertex frame;

[0048] Step 5: The optical strain, horizontal component of optical flow, and vertical component of optical flow of the vertex frame are used as input, and the micro-expression recognition model based on a convolutional neural network outputs the micro-expression category prediction result. The micro-expression categories include positive, negative, surprise, and other;

[0049] Specifically, the micro-expression recognition model based on convolutional neural network is as follows: Figure 2 As shown in the figure, it includes: 3 feature extraction modules, 2 fully connected layers and a softmax layer; the optical strain, horizontal component of optical flow and vertical component of optical flow of the vertex frame are normalized by the minimum-maximum normalization method, and the normalized results are input into the micro-expression recognition model; the three feature extraction modules use the same network structure and serve as three branches to extract features of the optical strain, horizontal component of optical flow and vertical component of optical flow of the vertex frame respectively, and obtain corresponding one-dimensional feature vectors; the three one-dimensional feature vectors are concatenated to achieve feature fusion, and the fused feature vector contains both spatiotemporal information about facial movement and intensity information about facial deformation; the fused feature vector is converted into dimension by two fully connected layers, and finally the softmax layer predicts the category of micro-expression and outputs the category prediction result;

[0050] More specifically, the feature extraction module is as follows Figure 3As shown in the figure, it includes: 2 convolutional models, a maximum pooling layer and a fully connected layer; the two convolutional models use the same network structure, which is composed of two 3×3 convolutional layers, the step size of the first convolutional layer is 1, and the step size of the second convolutional layer is 2; the input passes through the two convolutional modules and the maximum pooling layer in sequence to obtain a three-dimensional feature map, which is flattened and converted into a one-dimensional feature map, and then output after passing through the fully connected layer.

[0051] In this embodiment, the dimensions of the optical strain, horizontal component of optical flow, and vertical component of optical flow of the vertex frame after normalization are 48×48×1, the dimension of the one-dimensional feature vector output by the feature extraction module is 128, the dimension of the feature vector after fusion is 384, and the dimension of the feature vector after dimensional conversion is 4; the dimension of the three-dimensional feature map extracted in the feature extraction module is 6×6×16, and the dimension of the one-dimensional feature map after conversion is 576.

[0052] In this embodiment, the micro-expression recognition model is trained offline. The training process uses the Adam optimizer with multi-class cross entropy loss and sets the initial learning rate to 2×10 -4 If the loss value does not change within 2 training cycles, the learning rate is multiplied by 0.5; if the loss value does not change within 5 training cycles, the training is stopped early; the loss function loss is set to:

[0053]

[0054] Among them, C is the number of micro-expression categories, N is the number of training samples for each category, g i,j represents the sample label, p i,j Represents the model output prediction value;

[0055] The training data set uses a self-made data set. The data collection process is as follows: while the volunteers watch the stimulation video, a sampling video is collected through a video recorder; the sampling video is processed according to steps 1 to 4, the data processing results are used as sample input, and the sample labels are manually annotated to form the training samples.

[0056] The above description is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all disclosed features, or all steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A micro-expression recognition method based on convolutional neural network and optical flow features, characterized in that: The following steps are involved: Step 1: Use the face detection module to perform face detection on each image frame in the video segment and output the facial key point information; Step 2: Extract four facial regions of interest based on facial key point information: nose, mouth, left eye and eyebrow, right eye and eyebrow; Step 3: Use the TV_L1 optical flow calculation algorithm to calculate the optical strain of all image frames and the first image frame in the video segment, and calculate the optical strain value of each region of interest in the image frame, and add them as the optical flow feature value of the image frame; Step 4: Using the first frame of the video segment as the micro-expression start frame and the last frame as the micro-expression end frame, a binary search algorithm is used to search for the image frame with the largest optical flow eigenvalue in the image frame sequence of the video segment, and the image frame is used as the micro-expression vertex frame; Step 5: The optical strain, horizontal component of optical flow, and vertical component of optical flow of the vertex frame are used as input, and the micro-expression recognition model based on a convolutional neural network outputs the micro-expression category prediction result. The micro-expression categories include positive, negative, surprise, and other; The micro-expression recognition model based on a convolutional neural network consists of three feature extraction modules, two fully connected layers, and a softmax layer. The three feature extraction modules use the same network structure and extract features from the optical strain, horizontal component of optical flow, and vertical component of optical flow in the vertex frame, respectively, to obtain corresponding one-dimensional feature vectors. These three one-dimensional feature vectors are concatenated to achieve feature fusion. The fused feature vectors are then transformed through two fully connected layers for dimensionality conversion. Finally, a softmax layer predicts the micro-expression category and outputs the predicted category. The feature extraction module includes: 2 convolutional models, a maximum pooling layer and a fully connected layer; the two convolutional models use the same network structure, consisting of two 3×3 convolutional layers connected together, with the stride of the first convolutional layer being 1 and the stride of the second convolutional layer being 2; the input passes through the two convolutional modules and the maximum pooling layer in sequence to obtain a three-dimensional feature map, which is then flattened and converted into a one-dimensional feature map and output after passing through the fully connected layer.

2. The micro-expression recognition method based on convolutional neural network and optical flow features according to claim 1, characterized in that: In step 1, the face detection module includes the MTCNN face detection model and the LBF face key point detection model; the MTCNN face detection model performs face detection on each image frame of the video segment and outputs a stable face bounding box in combination with the C-OF tracking algorithm; the LBF face key point detection model extracts 68 facial key points based on the face bounding box and outputs them as face key point information.

3. The micro-expression recognition method based on convolutional neural network and optical flow features according to claim 1, characterized in that: In step 2, the facial region of interest extraction process is to determine the upper left vertex and the lower right vertex of the rectangular box corresponding to the nose, mouth, left eye and eyebrow, right eye and eyebrow through the facial key point information, and extract the corresponding facial region of interest from the rectangular box.

4. The micro-expression recognition method based on convolutional neural network and optical flow features according to claim 1, characterized in that: In step 3, the optical flow O of the current image frame relative to the first image frame is: O={(u(x,y),v(x,y))|x=1,2,…,X,y=1,2,…,Y}, Where u represents the horizontal component of the optical flow, v represents the vertical component of the optical flow, (x, y) represents the pixel coordinates; X and Y represent the width and height of the image frame respectively; The optical strain ε is calculated based on the horizontal component u and the vertical component v of the optical flow: Among them, (ε xx , ε yy ) is the normal optical strain component, (ε xy , ε yx ) is the tangential optical strain component; Calculate the optical strain value ε(x,y) of each pixel in the current image frame: The optical strain value of each region of interest is obtained by cumulative calculation, and then the optical strain values of the four regions of interest are added together to obtain the optical flow feature value of the current image frame.

Images