Micro-expression recognition method and apparatus

Abstract

The application discloses a micro-expression recognition method and device, and belongs to the technical field of image recognition. The method comprises the following steps: acquiring image information to be recognized; acquiring a first micro-expression data set and a second micro-expression data set, performing image preprocessing on the first micro-expression data set and the second micro-expression data set, and obtaining a full-face image, an eye image, a mouth image and a light flow image sequence; inputting the full-face image, the eye image, the mouth image and the light flow image sequence into a three-dimensional space-time attention convolution network respectively for extraction; performing feature fusion on the extracted image sequence; sending the full-face feature map, the eye feature map and the mouth feature map after the feature fusion to a category probability calculation model to obtain the probability of each category; performing model training by using the above steps to obtain a trained micro-expression recognition model; and classifying and recognizing human face micro-expression by using the trained micro-expression recognition model. The application can capture image detail information and improve micro-expression recognition efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image recognition, and particularly relates to a micro-expression recognition method and device. BACKGROUND

[0002] According to the duration, facial expressions can be divided into macro-expression and micro-expression. Micro-expression is a subtle and spontaneous facial expression produced when people want to hide their inner emotions. Micro-expression usually occurs within 0.04s to 0.2s, only appears in a small part of the face, and can reflect the true emotions of people and cannot deceive others. Therefore, micro-expression recognition is an important auxiliary tool for behavior analysis, and has important application value in criminal interrogation, clinical medicine, business negotiation and other fields. Unlike large-scale macro facial expressions, micro-expression has a short duration, small fluctuations, and is difficult to identify with the naked eye, which brings many challenges to the automatic detection and classification of micro-expression.

[0003] For the problem of micro-expression recognition, the mainstream method of computer vision technology is divided into two categories: traditional method and deep learning method. In the early stage, the traditional micro-expression recognition method mainly includes local binary pattern (Local Binary Patterns, hereinafter referred to as "LBP") and its improved method and optical flow feature and its improved method. The difference between the two methods is that the LBP-based algorithm usually uses LBP to represent the local texture of the image for classification, while the optical flow-based method extracts optical flow information for classification. Although the features extracted by the LBP and optical flow methods are proved to reflect the subtle movements of micro-expression, they improve the upper limit of the traditional method considering global feature information, but the LBP-based method encodes too much redundant information, resulting in high computational cost and weak generalization ability, and the method based on optical flow only considers the optical flow information between the peak frame and the end frame, but ignores the influence of spontaneous actions of human face (such as blinking). These methods may not naturally learn the subtle spatio-temporal changes, resulting in the loss of part of the facial information.

[0004] With the rapid development of computer vision and Graphics Processing Unit (hereinafter referred to as "GPU"), deep learning algorithms have been widely applied in this field. The network based on traditional CNN mostly extracts horizontal and vertical optical flow between the starting frame, peak frame and ending frame of micro-expression, but ignores the information of other offset frames. Many methods combine convolutional neural network (hereinafter referred to as "CNN") and long short term memory neural network (hereinafter referred to as "LSTM") to extract time and space features, but this will cause the network parameter quantity and running time to increase, and it is easy to cause overfitting for small sample data set such as micro-expression. In recent years, three-dimensional convolutional neural network (hereinafter referred to as "3D CNN") uses the advantage of joint extraction of space-time features, gradually replaces two-dimensional convolution in the field of micro-expression recognition (hereinafter referred to as "MER"), and has made significant improvement in MER research.

[0005] However, 3D CNN still has shortcomings in feature extraction. First, the effective information of micro-expression only exists in a specific area at a specific time, while the space-time information extracted by 3D CNN is mostly unimportant. In addition, simply stacking 3D CNN blocks will not only ignore the details of shallow images, but also may cause overfitting. SUMMARY

[0006] The purpose of the present application is to provide a micro-expression recognition method and system, which combines a three-dimensional residual module and an ST attention mechanism into a network of a double-flow pyramid fusion strategy to improve the importance of local fine information in the spatial, temporal and channel dimensions and maintain the diversity of features, so as to overcome the problem of insufficient extraction of key timing information and low recognition accuracy.

[0007] To solve the above problems, the first aspect of the present application provides a micro-expression recognition method, comprising:

[0008] S1, obtaining a first micro-expression data set and a second micro-expression data set, wherein the first micro-expression data set and the second micro-expression data set comprise original data frames in a preset video;

[0009] S2, image pre-processing the first micro-expression data set and the second micro-expression data set to obtain a fixed frame number of full-face image sequence, eye image sequence, mouth image sequence and optical flow image sequence;

[0010] S3, inputting the full face image sequence, the eye image sequence, the mouth image sequence and the optical flow image sequence into a three-dimensional space-time attention convolution network respectively to extract full face image feature maps, eye image feature maps, mouth image feature maps and optical flow image feature maps;

[0011] S4, inputting the full face image feature maps and the optical flow image feature maps into a dual-flow feature pyramid fusion module to perform feature fusion to obtain full face feature maps after feature fusion, inputting the eye image feature maps and the mouth image feature maps into an eye-mouth auxiliary module to calculate to obtain eye feature maps after feature fusion and mouth feature maps after feature fusion;

[0012] S5, sending the full face feature maps after feature fusion, the eye feature maps after feature fusion and the mouth feature maps after feature fusion to a class probability calculation module, calculating a cross-entropy loss function in the class probability calculation module, and obtaining a probability of each class according to the cross-entropy loss function, the class being a result of micro-expression recognition;

[0013] S6, performing model training by using steps S1-S5 to obtain a trained micro-expression recognition model;

[0014] S7, using the trained micro-expression recognition model to classify and recognize facial micro-expressions.

[0015] Optionally, the first micro-expression data set includes a high-frame-rate data subset, a normal data subset and an infrared data subset, wherein the high-frame-rate data subset is captured by using a high-frame-rate camera, the normal data subset is captured by using a normal camera, and the infrared data subset is captured by using an infrared camera.

[0016] The second micro-expression data set is captured by using a high-speed camera.

[0017] Optionally, the preprocessing in S2 includes:

[0018] S2.1, processing the first micro-expression data set and the second micro-expression data set respectively by using a time domain interpolation module to obtain an interpolated image sequence;

[0019] S2.2, recognizing facial key points by using a machine model algorithm library on the interpolated image sequence;

[0020] S2.3, using a 5-key-point model to perform registration by using a registration method of the machine model algorithm library, and cutting the face, eyes and mouth pictures by using the machine model algorithm library after registration to obtain a full face image sequence, an eye image sequence and a mouth image sequence respectively;

[0021] S2.4, performing optical flow feature extraction on the interpolated image sequence by using an optical flow method to obtain an optical flow image sequence.

[0022] Optionally, the full-face image sequence, the eye image sequence, the mouth image sequence and the optical flow image sequence are respectively input into a three-dimensional spatio-temporal attention convolutional network to extract full-face image feature maps, eye image feature maps, mouth image feature maps and optical flow image feature maps, which comprises:

[0023] The full-face image sequence is input into a face branch of the three-dimensional spatio-temporal attention convolutional network to extract full-face image feature maps;

[0024] The optical flow image sequence is input into an optical flow branch of the three-dimensional spatio-temporal attention convolutional network to extract optical flow image feature maps;

[0025] The eye image sequence is input into a double-eye branch of the three-dimensional spatio-temporal attention convolutional network to extract eye image feature maps;

[0026] The mouth image sequence is input into a mouth branch of the three-dimensional spatio-temporal attention convolutional network to extract mouth image feature maps.

[0027] Optionally, the full-face image sequence is input into a face branch of the three-dimensional spatio-temporal attention convolutional network to extract full-face image feature maps, which comprises:

[0028] The full-face image sequence is sequentially input into two three-dimensional single convolution feature distillation modules to obtain feature maps δ1;

[0029] The feature maps δ1 are input into a three-dimensional spatio-temporal attention module to obtain feature maps γ1;

[0030] The feature maps γ1 are input into a three-dimensional double convolution feature extraction module to obtain feature maps α1;

[0031] The feature maps α1 are sequentially input into two residual modules to obtain full-face feature maps β1.

[0032] Optionally, the optical flow image sequence is input into an optical flow branch of the three-dimensional spatio-temporal attention convolutional network to extract optical flow image feature maps, which comprises:

[0033] The optical flow image sequence is sequentially input into two three-dimensional single convolution feature distillation modules to obtain feature maps δ2;

[0034] The feature maps δ2 are input into a three-dimensional spatio-temporal attention module to obtain feature maps γ2;

[0035] The feature maps γ2 are input into a three-dimensional double convolution feature extraction module to obtain feature maps α2;

[0036] The feature maps α2 are sequentially input into two residual modules to obtain optical flow feature maps β2.

[0037] Optionally, inputting the eye image sequence into a binocular branch of the three-dimensional spatiotemporal attention convolutional network to extract an eye image feature map comprises:

[0038] The eye image sequence is sequentially input into two three-dimensional single convolution feature distillation modules to obtain a feature map δ3.

[0039] The feature map δ3 is input into a three-dimensional spatiotemporal attention module to obtain a feature map γ3.

[0040] The feature map γ3 is input into a three-dimensional double convolution feature extraction module to obtain a feature map α3.

[0041] The feature map α3 is sequentially input into two residual modules to obtain an eye feature map β3.

[0042] Optionally, inputting the mouth image sequence into a mouth branch of the three-dimensional spatiotemporal attention convolutional network to extract a mouth image feature map comprises:

[0043] The mouth image sequence is sequentially input into two three-dimensional single convolution feature distillation modules to obtain a feature map δ4.

[0044] The feature map δ4 is input into a three-dimensional spatiotemporal attention module to obtain a feature map γ4.

[0045] The feature map γ4 is input into a three-dimensional double convolution feature extraction module to obtain a feature map α4.

[0046] The feature map α4 is sequentially input into two residual modules to obtain a mouth feature map β4.

[0047] Optionally, the cross-entropy loss function is calculated, and the probability calculation formula of each category is as follows:

[0048]

[0049] Wherein, M represents the number of categories, yic represents a symbol function, if the true category of sample i is equal to c, 1 is taken, otherwise 0 is taken; pic represents the predicted probability that the observed sample i belongs to category c, N represents the number of folds (N-fold cross-validation is used for model training), and L i represents the sum of the loss functions corresponding to each fold.

[0050] Another aspect of the present application provides a micro-expression recognition device, the device comprises:

[0051] The model training module trains the micro-expression recognition model to obtain a trained micro-expression recognition model; the training of the micro-expression recognition model comprises:

[0052] S1, acquire a first micro-expression dataset and a second micro-expression dataset, the first micro-expression dataset and the second micro-expression dataset include original data frames in a preset video;

[0053] S2, image pre-processing is performed on the first micro-expression dataset and the second micro-expression dataset, to obtain a fixed frame number of full face image sequences, eye image sequences, mouth image sequences and optical flow image sequences;

[0054] S3, the full face image sequences, the eye image sequences, the mouth image sequences and the optical flow image sequences are respectively input into a three-dimensional space-time attention convolution network for extraction, to obtain full face image feature maps, eye image feature maps, mouth image feature maps and optical flow image feature maps;

[0055] S4, the full face image feature maps and the optical flow image feature maps are input into a double-flow feature pyramid fusion module for feature fusion to obtain full face feature maps after feature fusion, and the feature maps of the eye and mouth branches are input into an eye and mouth auxiliary module for calculation to obtain eye feature maps after feature fusion and mouth feature maps after feature fusion;

[0056] S5, the full face feature maps after feature fusion, the eye feature maps after feature fusion and the mouth feature maps after feature fusion are sent to a category probability calculation module, a cross-entropy loss function is calculated in the category probability module, and the probability of each category is obtained according to the cross-entropy loss function;

[0057] S6, model training is performed by using steps S1-S5, to obtain a trained micro-expression recognition model;

[0058] A micro-expression recognition module is adopted to classify and recognize human face micro-expressions by using the trained micro-expression recognition model.

[0059] The above technical solutions of the present application have the following beneficial technical effects:

[0060] The application extracts features by using the proposed four-branch three-dimensional space-time attention convolutional network as a whole, and the full face and optical flow branches enter the improved double-flow feature pyramid fusion module, cooperate with the eye and mouth feature input eye and mouth auxiliary module, perform feature map output fusion, complete the joint learning of space-time features and detail information, realize the capture of image detail information, and improve the micro-expression recognition performance and efficiency. The three-dimensional space-time attention convolutional network uses the mode of 3D convolution, avoids the omission of time sequence information of traditional convolution, and sets three-dimensional space-time attention modules inside, including SAM (space) and TAM (time), respectively, for double feature weight calibration from the space and time dimensions, amplifies the local subtle expression changes of micro-expression and the expression amplitude changes in the time dimension, improves the recognition accuracy based on multi-directional features, realizes adaptive space-time feature selection, and makes up for the problem of learning redundant features without preference of three-dimensional convolution. Then cooperate with the improved double-flow feature pyramid fusion module, the double-flow feature pyramid fusion module fuses the low-layer optical flow features and the high-layer full face image features, simultaneously utilizes the high resolution and motion information of the bottom optical flow features and the high semantic information of the high-layer face image features, shares semantic information in the double-flow structure, so that the double-flow information is effectively combined. Meanwhile, aiming at the rich micro-motion semantic information of the eye and mouth regions, the features extracted from the eyes and mouth are extracted again in the eye and mouth auxiliary structure, the multi-branch features are enhanced and fused, the cross-entropy loss is calculated in the category probability model, and the multi-branch features enhance the correction of the original prediction of the local details and optical flow motion information. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a micro-expression recognition method flowchart provided by an embodiment of the application.

[0062] Figure 2 is a model framework diagram of the micro-expression recognition method provided by an embodiment of the application.

[0063] Figure 3 is a structure diagram of the three-dimensional space-time attention model provided by an embodiment of the application.

[0064] Figure 4 is a double-flow feature pyramid fusion module structure diagram provided by an embodiment of the application.

[0065] Figure 5 is an eye and mouth auxiliary model structure diagram provided by an embodiment of the application. DETAILED DESCRIPTION

[0066] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments and drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present application. In addition, in the following description, the description of well-known structures and techniques is omitted to avoid unnecessary confusion of the concept of the present application.

[0067] Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0068] In the description of the present application, it should be noted that the terms "first", "second", "third" are only for descriptive purposes, and cannot be understood or implied as indicating or implying relative importance.

[0069] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0070] Technical terms:

[0071] Multi-branch spatial-temporal attention pyramid network (Multi-branch Spatial-temporal Attention Pyramid Net, hereinafter referred to as "MSAPNet"), the complete MSAPNet network model structure in the present application Figure 2 .

[0072] Three-dimensional spatial-temporal attention convolutional network (abbreviated as 3D STAC), the structure is referred to Figure 2 The left side of the first dashed line part; it is worth noting that the three-dimensional spatial-temporal attention convolutional network is different from the three-dimensional spatial-temporal attention module, and the specific relationship will be described in detail in the part of step S3.

[0073] Three-dimensional spatial-temporal attention module (hereinafter referred to as "3DSTAttention"), which will be explained in detail according to the specific content, please refer to Figure 3 .

[0074] Dual stream feature pyramid fusion module (hereinafter referred to as "Dual Stream-FPN"), which will be explained in detail according to the specific content, please refer to Figure 4 .

[0075] Eye mouth auxiliary module (Eye Mouth Auxiliary module, hereinafter referred to as "EMA"), please refer to Figure 5 .

[0076] Figure 1 is a flow chart of a micro-expression recognition method according to the first embodiment of the present application.

[0077] With reference to Figures 1-4 , the present application provides a micro-expression recognition method, comprising:

[0078] S1, acquiring a first micro-expression data set and a second micro-expression data set, the first micro-expression data set and the second micro-expression data set comprising original data frames in a preset video; for example, acquiring a first micro-expression data set SMIC and a second micro-expression data set CASMEII composed of original data frames from a starting frame to an ending frame of a facial micro-expression in a preset video;

[0079] S2, image pre-processing the first micro-expression data set and the second micro-expression data set to obtain a fixed-frame-number full-face image sequence, an eye image sequence, a mouth image sequence and an optical flow image sequence; for example, image pre-processing the acquired CASMEII and SMIC micro-expression data sets, including TIM interpolation fixed sample frame number, face key point detection, face alignment cropping, optical flow calculation, to obtain a fixed-frame-number full-face image sequence, an eye image sequence, a mouth image sequence and an optical flow image sequence;

[0080] S3, inputting the full-face image sequence, the eye image sequence, the mouth image sequence and the optical flow image sequence into a three-dimensional spatio-temporal attention convolution network (3D STAC for short) respectively for extraction to obtain a full-face image feature map, an eye image feature map, a mouth image feature map and an optical flow image feature map; for example, inputting the pre-processed four kinds of image sequences into a three-dimensional spatio-temporal attention convolution network 3D STAC respectively for extracting feature maps of each branch sample image;

[0081] S4, inputting the full-face image feature map and the optical flow image feature map into a dual-stream feature pyramid fusion module for feature fusion to obtain a full-face feature map after feature fusion, and inputting the feature maps of the eye and mouth branches into an eye-mouth auxiliary model for calculation to obtain an eye feature map after feature fusion and a mouth feature map after feature fusion; for example, inputting the full-face sequence and the optical flow sequence feature maps into a Dual Stream-FPN feature pyramid fusion strategy for feature fusion to obtain a full-face feature map after feature fusion, and inputting the feature maps of the eye and mouth branches into an eye-mouth auxiliary model EMA for calculation of the eye and mouth feature maps;

[0082] S5, send the full face feature map after feature fusion, the feature map of the eyes after feature fusion and the feature map of the mouth after feature fusion to a category probability calculation module, calculate a cross-entropy loss function in the category probability calculation module, and obtain the probability of each category according to the cross-entropy loss function, the category being the result of micro-expression recognition; for example, send the feature map after fusion of the three feature maps to a category probability calculation module (hereinafter referred to as a "head module") and calculate a cross-entropy loss function Cross-Entropy loss to obtain the category information of micro-expression;

[0083] S6, use steps S1-S5 to train a model to obtain a trained micro-expression recognition model; for example, use steps S1-S5 to train a model to obtain a multi-branch three-dimensional space-time attention pyramid network MSAPNet model (referred to as MSAPNet model).

[0084] S7, use the trained micro-expression recognition model to classify and recognize facial micro-expression, for example, input facial micro-expression test data into the trained MSAPNet model to classify micro-expression.

[0085] The micro-expression recognition method based on the multi-branch three-dimensional space-time attention pyramid network MSAPNet model provided by the application is based on the multi-branch three-dimensional space-time attention pyramid network MSAPNet model for recognition, and solves the problems of micro-expression local detail information deviation, weakening or even loss caused by subtle changes, insufficient data set samples and single branch.

[0086] The recognition method provided by the embodiment of the application is mainly applied to professional micro-expression recognition scenes such as judicial interrogation.

[0087] The application tests the detection effect of the MSAPNet model on the test set of the CASME II 3 classification and SMIC3 classification data set. The accuracy (Accuracy) / F1 score (F1-Scorce) is used as the standard for measuring accuracy, and the detection accuracy of the MSAPNet model reaches 95.92% / 95.02% and 73.17% / 73.07%.

[0088] The application uses the three-dimensional space-time attention convolution network 3D STAC as a whole, uses the improved structure of the dual-stream feature pyramid fusion module Dual Stream-FPN and the eye and mouth auxiliary model EMA to output the feature map, and uses the cross-entropy loss function Cross-Entropy loss to calculate the classification loss, solves the problems of micro-expression local detail information deviation, weakening or even loss caused by subtle changes, insufficient data set samples and single branch, corrects the micro-expression peak information in the time and space dimensions, and ensures the effective transmission and recognition of the detail information.

[0089] In an embodiment, the first micro-expression dataset (SMIC) comprises a high frame rate data (HS) subset, a normal data (VIS) subset and an infrared data (NIR) subset, wherein the HS subset is captured by using a high frame rate camera, the VIS subset is captured by using a normal camera, and the NIR subset is captured by using an infrared camera.

[0090] The CASMEII dataset is captured by using a high-speed camera.

[0091] For example, the micro-expression original dataset SMIC and CASMEII in step S1 are respectively from the micro-expression video stream data captured by the University of Oulu, Finland and the Institute of Psychology, Chinese Academy of Sciences. The two datasets are collected in a well-controlled laboratory environment, under appropriate experimental design and lighting conditions, requiring the subjects to keep their faces expressionless and their bodies still, eliminating the interference of non-emotional factors such as light and head movement, and stimulating the facial expressions of the participants, so as to obtain pure micro-expression samples and form the micro-expression dataset. The difference is that the SMIC dataset contains three data subsets, which are captured by using different types of cameras - the HS subset is captured by using a high frame rate camera, the VIS subset is captured by using a normal camera, and the NIR subset is captured by using an infrared camera. For the CASMEII dataset, a high-speed camera is used for capturing.

[0092] For the SMIC dataset, the micro-expression is divided into 107 positive samples, 116 negative samples and 83 surprise samples in total, which are three categories of 306 samples; and for the CASMEII dataset, the micro-expression is divided into 33 happy samples, 60 disgust samples, 25 surprise samples, 27 suppressed samples and 102 other samples in total, which are five categories of 247 samples. In addition, in order to facilitate comparison, we combine the five categories in the CASMEII dataset into three categories of positive (including happy samples), negative (including disgust and suppressed samples) and surprise, which are 33, 87 and 25 samples respectively, and a total of 145 samples.

[0093] In an embodiment, the preprocessing in S2 comprises:

[0094] S2.1, using a time domain interpolation module to process the first micro-expression dataset and the second micro-expression dataset respectively to obtain an interpolated image sequence;

[0095] S2.2, using a machine model algorithm library to recognize the face key points of the interpolated image sequence;

[0096] S2.3, by the registration method of the machine model algorithm library, registration is performed using a 5-key point model, and after registration, the machine model algorithm library is used to crop face, eye and mouth pictures, respectively to obtain full face image sequence, eye image sequence and mouth image sequence;

[0097] S2.4, the optical flow method is used to extract optical flow features from the interpolated image sequence to obtain an optical flow image sequence.

[0098] For example, the preprocessing in step S2 includes:

[0099] S2.1, the present application uses a temporal interpolation module (TIM) to increase the number of images contained in a micro-expression segment, thereby prolonging the duration of the micro-expression. The method first regards the video segment as a graph, and uses a node in the graph to represent a frame image; then, a graph embedding algorithm is used to embed the graph into a low-dimensional manifold, and finally, an image vector is substituted to calculate a high-dimensional continuous curve. Re-sampling on the curve can obtain an interpolated image sequence, which generally uses a group of 32 images to form an image sequence;

[0100] S2.2, for face key point detection, first, a machine model algorithm library (Dlib toolkit) is used to detect a face, and after the face is obtained, a Dlib toolkit is also used to obtain face key points;

[0101] S2.3, for face alignment function, a registration method "get_face_chip() method" in the Dlib toolkit is used to perform registration using a 5-key point model. After registration, the Dlib toolkit is also used to crop face, eye and mouth pictures, with sizes of 64x64, 64x32 and 64x32 respectively. In order to avoid the influence of image jitter after cropping, the present application performs normalization processing on the image. The mean value of the face video stream normalization is 0.3678, and the standard deviation is 0.1484; the mean value of the face optical flow normalization is [0.0291, 0.4986, 0.4978], and the standard deviation is [0.0451, 0.0673, 0.0474];

[0102] S2.4, the present application uses a TV-L1 optical flow method to extract optical flow features from the micro-expression, which is realized by the create method of the DualTVL1OpticalFlow class in the OpenCV function library. Specifically, the first frame of the micro-expression video frame sequence after adding a mask is taken as a baseline frame, and then the optical flow information of the starting frame and the ending frame after adding the mask is calculated in turn through the baseline frame. The calculation result is a two-dimensional vector field, which respectively represents the horizontal and vertical components of the motion of each pixel.

[0103] Through the preprocessing in step S2, four groups of video streams can be obtained: a face video stream (i.e., a full-face image sequence), a face optical flow video stream (i.e., an optical flow image sequence), a double-eye video stream (i.e., an eye image sequence), and a mouth video stream (i.e., a mouth image sequence). The above video streams are respectively input into corresponding branches to perform feature extraction.

[0104] In an embodiment, the full-face image sequence, the eye image sequence, the mouth image sequence, and the optical flow image sequence are respectively input into a three-dimensional spatio-temporal attention convolutional network to extract a full-face image feature map, an eye image feature map, a mouth image feature map, and an optical flow image feature map, including:

[0105] The full-face image sequence is input into a face branch of the three-dimensional spatio-temporal attention convolutional network to extract the full-face image feature map;

[0106] The eye image sequence is input into a double-eye branch of the three-dimensional spatio-temporal attention convolutional network to extract the eye image feature map;

[0107] The mouth image sequence is input into a mouth branch of the three-dimensional spatio-temporal attention convolutional network to extract the mouth image feature map;

[0108] The optical flow image sequence is input into a face optical flow branch of the three-dimensional spatio-temporal attention convolutional network to extract the optical flow image feature map.

[0109] In an embodiment, the full-face image sequence is input into the face branch of the three-dimensional spatio-temporal attention convolutional network to extract the full-face image feature map, including:

[0110] The full-face image sequence is sequentially input into two three-dimensional single convolution feature distillation (hereinafter referred to as “3D CBL”) modules to obtain a feature map δ1;

[0111] The feature map δ1 is input into a three-dimensional spatio-temporal attention (hereinafter referred to as “3DSTAttention”) module to obtain a feature map γ1;

[0112] The feature map γ1 is input into a three-dimensional double convolution feature extraction (hereinafter referred to as “3D CCD”) module to obtain a feature map α1;

[0113] The feature map α1 is sequentially input into two residual modules (hereinafter referred to as “Res_Block”) to obtain a full-face feature map β1.

[0114] For example, the three-dimensional spatio-temporal attention convolutional network 3D STAC in step S3 includes a face branch Layer1, a face optical flow branch Layer2, a double-eye branch Layer3, and a mouth branch Layer4.

[0115] The calculation method of the face branch Layer1 is as follows:

[0116] The full-face image sequence is sequentially passed through two three-dimensional single convolution feature distillation (hereinafter referred to as "3D CBL") modules to obtain a feature map δ1;

[0117] The feature map δ1 is passed through a three-dimensional space-time attention (hereinafter referred to as "3DSTAttention") module to obtain a feature map γ1;

[0118] The feature map γ1 is passed through a three-dimensional double convolution feature extraction (hereinafter referred to as "3D CCD") module to obtain a feature map α1;

[0119] The feature map α1 is sequentially passed through two residual modules (hereinafter referred to as "Res_Block") to obtain a full-face feature map β1.

[0120] The first 3DCBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 1x2x2, and padding of 1, a 3D batch normalization module with an output channel number of 64, and a randomized leaky ReLU (hereinafter referred to as "LeakyReLU"). The second 3DCBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 2x2x2, and padding of 1, a 3D batch normalization model with an output channel number of 128, and a LeakyReLU activation function model. The full-face image sequence is first passed through the first 3DCBL module and then passed through the second 3DCBL module to obtain the feature map δ1;

[0121] The 3DSTAttention module includes a temporal attention model and a spatial attention model, and the channel number of the feature map in the module remains unchanged. The temporal attention model is to copy the feature map into two identical parts, one part is passed through the maximum pooling to obtain the maximum pooling feature vector, and the other part is passed through the average pooling to obtain the average pooling vector. The two vectors are then added after passing through a 2D convolution module, a ReLU activation function, and a 2D convolution module, and then passing through a Sigmoid activation function to obtain a temporal feature vector. The spatial attention model is to copy the feature map into two identical parts, one part is passed through the maximum pooling to obtain the maximum pooling feature vector, and the other part is passed through the average pooling to obtain the average pooling vector. The two vectors are concatenated in the column dimension and then passed through a 3D convolution and a Sigmoid activation function to obtain a spatial feature vector. The feature map δ1 is multiplied by the temporal feature vector and the spatial feature vector respectively, and then added to the original feature map itself, to obtain the feature map γ1.

[0122] The 3D CCD module includes a 3D convolution model with 256 output channels, a 3x3x3 convolution kernel, a 2x1x2 stride, and a 1 padding, a 3D convolution model with 512 output channels, a 3x3x3 convolution kernel, a 2x1x2 stride, and a 1 padding, and a three-dimensional random inactivation (hereinafter referred to as “3D Dropout”) model with a random inactivation rate of 0.5, and the feature map γ1 sequentially passes through the above 3D convolution model and three-dimensional random inactivation model to obtain the feature map α1.

[0123] The first Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a 3x3x3 convolution kernel, a 1 stride, and a 1 padding, and the other branch has no model. The feature maps of the last two branches are connected by residual connection to obtain the output. The second Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a 3x3x3 convolution kernel, a 1 stride, and a 1 padding, and the other branch has no model. The feature maps of the last two branches are also connected by residual connection to obtain the output. In this process, the channel number of the feature map remains unchanged at 512, and the feature map α1 sequentially passes through the above two Res_Block modules to obtain the full-face feature map β1.

[0124] In an embodiment, inputting the optical flow image sequence into the face optical flow branch of the three-dimensional spatio-temporal attention convolutional network to extract the optical flow image feature map includes:

[0125] The optical flow image sequence sequentially passes through two three-dimensional single convolution feature distillation (hereinafter referred to as “3D CBL”) modules to obtain the feature map δ2;

[0126] The feature map δ2 passes through a three-dimensional spatio-temporal attention (hereinafter referred to as “3D ST Attention”) module to obtain the feature map γ2;

[0127] The feature map γ2 passes through a three-dimensional double convolution feature extraction (hereinafter referred to as “3D CCD”) module to obtain the feature map α2;

[0128] The feature map α2 sequentially passes through two residual modules (hereinafter referred to as “Res_Block”) to obtain the optical flow feature map β2.

[0129] For example, the calculation method of the face optical flow branch Layer2 is as follows:

[0130] The first 3DCBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 1x2x2, and padding of 1, a 3D batch normalization model with an output channel number of 64, and a randomized leaky ReLU (hereinafter referred to as "LeakyReLU"). The second 3DCBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 2x2x2, and padding of 1, a 3D batch normalization model with an output channel number of 128, and a LeakyReLU activation function model. The optical flow image sequence is first subjected to the first 3DCBL module and then subjected to the second 3DCBL module to obtain a feature map δ2.

[0131] The 3DSTAttention module includes a time attention model and a space attention model, and the number of channels of the feature map in the module is unchanged. The time attention model is to copy the feature map into two identical parts, one part is subjected to maximum pooling to obtain a maximum pooling feature vector, and the other part is subjected to average pooling to obtain an average pooling vector. The two vectors are then added after being subjected to a 2D convolution module, a rectified linear unit (hereinafter referred to as ReLU) activation function, and a 2D convolution module, and subjected to a Sigmoid activation function to obtain a time feature vector. The space attention model is to copy the feature map into two identical parts, one part is subjected to maximum pooling to obtain a maximum pooling feature vector, and the other part is subjected to average pooling to obtain an average pooling vector. The two vectors are spliced in the column dimension and then subjected to a 3D convolution and a Sigmoid activation function to obtain a space feature vector. The feature map δ4 is multiplied by the time feature vector and the space feature vector respectively and added to the original feature map itself, that is, the feature map γ2 is obtained.

[0132] The 3DCCD module includes a 3D convolution model with an output channel number of 256, a convolution kernel size of 3x3x3, a stride of 2x1x2, and padding of 1, a 3D convolution model with an output channel number of 512, a convolution kernel size of 3x3x3, a stride of 2x1x2, and padding of 1, and a three-dimensional random inactivation (hereinafter referred to as "3DDropout") model with a random inactivation rate of 0.5. The feature map γ4 is sequentially subjected to the above 3D convolution model and three-dimensional random inactivation model to obtain a feature map α2.

[0133] The first Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a convolution kernel size of 3x3x3, a stride of 1, and a padding of 1, and the other branch has no model. The feature maps of the last two branches are connected by residual connection to obtain the output. The second Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a convolution kernel size of 3x3x3, a stride of 1, and a padding of 1, and the other branch has no model. The feature maps of the last two branches are also connected by residual connection to obtain the output. In this process, the channel number of the feature map remains unchanged at 512, and the feature map a4 passes through the above two Res_Block modules in turn to obtain the eye feature map b2.

[0134] In an embodiment, inputting the eye image sequence into the dual-eye branch of the three-dimensional spatiotemporal attention convolutional network to extract the eye image feature map includes:

[0135] The eye image sequence passes through two three-dimensional single convolution feature distillation (hereinafter referred to as “3D CBL”) modules in turn to obtain the feature map d3;

[0136] The feature map d3 passes through a three-dimensional spatiotemporal attention module (hereinafter referred to as “3D ST Attention”) to obtain the feature map g3;

[0137] The feature map g3 passes through a three-dimensional double convolution feature extraction module (hereinafter referred to as “3D CCD”) to obtain the feature map a3;

[0138] The feature map a3 passes through two residual modules (hereinafter referred to as “Res_Block”) in turn to obtain the eye feature map b3.

[0139] For example, the calculation method of the dual-eye branch Layer3 is as follows:

[0140] Wherein, the first 3D CBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 1x2x2, and a padding of 1, a 3D batch normalization module with an output channel number of 64, and a Randomized LeakyReLU (hereinafter referred to as “LeakyReLU”). The second 3D CBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 2x2x2, and a padding of 1, a 3D batch normalization model with an output channel number of 128, and a LeakyReLU activation function model. The eye image sequence passes through the first 3D CBL module and then passes through the second 3D CBL module to obtain the feature map d3;

[0141] The 3DSTAttention module includes a time attention model and a space attention model, and the channel number of the feature map is unchanged inside the module. The time attention module is to copy the feature map into two identical parts, one of which is subjected to maximum pooling to obtain a maximum pooling feature vector, and the other is subjected to average pooling to obtain an average pooling vector. The two vectors are then added after passing through a 2D convolution module, a ReLU activation function and a 2D convolution module, and then passing through a Sigmoid activation function to obtain a time feature vector. The space attention module is to copy the feature map into two identical parts, one of which is subjected to maximum pooling to obtain a maximum pooling feature vector, and the other is subjected to average pooling to obtain an average pooling vector. The two vectors are concatenated in the column dimension and then passed through a 3D convolution and a Sigmoid activation function to obtain a space feature vector. The feature map δ3 is multiplied by the time feature vector and the space feature vector respectively, and then added to the original feature map itself, that is, the feature map γ3 is obtained.

[0142] The 3DCCD module includes a 3D convolution module with an output channel number of 256, a convolution kernel size of 3x3x3, a stride of 2x1x2, and a padding of 1, a 3D convolution model with an output channel number of 512, a convolution kernel size of 3x3x3, a stride of 2x1x2, and a padding of 1, and a three-dimensional random deactivation (hereinafter referred to as "3DDropout") model with a random deactivation rate of 0.5. The feature map γ2 is sequentially passed through the above-mentioned 3D convolution model and three-dimensional random deactivation model to obtain the feature map α3.

[0143] The first Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a convolution kernel size of 3x3x3, a stride of 1, and a padding of 1, and the other branch has no model. The feature maps of the last two branches are connected by residual connection to obtain the output. The second Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a convolution kernel size of 3x3x3, a stride of 1, and a padding of 1, and the other branch has no model. The feature maps of the last two branches are also connected by residual connection to obtain the output. In this process, the channel number of the feature map remains unchanged, and the feature map α3 is sequentially passed through the above-mentioned two Res_Block modules to obtain the eye feature map β3.

[0144] In an embodiment, inputting the mouth image sequence into the mouth branch of the three-dimensional spatio-temporal attention convolutional network to extract the mouth image feature map includes:

[0145] The mouth image sequence is sequentially passed through two three-dimensional single convolution feature distillation (hereinafter referred to as "3D CBL") modules to obtain the feature map δ4;

[0146] The feature map δ4 passes through a three-dimensional space-time attention (hereinafter referred to as "3DSTAttention") module to obtain a feature map γ4;

[0147] The feature map γ4 passes through a three-dimensional double convolution feature extraction (hereinafter referred to as "3D CCD") module to obtain a feature map α4;

[0148] The feature map α4 sequentially passes through two residual modules (hereinafter referred to as "Res_Block") to obtain a mouth feature map β4.

[0149] The calculation method of the mouth branch Layer4 is as follows:

[0150] The first 3DCBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 1x2x2, and padding of 1, a 3D batch normalization module with an output channel number of 64, and a randomized leaky ReLU (hereinafter referred to as "LeakyReLU"). The second 3DCBL module includes a 3D convolution model with a convolution kernel size of 3x3x3, a stride of 2x2x2, and padding of 1, a 3D batch normalization model with an output channel number of 128, and a LeakyReLU activation function model. The mouth image sequence is first passed through the first 3DCBL module and then passed through the second 3DCBL module to obtain the feature map δ4.

[0151] The 3DSTAttention module includes a temporal attention model and a spatial attention model, and the channel number of the feature map in the module remains unchanged. The temporal attention model is to copy the feature map into two identical parts, one part is subjected to maximum pooling to obtain a maximum pooling feature vector, and the other part is subjected to average pooling to obtain an average pooling vector. The two vectors are then added after passing through a 2D convolution module, a ReLU activation function, and a 2D convolution module, and then passing through a Sigmoid activation function to obtain a temporal feature vector. The spatial attention model is to copy the feature map into two identical parts, one part is subjected to maximum pooling to obtain a maximum pooling feature vector, and the other part is subjected to average pooling to obtain an average pooling vector. The two vectors are concatenated in the column dimension and then passed through a 3D convolution and a Sigmoid activation function to obtain a spatial feature vector. The feature map δ3 is multiplied by the temporal feature vector and the spatial feature vector respectively, and then added to the original feature map itself, thereby obtaining the feature map γ4.

[0152] The 3DCCD module includes a 3D convolution module with 256 output channels, a 3x3x3 convolution kernel, a 2x1x2 stride, and a 1 padding, a 3D convolution module with 512 output channels, a 3x3x3 convolution kernel, a 2x1x2 stride, and a 1 padding, and a three-dimensional random inactivation (hereinafter referred to as “3D Dropout”) model with a random inactivation rate of 0.5. The feature map γ3 sequentially passes through the above 3D convolution module and three-dimensional random inactivation model to obtain the feature map α4.

[0153] The first Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a 3x3x3 convolution kernel, a 1 stride, and a 1 padding. The other branch has no model on it. The feature maps of the last two branches are connected by residual connection to obtain the output. The second Res_Block includes two branches. One branch needs to pass through two 3D CBL models with a 3x3x3 convolution kernel, a 1 stride, and a 1 padding. The other branch has no model on it. The feature maps of the last two branches are also connected by residual connection to obtain the output. In this process, the channel number of the feature map remains unchanged at 512. The feature map α3 sequentially passes through the above two Res_Block modules to obtain the eye feature map β4.

[0154] Referring to Figure 5 For example, the dual stream feature pyramid fusion module (Dual Stream-FPN feature pyramid) and the eye and mouth auxiliary model (EMA) in step S4 have the following specific principles:

[0155] In the Dual Stream-FPN feature pyramid, the channel number of the feature map always remains 512. Among them, the face branch first passes through a 3D CCD module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1, then adds the face optical flow after passing through a 3D CCD module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1 and a 1x1 convolution module with a convolution kernel size of 1 to obtain the first layer of superimposed feature map. The first layer of superimposed feature map passes through a Res_Block module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1, then adds the face optical flow after passing through a Res_Block module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1 and a 1x1 convolution module with a convolution kernel size of 1 to obtain the second layer of superimposed feature map. The second layer of superimposed feature map passes through a Res_Block module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1, then adds the face optical flow after passing through a Res_Block module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1 and a 1x1 convolution module with a convolution kernel size of 1 to obtain the third layer of superimposed feature map. After the third layer of superimposed feature map is flattened, the feature vector fused with the face and the optical flow is obtained.

[0156] In the eye-mouth auxiliary module EMA, the channel number of the feature map always remains 512. Among them, the eye branch first passes through a 3D CCD module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1, then passes through two Res_Block modules with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1, and finally flattens the feature map to obtain the eye feature vector. The mouth branch is first passed through a 3D CCD module with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1, then passed through two Res_Block modules with a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1, and finally flattens the feature map to obtain the mouth feature vector.

[0157] For example, in step S5, the face optical flow feature vector, the eye feature vector and the mouth feature vector are first fused into a vector by splicing in the column dimension, and then sent to the category probability calculation module (head module). In the head module, the vector will pass through a fully connected layer with an output dimension of 512, a ReLU activation function and a fully connected layer with an output dimension of 3 or 5 (equal to the number of categories), that is, the probability of each category can be obtained.

[0158] In an embodiment, the cross-entropy loss function is calculated, and the probability calculation formula of each category is as follows:

[0159]

[0160] where M represents the number of classes; yic represents a sign function (0 or 1), which takes 1 if the true class of sample i is equal to c, and 0 otherwise; pic represents the predicted probability that the observed sample i belongs to class c; N represents the number of folds (N-fold cross-validation is used for model training), and L i represents the sum of the loss functions corresponding to each fold.

[0161] For example, in step (6), the video stream is first batched through a series of preprocessing operations, and then fed into the network for forward propagation, so that the probability belonging to a certain class (for 3 classification of positive, negative and surprised micro-expression) can be obtained. For example, an image sequence is input, and the model outputs three probability values (i.e. a 1x3 vector), and the micro-expression corresponding to the maximum probability is the recognition result of the model for the micro-expression.

[0162] Another embodiment of the present application provides a micro-expression recognition device, which comprises:

[0163] A model training module is configured to train the micro-expression recognition model to obtain a trained micro-expression recognition model; and the training of the micro-expression recognition model comprises:

[0164] S1, obtaining a first micro-expression data set and a second micro-expression data set, wherein the first micro-expression data set and the second micro-expression data set comprise original data frames in a preset video;

[0165] S2, performing image preprocessing on the first micro-expression data set and the second micro-expression data set to obtain a fixed number of full-face image sequences, eye image sequences, mouth image sequences and optical flow image sequences;

[0166] S3, inputting the full-face image sequences, the eye image sequences, the mouth image sequences and the optical flow image sequences into a three-dimensional spatio-temporal attention convolutional network respectively to extract full-face image feature maps, eye image feature maps, mouth image feature maps and optical flow image feature maps;

[0167] S4, inputting the full-face image feature maps and the optical flow image feature maps into a multi-branch spatio-temporal attention pyramid network to perform feature fusion to obtain full-face feature maps after feature fusion, and inputting the feature maps of the eye and mouth branches into an eye-mouth auxiliary model to perform calculation to obtain eye feature maps after feature fusion and mouth feature maps after feature fusion;

[0168] S5, sending the full-face feature maps after feature fusion, the eye feature maps after feature fusion and the mouth feature maps after feature fusion to a class probability calculation model, calculating a cross-entropy loss function in the class probability calculation model, and obtaining the probability of each class according to the cross-entropy loss function;

[0169] S6, training the model by using steps S1-S5 to obtain a trained micro-expression recognition model;

[0170] The micro-expression recognition module classifies and recognizes the facial micro-expression by using the trained micro-expression recognition model. The present application aims to protect a micro-expression recognition method. The present application uses the proposed four-branch 3D STAC network for feature extraction as a whole, and the full-face and optical flow branches enter the improved Dual Stream-FPN, and the eye and mouth features are input into the EMA module to perform feature map output fusion, complete the joint learning of spatial and temporal features and detailed information, realize the capture of image detailed information, and improve the micro-expression recognition performance and efficiency. The 3D STAC network uses the 3D convolution mode to avoid the omission of time sequence information of the traditional convolution, and sets a three-dimensional space-time attention model ST attention inside, including a time attention module (TAM) and a space attention module (SAM), which performs double feature weight calibration from the spatial and temporal dimensions, amplifies the local subtle expression changes of the micro-expression style and the expression amplitude changes in the time dimension, improves the recognition accuracy based on multi-directional features, realizes adaptive space-time feature selection, and makes up for the problem of learning redundant features without preference of three-dimensional convolution. Then, the improved Dual Stream-FPN is used, the double-flow feature pyramid fusion module fuses the low-layer optical flow features and the high-layer full-face image features, and simultaneously utilizes the high resolution and motion information of the bottom optical flow features and the high semantic information of the high-layer face image features, and shares the semantic information in the double-flow structure, so that the double-flow information is effectively combined. Meanwhile, aiming at the rich micro-motion semantic information of the eye and mouth regions, the features extracted from the eyes and mouth are extracted again in the eye and mouth auxiliary structure, and the multi-branch features are enhanced and fused, and the cross-entropy loss is calculated in the head. The multi-branch feature enhances the correction of the original prediction by the local details and optical flow motion information. The excellent performance on the CASME II and SMIC and composite data sets proves that the method can meet the task demand of micro-expression accurate recognition under different data sets.

[0171] It should be understood that the above specific embodiments of the present application are only used for illustrative or explanatory purposes of the principles of the present application, and do not constitute a limitation on the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

[0172] The application has been described herein with reference to the preferred embodiments. Obviously, modifications and alterations will occur to others upon reading and understanding the preceding detailed description. It is intended to include all such modifications and alterations insofar as they come within the scope of the application. It is also to be understood that the application is not to be limited to the specific embodiments disclosed and that modifications, changes and substitutions are intended to be constructed as falling within the scope of the appended claims and equivalents thereof.

[0173] Although the embodiments of this application have been described in detail, it should be understood that various changes, substitutions and alterations can be made hereto without departing from the spirit and scope of the application as defined by the appended claims.

[0174] Obviously, the above-described embodiments are only examples for clearly illustrating the application and are not intended to limit the embodiments. Based on the above description, those skilled in the art can further make various changes and modifications to the embodiments. There is no need and no possibility to enumerate all the embodiments. Any changes or modifications made by those skilled in the art should fall within the scope of the application.

[0175] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a system, or a computer program product. Thus, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0176] The application is described with reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 The flowchart and / or block diagrams can also be implemented by one or more of the flowcharts and / or blocks in the flowcharts and / or block diagrams. Figure 1 The flowchart and / or block diagrams can also be implemented by one or more of the flowcharts and / or blocks in the flowcharts and / or block diagrams.

[0177] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 The flowchart and / or block diagrams can also be implemented by one or more of the flowcharts and / or blocks in the flowcharts and / or block diagrams.Figure 1 the function specified in the one or more blocks.

[0178] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operational steps are performed on the computer or other programmable data processing device to generate a computer-implemented process, thus the instructions executed on the computer or other programmable data processing device provide a process for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0179] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by instructing relevant hardware through programs, and the programs can be stored in a computer-readable storage medium. When the program is executed, the flow of each embodiment of the above-mentioned method is included. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM) and the like.

[0180] The steps in the embodiment method of the present application can be adjusted, combined and reduced in sequence according to actual needs. The modules in the embodiment system of the present application can be combined, divided and reduced according to actual needs.

Claims

1. A micro-expression recognition method, characterized in that, The method comprises the following steps: S1, acquiring a first micro-expression dataset and a second micro-expression dataset, wherein the first micro-expression dataset and the second micro-expression dataset comprise original data frames in a preset video; S2, performing image preprocessing on the first micro-expression dataset and the second micro-expression dataset to obtain a fixed-frame-number full-face image sequence, an eye image sequence, a mouth image sequence and an optical flow image sequence; S3, inputting the full-face image sequence, the eye image sequence, the mouth image sequence and the optical flow image sequence into a three-dimensional spatiotemporal attention convolutional network respectively to extract a full-face image feature map, an eye image feature map, a mouth image feature map and an optical flow image feature map; S4, inputting the full-face image feature map and the optical flow image feature map into a dual-flow feature pyramid fusion module to perform feature fusion and obtain a full-face feature map after feature fusion, and inputting the eye image feature map and the mouth image feature map into an eye-mouth auxiliary module to calculate an eye feature map after feature fusion and a mouth feature map after feature fusion; S5, sending the full-face feature map after feature fusion, the eye feature map after feature fusion and the mouth feature map after feature fusion to a category probability calculation module, calculating a cross-entropy loss function in the category probability calculation module, and obtaining a probability of each category according to the cross-entropy loss function, wherein the category is a result of micro-expression recognition; S6, performing model training by using steps S1-S5 to obtain a trained micro-expression recognition model; S7, classifying and recognizing a human face micro-expression by using the trained micro-expression recognition model; In the step of inputting the full-face image feature map and the optical flow image feature map into the dual-flow feature pyramid fusion module to perform feature fusion and obtain the full-face feature map after feature fusion, the step comprises the following steps: inputting the full-face image feature map into a first 3DCCD module to obtain a first output; inputting the optical flow image feature map into a second 3DCCD module and a first convolution module in sequence to obtain a second output; adding the first output and the second output to obtain a first layer superimposed feature map; inputting the first layer superimposed feature map into a first Res_Block module to obtain a third output; inputting the optical flow image feature map into a second Res_Block module and a second convolution module in sequence to obtain a fourth output; adding the third output and the fourth output to obtain a second layer superimposed feature map; inputting the second layer superimposed feature map into a third Res_Block module to obtain a fifth output; inputting the optical flow image feature map into a fourth Res_Block module and a third convolution module in sequence to obtain a third layer superimposed feature map; flattening the third layer superimposed feature map to obtain the full-face feature map after feature fusion. The eye image feature map and the mouth image feature map are input into the eye-mouth auxiliary module for calculation to obtain a feature-fused eye feature map and a feature-fused mouth feature map, including: sequentially inputting the eye image feature map into a third 3D CCD module, a fifth Res_Block module and a sixth Res_Block module, and flattening and outputting to obtain the feature-fused eye feature map; sequentially inputting the mouth image feature map into a fourth 3D CCD module, a seventh Res_Block module and an eighth Res_Block module, and flattening and outputting to obtain the feature-fused mouth feature map. The first 3D CCD module to the fourth 3D CCD module have a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1; the first convolution module to the third convolution module are 1x1 convolutions with a convolution kernel size of 1; and the first Res_Block module to the eighth Res_Block module have a convolution kernel size of 3x3x3, a stride of 1 and a padding of 1.

2. The micro-expression recognition method of claim 1, wherein the first micro-expression dataset comprises a high-frame-rate data subset, a normal data subset and an infrared data subset, wherein the high-frame-rate data subset is captured using a high-frame-rate camera, the normal data subset is captured using a normal camera, and the infrared data subset is captured using an infrared camera. The second micro-expression dataset is captured using a high-speed camera.

3. The micro-expression recognition method of claim 1, wherein the preprocessing in S2 comprises: S2.1, processing the first micro-expression dataset and the second micro-expression dataset using a time domain interpolation module to obtain an interpolated image sequence; S2.2, recognizing face key points using a machine model algorithm library on the interpolated image sequence; S2.3, using a 5-key-point model for registration through a registration method of the machine model algorithm library, and cropping face, eye and mouth pictures after registration using the machine model algorithm library to obtain a full face image sequence, an eye image sequence and a mouth image sequence, respectively; S2.4, extracting optical flow features from the interpolated image sequence using an optical flow method to obtain an optical flow image sequence.

4. The micro-expression recognition method of claim 1, wherein inputting the full face image sequence, the eye image sequence, the mouth image sequence and the optical flow image sequence into a three-dimensional spatiotemporal attention convolutional network to extract full face image feature maps, eye image feature maps, mouth image feature maps and optical flow image feature maps comprises: inputting the full face image sequence into a face branch of the three-dimensional spatiotemporal attention convolutional network to extract full face image feature maps; inputting the optical flow image sequence into a face optical flow branch of the three-dimensional spatiotemporal attention convolutional network to extract optical flow image feature maps; inputting the eye image sequence into a double eye branch of the three-dimensional spatiotemporal attention convolutional network to extract eye image feature maps; inputting the mouth image sequence into a mouth branch of the three-dimensional spatiotemporal attention convolutional network to extract mouth image feature maps. ​ ​ ​ 5. The micro-expression recognition method of claim 4, wherein the full-face image sequence is input into a face branch of the three-dimensional spatio-temporal attention convolutional network to extract a full-face image feature map, including: sequentially inputting the full-face image sequence into two three-dimensional single convolution feature distillation modules to obtain a feature map δ1; inputting the feature map δ1 into a three-dimensional spatio-temporal attention module to obtain a feature map γ1; inputting the feature map γ1 into a three-dimensional double convolution feature extraction module to obtain a feature map α1; and sequentially inputting the feature map α1 into two residual modules to obtain a full-face feature map β1.

6. The micro-expression recognition method of claim 4, wherein the optical flow image sequence is input into an optical flow branch of the three-dimensional spatio-temporal attention convolutional network to extract an optical flow image feature map, including: sequentially inputting the optical flow image sequence into two three-dimensional single convolution feature distillation modules to obtain a feature map δ2; inputting the feature map δ2 into a three-dimensional spatio-temporal attention module to obtain a feature map γ2; inputting the feature map γ2 into a three-dimensional double convolution feature extraction module to obtain a feature map α2; and sequentially inputting the feature map α2 into two residual modules to obtain an optical flow feature map β2.

7. The micro-expression recognition method of claim 4, wherein the eye image sequence is input into a double eye branch of the three-dimensional spatio-temporal attention convolutional network to extract an eye image feature map, including: sequentially inputting the eye image sequence into two three-dimensional single convolution feature distillation modules to obtain a feature map δ3; inputting the feature map δ3 into a three-dimensional spatio-temporal attention module to obtain a feature map γ3; inputting the feature map γ3 into a three-dimensional double convolution feature extraction module to obtain a feature map α3; and sequentially inputting the feature map α3 into two residual modules to obtain an eye feature map β3.

8. The micro-expression recognition method of claim 4, wherein the mouth image sequence is input into a mouth branch of the three-dimensional spatio-temporal attention convolutional network to extract a mouth image feature map, including: sequentially inputting the mouth image sequence into two three-dimensional single convolution feature distillation modules to obtain a feature map δ4; inputting the feature map δ4 into a three-dimensional spatio-temporal attention module to obtain a feature map γ4; inputting the feature map γ4 into a three-dimensional double convolution feature extraction module to obtain a feature map α4; and sequentially inputting the feature map α4 into two residual modules to obtain a mouth feature map β4. The cross-entropy loss function is calculated, and the probability calculation formula of each category is as follows: The device includes: A model training module trains the micro-expression recognition model to obtain a trained micro-expression recognition model; the training of the micro-expression recognition model includes: S1, obtaining a first micro-expression data set and a second micro-expression data set, the first micro-expression data set and the second micro-expression data set including original data frames in a preset video; S2, image pre-processing is performed on the first micro-expression data set and the second micro-expression data set to obtain a fixed frame number of a full-face image sequence, an eye image sequence, a mouth image sequence, and an optical flow image sequence; ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 9. The micro-expression recognition method of claim 1, wherein, ​ where M denotes the number of classes, y ic denotes the indicator function, which takes 1 if the true class of sample i equals c, and 0 otherwise; p ic denotes the predicted probability that observation sample i belongs to class c, N denotes the number of folds, L i denotes the sum of the loss function for each fold.

10. A micro-expression recognition apparatus, characterized by, ​ ​ ​ ​ S3. Input the full-face image sequence, the eye image sequence, the mouth image sequence, and the optical flow image sequence into a three-dimensional spatiotemporal attention convolutional network for extraction to obtain full-face image feature maps, eye image feature maps, mouth image feature maps, and optical flow image feature maps; S4. Input the full-face image feature map and optical flow image feature map into the dual-flow feature pyramid fusion module to perform feature fusion and obtain the full-face feature map after feature fusion. Input the feature maps of the eye and mouth branches into the eye and mouth auxiliary module for calculation to obtain the feature map of the eye after feature fusion and the feature map of the mouth after feature fusion. S5. Send the full-face feature map after feature fusion, the eye feature map after feature fusion, and the mouth feature map after feature fusion to the category probability calculation module. Calculate the cross-entropy loss function in the category probability module and obtain the probability of each category based on the cross-entropy loss function. S6. Use steps S1-S5 to train the model and obtain a trained micro-expression recognition model. The process involves inputting the full-face image feature map and the optical flow image feature map into a dual-flow feature pyramid fusion module to perform feature fusion and obtain a fused full-face feature map. This includes: inputting the full-face image feature map into a first 3DCCD module to obtain a first output; inputting the optical flow image feature map sequentially into a second 3DCCD module and a first convolution module to obtain a second output; adding the first and second outputs to obtain a first layer of superimposed feature map; inputting the first layer of superimposed feature map into a first Res_Block module to obtain a third output; inputting the optical flow image feature map sequentially into a second Res_Block module and a second convolution module to obtain a fourth output; adding the third and fourth outputs to obtain a second layer of superimposed feature map; inputting the second layer of superimposed feature map into a third Res_Block module to obtain a fifth output; inputting the optical flow image feature map sequentially into a fourth Res_Block module and a third convolution module to obtain a third layer of superimposed feature map; and flattening the third layer of superimposed feature map to obtain the fused full-face feature map. The feature maps of the eye and mouth images are input into the eye-mouth auxiliary module for calculation, resulting in feature maps of the eyes and mouth after feature fusion. This includes: inputting the eye image feature map sequentially into the third 3DCCD module, the fifth Res_Block module, and the sixth Res_Block module, and flattening it to output the feature map of the eyes after feature fusion; and inputting the mouth image feature map sequentially into the fourth 3DCCD module, the seventh Res_Block module, and the eighth Res_Block module, and flattening it to output the feature map of the mouth after feature fusion. The first to the fourth 3DCCD modules have a kernel size of 3×3×3, a stride of 1, and padding of 1, respectively; the first to the third convolutional modules are 1×1 convolutions with a kernel size of 1; the first to the eighth Res_Block modules have a kernel size of 3×3×3, a stride of 1, and padding of 1, respectively. The micro-expression recognition module adopts the trained micro-expression recognition model to classify and recognize the human face micro-expression.

Images