Micro-expression recognition method, system and device and computer readable medium

Abstract

The invention discloses a micro-expression recognition method, system and device and a computer readable medium, and belongs to the technical field of deep learning algorithms, and the method comprises the steps: taking a motion optical flow graph between a start frame and a climax frame in a micro-expression sequence as the input of a network model; the optical flow feature map passes through a picture embedding layer, the picture embedding layer segments the optical flow feature map into image blocks, and each image block comprises motion information of different face units; in the visual Transform structure, attention calculation is carried out on the Patch Token in each layer of encoder through a selection learning module, and the selected Token and the global classification Token are connected and input into the last layer of Transform encoder; through cross entropy loss and contrast loss, classification and identification of micro-expression emotion types are realized. According to the method, the problem of micro-expression fine-grained recognition can be solved, and discriminative micro-expression motion features can be learned in limited samples.

Description

Technical Field

[0001] The present invention relates to the field of deep learning algorithm technology, and specifically to a micro-expression recognition method, system, device and computer-readable medium. Background Art

[0002] Micro-expression is a short and weak facial expression. It often appears when people intentionally or unintentionally hide their true emotions, providing clues for insight into people's true psychology or emotions. This expression has important application value in many fields such as criminal investigation and trial, security protection, and psychological counseling. At present, micro-expression recognition technology mainly includes methods based on facial texture, methods based on facial optical flow changes, and motion feature methods based on deep learning. Recognition algorithms based on hand-made facial texture and optical flow changes often extract texture information or edge information, and do not make full use of information. For recognition algorithms based on deep learning features, the input video sequence often carries redundant information such as identity and appearance, and cannot effectively learn motion features containing micro-expression identification. Micro-expression has local characteristics, which means that only some facial muscles will change when emotions occur. Although convolutional neural networks can capture these local features, they ignore the long-distance dependencies between different facial motion units. In addition, when extracting motion features from micro-expression video sequences, redundant information such as the identity and appearance of the subject is often mixed in, which is not conducive to the model learning key motion representations. Summary of the invention

[0003] The technical task of the present invention is to address the above shortcomings and provide a micro-expression recognition method, system, device and computer-readable medium, which can solve the problem of fine-grained recognition of micro-expressions, integrate global facial motion information, and learn discriminative micro-expression motion features in limited samples.

[0004] The technical solution adopted by the present invention to solve its technical problem is:

[0005] A micro-expression recognition method uses a motion optical flow map between a start frame and a climax frame in a micro-expression sequence as an input to a network model;

[0006] First, the optical flow feature map passes through the image embedding layer, which divides the optical flow feature map into image blocks. Each image block contains the motion information of different facial units.

[0007] Secondly, in the visual Transformer structure, the attention calculation of the patch token in each layer of the encoder is realized by selecting the learning module, and the selected token is connected with the global classification token and input into the last layer of the Transformer encoder;

[0008] Finally, in order to expand the representation distance between features of samples of different categories and reduce the representation distance of features of samples of the same category, the classification and recognition of micro-expression emotion types are achieved through cross entropy loss and contrast loss.

[0009] Furthermore, the image embedding layer is a process of extracting features by a convolutional neural network, mapping the micro-expression optical flow map area calculated by the convolution kernel into high-dimensional features, and the divided Patch Token meets the requirements of the Transformer serialization input.

[0010] Furthermore, the learning selection module performs attention calculation on the encoded features in each layer of the encoder. The higher the attention weight score, the more obvious the micro-expression motion area features corresponding to the token are, that is, it contains key identification information, which is conducive to classification.

[0011] Furthermore, the contrast loss learns the effective representation of data by comparing the similarities and differences between samples, shortening the distance between micro-expression samples under the same emotion category and increasing the distance between samples of different categories.

[0012] Furthermore, the specific implementation of this method includes:

[0013] 1) Data preprocessing:

[0014] Preprocessing of micro-expression videos, including obtaining video frame sequences, face detection and positioning, face alignment, and extracting optical flow motion feature maps between the start frame and the climax frame;

[0015] 2) Feature Learning:

[0016] Extract motion features from the micro-expression optical flow map. For the optical flow map, in order to preserve the local adjacent structure, a sliding window is used to generate overlapping patches.

[0017] Feed the trainable Patch block into the Transformer encoder to map the features to a latent D-dimensional embedding space, while adding the learnable position embedding to the Patch block to preserve the position information;

[0018] In order to make full use of the attention information, the feature selection learning module is used in each layer of the Transformer encoder to multiply the weights of all layers before the last layer, and then select the A_K Patch Tokens with the largest weights as the input of the last layer; the original entire input sequence is replaced with the labeled Patch blocks corresponding to the areas containing key micro-expression motion information, and the classification Token is connected as input to the last Transformer layer, which not only retains the global information, but also makes the last layer of Transformer encoder focus on the subtle differences between different subcategories, while reducing the areas with less common feature discrimination for invalid information such as background;

[0019] 3) Identification optimization:

[0020] The classification learning of features is supervised by the cross entropy loss function and the contrast loss function. The overall network model is not able to fully supervise the classification learning of features because the changes in micro-expressions are very slight and the differences between categories are small. It is easy to identify one category as an emotion label of another category. The contrast loss function is added to minimize the similarity of classification tokens corresponding to different labels and maximize the similarity of classification tokens of samples with the same label.

[0021] 4) Classification and Identification:

[0022] Use the trained Transformer recognition model to classify and recognize the test set.

[0023] Furthermore, the data preprocessing,

[0024] Acquiring a video frame sequence is to perform frame processing on the video containing micro-expressions, obtain a video frame sequence and store it;

[0025] Face detection and localization uses the Dlib visual library to detect and localize faces in a video frame sequence, giving the number of faces in the detected video frame and the distance between the face and the image boundary.

[0026] Face alignment uses the Dlib visual library to determine 68 key facial feature points, complete face segmentation and achieve face correction;

[0027] The optical flow feature map is obtained by selecting the starting frame and the climax frame in a segment of RGB sequence according to the dataset annotation table, and calculating the motion optical flow between the starting frame and the climax frame using the optical flow algorithm, and finally constructing the optical flow feature map;

[0028] Finally, the preprocessed optical flow motion map is divided into a test set and a training set.

[0029] Furthermore, in order to prevent the loss from being easily affected by negative samples (samples of different categories with very small similarity), the contrast loss function sets a threshold a. Under the balance of the constant threshold a, only the contrast loss of negative samples with similarity greater than a will take effect.

[0030] The present invention also claims protection for a micro-expression recognition system, comprising:

[0031] A data preprocessing module is used to preprocess the micro-expression video, obtain the starting frame and climax frame of a micro-expression picture sequence, and calculate the optical flow motion map between the starting frame and the climax frame using an optical flow algorithm;

[0032] The feature learning module is used to feed the optical flow map into the visual Transformer network. After feature learning, the area with discriminative information of micro-expression facial muscle movement is obtained;

[0033] The classification and recognition module optimizes classification and recognition by jointly constraining the cross entropy loss function and the contrast loss function; the trained Transformer recognition model is used to classify and recognize the test set;

[0034] The system specifically realizes micro-expression recognition through the above-mentioned micro-expression recognition method.

[0035] The present invention also claims protection for a micro-expression recognition device, comprising: at least one memory and at least one processor;

[0036] The at least one memory is used to store a machine-readable program;

[0037] The at least one processor is used to call the machine-readable program to implement the above method.

[0038] The present invention also claims protection for a computer-readable medium having computer instructions stored thereon, which implement the above method when executed by a processor.

[0039] Compared with the prior art, the micro-expression recognition method, system, device and computer-readable medium of the present invention have the following beneficial effects:

[0040] The present invention mainly designs a micro-expression recognition method based on visual Transformer feature selection and fusion. Firstly, according to the starting frame and climax frame of the micro-expression sequence, the motion optical flow map between the two frames is calculated by the optical flow algorithm; secondly, the attention weights of all previous layers are calculated in the Transform encoder through the learning selection module, and recursively applied to the original attention weights in all layers, so as to converge to the Token with larger weight in the last layer of encoder, and select the motion area with significant change in the optical flow map; finally, the network model is learned under the joint constraints of contrast loss and cross entropy loss to effectively obtain and fuse more discriminative micro-expression facial motion unit change features. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flowchart of a micro-expression recognition method provided by an embodiment of the present invention;

[0042] Figure 2 It is a diagram of a visual Transformer classification and recognition algorithm model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with specific embodiments.

[0044] The embodiment of the present invention provides a micro-expression recognition method, which uses a motion optical flow map between a start frame and a climax frame in a micro-expression sequence as an input of a network model;

[0045] First, the optical flow feature map passes through the image embedding layer, which divides the optical flow feature map into image blocks. Each image block contains the motion information of different facial units.

[0046] Secondly, in the visual Transformer structure, the attention calculation of the patch token in each layer of the encoder is realized by selecting the learning module, and the selected token is connected with the global classification token and input into the last layer of the Transformer encoder;

[0047] Finally, in order to expand the representation distance between features of samples of different categories and reduce the distance between features of samples of the same category, in addition to the cross-entropy loss, the contrast loss is also introduced to achieve the classification and recognition of micro-expression emotion types.

[0048] Among them, the image embedding layer is the process of extracting features by the convolutional neural network, mapping the micro-expression optical flow map area calculated by the convolution kernel into high-dimensional features, and the divided Patch Token meets the requirements of Transformer serialization input.

[0049] The learning selection module is a process of calculating the attention of the encoded features in each layer of the encoder. The higher the attention weight score, the more obvious the micro-expression motion area features corresponding to the token are, and it contains key identification information, which is conducive to classification.

[0050] The contrast loss is to learn the effective representation of data by comparing the similarities and differences between samples, shorten the distance between micro-expression samples under the same emotion category, and increase the distance between samples of different categories.

[0051] The specific implementation process of this method is as follows:

[0052] 1. Data preprocessing:

[0053] The micro-expression video is preprocessed, including obtaining video frame sequence, face detection and positioning, face alignment, and extracting optical flow motion feature map between the start frame and the climax frame.

[0054] Obtaining a video frame sequence is to perform frame processing on the video containing micro-expressions, obtain a video frame sequence and store it; face detection and positioning is to use the Dlib visual library to detect and locate the face in the video frame sequence, and give the number of faces in the detected video frame and the distance between the face and the image boundary; face alignment is to use the Dlib visual library to determine the 68 key feature points of the face, complete face segmentation and realize face correction; obtaining an optical flow feature map is to select the starting frame and climax frame in a RGB sequence according to the data set annotation table, and use the optical flow algorithm to calculate the motion optical flow between the starting frame and the climax frame, and finally construct the optical flow feature map. Finally, the pre-processed optical flow motion map is divided into a test set and a training set.

[0055] 2. Feature Learning:

[0056] Extract motion features from the micro-expression optical flow map. For the optical flow map, in order to preserve the local adjacent structure, a sliding window is used to generate overlapping patches.

[0057] The trainable Patch block is fed into the Transformer encoder to map the features into a latent D-dimensional embedding space, while the learnable position embedding is added to the Patch block to preserve the position information.

[0058] In order to make full use of the attention information, a feature selection learning module is passed through each layer of the Transformer encoder, the weights of all layers before the last layer are multiplied, and then the A_K Patch Tokens with the largest weights are selected as the input of the last layer; the original entire input sequence is replaced with the labeled Patch blocks corresponding to the areas containing key micro-expression motion information, and the classification Token is connected as input to the last Transformer layer, which not only retains the global information, but also enables the last layer of Transformer encoder to focus on the subtle differences between different subcategories, while reducing the areas with less discernible common features such as invalid information such as background.

[0059] 3. Identification optimization:

[0060] The classification learning of features is supervised by the cross entropy loss function and the contrast loss function. The overall network model is not sufficient to fully supervise the classification learning of features in the simple cross entropy loss. Because the changes in micro-expressions are very weak and the differences between categories are very small, it is easy to identify one of the categories as the emotional label of other categories. The contrast loss function is added to minimize the similarity of classification tokens corresponding to different labels and maximize the similarity of classification tokens of samples with the same label. In order to prevent the loss from being easily affected by negative samples (samples of different categories with very little similarity), a threshold a is set. Under the balance of the constant threshold a, only the contrast loss of negative samples with a similarity greater than a will take effect.

[0061] 4. Classification and identification:

[0062] Use the trained Transformer recognition model to classify and recognize the test set.

[0063] Transformer is a deep learning model architecture for natural language processing (NLP) and other sequence-to-sequence tasks. The Transformer architecture introduces a self-attention mechanism, which makes it perform well when processing sequence data. The self-attention mechanism is the core component of the visual Transformer, which captures global information by calculating the correlation between each element in the input sequence and other elements. This mechanism enables the model to focus on key areas in the image and improve recognition accuracy. Compared with traditional neural networks, the visual Transformer provides powerful representation capabilities that can capture global and local information in the image.

[0064] The embodiment of the present invention further provides a micro-expression recognition system, comprising:

[0065] A data preprocessing module is used to preprocess the micro-expression video, obtain the starting frame and climax frame of a micro-expression picture sequence, and calculate the optical flow motion map between the starting frame and the climax frame using an optical flow algorithm;

[0066] The feature learning module is used to feed the optical flow map into the visual Transformer network. After feature learning, the area with discriminative information of micro-expression facial muscle movement is obtained;

[0067] The classification and recognition module optimizes classification and recognition by jointly constraining the cross entropy loss function and the contrast loss function; the trained Transformer recognition model is used to classify and recognize the test set;

[0068] The system specifically implements micro-expression recognition through the micro-expression recognition method described in the above embodiment.

[0069] The data preprocessing module:

[0070] The micro-expression video is preprocessed, including obtaining video frame sequence, face detection and positioning, face alignment, and extracting optical flow motion feature map between the start frame and the climax frame.

[0071] Obtaining a video frame sequence is to perform frame processing on the video containing micro-expressions, obtain a video frame sequence and store it; face detection and positioning is to use the Dlib visual library to detect and locate the face in the video frame sequence, and give the number of faces in the detected video frame and the distance between the face and the image boundary; face alignment is to use the Dlib visual library to determine the 68 key feature points of the face, complete face segmentation and realize face correction; obtaining an optical flow feature map is to select the starting frame and climax frame in a RGB sequence according to the data set annotation table, and use the optical flow algorithm to calculate the motion optical flow between the starting frame and the climax frame, and finally construct the optical flow feature map. Finally, the pre-processed optical flow motion map is divided into a test set and a training set.

[0072] The feature learning module:

[0073] Extract motion features from the micro-expression optical flow map. For the optical flow map, in order to preserve the local adjacent structure, a sliding window is used to generate overlapping patches.

[0074] The trainable Patch block is fed into the Transformer encoder to map the features into a latent D-dimensional embedding space, while the learnable position embedding is added to the Patch block to preserve the position information.

[0075] In order to make full use of the attention information, a feature selection learning module is passed through each layer of the Transformer encoder, the weights of all layers before the last layer are multiplied, and then the A_K Patch Tokens with the largest weights are selected as the input of the last layer; the original entire input sequence is replaced with the labeled Patch blocks corresponding to the areas containing key micro-expression motion information, and the classification Token is connected as input to the last Transformer layer, which not only retains the global information, but also enables the last layer of Transformer encoder to focus on the subtle differences between different subcategories, while reducing the areas with less discernible common features such as invalid information such as background.

[0076] The classification recognition module:

[0077] The classification learning of features is supervised by the cross entropy loss function and the contrast loss function. The overall network model is not sufficient to fully supervise the classification learning of features in the simple cross entropy loss. Because the changes in micro-expressions are very weak and the differences between categories are very small, it is easy to identify one of the categories as the emotional label of other categories. The contrast loss function is added to minimize the similarity of classification tokens corresponding to different labels and maximize the similarity of classification tokens of samples with the same label. In order to prevent the loss from being easily affected by negative samples (samples of different categories with very little similarity), a threshold a is set. Under the balance of the constant threshold a, only the contrast loss of negative samples with a similarity greater than a will take effect.

[0078] Use the trained Transformer recognition model to classify and recognize the test set.

[0079] The present invention also claims protection for a micro-expression recognition device, comprising: at least one memory and at least one processor;

[0080] The at least one memory is used to store a machine-readable program;

[0081] The at least one processor is used to call the machine-readable program to implement the micro-expression recognition method described in the above embodiment.

[0082] The embodiment of the present invention further provides a computer-readable medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the micro-expression recognition method described in the above embodiment is implemented. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0083] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0084] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0085] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0086] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0087] The present invention is shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the code review methods in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A micro-expression recognition method, characterized in that: Use the motion optical flow map between the start frame and the climax frame in a micro-expression sequence as the input of the network model; First, the optical flow feature map passes through the image embedding layer, which divides the optical flow feature map into image blocks. Each image block contains the motion information of different facial units. Secondly, in the visual Transformer structure, the attention calculation of the PatchToken in each layer of the encoder is realized by selecting the learning module, and the selected Token is connected with the global classification Token and input into the last layer of the Transformer encoder; Finally, the classification and recognition of micro-expression emotion types are achieved through cross entropy loss and contrast loss.

2. A micro-expression recognition method according to claim 1, characterized in that: The image embedding layer is a process of extracting features by a convolutional neural network, mapping the micro-expression optical flow map area calculated by the convolution kernel into high-dimensional features, and the divided Patch Token meets the requirements of the Transformer serialization input.

3. A micro-expression recognition method according to claim 1, characterized in that: The learning selection module performs attention calculation on the encoded features in each layer of the encoder. The higher the attention weight score, the more obvious the micro-expression motion area features corresponding to the token are, that is, it contains key identification information, which is conducive to classification.

4. A micro-expression recognition method according to claim 1, characterized in that: The contrast loss is to learn the effective representation of data by comparing the similarities and differences between samples, shorten the distance between micro-expression samples under the same emotion category, and increase the distance between samples of different categories.

5. A micro-expression recognition method according to claim 1, characterized in that: The specific implementation of this method includes: 1) Data preprocessing: Preprocessing of micro-expression videos, including obtaining video frame sequences, face detection and positioning, face alignment, and extracting optical flow motion feature maps between the start frame and the climax frame; 2) Feature Learning: Extract motion features from the micro-expression optical flow map. For the optical flow map, use a sliding window to generate overlapping patches. Feed the trainable Patch block into the Transformer encoder to map the features to a latent D-dimensional embedding space, while adding the learnable position embedding to the Patch block to preserve the position information; In each layer of the Transformer encoder, the weights of all layers before the last layer are multiplied through the feature selection learning module, and then the A_K Patch Tokens with the largest weights are selected as the input of the last layer; the original entire input sequence is replaced with the labeled Patch blocks corresponding to the areas containing key micro-expression movement information, and the classification Token is connected as the input to the last Transformer layer; 3) Identification optimization: The classification learning of features is supervised by the cross entropy loss function and the contrast loss function, minimizing the similarity of classification tokens corresponding to different labels and maximizing the similarity of classification tokens of samples with the same label; 4) Classification and Identification: Use the trained Transformer recognition model to classify and recognize the test set.

6. A micro-expression recognition method according to claim 5, characterized in that , the data preprocessing, Acquiring a video frame sequence is to perform frame processing on the video containing micro-expressions, obtain a video frame sequence and store it; Face detection and localization uses the Dlib visual library to detect and localize faces in a video frame sequence, giving the number of faces in the detected video frame and the distance between the face and the image boundary. Face alignment uses the Dlib visual library to determine 68 key facial feature points, complete face segmentation and achieve face correction; The optical flow feature map is obtained by selecting the starting frame and the climax frame in a segment of RGB sequence according to the dataset annotation table, and calculating the motion optical flow between the starting frame and the climax frame using the optical flow algorithm, and finally constructing the optical flow feature map; Finally, the preprocessed optical flow motion map is divided into a test set and a training set.

7. A micro-expression recognition method according to claim 5, characterized in that: The contrast loss function sets a threshold a. Under the balance of the constant threshold a, only the contrast loss of negative samples with similarity greater than a will take effect.

8. A micro-expression recognition system, characterized in that: include: A data preprocessing module is used to preprocess the micro-expression video, obtain the starting frame and climax frame of a micro-expression picture sequence, and calculate the optical flow motion map between the starting frame and the climax frame using an optical flow algorithm; The feature learning module is used to feed the optical flow map into the visual Transformer network. After feature learning, the area with discriminative information of micro-expression facial muscle movement is obtained; The classification and recognition module optimizes classification and recognition by jointly constraining the cross entropy loss function and the contrast loss function; the trained Transformer recognition model is used to classify and recognize the test set; The system specifically realizes micro-expression recognition through the micro-expression recognition method described in any one of claims 1 to 7.

9. A micro-expression recognition device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to implement the method according to any one of claims 1 to 7.

10. A computer-readable medium, characterized in that The computer readable medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 7.

Images