A spatio-temporal region three-stream micro-expression recognition method based on transformer and saliency map

Abstract

The application discloses a kind of spatio-temporal region three-flow microexpression recognition methods based on Transformer and saliency map, comprising: using ResNet18 from optical flow map Extraction optical flow feature, it is passed to self-attention module and generates saliency map;Saliency map is multiplied with the peak frame point of microexpression sequence, leave the key microexpression part in peak frame, after being extracted spatial flow feature using ResNet18 is passed to self-attention module;Using Vision Transformer extracts area position feature from microexpression starting frame, including face, five features area position information;Optical flow feature, spatial flow feature and area position feature are input into Transformer, and feature fusion is carried out using multi-head attention mechanism, finally input into softmax layer and classified.The application makes full use of the starting frame and peak frame of microexpression sequence, strengthens the association between each feature and makes the network pay more attention to the subtle motion changes of facial expression.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence emotion recognition, and relates to a micro-expression recognition method based on a spatio-temporal region three-flow micro-expression recognition method of a Transformer and a saliency map. BACKGROUND

[0002] Micro-expression recognition is a branch of emotion recognition that focuses on detecting and analyzing extremely brief and difficult-to-detect facial expression changes that usually last less than a second. Micro-expression recognition technology has broad application potential in many fields, such as lie detection. However, micro-expression recognition also faces not small challenges, such as accuracy and efficiency problems.

[0003] In view of the accuracy problem of micro-expression recognition, previous researches usually only independently focus on single independent feature extraction, and finally fuse different features. These methods often have difficulty in capturing the relationship between different features, so that the network is not sufficient for data utilization, resulting in unsatisfactory micro-expression recognition effect. The features extracted by micro-expression recognition are usually divided into two categories: the first category is spatial features, such as the information covered by the facial micro-expression peak frame, including key point position, texture features and shape features. The second category is time features, which can be generally represented by optical flow features. Optical flow features refer to the motion vector of pixels when tracking the same object or feature point between consecutive frames of images. The extraction of spatial features can generally use convolutional networks to extract after simple cropping and amplification of data. Optical flow features can capture the motion information of facial key points at different time points, so they can effectively capture the dynamic features in micro-expression, which is very important for micro-expression recognition. In addition, facial region position features are beneficial to positioning the position information of the salient region containing facial features.

[0004] ResNet18 network has advantages in micro-expression recognition, especially in processing static image data in micro-expression: first, ResNet18 is a relatively deep convolutional neural network, which has strong feature extraction capability. Second, ResNet18 uses the structure of residual block, which can help to alleviate the problem of gradient disappearance, allowing the network to better train deep models. The saliency map is an important research direction in computer vision, which can be used to determine the salient regions in the image, so the saliency map can help the system focus on the part of the face area with the most micro-expression key information. The self-attention module can help the model focus on the important part of the current task, in addition to this mechanism, it can also capture the relationship between elements in the sequence, using this module can help improve the accuracy of the model. The input of ViT is an image, which is usually divided into fixed-size non-overlapping image blocks. These image blocks are flattened into one-dimensional vectors to form a sequence, and ViT can extract the features and position relationships of each image block, which can be used to extract the position relationships of each salient region of the face.

[0005] Therefore, it is necessary to provide a spatio-temporal region three-flow micro-expression recognition method based on Transformer and saliency map to solve the above problems. SUMMARY

[0006] The main purpose of the present application is to propose a spatio-temporal region three-flow network architecture based on Transformer and saliency map for micro-expression recognition.

[0007] To achieve the above purpose, the present application provides a spatio-temporal region three-flow micro-expression recognition method based on Transformer and saliency map, comprising the following steps:

[0008] Step one, using ResNet18 to extract optical flow features from the optical flow map, passing the optical flow features to the self-attention module, and generating a saliency map;

[0009] Step two, point-multiply the saliency map with the peak frame of the micro-expression sequence, leaving the key micro-expression part in the peak frame, and passing the spatial flow features extracted by the ResNet18 to the self-attention module;

[0010] Step three, using Vision Transformer to extract region position features from the starting frame of the micro-expression database, the region position features containing face and facial feature region position information;

[0011] Step four, input the optical flow features, spatial flow features and region position features into the Transformer, and use the multi-head attention mechanism for feature fusion.

[0012] As a further improvement of the present application, in step one, the optical flow map is input into the ResNet18 network, first through a convolutional layer with a convolution kernel size of 7*7, a step of 2, padding of 3, and an output channel of 64, specifically:

[0013] conv1: k=7, s=2, padding=3, d=64,

[0014] Wherein, k represents the size of the convolution kernel, s represents the step, padding represents padding, and d represents the number of output channels;

[0015] The parameter design of the first module 3*3 convolutional layer is as follows:

[0016] conv2_1: k=3, s=1, padding=1, d=64,

[0017] conv2_2: k=3, s=1, padding=1, d=64,

[0018] conv2_3: k=3, s=1, padding=1, d=64,

[0019] conv2_4: k=3, s=1, padding=1, d=64,

[0020] That is, the size of the convolution kernel is 3*3, the step is 1, and the padding is 1;

[0021] The second module first passes through a 1*1 convolutional layer and down-sampling, and the 3*3 convolutional layer parameter design is as follows:

[0022] conv2_1: k=3, s=2, padding=1, d=128,

[0023] conv2_2: k=3, s=1, padding=1, d=128,

[0024] conv2_3: k=3, s=1, padding=1, d=128,

[0025] conv2_4: k=3, s=1, padding=1, d=128,

[0026] The third module 3*3 convolutional layer has the same parameters as the convolutional layer of the second module, and also performs 1*1 convolution and down-sampling;

[0027] The fourth module 3*3 convolutional layer also has the same parameters as the convolutional layer of the second module, and performs 1*1 convolution but does not perform down-sampling.

[0028] As a further improvement of the application, in the second step, the saliency map is multiplied with the points of the peak frame to highlight the important part where micro-expression occurs, and the important part is taken as the input of the third path to extract features using the ResNet18 and the self-attention module.

[0029] As a further improvement of the application, in the third step, the Vision Transformer uses a self-attention mechanism to calculate the attention weight of each position, and in the Vision Transformer, the position information of the input image is processed through position embedding, and the calculation formula of the position embedding is:

[0030]

[0031] where i is the position index, j is the index of the embedding dimension, and d is the embedding dimension.

[0032] As a further improvement of the application, in the third step, the Transformer includes the self-attention module, which calculates the relationship and weight between the elements in the input sequence, and the weight calculation formula of the self-attention module is as follows: for the element at position i, the self-attention weight is calculated.

[0033]

[0034] where Q i represents the query vector of the position, K i represents the key vector of the position, d k is the dimension of the key.

[0035] The self-attention output F i is calculated as:

[0036]

[0037] where V i represents the value matrix.

[0038] As a further improvement of the application, the self-attention weight is regarded as a saliency score, and the saliency score is mapped to (0, 1) using a Sigmoid function to generate a saliency map, and the formula of the Sigmoid function is as follows:

[0039] Sigmoid(x) = 1 / (1+e^-x).

[0040] As a further improvement of the application, in the fourth step, the features output in the first step the features output in the second step and the features output in the third step Input into the Transformer, utilize the multi-head mutual attention mechanism fusion, as follows:

[0041]

[0042]

[0043]

[0044]

[0045] Wherein, Indicate the emotional features that the time feature matrix O obtained by the i-th mutual attention mechanism needs to pay attention to the space feature matrix S, Indicate the emotional features that the space feature matrix S obtained by the i-th mutual attention mechanism needs to pay attention to the time feature matrix O, then And The time and space feature matrix M output by the i-th mutual attention module is obtained by splicing O_S ; Similarly, calculate the optical flow feature and the district feature output by the multi-head mutual attention module And the output of the multi-head mutual attention module of the space feature and the district feature Finally, splice multiple outputs to obtain a complete output:

[0046]

[0047] As a further improvement of the application, in step four, respectively utilize the optical flow map generated by the peak frame and the starting frame, the point multiplication of the peak frame and the saliency map, the starting frame as input to extract the time feature, the space feature and the district feature, input the result to the Transformer for fusion, and send the fused vector to the softmax layer for multi-classification.

[0048] The beneficial effects of the application are as follows:

[0049] The application proposes a new multi-stream micro-expression recognition network architecture to learn different micro-expression features, including optical flow features, spatial features, district features, and fusion of multiple features; through the above steps, the starting frame and the peak frame of the micro-expression sequence are utilized to the greatest extent, the saliency map and the self-attention module are introduced to make the model more focused on the subtle part of the facial micro-expression, and the interaction between different features is strengthened by using the Transformer fusion, so that a micro-expression recognition method with higher accuracy is realized. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the method flow chart of the application;

[0051] Figure 2 is a residual network structure diagram used by the present application;

[0052] Figure 3 is a Vision Transformer structure diagram;

[0053] Figure 4 is a Transformer Encorder structure diagram;

[0054] Figure 5 is a multi-head attention mechanism unit structure diagram. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in detail below in combination with the drawings and specific embodiments.

[0056] Here, it should be noted that, in order to avoid the present application being obscured by unnecessary details, only structures and / or processing steps closely related to the scheme of the present application are shown in the drawings, and other details not closely related to the present application are omitted.

[0057] In addition, it should also be noted that the term “comprises”, “includes” or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device.

[0058] The embodiments of the present application will be further described in detail below in combination with the drawings of the specification.

[0059] The present application proposes a spatio-temporal region three-stream micro-expression recognition method based on Transformer and saliency map on the basis of existing micro-expression recognition research, and the main steps are as follows Figure 1As shown in the figure. First, the micro-expression library needs to be preprocessed. If the data set contains video, the video data needs to be converted into picture sequence by frame. Second, the micro-expression picture sequence needs to be preprocessed such as cropping and enlarging. For the first stream, the optical flow map between the starting frame and the peak frame needs to be calculated as the input, and the feature extraction is performed through ResNet18 and self-attention module. The saliency map is generated by mapping the importance score to the interval (0, 1). For the second stream, the peak frame is multiplied with the saliency map, and the multiplied map is input into the ResNet18 network and the self-attention module for feature extraction. For the third stream, the position of the salient region (facial features, etc.) needs to be learned, so the starting frame picture needs to be scaled to match the feature size of the other two streams. Then the image is segmented into multiple patches, each patch is flattened into a vector, and the position information of each position patch is added. Then all the patches are sent to the encoder to learn the relationship between the patches. Finally, the output is reshaped. The three output streams are reshaped and sent to the Transformer for feature fusion, and finally the classification result is obtained through the softmax layer.

[0060] As shown in the figure, the present application proposes a three-stream micro-expression recognition method based on Transformer and saliency map, which comprises the following steps: Figure 1

[0061] Step one, the first stream takes the optical flow map as the input, uses ResNet18 to extract the optical flow feature, and then transmits the feature data to the self-attention module to generate the saliency map.

[0062] Step two, multiply the saliency map with the peak frame of the micro-expression sequence to highlight the key expression part, and then transmit the spatial stream feature extracted by ResNet18 to the self-attention module.

[0063] Step three, for the starting frame in the micro-expression database, use Vision Transformer (ViT) to extract the region position feature, which contains the position information of the salient region such as facial features.

[0064] Step four, input the three extracted features into the Transformer, use the multi-head attention network to fuse the features, and output to the softmax layer to get the classification result.

[0065] Each step will be described one by one in combination with the drawings.

[0066] ​The self-attention module is used in feature extraction in steps 1 and 2, and is also used in the VisionTransformer in step 3. Self-attention is used to calculate the relationship and weight between elements in the input sequence. The self-attention weight calculation formula is as follows: For the element at position i, the self-attention weight is calculated as follows:

[0067]

[0068] where Q i The query vector representing the position, indicating the element to be associated. i A key vector representing the position of the other elements. k is the dimension of the key. This formula expresses the correlation between position i and all other positions.

[0069] Calculate the self-attention output F i :

[0070]

[0071] Where V i To generate a saliency map, the self-attention network assigns weights to each feature, treating these weights as saliency scores. Depending on the degree of attention paid to different parts of the input features, the self-attention network learns different weight distributions to assign weights to each feature, treating these weights as saliency scores. The saliency map is generated by mapping the scores to the interval (0, 1) using the Sigmoid function. The Sigmoid function formula is as follows:

[0072] Sigmoid(x)=1 / (1+e^-x),

[0073] The multi-head attention module performs multi-head parallel calculations, connecting the outputs of multiple attention heads together to form a comprehensive output. Step 4: The features output in step 1 Features output in step 2 Features output in step 3 The input is fed into the Transformer and fused using the multi-head mutual attention mechanism to help the model better capture the interactive relationship between features, as follows:

[0074]

[0075]

[0076]

[0077]

[0078] denotes the sentiment feature that the spatial feature matrix S obtained by the i-th mutual attention mechanism needs to pay attention to relative to the time feature matrix O, and then denotes the sentiment feature that the spatial feature matrix S obtained by the i-th mutual attention mechanism needs to pay attention to relative to the time feature matrix O, and then and concatenate to obtain the time and spatial feature matrix M output by the i-th mutual attention module O_S ; similarly, the output of the multi-head mutual attention module for the optical flow feature and the zone feature and the output of the multi-head mutual attention module for the spatial feature and the zone feature Finally, the multiple outputs are spliced to obtain the complete output:

[0079]

[0080] The ResNet18 network module in steps one and two is as shown in Figure 2 . The input of the second stream is the optical flow map obtained by preprocessing the micro-expression database. The optical flow map calculates the motion vector through the change between two frames of micro-expression pixels. Each pixel has a corresponding motion vector, which represents the direction and size of the displacement of the pixel. The method of arrow or color coding is used to represent this vector to form an optical flow field. The least squares method is used to solve the optical flow vector of each local area to obtain the dense optical flow field of the entire image. The size of the input data is 3*224*224, which is input into the ResNet18 network. The residual network is as shown in Figure 2 , which first passes through a convolutional layer. The convolutional layer has a convolution kernel size of 7*7, a step size of 2, a padding of 3, and an output channel of 64. The network structure is as shown in Figure 1 .

[0081] conv1: k=7, s=2, padding=3, d=64,

[0082] where k represents the size of the convolution kernel, s represents the step size, padding represents padding, and d represents the number of output channels. The size of the output data is 64*112*112. The parameter design of the first module 3*3 convolutional layer is as follows:

[0083] conv2_1: k=3, s=1, padding=1, d=64,

[0084] conv2_2: k=3, s=1, padding=1, d=64,

[0085] conv2_3: k=3, s=1, padding=1, d=64,

[0086] conv2_4: k=3, s=1, padding=1, d=64,

[0087] That is, the size of the convolution kernel is 3*3, the step is 1, and the padding is 1. Finally, the output data size of the two convolution layers is 64*56*56, and this layer does not change the size and channel number of the data. The second module first passes through a 1*1 convolution layer and down-sampling, and the design of the convolution layer is as follows:

[0088] Kernel size: 1

[0089] s: 2

[0090] padding: 0

[0091] The design of the 3*3 convolution layer parameters is as follows:

[0092] conv2_1: k=3, s=2, padding=1, d=128,

[0093] conv2_2: k=3, s=1, padding=1, d=128,

[0094] conv2_3: k=3, s=1, padding=1, d=128,

[0095] conv2_4: k=3, s=1, padding=1, d=128,

[0096] After down-sampling, the output data size is 128*28*28. The 3*3 convolution layer of the third module is the same as that of the second module, and 1*1 convolution and down-sampling are also performed. The output data size is 256*14*14. The fourth 3*3 convolution layer is also the same as the convolution layer of the second module, and 1*1 convolution is performed but not down-sampling. The final output data size is 512*14*14.

[0097] In step two, the dot product of the significant map generated in step one and the peak frame is used as the input of the third path to highlight the key part of the micro-expression, as shown in Figure 1 The use of the significant map makes the network pay more attention to the subtle changes of the human face.

[0098] In step three, the Vision Transformer is used, and the structure of the Vision Transformer is as shown in Figure 4 The Transformer encoding module used is as shown in Figure 5As shown. In order for the model to understand the position information of the block, the ViT introduces the position embedding, and the position information of each block is embedded into the corresponding feature vector, so that the model understands the relative position of the block. In ViT, the position information of the input image is usually processed by position embedding, and the calculation formula of the position embedding is:

[0099]

[0100] Where i is the position index, j is the index of the embedding dimension, and d is the embedding dimension. The position embedding will be added to the input feature to consider the position information.

[0101] The micro-expression video is divided into a picture sequence, a video-based classification task is converted into a picture-based task, only the starting frame and the peak frame of the micro-expression sequence are selected, the difficulty of the task is reduced, the dot product of the optical flow map generated by the peak frame and the starting frame, the peak frame and the saliency map is used, and the starting frame is used as input to extract time features, spatial features and location features. The results are input into the Transformer to use the multi-head mutual attention mechanism for fusion.

[0102] In summary, the present application proposes a new multi-stream micro-expression recognition network architecture to learn different micro-expression features, including optical flow features, spatial features, location features, and multi-feature fusion. Through the above steps, the present application makes full use of the starting frame and the peak frame of the micro-expression sequence, introduces the self-attention module and the saliency map to make the model more focused on the subtle part of the facial micro-expression, and uses the multi-head mutual attention mechanism to strengthen the interaction between different features, thereby realizing a micro-expression recognition method with higher accuracy.

[0103] The above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application.

Claims

1. A three-stream micro-expression recognition method based on Transformer and saliency map in spatiotemporal regions, characterized by: The steps include: Step 1: Use ResNet18 to extract optical flow features from the optical flow map, pass the optical flow features to the self-attention module, and generate a saliency map; Step 2: Perform a dot product between the saliency map and the peak frame of the micro-expression sequence, retain the key micro-expression part in the peak frame, extract the spatial stream features using the ResNet18, and pass them to the self-attention module; Step 3: Use Vision Transformer to extract regional position features from the starting frame of the micro-expression database, where the regional position features include the position information of the face and facial features. Step 4: Input the optical flow features, the spatial flow features, and the regional position features into the Transformer, and perform feature fusion using a multi-head attention mechanism; In step 3, the Transformer includes the self-attention module, which calculates the relationship and weight between the internal elements of the input sequence. The weight calculation formula of the self-attention module is as follows: Elements of , calculate the self-attention weight: ， in, The query vector representing the location, A key vector representing the position, is the dimension of the key; Calculating self-attention output : ， Among them, represents the value matrix; The fourth step is to convert the features output in the first step into , the features output in step 2 And the features output in step 3 The input is fed into the Transformer and fused using the multi-head mutual attention mechanism, as follows: ， ， ， ， in, Indicates that through the Mutual attention mechanism, the resulting temporal feature matrix For the spatial feature matrix The emotional characteristics that need attention, Indicates that through The spatial feature matrix obtained by the mutual attention mechanism Relative to the time feature matrix The emotional features that need to be paid attention to, and then and The splicing obtains the time and space feature matrix through the The output of the mutual attention module ; Similarly, calculate the output of optical flow features and location features through the multi-head mutual attention module , and the output of spatial features and location features obtained through the multi-head mutual attention module , and finally concatenate multiple outputs to get the complete output: 。 2. The method for spatiotemporal micro-expression recognition based on a Transformer and saliency map according to claim 1, characterized in that: In step 1, the optical flow map is input into the ResNet18 network, and first passes through a convolutional layer. The convolution kernel size of the convolutional layer is 7*7, the step size is 2, the padding is 3, and the output channel is 64, specifically: ， in, represents the size of the convolution kernel, represents the step length, Indicates filling, Indicates the number of output channels; The parameters of the first module 3*3 convolutional layer are designed as follows: ， ， ， ， That is, the size of the convolution kernel is 3*3, the stride is 1, and the padding is 1; The second module first passes through a 1*1 convolution layer and downsampling. The parameters of the 3*3 convolution layer are designed as follows: ， ， ， ， The 3*3 convolutional layer of the third module has the same parameters as the convolutional layer of the second module, and also performs 1*1 convolution and downsampling; The 3*3 convolution layer of the fourth module also has the same parameters as the convolution layer of the second module, performing 1*1 convolution but not downsampling.

3. The method for spatiotemporal micro-expression recognition based on a Transformer and saliency map according to claim 1, characterized in that: In the step 2, the saliency map is multiplied by the point of the peak frame to highlight the important part of the micro-expression, and the important part is used as the input of the third path to extract features using the ResNet18 and the self-attention module.

4. The method for spatiotemporal micro-expression recognition based on a Transformer and saliency map according to claim 1, characterized in that: In step 3, the Vision Transformer uses a self-attention mechanism to calculate the attention weight of each position. In the Vision Transformer, the position information of the input image is processed by position embedding. The calculation formula of the position embedding is: ， in, is the position index, is the index of the embedding dimension, is the embedding dimension.

5. The method for spatiotemporal micro-expression recognition based on a Transformer and saliency map according to claim 1, characterized in that: The self-attention weight is regarded as a saliency score, and the saliency score is mapped to (0, 1) using the Sigmoid function to generate a saliency map. The Sigmoid function formula is as follows: 。 6. The method for spatiotemporal micro-expression recognition based on a Transformer and saliency map according to claim 1, characterized in that: In step 4, the optical flow map generated by the peak frame and the starting frame, the dot product of the peak frame and the saliency map, and the starting frame are used as input to extract temporal features, spatial features, and location features. The results are input into the Transformer for fusion, and the fused vector is sent to the softmax layer for multi-classification.

Images