Micro-expression recognition method based on fast and slow motion double-branch network

Abstract

The invention belongs to the technical field of computer vision, and discloses a micro-expression recognition method based on a fast and slow motion double-branch network. Carrying out space normalization and time normalization processing on the micro-expression video clip, and extracting fast motion and slow motion from the micro-expression video clip; designing a fast and slow motion double-branch network to obtain fast motion features and slow motion features; the fast motion features and the slow motion features are fused in a feature addition mode, and fused features are obtained; and a classifier module is adopted to classify the fusion features, and micro-expression recognition is achieved. The fast motion branch focuses on overall motion, and long-term and overall motion information is captured; the slow motion branch focuses on detailed motion, captures short-term and detailed motion information, and obtains performance improvement. And designing a space-time modeling mechanism for adding accumulative attention, and adding an attention marking module at each stage to realize accumulative attention. In addition, the motion is represented by pixel difference, the implementation process is simplified, and the calculation complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a micro-expression recognition method based on a fast-slow motion dual-branch network. Background Art

[0002] Facial expressions are divided into two categories: macro-expressions and micro-expressions. The main difference between them is the duration and intensity. Macro-expressions are the most common and well-known expressions in daily life. When a macro-expression occurs, people can generally clearly observe and distinguish the corresponding emotion type. This is because when a macro-expression occurs, it covers a large area of ​​the face, such as the mouth, eyes, cheeks, and chin, and lasts for a long time, usually 0.5 to 4 seconds, which can be easily recognized and interpreted by others. Micro-expressions are fast, spontaneous, low-intensity facial expressions that usually appear when people try to hide their true feelings, especially under high pressure, tension, and anxiety. Micro-expressions are unconscious local movements that cover a small facial area, mainly concentrated in the mouth and eye areas, with a short duration and low movement intensity. Its typical duration is between 0.065 and 0.5 seconds, and it is difficult to recognize with the naked eye. The brevity and concealment of micro-expressions make it difficult to disguise, but it can reflect the individual's true emotional state or inner conflict, and has a higher credibility in emotional expression. It can be used in psychology, criminal investigation, education and other fields.

[0003] Micro-expression recognition is a facial recognition technology that processes micro-expression video sequences to achieve three-classification or five-classification. Micro-expression recognition can be divided into traditional methods and deep learning methods. Traditional methods have advantages in comprehensibility, but have limitations in processing complex micro-expression features and improving recognition accuracy. Deep learning methods perform better in recognition accuracy and robustness through automatic feature extraction and complex pattern recognition.

[0004] In some studies, such as "Gan YS, Liong ST, Yau WC, et al. OFF-ApexNet on micro-expression recognition system[J]. Signal Processing: Image Communication, 2019, 74: 129-139.", "Liong ST, Gan YS, See J, et al. Shallow triple stream three-dimensional cnn(ststnet)for micro-expression recognition[C] / / 2019 14th IEEEinternational conference on automatic face&gesture recognition(FG 2019). IEEE, 2019: 1-5." and "Li H, Sui M, Zhu Z, et al. MMNet: Muscle Motion-guided Network for Micro-expression Recognition[J]. In Proc. 31st Int'l Joint Conf. Artificial Intelligence(IJCAI)", motion features are directly extracted from the start and apex frames of micro-expressions, showing the simplicity and effectiveness of this method. However, this method inevitably ignores the details of the motion information from the vertex to the end frame.

[0005] "Zhi R, Xu H, Wan M, et al. Combining 3D convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition [J]. IEICE TRANSACTIONS on Information and Systems, 2019, 102 (5): 1054-1064." and "Hong J, Lee C, Jung H. Late fusion-based video transformer for facial micro-expression recognition [J]. Applied Sciences, 2022, 12 (3): 1169." Extracting motion information from adjacent frames of the entire sequence can produce richer motion information, but these methods do not fully utilize the annotated vertex frames and introduce redundant action representations. In addition, the above methods often rely on manually designed optical flow features. The optical flow calculation is very complex. It is necessary to calculate the pixel changes between every two frames, and match and interpolate to determine the direction and speed of the optical flow. This complexity makes it difficult for optical flow-based methods to meet real-time requirements.

[0006] Therefore, the present invention proposes a dual-branch micro-expression classification network that integrates fast and slow micro-expression motions, uses pixel differences to represent micro-expression motions, and integrates the overall motion and detail motion features of micro-expressions to solve the above problems. Summary of the invention

[0007] The purpose of this invention is to propose a micro-expression recognition method based on a fast-slow motion dual-branch network, and propose a fast-slow motion dual-branch network that integrates fast and slow motion. The fast motion branch focuses on the overall motion, captures long-term, overall motion information, and avoids being disturbed by redundant local motion information; the slow motion branch focuses on detail motion, captures short-term, detail motion information, and makes up for the dynamic information lost by the overall motion, thereby improving the performance. A spatiotemporal modeling mechanism that adds cumulative attention is designed to guide the model to pay more attention to the region of interest.

[0008] In order to achieve the above task, the present invention adopts the following technical solution: a micro-expression recognition method based on a fast and slow motion dual-branch network, comprising the following steps:

[0009] Step 1: Data preprocessing: spatial and temporal normalization of micro-expression video clips to ensure that the input micro-expression sequence frame height, width and number are uniform;

[0010] Step 2: Extract fast and slow motions. Extract fast and slow motions from a given micro-expression video clip. Fast motion is defined as the pixel difference between the vertex frame and the starting frame, and slow motion is defined as the pixel difference between two adjacent frames of the entire micro-expression sequence.

[0011] Step 3: motion feature extraction: design a fast and slow motion dual-branch network, extract motion features in two branches respectively, and obtain fast motion features and slow motion features;

[0012] Step 4: feature fusion, which combines the fast motion features with the slow motion features by adding features to obtain fused features;

[0013] Step 5: Feature classification: Use the classifier module to classify the fused features to achieve micro-expression recognition.

[0014] The fast-slow motion dual-branch network includes a fast motion branch and a slow motion branch, and the two branches respectively include a sequentially connected spatiotemporal block module, a first stage module, a second stage module, and a third stage module;

[0015] The spatiotemporal block module divides the micro-expression video preprocessed in step 1 into G spatiotemporal blocks;

[0016] The first stage module includes a linear projection module, two hierarchical visual attention modules and a first tag attention module connected in sequence; the spatiotemporal block is input into the linear projection module to form a spatiotemporal tag;

[0017] The second stage module includes a first tag merging module, two hierarchical visual attention modules and a second tag attention module connected in sequence; the first tag merging module is connected to the first tag attention module; the second tag attention module is connected to the first tag attention module;

[0018] The third stage module includes a second tag merging module, six hierarchical visual attention modules and a third tag attention module connected in sequence; the second tag merging module is connected to the second tag attention module; the third tag attention module is connected to the second tag attention module.

[0019] In the slow motion branch, the extracted slow motion is first divided into multiple spatiotemporal blocks, each of which is a feature vector with the same dimension, also called a spatiotemporal marker; the linear projection module projects the feature of each spatiotemporal marker onto dimension, after being input into two hierarchical visual attention modules, it is output into the first marking attention module; C is the feature size of the final output mark of the slow motion branch; the output of the first marking attention module is respectively input into the first marking merging module and the second marking attention module; the output feature of the first marking merging module is input into the two hierarchical visual attention modules and then into the second marking attention module; the output feature of the second marking attention module is respectively input into the second marking merging module and the third marking attention module; the output feature of the second marking merging module is input into the six hierarchical visual attention modules and then into the third marking attention module; the third marking attention module outputs the slow motion feature; the marking merging module in each stage performs 2 times spatial downsampling, so that the number of spatiotemporal blocks in each stage becomes 1 / 4 of the original, and the dimension becomes 2 times of the original; after the three-stage modules of the fast and slow motion dual-branch network, a slow motion feature vector with a dimension size of N×C is output, where N is the number of spatiotemporal labels, T is the time length, H is the video height, and W is the video width.

[0020] The tag attention module adds an attention weight to each spatiotemporal tag of its input to achieve weighted spatiotemporal tags. Let TK be the input spatiotemporal tag, TK max Indicates the maximum value of TK in the channel dimension, TK avg represents the average value of TK in the channel dimension, TK last_atten represents the marked attention weight of the previous stage, MaxPool represents the maximum pooling layer, FC represents the matrix multiplication of the fully connected layer, Sig represents the sigmoid function, and the attention weight TK of the spatiotemporal mark atten The initial calculation is as follows:

[0021] TK atten =Sig(FC([TK max ;TK avg ])) (1)

[0022] The symbol “;” indicates the concatenation operation in the channel dimension;

[0023] Combined with the marked attention weights of the previous stage, the final attention weight TK of this stage is obtained atten ;

[0024] TK atten =TK atten ⊙DownSampling(TK last_atten ) (2)

[0025] Among them, DownSampling is a downsampling operation. last_atten Downsample 2× spatially;

[0026] Finally, the input token and the attention weight TK of this stage atten Perform element-level multiplication to obtain the weighted label TK' output at this stage:

[0027] TK'=TK⊙TK atten (3)

[0028] Here, the symbol “⊙” represents an element-wise multiplication operation.

[0029] The classifier module consists of a fully connected layer and a Softmax layer, and optimizes the fast and slow motion dual-branch network through a weighted cross entropy loss function;

[0030] The input dimension of the fully connected layer is C in =C×N, output dimension C out Determined by the number of categories;

[0031] For a classification problem with K categories, the formula for weighted cross entropy loss is as follows:

[0032]

[0033] Where y = [y1, y2, ..., y K ] is the model’s predicted probability for each category; t = [t1, t2, ..., t K ] is the one-hot encoding vector of the actual label; w = [w1, w2, ..., W K ] is the weight vector of the category.

[0034] The spatial normalization includes face alignment, face cropping and scale unification; the temporal normalization unifies the number of frames of micro-expression video samples by linear interpolation;

[0035] The specific steps of spatial normalization are as follows:

[0036] Detect the facial key points of the first frame of the micro-expression video sequence, calculate the affine transformation matrix through the Left Outer Eye, Right Outer Eye, Nose Tip key points and the corresponding key points of the standard template face, and apply the affine transformation matrix to each frame of the micro-expression video in the sequence;

[0037] Determine the cropping boundary based on the coordinates of key points on the chin, eyebrows, and eyes to crop the face;

[0038] Unify the frame height and frame width to H and W respectively;

[0039] Time normalization is to linearly interpolate the three key frames of the micro-expression sample to the T+1 frame; the key frames are divided into a start frame, a vertex frame, and an end frame.

[0040] In step 2, the micro-expression fast motion and slow motion are extracted, and it is assumed that there are T+1 frames in the micro-expression sequence, which are represented by {F1, F2, F3, ..., F T+1}, where F i is the i-th frame in the sequence, and the shape of each frame tensor is expressed as H×W×3; F onset Indicates the start frame of the sequence, F apex Represents a sequence vertex frame;

[0041] Fast motion is defined as the pixel difference between the vertex frame and the starting frame, that is, Fast_Motion = F apex -F onset , whose tensor shape is H×W×3; slow motion is defined as the pixel difference between two adjacent frames in the entire sequence, that is, Slow_Motion = {F2-F1, F3-F2, F4-F3, ..., F T+1 -F T}, whose tensor shape is T×H×W×3.

[0042] In step 4, the fast motion features and slow motion features are fused; the feature dimensions of fast motion and slow motion are different, namely H×W×3 and T×H×W×3, and the size of each spatial block is set to 4×4×3, and the size of each spatiotemporal block is set to 8×4×4×3. After three stages of learning of the dual-branch network, the number of spatial markers generated by fast motion is finally The number of spatiotemporal markers generated by slow motion The spatiotemporal markers at the same spatial position are averagely pooled in the time dimension to ensure that the number of spatiotemporal markers is equal to the number of spatial markers, and the feature addition method is selected for fusion.

[0043] The micro-expression recognition method based on the fast-slow motion dual-branch network proposed in the present invention integrates fast motion and slow motion, and designs a module for adding cumulative attention to give more attention to the region, thereby improving the performance of micro-expression recognition.

[0044] Beneficial effects of the present invention: The present invention proposes a micro-expression recognition method based on a fast-slow motion dual-branch network, wherein a fast-slow motion dual-branch network is proposed, wherein the fast motion branch focuses on the overall motion and captures long-term, overall motion information; the slow motion branch focuses on the detail motion and captures short-term, detailed motion information, thereby achieving performance improvement. The present invention designs a spatiotemporal modeling mechanism for adding cumulative attention, and adds a tag attention module at each stage to achieve cumulative attention. Moreover, the present invention represents motion with pixel difference, which simplifies the implementation process and reduces computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1This is a flow chart of a micro-expression recognition method based on a fast and slow motion dual-branch network;

[0046] Figure 2 It is a flowchart of spatial preprocessing of face frames;

[0047] Figure 3 It is a double-branch network flow chart for fast and slow motion;

[0048] Figure 4 It is a schematic diagram of the slow motion branch;

[0049] Figure 5 Schematic diagram of the marked attention module of the third stage module;

[0050] Figure 6 Schematic diagram of fast motion spatiotemporal markers being average pooled in the time dimension. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] The present invention will be further described below with reference to the accompanying drawings and examples:

[0053] Combination Figure 1 , Figure 1 The flowchart of the micro-expression recognition method based on the fast and slow motion dual-branch network provided by the present invention is used to illustrate a specific embodiment of the micro-expression recognition method based on the fast and slow motion dual-branch network provided by the present invention, including:

[0054] S1: Perform spatial normalization processing on the micro-expression video to obtain the first video;

[0055] S2: Performing time normalization processing on the first video to obtain a second video;

[0056] S3: Construct a fast and slow motion dual-branch network;

[0057] S4: input the second video into the fast and slow motion dual-branch network and output a classification label;

[0058] S5: Use the classifier module to identify the classification mark.

[0059] In some optional embodiments, referring to Figure 2 , Figure 2The figure is a flow chart of spatial normalization, which performs spatial normalization processing on micro-expression videos, including:

[0060] The micro-expression video includes M frames of images. After face alignment is performed on each frame of the image, natural face cropping is performed to obtain a micro-expression image, and the micro-expression image is adjusted to a preset size.

[0061] It is understandable that the micro-expression video can be regarded as an image sequence, and the face alignment adopts non-rigid transformation face alignment, including:

[0062] The facial key point detection algorithm is used to locate the coordinates of facial key points in each frame of the micro-expression video.

[0063] Select the key point coordinates corresponding to Left Outer Eye, Right Outer Eye, and Nose Tip in the first frame as the alignment reference points.

[0064] The selected key points are matched with the corresponding points on the standard template face, and the affine transformation matrix is ​​calculated.

[0065] Using the calculated affine transformation matrix, a non-rigid transformation is performed on each frame of the image. This transformation will make the key points in the image consistent with the corresponding points of the standard template face, thereby achieving face alignment.

[0066] Specifically, for face cropping, it includes:

[0067] For the first face frame of the micro-expression video, the facial key point detection algorithm is used to determine the key point coordinates. The set of key points can be expressed as Landmarks = [(x0, y0), (x1, y1), ..., (x 67 ,y 67 )], where (x i ,y i ) represents the two-dimensional coordinates of the i-th facial key point.

[0068] Determine a rectangular area by two coordinate points. l and t Determine the upper left coordinate. Therefore, the upper left coordinate A(x l ,y t ). According to x r and b Determine the lower right coordinate B(x r ,y b ). Among them, x l =min(x0,x1,...,x 67 ),y t =min(y0,y1,...,y 67 ),x r=max(x0,x1,...,x 67 ),y b =max(y0, y1, ..., y 67 ).

[0069] Each frame of the entire micro-expression video is cropped by a rectangular frame determined by coordinate points A and B.

[0070] Specifically, the choice of scale is related to the dual-branch network structure and the face resolution in the data set. In this embodiment, the aligned and cropped faces are scaled to 224×224, that is, H=W=224, which meets the network input requirements and can provide more detailed information.

[0071] In some optional implementations, specifically, performing time normalization processing on the first video includes:

[0072] Setting the normalization value of the first video to a preset value;

[0073] The first video includes M frames of micro-expression images, and it is determined whether M is equal to a preset value;

[0074] If M is equal to the preset value, no processing is required;

[0075] If M is not equal to the preset value, a 3D linear interpolation sampling operation is performed on the first video so that the second video has a micro-expression video of a frame where the preset value is located.

[0076] In this embodiment, the preset value is T+1, where T is 8. Of course, it is not limited to this. The value of the preset value can be adjusted according to actual needs, and this embodiment does not impose specific restrictions on this. The micro-expression video after spatial normalization and temporal normalization has unified spatial and temporal features, which facilitates the dual-branch network to extract and analyze the features of micro-expressions.

[0077] In this embodiment, refer to Figure 3 , Figure 3 This is a schematic diagram of a dual-branch network structure, where the two branch inputs, namely fast motion and slow motion, are represented as follows:

[0078] There are 9 frames in the second micro-expression video, represented as {F1, F2, F3, ..., F9}, where F i is the i-th frame in the video sequence, and the shape of each frame tensor is 224×224×3. onset Indicates the starting frame of the micro-expression motion in the sequence, F apex Vertex frames representing micro-expression motion in a sequence.

[0079] Fast motion is defined as the pixel difference between the vertex frame and the starting frame, i.e., M fast =F apex -Fonset , whose tensor shape is 224×224×3.

[0080] Slow motion is defined as the pixel difference between two adjacent frames in the entire sequence, i.e., M slow ={F2-F1, F3-F2, F4-F3, ..., F9-F8}, whose tensor shape is 8×224×224×3.

[0081] In this embodiment, refer to Figure 4 This module is based on the hierarchical visual attention module and adds a tag attention module at each stage to achieve cumulative attention. The following takes the slow motion branch as an example:

[0082] The extracted slow motion is divided into multiple spatiotemporal blocks. The size of the slow motion is 8×224×224×3, and the size of each spatiotemporal block is 8×4×4×3, then it can be divided into Each spatiotemporal block can be flattened into a 384-dimensional feature vector (also called label).

[0083] Feature Projection. Apply a linear embedding layer to project each labeled feature into Dimension, in this embodiment, the hyperparameter C is set to 512.

[0084] Output feature vector. After the three stages of spatiotemporal feature learning of the dual-branch network, the output dimension is N×C feature vector, where N is the number of spatiotemporal tags.

[0085] Add cumulative attention. Add a tag attention module at each stage, add an attention weight to each tag, and achieve weighted tags.

[0086] Specifically, the implementation details of the tag attention are as follows Figure 5 As shown, the marked attention weight TK atten The initial calculation is as follows:

[0087] TK atten =Sig(FC([TK max ;TK avg ])) (1)

[0088] Among them, TK is the input tag, TK max Indicates the maximum value of TK in the channel dimension, TK avg represents the average value of TK in the channel dimension, TK last_atten Represents the attention weight of the previous stage, MaxPool represents the maximum pooling layer, FC represents the matrix multiplication of the fully connected layer, Sig represents the sigmoid function, and the symbol “;” represents the concatenation operation in the channel dimension.

[0089] Then, combined with the attention weight of the previous stage, the final attention weight TK of this stage is obtained atten .

[0090] TK atten =TK atten ⊙DownSampling(TK last_atten ) (2)

[0091] Since the hierarchical visual attention module performs 2× downsampling in space at each stage, the present invention also needs to perform 2× downsampling on TK. last_atten Downsample 2× spatially.

[0092] Finally, the input token and the attention weight TK of this stage atten Perform element-level multiplication to obtain the output label TK' at this stage:

[0093] TK'=TK⊙TK atten (3)

[0094] Here, the symbol “⊙” represents an element-wise multiplication operation.

[0095] In some optional embodiments, such as Figure 4 As shown in Figure 2, richer information is extracted by fusing fast motion features and slow motion features.

[0096] Since the feature dimensions of fast motion and slow motion are different, namely 224×224×3 and 8×224×224×3, in this embodiment, the size of each spatial block is 4×4×3, and the size of each spatiotemporal block is 8×4×4×3, then after three stages of learning of the dual-branch network, the number of spatial markers generated by fast motion is finally N fast =224 / 16×224 / 16, the number of spatiotemporal markers generated by slow motion N slow =N= . Figure 6 As shown, the present invention performs average pooling on the spatiotemporal markers of the same spatial position in the time dimension, ensures that the number of spatiotemporal markers is equal to the number of spatial markers, and selects the feature addition method for fusion. The specific steps of the average pooling operation of the spatiotemporal markers in the time dimension are:

[0097] The spatiotemporal markers at the same position in space perform feature accumulation operations to obtain the accumulated feature vector F i，sum , where i is the spatial position index of the spatiotemporal marker and j is the number of spatiotemporal markers at that position:

[0098]

[0099] The accumulated feature vector F i，sumDivided by the number of spatiotemporal markers in the time dimension 8, we get the average feature vector F of the i-th spatiotemporal marker in space i，avg :

[0100]

[0101] For the slow motion spatiotemporal marker and the fast motion output marker at the same position in space, the two motion features are fused by feature addition.

[0102] After feature fusion, a micro-expression video outputs a tensor of size N×C through a dual-branch network, where N is the output of the output tag and C is the feature dimension size of each tag.

[0103] In this embodiment, if Figure 3 As shown in Figure 1, the output tags of the dual-branch network are used as the input of the classifier module to achieve micro-expression recognition. Specifically,

[0104] Flatten the tensor output by the two-branch network in the previous stage into a vector of length N×C;

[0105] Build the classifier module:

[0106] The classifier consists of a fully connected layer, and the model is optimized by the cross entropy loss function.

[0107] The input dimension of the fully connected layer is C in =N×C, output dimension C out Determined by the number of categories. In this example, the input dimension is: C in =196×512.

[0108] For a classification problem with 5 classes, the formula for weighted cross entropy loss is:

[0109]

[0110] For a classification problem with 3 classes, the formula for weighted cross entropy loss is:

[0111]

[0112] Where y = [y1, y2, ..., y K ] is the model’s predicted probability for each category; t = [t1, t2, ..., t K ] is the one-hot encoding vector of the actual label; W = [w1, w2, ..., w K ] is the weight vector of the category.

[0113] In terms of evaluation indicators, because the current micro-expression database has a serious sample category imbalance problem, it is necessary to introduce the unweighted average recall rate (UAR) and the unweighted F1-score (UF1) to comprehensively evaluate the algorithm performance.

[0114]

[0115]

[0116] Where C is the number of micro-expression categories, c≤C, N c TP represents the total number of samples whose true micro-expression category is the cth category, and N represents the total number of all samples. c Indicates the correctly classified positive examples (True Positive, that is, samples whose real label and predicted label category are both class c), FP c Indicates the misclassified positive examples (False Positive, i.e., samples whose true label category is the cth category and whose predicted label category is the rest of the categories), FN c Represents a negative example that is misclassified (False Negative, that is, the true label category is not the c-th category, and the predicted expression is the c-th category).

[0117] The present invention conducts a three-category (surprise, positive, negative) experiment and a five-category (surprise, happiness, depression, disgust, and others) experiment on the public CASME II micro-expression dataset, and obtains better results of UF1 of 0.94, UAR of 0.95 and UF1 of 0.84, UAR of 0.86 respectively.

[0118] In summary, the present invention provides a micro-expression recognition method based on a fast-slow motion dual-branch network. By fusing the fast-motion and slow-motion features of micro-expressions, efficient and simplified micro-expression recognition is achieved, and the effectiveness of the method provided by the present invention is verified on a micro-expression dataset.

Claims

1. A micro-expression recognition method based on a fast and slow motion dual-branch network, characterized in that: The following steps are involved: Step 1: Data preprocessing: spatial and temporal normalization of micro-expression video clips to ensure that the input micro-expression sequence frame height, width and number are uniform; Step 2: Extract fast and slow motions. Extract fast and slow motions from a given micro-expression video clip. Fast motion is defined as the pixel difference between the vertex frame and the starting frame, and slow motion is defined as the pixel difference between two adjacent frames of the entire micro-expression sequence. Step 3: motion feature extraction: design a fast and slow motion dual-branch network, extract motion features in two branches respectively, and obtain fast motion features and slow motion features; Step 4: feature fusion, which combines fast motion features with slow motion features by adding features to obtain fused features; Step 5: Feature classification: Use the classifier module to classify the fused features to achieve micro-expression recognition.

2. The micro-expression recognition method based on the fast and slow motion dual-branch network according to claim 1 is characterized in that: The fast-slow motion dual-branch network includes a fast motion branch and a slow motion branch, and the two branches respectively include a sequentially connected spatiotemporal block module, a first stage module, a second stage module, and a third stage module; The spatiotemporal block module divides the micro-expression video preprocessed in step 1 into G spatiotemporal blocks; The first stage module includes a linear projection module, two hierarchical visual attention modules and a first tag attention module connected in sequence; the spatiotemporal block is input into the linear projection module to form a spatiotemporal tag; The second stage module includes a first tag merging module, two hierarchical visual attention modules and a second tag attention module connected in sequence; The first tag merging module is connected to the first tag attention module; the second tag attention module is connected to the first tag attention module; The third stage module includes a second tag merging module, six hierarchical visual attention modules and a third tag attention module connected in sequence; The second tag merging module is connected to the second tag attention module; the third tag attention module is connected to the second tag attention module.

3. The micro-expression recognition method based on the fast and slow motion dual-branch network according to claim 2 is characterized in that: In the slow motion branch, the extracted slow motion is first divided into multiple spatiotemporal blocks, each of which is flattened into spatiotemporal markers with the same dimension; the linear projection module projects the features of each spatiotemporal marker onto dimension, after being input into two hierarchical visual attention modules, it is output to the first tag attention module; C is the feature size of the final output mark of the slow motion branch; The outputs of the first tag attention module are respectively input into the first tag merging module and the second tag attention module; The output features of the first tag merging module are input into two hierarchical visual attention modules and then into the second tag attention module; The features output by the second tag attention module are respectively input into the second tag merging module and the third tag attention module; The output features of the second tag merging module are input into six hierarchical visual attention modules and then into the third tag attention module; The third marking attention module outputs slow motion features; The tag merging module at each stage performs 2x spatial downsampling, so that the number of spatiotemporal blocks at each stage becomes 1 / 4 of the original, and the dimension becomes 2x the original; after the three-stage module of the fast-slow motion dual-branch network, a slow motion feature vector with a dimension size of N×C is output, where N is the number of spatiotemporal tags, T is the time length, H is the video height, and W is the video width.

4. The micro-expression recognition method based on the fast and slow motion dual-branch network according to claim 2 or 3, characterized in that: The tag attention module adds an attention weight to each spatiotemporal tag of its input to achieve weighted spatiotemporal tags. Let TK be the input spatiotemporal tag, TK max Indicates the maximum value of TK in the channel dimension, TK avg represents the average value of TK in the channel dimension, TK last_atten represents the marked attention weight of the previous stage, MaxPool represents the maximum pooling layer, FC represents the matrix multiplication of the fully connected layer, Sig represents the sigmoid function, and the attention weight TK of the spatiotemporal mark atten The initial calculation is as follows: TK atten =Sig(FC([TK max ;TK avg ])) (1) The symbol ";" indicates the concatenation operation in the channel dimension; Combined with the marked attention weights of the previous stage, the final attention weight TK of this stage is obtained atten ; <h2 style=";text-align:left;direction:ltr">TK<h2 style=";text-align:left;direction:ltr"> atten <h2 style=";text-align:left;direction:ltr"> =TK<h2 style=";text-align:left;direction:ltr"> atten <h2 style=";text-align:left;direction:ltr"> ⊙DownSampling(TK<h2 style=";text-align:left;direction:ltr"> last_atten <h2 style=";text-align:left;direction:ltr"> ) (2) Among them, DownSampling is a downsampling operation. last_atten Downsample 2× spatially; Finally, the input token and the attention weight TK of this stage atten Perform element-level multiplication to obtain the weighted label TK' output at this stage: TK’=TK⊙TK atten (3) Here, the symbol "⊙" represents element-wise multiplication operation.

5. The micro-expression recognition method based on the fast and slow motion dual-branch network according to claim 4 is characterized in that: The classifier module consists of a fully connected layer and a Softmax layer, and optimizes the fast and slow motion dual-branch network through a weighted cross entropy loss function; The input dimension of the fully connected layer is C in =C×N, output dimension C out Determined by the number of categories; For a classification problem with K categories, the formula for weighted cross entropy loss is as follows: Where y = [y1, y2, ..., y K ] is the model’s predicted probability for each category; t = [t1, t2, ..., t K ] is the one-hot encoded vector of the actual label; w = [w1, w2, ..., w K ] is the weight vector of the category.

6. The micro-expression recognition method based on the fast and slow motion dual-branch network according to claim 4 is characterized in that: The spatial normalization includes face alignment, face cropping and scale unification; the temporal normalization unifies the number of frames of micro-expression video samples by linear interpolation; The specific steps of spatial normalization are as follows: Detect the facial key points of the first frame of the face in the micro-expression video sequence, calculate the affine transformation matrix through the Left Outer Eye, Right Outer Eye, Nose Tip key points and the corresponding key points of the standard template face, and apply the affine transformation matrix to each frame of the micro-expression video in the sequence; Determine the cropping boundary based on the coordinates of key points on the chin, eyebrows, and eyes to crop the face; Unify the frame height and frame width to H and W respectively; Time normalization is to linearly interpolate the three key frames of the micro-expression sample to the T+1 frame; the key frames are divided into a start frame, a vertex frame, and an end frame.

7. The micro-expression recognition method based on the fast and slow motion dual-branch network according to claim 4 is characterized in that: In step 2, the micro-expression fast motion and slow motion are extracted, and it is assumed that there are T+1 frames in the micro-expression sequence, which are represented by {F1, F2, F3, ..., F T+1 }, where F i is the i-th frame in the sequence, and the shape of each frame tensor is expressed as H×W×3; F onset Indicates the start frame of the sequence, F apex Represents a sequence vertex frame; Fast motion is defined as the pixel difference between the vertex frame and the starting frame, that is, Fast_Motion = F apex -F onset , whose tensor shape is H×W×3; slow motion is defined as the pixel difference between two adjacent frames in the entire sequence, that is, Slow_Motion={F2-F1,F3-F2,F4-F3,…,F T+1 -F T }, whose tensor shape is T×H×W×3.

8. The micro-expression recognition method based on the fast and slow motion dual-branch network according to claim 4 is characterized in that: In the step 4, fast motion features and slow motion features are fused; The feature dimensions of fast motion and slow motion are different, which are H×W×3 and T×H×W×3 respectively. The size of each spatial block is set to 4×4×3, and the size of each spatiotemporal block is set to 8×4×4×3. After three stages of learning of the dual-branch network, the number of spatial markers generated by fast motion is finally The number of spatiotemporal markers generated by slow motion The spatiotemporal markers at the same spatial position are averagely pooled in the time dimension to ensure that the number of spatiotemporal markers is equal to the number of spatial markers, and the feature addition method is selected for fusion.

Images