A Micro-expression Recognition Method Based on Fast and Slow Motion Dual-Branch Network

Abstract

This invention belongs to the field of computer vision technology and discloses a micro-expression recognition method based on a fast and slow motion dual-branch network. The method involves spatially and temporally normalizing micro-expression video clips to extract fast and slow motion; designing a fast and slow motion dual-branch network to obtain fast and slow motion features; fusing the fast and slow motion features through feature addition to obtain fused features; and using a classifier module to classify the fused features to achieve micro-expression recognition. This invention improves performance by having the fast motion branch focus on overall motion, capturing long-term, holistic motion information, while the slow motion branch focuses on detailed motion, capturing short-term, detailed motion information. A spatiotemporal modeling mechanism with cumulative attention is designed, adding a labeled attention module at each stage to achieve cumulative attention. Furthermore, this invention represents motion using pixel differences, simplifying the implementation process and reducing computational complexity.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a micro-expression recognition method based on a fast and slow motion dual-branch network. Background Technology

[0002] Facial expressions are divided into two categories: macro-expressions and micro-expressions. Their main difference lies in duration and intensity. Macro-expressions are the most common and well-known expressions in daily life. When a macro-expression occurs, people can generally observe and distinguish its corresponding emotional type. This is because macro-expressions cover a large area of ​​the face, such as the mouth, eyes, cheeks, and chin, and last for a long time, usually 0.5 to 4 seconds, making them easily recognizable and interpretable. Micro-expressions are rapid, spontaneous, and low-intensity facial expressions, often appearing when people are trying to hide their true feelings, especially under pressure, tension, or anxiety. Micro-expressions are unconscious, localized movements covering a small area of ​​the face, mainly concentrated in the mouth and eye areas. They are short-lived and low-intensity, typically lasting between 0.065 and 0.5 seconds, making them difficult to detect with the naked eye. The transience and subtlety of micro-expressions make them difficult to fake, but they can reflect an individual's true emotional state or inner conflict, possessing higher credibility in emotional expression and applicable in fields such as psychology, criminal investigation, and education.

[0003] Micro-expression recognition is a facial recognition technology that processes micro-expression video sequences to achieve three- or five-category classification. Micro-expression recognition can be divided into traditional methods and deep learning methods. Traditional methods have advantages in interpretability but limitations in handling complex micro-expression features and improving recognition accuracy. Deep learning methods, through automatic feature extraction and complex pattern recognition, outperform traditional methods in terms of recognition accuracy and robustness.

[0004] In some studies, such as “Gan YS,Liong ST,Yau WC,et al.OFF-ApexNet on micro-expression recognition system[J].Signal Processing:Image Communication,2019,74:129-139.”, “Liong ST,Gan YS,See J,et al.Shallow triple stream three-dimensional cnn(ststnet)for micro-expression recognition[C] / / 2019 14th IEEE international conference on automatic face & gesture recognition(FG 2019).IEEE,2019:1-5.”, and “Li H,Sui M,Zhu Z,et al.MMNet:Muscle Motion-guided Network for Micro-expression Recognition[J].In Proc.31st Int'l Joint Conf.ArtificialIntelligence(IJCAI)”, motion features are extracted directly from the start and vertex frames of micro-expressions, demonstrating the simplicity and effectiveness of this method. However, this method inevitably ignores the details of motion information from the vertex to the end frame.

[0005] The methods “Zhi R, Xu H, Wan M, et al. Combining 3D convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition[J]. IEICE TRANSACTIONS on Information and Systems, 2019, 102(5): 1054-1064.” and “Hong J, Lee C, Jung H. Late fusion-based video transformer for facial micro-expression recognition[J]. Applied Sciences, 2022, 12(3): 1169.” extract motion information from adjacent frames of the entire sequence, which can generate richer motion information. However, these methods do not make full use of the annotated vertex frames, which introduces redundant action representations. In addition, the above methods often rely on hand-designed optical flow features. The computational complexity of optical flow is very high. It is necessary to calculate the pixel changes between every two frames and perform matching and interpolation to determine the direction and speed of optical flow. This complexity makes it difficult for optical flow-based methods to meet the requirements of real-time performance.

[0006] Therefore, this invention proposes a dual-branch micro-expression classification network that integrates fast and slow micro-expression movements, uses pixel differences to represent micro-expression movements, and integrates overall micro-expression movement and detail movement features to solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to propose a micro-expression recognition method based on a fast-slow motion dual-branch network. This method integrates fast and slow motion, with the fast motion branch focusing on overall motion, capturing long-term, holistic motion information and avoiding interference from redundant local motion information. The slow motion branch focuses on detailed motion, capturing short-term, detailed motion information to compensate for dynamic information lost in the overall motion, thereby improving performance. Furthermore, a spatiotemporal modeling mechanism that adds cumulative attention is designed to guide the model to give more attention to regions of interest.

[0008] To achieve the above tasks, the present invention adopts the following technical solution: a micro-expression recognition method based on a fast and slow motion dual-branch network, comprising the following steps:

[0009] Step 1: Data preprocessing. Spatial and temporal normalization is performed on the micro-expression video clips to ensure that the height, width, and number of the input micro-expression sequence frames are uniform.

[0010] Step 2, Fast and Slow Motion Extraction: Extract fast and slow motion from the given micro-expression video clip. Fast motion is defined as the pixel difference between the vertex frame and the starting frame, and slow motion is defined as the pixel difference between two adjacent frames in the entire micro-expression sequence.

[0011] Step 3: Motion feature extraction. Design a dual-branch network for fast and slow motion, and extract motion features in the two branches to obtain fast motion features and slow motion features.

[0012] Step four, feature fusion: fast motion features and slow motion features are fused by adding features together to obtain fused features;

[0013] Step 5: Feature classification. A classifier module is used to classify the fused features to achieve micro-expression recognition.

[0014] The fast and slow motion dual-branch network includes a fast motion branch and a slow motion branch, and the two branches respectively include a spatiotemporal block module, a first stage module, a second stage module, and a third stage module connected in sequence.

[0015] The spatiotemporal block module divides the micro-expression video after preprocessing in step one into G spatiotemporal blocks;

[0016] The first stage module includes a linear projection module, two hierarchical visual attention modules, and a first marker attention module connected in sequence; the spatiotemporal block is input to the linear projection module to form a spatiotemporal marker;

[0017] The second stage module includes a first label merging module, two hierarchical visual attention modules, and a second label attention module connected in sequence; the first label merging module is connected to the first label attention module; the second label attention module is connected to the first label attention module.

[0018] The third-stage module includes a second label merging module, six hierarchical visual attention modules, and a third label attention module connected in sequence; the second label merging module is connected to the second label attention module; and the third label attention module is connected to the second label attention module.

[0019] In the slow motion branch, the extracted slow motion is first divided into multiple spatiotemporal blocks, each of which is a feature vector of equal dimension, also known as a spatiotemporal marker; the linear projection module projects the features of each spatiotemporal marker onto... The dimension, after being input into two hierarchical visual attention modules, is output to the first label attention module; C is the feature size of the final output label of the slow motion branch; the output of the first label attention module is input into the first label merging module and the second label attention module respectively; the output features of the first label merging module are input into two hierarchical visual attention modules and then into the second label attention module; the output features of the second label attention module are input into the second label merging module and the third label attention module respectively; the output features of the second label merging module are input into six hierarchical visual attention modules and then into the third label attention module; the third label attention module outputs slow motion features; the label merging module in each stage performs 2x spatial downsampling, so that the number of spatiotemporal blocks in each stage becomes 1 / 4 of the original, and the dimension becomes twice the original; after passing through the three stages of the fast and slow motion dual-branch network, the output slow motion feature vector with a dimension of N×C is generated, where N is the number of spatiotemporal labels. T represents the duration, H represents the video height, and W represents the video width.

[0020] The labeling attention module adds an attention weight to each spatiotemporal label it inputs, thereby weighting the spatiotemporal labels; let TK be the input spatiotemporal label, TK max TK represents the maximum value of TK in the channel dimension. avg TK represents the average value of TK across the channel dimension. last_atten The attention weights for the labels in the previous stage are represented by: MaxPool (max pooling layer), FC (fully connected layer) (matrix multiplication), Sig (sigmoid function), and TK (attention weights for spatiotemporal labels). atten Preliminary calculations are as follows:

[0021] TK atten =Sig(FC([TK max TK avg ])) (1)

[0022] The symbol “;” indicates a concatenation operation in the channel dimension.

[0023] Combining the labeled attention weights from the previous stage, we obtain the final attention weights TK for this stage. atten ;

[0024] TK atten =TK atten ⊙DownSampling(TK last_atten (2)

[0025] DownSampling is the downsampling operation, applied to TK. last_atten Perform 2× downsampling in space;

[0026] Finally, the input labels and the attention weights TK for this stage. atten Perform element-wise multiplication to obtain the weighted label TK' of this stage's output:

[0027] TK'=TK⊙TK atten (3)

[0028] The symbol “⊙” represents element-wise multiplication.

[0029] The classifier module consists of a fully connected layer and a Softmax layer, and the fast and slow motion dual-branch network is optimized by a weighted cross-entropy loss function.

[0030] The input dimension C of the fully connected layer in =C×N, output dimension C out Determined by the number of categories;

[0031] For a classification problem with K categories, the formula for weighted cross-entropy loss is as follows:

[0032]

[0033] Where y = [y1, y2, ..., y K ] represents the model's predicted probability for each category; t = [t1, t2, ..., t3]. K [w1, w2, ..., W] is the one-hot encoded vector of the actual label; w = [w1, w2, ..., W] K ] is the weight vector of the category.

[0034] The spatial normalization includes face alignment, face cropping, and scale unification; the temporal normalization unifies the frame number of micro-expression video samples through linear interpolation.

[0035] The specific steps of spatial normalization are as follows:

[0036] The facial key points of the first frame of the micro-expression video sequence are detected. The affine transformation matrix is ​​calculated by comparing the key points of Left Outer Eye, Right Outer Eye, and Nose Tip with the key points of the standard template face. The affine transformation matrix is ​​then applied to each frame of the micro-expression video sequence.

[0037] Determine the cropping boundaries based on the coordinates of key points on the chin, eyebrows, and eyes, and then perform face cropping.

[0038] The frame height and frame width are standardized as H and W, respectively.

[0039] Temporal normalization involves linearly interpolating three key frames of a micro-expression sample to frame T+1; these key frames are categorized as a start frame, a vertex frame, and an end frame.

[0040] In step two, the fast and slow movements of micro-expressions are extracted. The micro-expression sequence is set to have T+1 frames, represented as {F1, F2, F3, ..., F...}. T+1}, where F i It is the i-th frame in the sequence, and the tensor shape of each frame is represented as H×W×3; F onset Indicates the start frame of the sequence, F apex Represents a sequence of vertex frames;

[0041] Fast motion is defined as the pixel difference between the vertex frame and the starting frame, i.e., Fast_Motion = F. apex -F onset Its tensor shape is H×W×3; slow motion is defined as the pixel difference between two adjacent frames in the entire sequence, i.e., Slow_Motion={F2-F1, F3-F2, F4-F3, ..., F T+1 -F T Its tensor shape is T×H×W×3.

[0042] In step four, the fast motion features and slow motion features are fused. The fast and slow motion features have different dimensions, H×W×3 and T×H×W×3 respectively. Each spatial block size is set to 4×4×3, and each spatiotemporal block size is set to 8×4×4×3. After three stages of learning by the dual-branch network, the final number of spatial labels generated by the fast motion is... Number of spatiotemporal markers generated by slow motion Spatiotemporal labels at the same spatial location are averaged in the time dimension to ensure that the number of spatiotemporal labels equals the number of spatial labels, and features are fused by adding features together.

[0043] The micro-expression recognition method based on a fast and slow motion dual-branch network proposed in this invention integrates fast and slow motion, and designs a module to add cumulative attention, giving more attention to the region, thereby improving the performance of micro-expression recognition.

[0044] The beneficial effects of this invention are as follows: This invention proposes a micro-expression recognition method based on a fast and slow motion dual-branch network. The fast motion branch focuses on overall motion, capturing long-term, holistic motion information; the slow motion branch focuses on detailed motion, capturing short-term, detailed motion information, thus improving performance. This invention also designs a spatiotemporal modeling mechanism that incorporates cumulative attention, adding a labeled attention module at each stage to achieve cumulative attention. Furthermore, this invention represents motion using pixel differences, simplifying the implementation process and reducing computational complexity. Attached Figure Description

[0045] Figure 1This is a flowchart of a micro-expression recognition method based on a fast and slow motion dual-branch network;

[0046] Figure 2 Flowchart of face frame space preprocessing;

[0047] Figure 3 Flowchart of a two-branch network for fast and slow motion;

[0048] Figure 4 This is a schematic diagram of the slow motion branch;

[0049] Figure 5 A schematic diagram of the labeling attention module in the third stage module;

[0050] Figure 6 This is a schematic diagram of the average pooling of the spatiotemporal markers for fast-motion events in the time dimension. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] The present invention will be further described below with reference to the accompanying drawings and examples:

[0053] Combination Figure 1 , Figure 1 This is a flowchart illustrating a specific embodiment of the micro-expression recognition method based on a fast and slow motion dual-branch network provided by the present invention, including:

[0054] S1: Spatial normalization is performed on the micro-expression video to obtain the first video;

[0055] S2: Perform time normalization on the first video to obtain the second video;

[0056] S3: Construct a dual-branch network for fast and slow motion;

[0057] S4: Input the second video into the fast and slow motion dual-branch network and output classification labels;

[0058] S5: Use a classifier module to identify classification tags.

[0059] In some alternative embodiments, refer to Figure 2 , Figure 2This is a flowchart illustrating a spatial normalization process for micro-expression videos, including:

[0060] The micro-expression video consists of M frames of images. After aligning the faces in each frame, the micro-expression images are obtained by natural face cropping and then adjusted to a preset size.

[0061] It is understandable that micro-expression videos can be viewed as an image sequence, and face alignment uses non-rigid transformation face alignment, including:

[0062] The facial key point coordinates are located for each frame of the micro-expression video using a facial key point detection algorithm.

[0063] Select the keypoint coordinates corresponding to Left Outer Eye, Right Outer Eye, and Nose Tip in the first frame as alignment reference points.

[0064] The selected key points are matched with the corresponding points on the standard template face, and the affine transformation matrix is ​​calculated.

[0065] Using the calculated affine transformation matrix, a non-rigid transformation is performed on each frame of the image. This transformation will make the key points in the image consistent with the corresponding points of the standard template face, thereby achieving face alignment.

[0066] Specifically, for face cropping, this includes:

[0067] For the first frame of a micro-expression video, a facial landmark detection algorithm is used to determine the coordinates of key points. The set of key points can be represented as Landmarks=[(x0,y0),(x1,y1),...,(x 67 ,y 67 )], where (x i ,y i ) represents the two-dimensional coordinates of the i-th facial key point.

[0068] A rectangular region is defined by two coordinate points. Based on x... l and y t Determine the top-left coordinate. Therefore, the obtained top-left coordinate is A(x). l ,y t According to x) r and y b Determine the lower right coordinate B(x) r ,y b ), where x l =min(x0,x1,...,x) 67 ),y t =min(y0,y1,...,y 67 ),x r=max(x0,x1,...,x 67 ),y b =max(y0, y1, ..., y 67 ).

[0069] Each frame of the entire micro-expression video is cropped using a rectangle defined by coordinate points A and B.

[0070] Specifically, the choice of scale is related to the dual-branch network structure and the face resolution in the dataset. In this embodiment, the aligned and cropped face is scaled to 224×224, that is, H=W=224, which satisfies the network input requirements while providing more detailed information.

[0071] In some optional implementations, specifically, time normalization is performed on the first video, including:

[0072] Set the normalization value of the first video to the preset value;

[0073] The first video contains M frames of micro-expression images; determine whether M is equal to a preset value.

[0074] If M equals the preset value, no processing is required;

[0075] If M is not equal to the preset value, perform 3D linear interpolation sampling on the first video to make the second video have the micro-expression video of the frame where the preset value is located.

[0076] In this embodiment, the preset value is T+1, where T is 8. Of course, it is not limited to this; the value of the preset value can be adjusted according to actual needs. This embodiment does not impose specific restrictions on this. Micro-expression videos that have undergone spatial and temporal normalization have uniform spatial and temporal characteristics, which facilitates the dual-branch network to extract and analyze micro-expression features.

[0077] In this embodiment, refer to Figure 3 , Figure 3 The diagram below illustrates a two-branch network structure, where the two branch inputs, namely fast motion and slow motion, are represented as follows:

[0078] The second video of micro-expressions contains 9 frames, represented as {F1, F2, F3, ..., F9}, where F... i It is the i-th frame in the video sequence, and the tensor shape of each frame is represented as 224×224×3. Here, F... onset F represents the starting frame of micro-expression movements in the sequence. apex Represents the vertex frame of micro-expression motion in the sequence.

[0079] Fast motion is defined as the pixel difference between the vertex frame and the start frame, i.e., M. fast =F apex -Fonset Its tensor shape is 224×224×3.

[0080] Slow motion is defined as the pixel difference between two adjacent frames in the entire sequence, i.e., M. slow = {F2-F1, F3-F2, F4-F3, ..., F9-F8}, whose tensor shape is 8×224×224×3.

[0081] In this embodiment, refer to Figure 4 This module is based on a hierarchical visual attention module, adding a labeled attention module at each stage to achieve cumulative attention. The following explanation uses the slow-motion branch as an example:

[0082] The extracted slow motion is divided into multiple spatiotemporal blocks. The slow motion size is 8×224×224×3, and each spatiotemporal block is 8×4×4×3. Therefore, the blocks can be divided into... Each spatiotemporal block can be flattened into a 384-dimensional feature vector (also called a label).

[0083] Feature projection. A linear embedding layer is applied to project the features of each label onto... In this embodiment, the hyperparameter C is set to 512.

[0084] Output feature vector. After three stages of spatiotemporal feature learning in this dual-branch network, the output feature vector has a dimension of N×C, where N is the number of spatiotemporal labels.

[0085] Add cumulative attention. Add a label attention module at each stage, adding an attention weight to each label to achieve weighted labeling.

[0086] Specifically, the implementation details of label attention are as follows: Figure 5 As shown, the attention weights TK of the labels atten Preliminary calculations are as follows:

[0087] TK atten =Sig(FC([TK max TK avg ])) (1)

[0088] Where TK is the input marker, TK max TK represents the maximum value of TK in the channel dimension. avg TK represents the average value of TK across the channel dimension. last_atten The attention weights of the previous stage are represented by , MaxPool represents the max pooling layer, FC represents the matrix multiplication of the fully connected layer, Sig represents the sigmoid function, and the symbol ";" represents the concatenation operation in the channel dimension.

[0089] Then, combining the attention weights from the previous stage, we obtain the final attention weight TK for this stage. atten .

[0090] TK atten =TK atten ⊙DownSampling(TK last_atten (2)

[0091] Since the hierarchical visual attention module performs 2× downsampling in space at each stage, this invention also needs to perform TK... last_atten Spatial 2× downsampling.

[0092] Finally, the input labels and the attention weights TK for this stage. atten Perform element-wise multiplication to obtain the output marker TK' for this stage:

[0093] TK'=TK⊙TK atten (3)

[0094] The symbol “⊙” represents element-wise multiplication.

[0095] In some alternative embodiments, such as Figure 4 As shown, richer information can be extracted by fusing fast motion features and slow motion features.

[0096] Since the feature dimensions of fast motion and slow motion are different, namely 224×224×3 and 8×224×224×3 respectively, in this embodiment, the size of each spatial block is 4×4×3 and the size of each spatiotemporal block is 8×4×4×3. Therefore, after three stages of learning by the dual-branch network, the final number of spatial labels generated by fast motion is N. fast = 224 / 16 × 224 / 16, the number of spacetime markers N generated by slow motion slow =N= One. For example. Figure 6 As shown, this invention performs average pooling on spatiotemporal markers at the same spatial location along the time dimension to ensure that the number of spatiotemporal markers equals the number of spatial markers, and selects a feature addition method for fusion. The specific steps of the average pooling operation on the time dimension of spatiotemporal markers are as follows:

[0097] The spatiotemporal markers at the same location in space are used to perform feature accumulation operations to obtain the accumulated feature vector F. i，sum Where i is the spatial location index of the spatiotemporal marker, and j is the number of spatiotemporal markers at that location:

[0098]

[0099] Accumulated feature vector F i，sumDividing by the number of spatiotemporal markers in the time dimension (8), we obtain the average eigenvector F of the i-th spatiotemporal marker in space. i，avg :

[0100]

[0101] For slow-motion spatiotemporal markers and fast-motion output markers at the same location in space, the two motion features are fused by feature addition.

[0102] After feature fusion, a micro-expression video outputs a tensor of size N×C through a two-branch network, where N is the output of the label and C is the feature dimension of each label.

[0103] In this embodiment, as Figure 3 As shown, the output labels of the dual-branch network are used as input to the classifier module to achieve micro-expression recognition. Specifically,

[0104] Flatten the tensor output by the previous two-branch network into a vector of length N×C;

[0105] Module for building classifiers:

[0106] The classifier consists of a fully connected layer and the model is optimized using the cross-entropy loss function.

[0107] The input dimension C of the fully connected layer in = N×C, output dimension C out The number of categories determines the input dimension. In this example, the input dimension is: C in =196×512.

[0108] For a classification problem with 5 categories, the formula for weighted cross-entropy loss is:

[0109]

[0110] For a classification problem with 3 categories, the formula for weighted cross-entropy loss is:

[0111]

[0112] Where y = [y1, y2, ..., y K ] represents the model's predicted probability for each category; t = [t1, t2, ..., t3]. K [w1, w2, ..., w] is the one-hot encoded vector of the actual label; W = [w1, w2, ..., w] K ] is the weight vector of the category.

[0113] In terms of evaluation metrics, since the current micro-expression database has a serious sample class imbalance problem, it is necessary to introduce unweighted average recall (UAR) and unweighted F1-score (UF1) to comprehensively evaluate the algorithm performance.

[0114]

[0115]

[0116] Where C is the number of micro-expression categories, c≤C, N c TP represents the total number of samples whose true micro-expression category is class c, and N represents the total number of all samples. c A positive example is a correctly classified instance (a sample whose true label and predicted label are both in class c). c FN represents a false positive (i.e., a sample whose true label is class c but whose predicted label is any of the other classes). c This represents a false negative (i.e., the true label category is not class c, but the predicted expression is class c).

[0117] This invention conducted three-class (surprise, positive, negative) and five-class (surprise, happiness, depression, disgust, other) experiments on the publicly available CASME II microexpression dataset, achieving better results with UF1 of 0.94 and UAR of 0.95 and UF1 of 0.84 and UAR of 0.86, respectively.

[0118] In summary, this invention provides a micro-expression recognition method based on a fast and slow motion dual-branch network. By fusing the fast and slow motion features of micro-expressions, it achieves efficient and simplified micro-expression recognition. The effectiveness of the method provided by this invention is verified on a micro-expression dataset.

Claims

1. A micro-expression recognition method based on a fast and slow motion dual-branch network, characterized in that, Includes the following steps: Step 1: Data preprocessing. Spatial and temporal normalization is performed on the micro-expression video clips to ensure that the height, width, and number of the input micro-expression sequence frames are uniform. Step 2, Fast and Slow Motion Extraction: Extract fast and slow motion from the given micro-expression video clip. Fast motion is defined as the pixel difference between the vertex frame and the starting frame, and slow motion is defined as the pixel difference between two adjacent frames in the entire micro-expression sequence. Step 3: Motion feature extraction. Design a dual-branch network for fast and slow motion, and extract motion features in the two branches to obtain fast motion features and slow motion features. Step four, feature fusion: fast motion features and slow motion features are fused by adding features together to obtain fused features; Step 5: Feature classification. A classifier module is used to classify the fused features to achieve micro-expression recognition. The fast and slow motion dual-branch network includes a fast motion branch and a slow motion branch, and the two branches respectively include a spatiotemporal block module, a first stage module, a second stage module, and a third stage module connected in sequence. The spatiotemporal block module divides the micro-expression video after preprocessing in step one into G spatiotemporal blocks; The first stage module includes a linear projection module, two hierarchical visual attention modules, and a first marker attention module connected in sequence; the spatiotemporal block is input to the linear projection module to form a spatiotemporal marker; The second stage module includes a first label merging module, two hierarchical visual attention modules, and a second label attention module connected in sequence; The first label merging module is connected to the first label attention module; the second label attention module is connected to the first label attention module. The third-stage module includes a second marker merging module, six hierarchical visual attention modules, and a third marker attention module connected in sequence. The second label merging module is connected to the second label attention module; the third label attention module is connected to the second label attention module.

2. The micro-expression recognition method based on a fast and slow motion dual-branch network according to claim 1, characterized in that, In the slow motion branch, the extracted slow motion is first divided into multiple spatiotemporal blocks, and each spatiotemporal block is flattened into a spatiotemporal marker of equal dimension; the linear projection module projects the features of each spatiotemporal marker onto... The dimension, after being input into two hierarchical visual attention modules, is output to the first labeled attention module; C represents the feature size of the final output label of the slow motion branch; The output of the first label attention module is input to the first label merging module and the second label attention module, respectively. The output features of the first label merging module are input into two hierarchical visual attention modules, and then into the second label attention module; The features output by the second label attention module are respectively input into the second label merging module and the third label attention module; The output features of the second label merging module are input into the six hierarchical visual attention modules, and then into the third label attention module; The third labeling attention module outputs slow motion features; In each stage, the label merging module performs a 2x spatial downsampling, reducing the number of spatiotemporal blocks in each stage to 1 / 4 and the dimension to twice the original. After passing through the three stages of the fast and slow motion dual-branch network, the output dimension is... The slow motion feature vector, where N is the number of spatiotemporal markers. = T represents the duration, H represents the video height, and W represents the video width.

3. The micro-expression recognition method based on a fast and slow motion dual-branch network according to claim 2, characterized in that, The labeling attention module adds an attention weight to each spatiotemporal label it inputs, thereby weighting the spatiotemporal labels; let... It is the input spatiotemporal marker. express The maximum value in the channel dimension. express The average value in the channel dimension. This represents the labeled attention weights from the previous stage. The expression represents a max-pooling layer, FC represents matrix multiplication in a fully connected layer, Sig represents the sigmoid function, and spatiotemporally labeled attention weights. Preliminary calculations are as follows: ； Among them, the symbol " "Indicates a splicing operation at the channel level; Combining the labeled attention weights from the previous stage, we obtain the final attention weights for this stage. ; ； DownSampling is the downsampling operation, which is used for... Perform 2× downsampling in space; Finally, the input labels and attention weights for this stage. Perform element-wise multiplication to obtain the weighted labels of the output at this stage. : ； The symbol "⊙" represents element-wise multiplication.

4. The micro-expression recognition method based on a fast and slow motion dual-branch network according to claim 2, characterized in that, The classifier module consists of a fully connected layer and a Softmax layer, and the fast and slow motion dual-branch network is optimized by a weighted cross-entropy loss function. Input dimensions of fully connected layers Output dimension Determined by the number of categories; For those with For a classification problem with 1 category, the formula for the weighted cross-entropy loss is as follows: ； in, It is the model's predicted probability for each category; It is the one-hot encoded vector of the actual label; It is the weight vector of the category.

5. The micro-expression recognition method based on a fast and slow motion dual-branch network according to claim 3, characterized in that, The spatial normalization includes face alignment, face cropping, and scale unification; the temporal normalization unifies the frame number of micro-expression video samples through linear interpolation. The specific steps of spatial normalization are as follows: Detect the facial key points of the first frame of the face in the micro-expression video sequence, calculate the affine transformation matrix by comparing the key points of Left Outer Eye, Right Outer Eye, and Nose Tip with the corresponding key points of the standard template face, and apply the affine transformation matrix to each frame of the micro-expression video sequence. Determine the cropping boundaries based on the coordinates of key points on the chin, eyebrows, and eyes, and then perform face cropping. The frame height and frame width are standardized as H and W, respectively. Temporal normalization is the linear interpolation of the three keyframes of a micro-expression sample to... The keyframes are divided into start frame, vertex frame, and end frame.

6. The micro-expression recognition method based on a fast and slow motion dual-branch network according to claim 3, characterized in that, In step two, fast and slow movements of micro-expressions are extracted, and the micro-expression sequence is defined as containing... Frame, represented as ,in It is the first in the sequence Frame, the tensor shape of each frame is represented as ; Indicates the start frame of the sequence. Represents a sequence of vertex frames; Fast motion is defined as the pixel difference between the vertex frame and the starting frame, i.e. Its tensor shape is Slow motion is defined as the pixel difference between two adjacent frames in the entire sequence, i.e. Its tensor shape is .

7. The micro-expression recognition method based on a fast and slow motion dual-branch network according to claim 3, characterized in that, In step four, the fast motion features and slow motion features are fused; the feature dimensions of fast motion and slow motion are different, namely... and Set the size of each space block to Each spacetime block is [size missing] After three stages of learning using a dual-branch network, the final number of spatial labels generated by fast motion is... The number of spatiotemporal markers generated by slow motion ; Spatiotemporal labels at the same spatial location are averaged in the time dimension to ensure that the number of spatiotemporal labels equals the number of spatial labels, and features are fused by adding features together.

Images