Micro-expression recognition method based on spatio-temporal features and attention mechanism

Abstract

The application provides a micro-expression recognition method based on space-time features and an attention mechanism, comprising: performing spatial normalization processing on a micro-expression video to obtain a first video; performing time normalization processing on the first video to obtain a second video; constructing a backbone network based on hierarchical visual self-attention and label selection; inputting the second video into the backbone network to output a classification label; and classifying and identifying the classification label by using a classifier module. By constructing the backbone network based on hierarchical visual self-attention and label selection, the weighting operation is continuously performed at each stage, and the network is explicitly required to pay different degrees of attention to different regions of the face; the labels that are important for micro-expression recognition are extracted, and the feature representation of these labels is highlighted, thereby improving the accuracy and robustness of the classification task, and thereby improving the performance of the micro-expression recognition method.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to a micro-expression recognition method based on spatiotemporal features and attention mechanisms. Background Technology

[0002] Facial expressions can generally be divided into macro-expressions and micro-expressions. Macro-expressions are the most common and well-known expressions in daily life. When a macro-expression occurs, people can usually clearly observe and distinguish its corresponding emotional type. This is because macro-expressions last for a relatively long time (0.5 to 2 seconds), cover a large area of ​​facial movement (chin, mouth, nose, cheeks, eyes, eyebrows, etc.), and involve relatively strong muscle movements. In contrast, micro-expressions are brief, spontaneous facial expressions, usually appearing when people are trying to hide their true feelings, especially under high pressure, tension, or anxiety. Micro-expressions have a short duration, generally considered to be no more than 0.5 seconds, with some studies using 0.2 seconds as a threshold. Furthermore, micro-expression movements cover a small area of ​​facial movement, mainly concentrated around the mouth and eyes, and involve small muscle movements. These characteristics make micro-expressions difficult to fake; therefore, micro-expressions are considered to express genuine human emotions, and their emotional expression is more credible than that of macro-expressions.

[0003] Micro-expression recognition is a process of classifying facial expressions (e.g., positive, negative, surprised) by extracting features from facial muscle movements in micro-expression sequences. Traditional micro-expression recognition methods typically begin with detailed processing of the face, such as face detection, face alignment, illumination correction, and region of interest (ROI) cropping. Then, feature extraction is performed, including shape features, texture features (e.g., local binary patterns), and dynamic features (e.g., local binary patterns in 3D space, optical flow), which represent key information about the micro-expression. Finally, a classifier is used for micro-expression recognition; commonly used classifiers include support vector machines, K-nearest neighbors, and decision trees. However, traditional methods have some drawbacks: first, they are weak at capturing subtle changes and dynamic features in micro-expressions, failing to fully utilize deeper information; second, feature selection and extraction often require manual design and adjustment, relying on the knowledge and experience of domain experts, thus limiting effectiveness. In recent years, deep learning-based methods have been widely used in micro-expression recognition. These methods can directly learn more abstract and higher-level feature representations from raw data, eliminating the need for manually designed feature extraction methods and reducing the complexity of manual feature engineering.

[0004] Some deep learning methods, based on single-frame face images, extract and classify spatial features using convolutional and fully connected networks, achieving some success. However, these methods do not fully utilize the temporal motion information of micro-expressions. To fully leverage the temporal and spatial features of micro-expressions, some works based on micro-expression video frame sequences use 3D convolutional networks and long short-term memory networks for extracting spatiotemporal features, achieving better results. However, existing methods based on spatiotemporal feature extraction from micro-expression frame sequences share a common problem: they still rely on manually designed optical flow features. Optical flow computation is highly complex, requiring calculation of pixel changes between every two frames, and matching and interpolation to determine the direction and velocity of the optical flow. This complexity makes it difficult for optical flow-based methods to meet real-time requirements.

[0005] Patent document CN105913038A discloses a video-based dynamic micro-expression recognition method. Its drawback lies in the extensive reliance on manual methods for feature extraction, such as video segmentation, optical flow extraction, and HOG3D (Histogram of Oriented 3D Gradients) feature extraction, all of which depend on expert experience. Patent document CN106570474A discloses a micro-expression recognition method based on a 3D convolutional neural network, and patent document CN113496217A discloses a micro-expression recognition method based on facial motion units and optical flow. These deep learning-based methods assume the acquisition of local geometric information such as optical flow or motion units. However, in practice, limitations in data quality and computational resources may prevent the acquisition of additional local geometric information, and facial motion unit detection itself is a challenging task.

[0006] Therefore, finding a more effective method to extract the spatiotemporal features of micro-expressions is an important direction. This method should reduce reliance on manual methods and be able to automatically learn more abstract and higher-level feature representations from raw data to improve the performance and practicality of micro-expression recognition. This invention provides a micro-expression recognition method based on spatiotemporal features and an attention mechanism. Summary of the Invention

[0007] To address the shortcomings mentioned above, this invention provides a micro-expression recognition method based on spatiotemporal features and attention mechanisms. The invention primarily utilizes spatial normalization processing on micro-expression videos to obtain a first video; temporal normalization processing on the first video to obtain a second video; constructing a backbone network based on hierarchical visual self-attention and label selection; inputting the second video into the backbone network and outputting classification labels; and employing a classifier module to classify and recognize the classification labels, thereby avoiding reliance on manual optical flow features and facial action unit detection tasks.

[0008] The technical means employed in this invention are as follows: This invention provides a micro-expression recognition method based on spatiotemporal features and attention mechanisms, comprising: Spatial normalization was performed on the micro-expression video to obtain the first video; The first video is time-normalized to obtain the second video; Construct a backbone network based on hierarchical visual self-attention and label selection; The second video is input into the backbone network, and classification labels are output. A classifier module is used to classify and identify the classification tags.

[0009] Furthermore, the spatial normalization processing of the micro-expression video includes: The micro-expression video includes M frames of images. Each frame of the image is aligned with a face and then cropped to obtain a micro-expression image. The micro-expression image is then adjusted to a preset size.

[0010] Further, the time normalization processing of the first video includes: The normalization value of the first video is set to a preset value; The first video includes M frames of micro-expression images, and it is determined whether M is equal to the preset value; If M equals the preset value, no processing is required; If M is less than the preset value, perform 3D linear interpolation upsampling on the first video so that the second video has the micro-expression image of the preset value frame; If M is greater than the preset value, the first video is divided into S semantic segments, where S is Z times the preset value, 0 < Z ≤ 1, and Q consecutive frames of micro-expression images are taken from each semantic segment to form the second video, where Q × S is equal to the preset value.

[0011] Furthermore, the construction of the backbone network based on hierarchical visual self-attention and label selection includes: Connect the spacetime block module, the first stage module, the second stage module, the third stage module, and the fourth stage module in sequence; The spatiotemporal block module divides the second video into G spatiotemporal blocks. The first stage module includes a linear transformation layer, a first hierarchical visual attention module, a second hierarchical visual attention module, and a first marker attention module connected in sequence, wherein the linear transformation layer is connected to the spatiotemporal blocks; The second stage module includes a first label merging module, a third hierarchical visual attention module, a fourth hierarchical visual attention module, and a second label attention module connected in sequence, with the first label merging module connected to the first label attention module; The third stage module includes a second label merging module, a fifth hierarchical visual attention module, a sixth hierarchical visual attention module, a seventh hierarchical visual attention module, an eighth hierarchical visual attention module, a ninth hierarchical visual attention module, a tenth hierarchical visual attention module, and a third label attention module connected in sequence, with the second label merging module connected to the second label attention module; The fourth stage module includes a third label merging module, an eleventh-level visual attention module, a twelfth-level visual attention module, and a fourth label attention module connected in sequence. The third label merging module is connected to the third label attention module, and the fourth label attention module is connected to the label selection module.

[0012] Further, the step of inputting the second video into the backbone network and outputting classification labels includes: The second video is input into the backbone network, passes through the spatiotemporal block module, the first stage module, the second stage module, the third stage module, and the fourth stage module, and outputs a classification label; The tag selection module obtains the classification tag based on multiple tags output by the fourth-stage module, including: Calculate the average of the multiple marks to obtain a first average; Calculate the L2 norm for each of the multiple labels, sort the obtained L2 norms from largest to smallest, and extract the top K L2 norms according to the hyperparameters; The second average is obtained by calculating the average of the K L2 norms; The classification label is obtained by concatenating the first average value and the second average value along the feature dimension.

[0013] Furthermore, the classification loss function of the classifier module is calculated in the following manner: ; in, For the classification loss function value, To belong to category The probability, The first parameter is used to adjust the category. The weights of the loss function, The second parameter controls the shape of the focus loss function.

[0014] Furthermore, the contrastive loss function of the classifier module is calculated as follows: ; in, To compare the loss function values, For batch size, and These are feature representations corresponding to different samples. and These are the labels corresponding to different samples. yes and Dot product.

[0015] Compared with the prior art, the present invention has the following advantages: 1. The micro-expression recognition method based on spatiotemporal features and attention mechanism provided by this invention recognizes micro-expressions by using pure RGB frame sequences, thus eliminating the limitation of relying on manual features such as optical flow.

[0016] 2. The micro-expression recognition method based on spatiotemporal features and attention mechanism provided by the present invention can better correct the pose and expression changes in face images by adopting a non-rigid transformation face alignment method, making the faces between different frames more consistent, which helps to reduce the deformation caused by changes in head pose and expression, thereby providing clearer and more comparable face features.

[0017] 3. The micro-expression recognition method based on spatiotemporal features and attention mechanism provided by this invention can not only reduce artifacts and errors caused by frame interpolation and improve the accuracy of the algorithm through time normalization strategy, but also speed up the preprocessing.

[0018] 4. The micro-expression recognition method based on spatiotemporal features and attention mechanism provided by the present invention constructs a backbone network based on hierarchical visual self-attention and label selection. Its label attention module continuously performs weighted operations at each stage, explicitly requiring the network to give different degrees of attention to different regions of the face.

[0019] 5. The micro-expression recognition method based on spatiotemporal features and attention mechanism provided by this invention has a label selection module that can extract labels that make important contributions to micro-expression recognition and highlight the feature representation of these labels, thereby improving the accuracy and robustness of the classification task and thus improving the performance of the micro-expression recognition method.

[0020] 6. The micro-expression recognition method based on spatiotemporal features and attention mechanism provided by this invention performs joint training based on contrastive feature learning and classification learning, realizing the effective utilization of low-level features, thereby improving the performance of the backbone network. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a micro-expression recognition method based on spatiotemporal features and attention mechanisms provided by the present invention.

[0023] Figure 2 This is a flowchart illustrating a spatial normalization process.

[0024] Figure 3 A flowchart illustrating time normalization.

[0025] Figure 4 This is a schematic diagram of a network backbone structure.

[0026] Figure 5 A schematic diagram of a labeled attention module.

[0027] Figure 6 A labeled partitioning diagram for a frame of micro-expression image.

[0028] Figure 7 This is a schematic diagram illustrating the model training for micro-expression recognition based on spatiotemporal features and attention mechanisms provided by this invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Combination Figure 1 , Figure 1 This is a flowchart illustrating a micro-expression recognition method based on spatiotemporal features and attention mechanisms provided by the present invention, to illustrate a specific embodiment of the micro-expression recognition method based on spatiotemporal features and attention mechanisms provided by the present invention, including: S1: Spatial normalization is performed on the micro-expression video to obtain the first video; S2: Perform time normalization on the first video to obtain the second video; S3: Construct a backbone network based on hierarchical visual self-attention and label selection; S4: Input the second video into the backbone network and output classification labels; S5: Use a classifier module to classify and identify the classification tags.

[0032] In some alternative embodiments, refer to Figure 2 , Figure 2 This is a flowchart illustrating a spatial normalization process for micro-expression videos, including: The micro-expression video consists of M frames of images. After aligning the faces in each frame, the micro-expression images are obtained by natural face cropping and then adjusted to a preset size.

[0033] It is understandable that micro-expression videos can be viewed as an image sequence, and face alignment uses non-rigid transformation face alignment, including: The facial key point coordinates are located for each frame of the micro-expression video using a facial key point detection algorithm.

[0034] Select the keypoint coordinates corresponding to Left Outer Eye, Right Outer Eye, and Nose Tip in the first frame as alignment reference points.

[0035] The selected key points are matched with the corresponding points on the standard template face, and the affine transformation matrix is ​​calculated.

[0036] Using the calculated affine transformation matrix, a non-rigid transformation is performed on each frame of the image. This transformation will make the key points in the image (such as eyes and nose) consistent with the corresponding points on the standard template face, thereby achieving face alignment.

[0037] Specifically, natural face cropping includes: For the first frame of a micro-expression video, a facial landmark detection algorithm is used to determine the coordinates of key points. The set of key points can be represented as follows: ,in, Indicates the first Two-dimensional coordinates of key facial features.

[0038] A rectangular area is defined by two coordinate points. and To determine the top-left coordinates and obtain more information about the area above the eyebrows, we will... minus , That is, half the distance between the 36th and 18th keypoints. Therefore, the obtained top-left coordinates... .according to and Determine the bottom right coordinates .in, .

[0039] For each frame of the entire micro-expression video, the coordinates are used to... and The defined rectangle is then cropped.

[0040] Specifically, the choice of scale is related to the backbone network structure and the face resolution in the dataset. Publicly available micro-expression datasets have limitations on the size of faces, typically with the shortest side around 230 pixels. In this embodiment, the aligned and cropped face is scaled to 224×224, i.e., H=W=224, which satisfies the backbone network input requirements while providing more detailed information.

[0041] In some optional implementations, refer to Figure 3 , Figure 3 This is a flowchart illustrating a time normalization process. The first video undergoes time normalization, including: Set the normalization value of the first video to the preset value; The first video contains M frames of micro-expression images; determine whether M is equal to a preset value. If M equals the preset value, no processing is required; If M is less than the preset value, perform 3D linear interpolation upsampling on the first video to make the second video have micro-expression images of the preset value frames; If M is greater than the preset value, the first video is divided into S semantic segments, where S is Z times the preset value, 0 < Z ≤ 1. A second video is formed by taking Q consecutive micro-expression images from each semantic segment, where Q × S is equal to the preset value.

[0042] Understandably, referring to Figure 3 In this embodiment, the preset value is 16, but it is not limited to this. The value of the preset value can be adjusted according to actual needs, and this embodiment does not impose specific restrictions on it. The first video is divided into S consecutive semantic segments. The value of S can be 1 / 2, 1 / 4, etc., such as 16, 8, 4, etc. The specific value of S can be modified according to actual needs or the value of the preset value, and this embodiment does not impose specific restrictions on it. Specifically, when the backbone network is trained, consecutive Q-frame micro-expression images are randomly selected from each semantic segment to form the second video; when the backbone network is tested, the Q-frame micro-expression images include the micro-expression images corresponding to the center position of the corresponding semantic segment. The micro-expression videos after spatial and temporal normalization have uniform spatial and temporal characteristics, which facilitates the backbone network to extract and analyze micro-expression features.

[0043] In some alternative embodiments, refer to Figure 4 and Figure 5 , Figure 4 This is a schematic diagram of a network backbone structure. Figure 5 A schematic diagram of the label attention module is provided. A backbone network based on hierarchical visual self-attention and label selection is constructed, including: Connect the spacetime block module, the first stage module, the second stage module, the third stage module, and the fourth stage module in sequence; The spatiotemporal block module divides the second video into G spatiotemporal blocks.

[0044] The first-stage module includes a linear transformation layer, a first-level visual attention module, a second-level visual attention module, and a first-marking attention module connected in sequence. The linear transformation layer is connected to the spatiotemporal block. The second-stage module includes a first label merging module, a third-level hierarchical visual attention module, a fourth-level hierarchical visual attention module, and a second label attention module connected in sequence. The first label merging module is connected to the first label attention module. The third-stage module includes the second label merging module, the fifth hierarchical visual attention module, the sixth hierarchical visual attention module, the seventh hierarchical visual attention module, the eighth hierarchical visual attention module, the ninth hierarchical visual attention module, the tenth hierarchical visual attention module, and the third label attention module, which are connected in sequence. The second label merging module is connected to the second label attention module. The fourth stage module includes the third label merging module, the eleventh hierarchical visual attention module, the twelfth hierarchical visual attention module, and the fourth label attention module, which are connected in sequence. The third label merging module is connected to the third label attention module, and the fourth label attention module is connected to the label selection module.

[0045] Understandably, the size of the second video is T×H×W×3, where T is the number of frames in the micro-expression images of the second video, H is the height dimension of the second video, W is the width dimension of the second video, and 3 represents the number of color channels (R, G, B). With the second video size being 16×224×224×3, and the spatiotemporal patch (3D patch) size set to 2×4×4×3, 16 / 2×224 / 4×224 / 4=25088 spatiotemporal patches can be generated. Each spatiotemporal patch, when flattened, is a vector with dimensions 2×4×4×3=96.

[0046] The labels are obtained by linearly transforming spatiotemporal blocks. The label attention module adds an explicit attention weight to each label, thus weighting the labels. (Refer to...) Figure 5 Let T be the input marker. This represents the maximum value of T in the channel dimension. This represents the average value of T over the channel dimension. FC indicates matrix multiplication in a fully connected layer, Sig represents the sigmoid function, and the label indicates the attention weights. Calculated as follows: ; The symbol “;” indicates a splicing operation in the channel dimension.

[0047] Finally, through the labeled attention weights Perform element-wise multiplication with the input token to obtain the output token. :

[0048] The symbol “⊙” represents element-wise multiplication.

[0049] Specifically, the backbone network is input to 25,088 spatiotemporal blocks, each with dimensions of 2×4×4×3. It undergoes four stages of modeling at different scales, each with a different configuration, such as... Figure 4 As shown. The marker merging module achieves multi-scale modeling through downsampled frames, with the four stages corresponding to downsampling of 4×, 8×, 16×, and 32× respectively.

[0050] Finally, the output dimension of the fourth stage module is 392×768, which means that 392 tags are output, and each tag has a dimension of 768.

[0051] In some alternative embodiments, reference continues to be made to... Figure 4 , Figure 5 and Figure 6 , Figure 6 To generate a labeled partition map for a single frame of micro-expression image, a second video is input into the backbone network, which outputs multiple classification labels, including: The second video is input into the backbone network, and after passing through the spatiotemporal block module, the first stage module, the second stage module, the third stage module, and the fourth stage module, multiple tags are output. The tag selection module obtains the classification tag based on multiple tags, including: Calculate the average of multiple marks to obtain the first average; Calculate the L2 norm for multiple labels, sort the obtained L2 norms from largest to smallest, and extract the top K L2 norms based on hyperparameters; The second average is obtained by calculating the average of the K L2 norms; The classification label is obtained by concatenating the first and second average values ​​along the feature dimension.

[0052] It should be noted that the label selection module can extract labels with a large L2 norm. These labels often correspond to important feature information and are further used to generate classification labels. The purpose of this is to highlight features that are beneficial to micro-expression classification, thereby improving the accuracy and robustness of the model.

[0053] Understandably, all the tags output by the fourth-stage module are used as input to the tag selection module to calculate the mean of the 392 tags, denoted as . , It includes global information. The L2 norm is calculated for 392 labels, and the L2 norm for each label is... The labels are then sorted based on their L2 norm. Let the hyperparameter K = 196, and select the top 196 labels with the largest L2 norm. The mean of these 196 labels is calculated along the feature dimension and denoted as . .Will and The classification label is obtained by concatenating the features along the dimension, which is 768+768=1536.

[0054] For a micro-expression video, the backbone network outputs a feature vector with a dimension of 1536.

[0055] The selection of the hyperparameter K in the marker selection module can be based on the number of markers occupying the high-frequency action regions of micro-expressions in the face. The backbone network performs hierarchical downsampling of the markers in four stages, achieving a 32-fold downsampling in space. The specific division is as follows: Figure 6As shown, in each frame, the micro-expression areas such as the eyes, eyebrows, and mouth occupy approximately 23 markers. Because of the 2x downsampling in time, the input 16 frames are downsampled to 8 frames, resulting in 184 markers corresponding to high-frequency micro-expression regions within those 8 frames. To increase the algorithm's fault tolerance, the hyperparameter K is increased to 196, half the total number of markers. This ensures that all markers occupied by high-frequency micro-expression regions are included, and provides additional markers to handle potential variations or errors.

[0056] In some alternative embodiments, refer to Figure 7 , Figure 7 This diagram illustrates the model training for micro-expression recognition based on spatiotemporal features and attention mechanisms provided by this invention. Preprocessed micro-expression videos are spatiotemporally modeled using a backbone network based on hierarchical visual self-attention and label selection, further generating classification labels. These classification labels are used for both contrastive feature learning and classification learning.

[0057] The loss function used for training is They have equal weighting coefficients. Among them, Indicates focal loss. It is the focus loss between the predicted label and the real label. The comparison loss function is represented. Validation employs leave-one-subject-out (LOSO) cross-validation, which avoids the classification module's dependence on human input, accurately assesses the generalization ability of the classification module, and improves the model's accuracy and reliability. LOSO cross-validation is a special cross-validation technique where all facial expression samples from each subject in the dataset are used as the test set, while samples from other subjects form the training set.

[0058] In terms of evaluation metrics, since the current micro-expression database has a serious sample class imbalance problem, it is necessary to introduce unweighted average recall (UAR) and unweighted F1-score (UF1) to comprehensively evaluate the algorithm performance.

[0059] ; ; in, It is the number of micro-expression categories. , This indicates that the true micro-expression category is number one. Total number of samples in each class This represents the total number of all samples. A positive example is one that is correctly classified (i.e., both the true label and the predicted label belong to the same category). (samples of the class), This represents a false positive (i.e., the true label is the first positive). (Class, samples whose predicted label category is one of the other categories), A false negative is a case where the true label is not the correct one. Class, predicting the expression as the first kind).

[0060] This invention conducted a three-classification (surprise, positivity, negativity) experiment on a publicly available micro-expression dataset, achieving a good result with a UF1 of 0.84 and a UAR of 0.86.

[0061] In summary, this invention provides a micro-expression recognition method based on spatiotemporal features and attention mechanisms, which avoids dependence on manual optical flow features and achieves efficient and simplified micro-expression recognition. The effectiveness of the method provided by this invention is verified on a micro-expression dataset.

[0062] In some optional embodiments, the classification loss function of the classifier module is calculated as follows: ; in, For the classification loss function value, To belong to category The probability, The first parameter is used to adjust the category. The weights of the loss function, The second parameter controls the shape of the focus loss function.

[0063] Understandably, the goal of classification learning is to categorize input samples into different predefined classes by learning a classifier. The classifier module consists of a fully connected layer and a softmax layer. The input dimension of the fully connected layer... The output dimension is determined by the dimension of the classification label. The number of categories is determined by the number of categories. The fully connected layer performs linear mapping and dimensionality transformation on the input features, while the softmax layer transforms the mapped features into probability distributions for each category using the softmax function, which are then used for classification decisions.

[0064] The classification loss function is also an important component of classification tasks. This invention uses the focus loss function. Compared to cross-entropy loss, it can address the class imbalance problem in the dataset and improve the classification accuracy for samples of the minority classes. Specifically, when... It degenerates into the cross-entropy loss function.

[0065] In some optional embodiments, the contrastive loss function of the classifier module is calculated as follows: ; in, To compare the loss function values, For batch size, and These are feature representations corresponding to different samples. and These are the labels corresponding to different samples. yes and Dot product.

[0066] Understandably, the goal of contrastive feature learning is to learn more discriminative feature representations by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs, thereby improving the performance of classification tasks. In each batch of training data, positive and negative samples need to be selected. Positive samples are different samples within the same class, while negative samples are samples between different classes. The features of each sample are represented using the classification label output from step four. By calculating the feature similarity between positive and negative samples, the loss function for contrastive learning can be defined.

[0067] The micro-expression recognition method based on spatiotemporal features and attention mechanisms provided by this invention eliminates the limitations of relying on manual features such as optical flow by using pure RGB frame sequences for micro-expression recognition. By employing a non-rigid transformation face alignment method, it can better correct pose and expression changes in facial images, making faces more consistent across different frames and helping to reduce distortions caused by changes in head pose and expression, thus providing clearer and more comparable facial features. The temporal normalization strategy not only reduces artifacts and errors caused by frame interpolation, improving algorithm accuracy, but also accelerates preprocessing. By constructing a backbone network based on hierarchical visual self-attention and label selection, its label attention module continuously performs weighted operations at each stage, explicitly requiring the network to give different levels of attention to different regions of the face. The label selection module can extract labels that significantly contribute to micro-expression recognition and highlight the feature representations of these labels, thereby improving the accuracy and robustness of classification tasks and thus improving the performance of the micro-expression recognition method. Joint training based on contrastive feature learning and classification learning enables effective utilization of low-level features, thereby improving the performance of the backbone network.

[0068] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0069] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A micro-expression recognition method based on spatiotemporal features and attention mechanisms, characterized in that, include: Spatial normalization was performed on the micro-expression video to obtain the first video; The first video is time-normalized to obtain the second video; Construct a backbone network based on hierarchical visual self-attention and label selection, including: Connect the spacetime block module, the first stage module, the second stage module, the third stage module, and the fourth stage module in sequence; The spatiotemporal block module divides the second video into G spatiotemporal blocks; The first stage module includes a linear transformation layer, a first hierarchical visual attention module, a second hierarchical visual attention module, and a first marker attention module connected in sequence, wherein the linear transformation layer is connected to the spatiotemporal block; The second stage module includes a first label merging module, a third hierarchical visual attention module, a fourth hierarchical visual attention module, and a second label attention module connected in sequence, with the first label merging module connected to the first label attention module; The third stage module includes a second label merging module, a fifth hierarchical visual attention module, a sixth hierarchical visual attention module, a seventh hierarchical visual attention module, an eighth hierarchical visual attention module, a ninth hierarchical visual attention module, a tenth hierarchical visual attention module, and a third label attention module connected in sequence, with the second label merging module connected to the second label attention module; The fourth stage module includes a third label merging module, an eleventh-level hierarchical visual attention module, a twelfth-level hierarchical visual attention module, and a fourth label attention module connected in sequence. The third label merging module is connected to the third label attention module, and the fourth label attention module is connected to the label selection module. The second video is input into the backbone network, and classification labels are output, including: The second video is input into the backbone network, passes through the spatiotemporal block module, the first stage module, the second stage module, the third stage module, and the fourth stage module, and outputs a classification label; The tag selection module obtains the classification tag based on the multiple tags output by the fourth stage module, including: Calculate the average of the multiple marks to obtain a first average; Calculate the L2 norm for each of the multiple labels, sort the obtained L2 norms from largest to smallest, and extract the top K L2 norms according to the hyperparameters; The second average is obtained by calculating the average of the K L2 norms; The classification label is obtained by concatenating the first average value and the second average value along the feature dimension; A classifier module is used to classify and identify the classification tags.

2. The micro-expression recognition method based on spatiotemporal features and attention mechanism according to claim 1, characterized in that, The spatial normalization processing of the micro-expression video includes: The micro-expression video includes M frames of images. Each frame of the image is aligned with a face and then cropped to obtain a micro-expression image. The micro-expression image is then adjusted to a preset size.

3. The micro-expression recognition method based on spatiotemporal features and attention mechanism according to claim 1, characterized in that, The time normalization process for the first video includes: The normalization value of the first video is set to a preset value; The first video includes M frames of micro-expression images, and it is determined whether M is equal to the preset value; If M equals the preset value, no processing is required; If M is less than the preset value, perform 3D linear interpolation upsampling on the first video so that the second video has the micro-expression image of the preset value frame; If M is greater than the preset value, the first video is divided into S semantic segments, where S is Z times the preset value, 0 < Z ≤ 1, and Q consecutive frames of micro-expression images are taken from each semantic segment to form the second video, where Q × S is equal to the preset value.

4. The micro-expression recognition method based on spatiotemporal features and attention mechanism according to claim 1, characterized in that, The classification loss function of the classifier module is calculated in the following manner: ； in, For the classification loss function value, To belong to category The probability, The first parameter is used to adjust the category. The weights of the loss function, The second parameter controls the shape of the focus loss function.

5. The micro-expression recognition method based on spatiotemporal features and attention mechanism according to claim 1, characterized in that, The contrastive loss function of the classifier module is calculated as follows: ； in, To compare the loss function values, For batch size, and These are feature representations corresponding to different samples. and These are the labels corresponding to different samples. yes and Dot product.

Images