A micro-expression recognition method, system, device and medium based on space-time facial region relationship

Abstract

This invention belongs to the field of facial expression recognition technology and discloses a method, system, device, and medium for micro-expression recognition based on spatiotemporal facial region relationships. The method includes: acquiring micro-expression video images; performing frame sampling processing on the micro-expression video images to obtain an image sequence; extracting regions of interest (ROIs) including facial key point location information from the image sequence to obtain preprocessed image data; and inputting the preprocessed image data into a micro-expression recognition model for classification and prediction to obtain a micro-expression recognition result. The micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence. The feature extraction module includes a parallel spatiotemporal feature extraction sub-module and a region relationship extraction sub-module. The technical solution of this invention significantly improves the accuracy and generalization ability of micro-expression recognition.

Description

Technical Field

[0001] This invention belongs to the field of facial expression recognition technology, and in particular relates to a method, system, device and medium for micro-expression recognition based on spatiotemporal facial region relationships. Background Technology

[0002] Microexpressions are rapid, subtle, and difficult-to-detect facial expressions, typically lasting only a very short time, that can reveal an individual's true emotions or intentions that they are trying to conceal. The value of microexpressions lies in their spontaneity; their non-performable nature makes them a valuable facial movement feature for measuring genuine emotional expression. Therefore, microexpression recognition has significant applications in fields such as emotion analysis and psychological state assessment. In recent years, with the development of computer vision and machine learning technologies, microexpression recognition has gradually become a research hotspot.

[0003] The core challenge of micro-expression recognition lies in the extremely small amplitude and subtle changes in micro-expression movements, making it very difficult to accurately extract micro-expression features from video sequences. Existing micro-expression recognition technologies suffer from the following shortcomings: First, in terms of feature extraction, micro-expression recognition methods using video as training input suffer from redundant input information and interference from facial identity information; while image-based micro-expression recognition methods have achieved good results, current research mainly focuses on spatial feature extraction of micro-expressions, neglecting the differences in the contribution of micro-expressions to different regions of the face; second, existing technologies fail to fully model the correlation between different facial regions, leading to a separation between local and overall features, limiting the improvement of recognition accuracy; furthermore, simple image preprocessing methods cannot effectively highlight key areas of micro-expressions, and feature extraction efficiency is low, making it difficult to meet practical application needs. Summary of the Invention

[0004] The purpose of this invention is to provide a micro-expression recognition method, system, device, and medium based on spatiotemporal facial region relationships to solve the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides a micro-expression recognition method based on spatiotemporal facial region relationships, comprising:

[0006] Acquire micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, extract the region of interest including facial key point location information from the image sequence to obtain preprocessed image data;

[0007] The preprocessed image data is input into the micro-expression recognition model for classification and prediction to obtain the micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module and a prediction module connected in sequence, and the feature extraction module includes a spatiotemporal feature extraction sub-module and a regional relationship extraction sub-module set in parallel.

[0008] Optionally, extracting the region of interest, including facial key point location information, from the image sequence specifically includes:

[0009] Face detection is performed on the starting frame and peak values ​​of the image sequence to obtain facial key point location information;

[0010] The optical flow information between the start frame and the peak frame is calculated based on the TV-L1 method. The optical flow information includes vertical optical flow, horizontal optical flow and optical flow strain. The three single-channel optical flow information are stacked to form a three-channel visual representation of micro-expression motion, and a micro-expression optical flow motion image is obtained.

[0011] In the micro-expression optical flow motion image, regions of interest including facial key point location information are extracted. The extracted regions of interest are aggregated to obtain the input data of the micro-expression recognition model.

[0012] Each micro-expression video image is processed repeatedly to construct preprocessed image data.

[0013] Optionally, the training process of the micro-expression recognition model specifically includes:

[0014] Obtain a training dataset, which includes micro-expression video images and corresponding micro-expression recognition labels;

[0015] Construct an initial micro-expression recognition model;

[0016] Data validation: Select one subject's data from the training data as the test set, and use the data from all remaining subjects to train the initial micro-expression recognition model. Iterate through all the subject data in the training dataset and calculate the average of all test results to measure the overall performance of the initial micro-expression recognition model.

[0017] After data verification, the class distribution balance performance of the initial micro-expression recognition model is evaluated based on preset indicators, and the model is optimized based on the evaluation results. The preset indicators include unweighted F1 score and unweighted average recall.

[0018] The micro-expression recognition task is performed on the image to be recognized based on the trained micro-expression recognition model.

[0019] Optionally, the processing procedure of the micro-expression recognition model specifically includes:

[0020] The preprocessed micro-expression video image is input into the spatiotemporal feature extraction submodule and the regional relationship extraction submodule to extract spatiotemporal features and regional relationship features. The extracted features are input into the feature fusion module for feature fusion to obtain fused features. The fused features are then input into the prediction module for classification prediction to obtain the corresponding micro-expression recognition result.

[0021] Optionally, the processing procedure of the spatiotemporal feature extraction submodule specifically includes:

[0022] The micro-expression video images are mapped into a high-dimensional feature space through the embedding operation. After the embedding operation is completed, the image partitioning operation is performed. Each layer divides the micro-expression video image into multiple small images. In each layer, the Transformer encoder is used to extract local features from the small images. The extracted local features are aggregated through the convolution operation to obtain the spatiotemporal features.

[0023] Optionally, the processing procedure of the region relationship extraction submodule specifically includes:

[0024] In the region relationship extraction submodule, the relationship between facial regions in micro-expression video images is extracted using the visual Transformer model to obtain region relationship features.

[0025] Optionally, the processing procedure of the feature fusion module specifically includes:

[0026] The output of the region relation extraction submodule is aligned with the output of the spatiotemporal feature extraction submodule through tensor embedding, reshaping, and dimensionality reduction operations. After alignment, the outputs of the two modules are fused by weighted summation and used as the input of the prediction module.

[0027] A micro-expression recognition system based on spatiotemporal facial region relationships includes:

[0028] The data acquisition module is used to acquire micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, and extract the region of interest including facial key point location information from the image sequence to obtain preprocessed image data.

[0029] The micro-expression recognition module is used to input preprocessed image data into the micro-expression recognition model for classification and prediction to obtain micro-expression recognition results. The micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence. The feature extraction module includes a spatiotemporal feature extraction sub-module and a regional relationship extraction sub-module set in parallel.

[0030] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to enable the electronic device to perform a micro-expression recognition method based on spatiotemporal facial region relationships.

[0031] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the micro-expression recognition method based on spatiotemporal facial region relationships.

[0032] The technical effects of this invention are as follows:

[0033] The spatiotemporal facial region relationship network provided by this invention fully explores the local dynamic features and global regional relationships of micro-expressions by fusing a spatiotemporal feature extraction module and a region relationship extraction module, effectively enhancing the expressive power of micro-expressions. The SEM module uses a multi-level structure to finely extract the spatiotemporal features of micro-expressions, while the REM module models the correlation between facial regions through a visual Transformer. The combination of these two modules and the integration of information through a feature fusion module significantly improves the accuracy and generalization ability of micro-expression recognition. It performs excellently when dealing with small sample datasets and tasks with significant individual differences, providing a new and effective solution for the field of micro-expression recognition.

[0034] The STRNet model in this application employs a parallel structure of spatiotemporal feature extraction and region relation extraction modules, combined with a feature fusion module. This not only extracts spatiotemporal features of micro-expressions but also models the correlations between facial regions, enhancing feature representation capabilities. Compared to methods with a single feature extraction module, performance is significantly improved. On the CASMEII dataset, UF1 and UAR reach 0.9764 and 0.9792 respectively, outperforming other state-of-the-art methods. On the CASME3 dataset, UF1 and UAR reach 0.5848 and 0.5601 respectively, an improvement of approximately 2% over the next best method. Furthermore, this invention utilizes leave-one-out object validation training, effectively reducing the impact of individual differences and improving model robustness. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0037] Figure 1 This is a schematic diagram of the image preprocessing process in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the STRNet model structure in an embodiment of the present invention;

[0039] Figure 3This is the confusion matrix analysis result of the Cross Datasets Evaluation (CDE) task in this embodiment of the invention;

[0040] Figure 4 These are different sample images of the surprise category in the SMIC dataset in this embodiment of the invention;

[0041] Figure 5 This is a flowchart illustrating the identification process in an embodiment of the present invention. Detailed Implementation

[0042] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0043] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0044] Unless otherwise stated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. While only preferred methods have been described herein, any methods similar or equivalent to those described herein may be used in the implementation or testing of this invention. All references to this specification are incorporated by way of citation to disclose and describe the methods associated with those references. In the event of any conflict with any incorporated reference, the content of this specification shall prevail.

[0045] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0046] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] like Figure 1 - Figure 5 As shown, this embodiment provides a micro-expression recognition method based on spatiotemporal facial region relationships, including: acquiring micro-expression video images; performing frame sampling processing on the micro-expression video images to obtain an image sequence; extracting regions of interest including facial key point location information from the image sequence to obtain preprocessed image data; inputting the preprocessed image data into a micro-expression recognition model for classification and prediction to obtain micro-expression recognition results; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence, and the feature extraction module includes a spatiotemporal feature extraction sub-module and a region relationship extraction sub-module set in parallel.

[0049] Current research primarily focuses on spatial feature extraction from micro-expression motion images, neglecting the varying contributions of micro-expressions to different facial regions. This, to some extent, limits the improvement of micro-expression recognition accuracy. Therefore, there is an urgent need to explore research directions that can integrate multiple micro-expression information and comprehensively apply them to emotion classification.

[0050] To address the aforementioned issues, this embodiment proposes a spatiotemporal and region relationship network for micro-expression recognition (STRNet). This model, based on traditional spatial feature extraction from micro-expression motion images, introduces correlation information between different facial regions and fuses it with existing spatial features. Experiments were conducted not only on publicly available datasets (SMIC, SAMM, CASME II, and CASME3) but also on a self-built food sensory micro-expression dataset for training and validation. Experimental results demonstrate that this model exhibits superior performance in micro-expression recognition tasks.

[0051] To meet the above requirements, this embodiment will elaborate on three aspects: facial data input, facial motion data processing, and sensory recognition result display.

[0052] (1) Face Data Input: In this embodiment, there are two input methods: one is real-time data acquisition via camera, and the other is non-real-time input via video file. Currently, common cameras on the market typically acquire images at a rate of 30 frames per second. Image processing algorithms are slower than sequence processing algorithms, mainly because a single frame contains multi-dimensional information such as width, height, and channels. Meanwhile, the change time for micro-expressions is approximately 0.5 seconds, and in 30 frames of data, only the motion changes between every 15 frames need to be calculated. Furthermore, real-time display of emotions may cause screen flickering due to frame rate fluctuations, affecting the user experience, which is less ideal than the overall display effect after recording. Therefore, in real-time camera input mode, the software records a video after starting the recording and recognition function and processes it using algorithms at a frequency of 15 frames per second; the same processing strategy is used in video file input mode.

[0053] (2) Facial Motion Data Processing: The facial motion data processing module mainly processes the image data containing facial information received by the software, generating a sequence containing the algorithm's inference results. After receiving the micro-expression video image, the system creates a new thread to execute the algorithm inference. First, frame sampling significantly reduces the computational load; the sampled image sequence is then processed according to the micro-expression recognition algorithm preprocessing flow, i.e., firstly, face detection is performed using MTCNN, then affine transformation is performed based on facial feature points to achieve face alignment, and then the face is cropped according to the distance between the inner corners of the eyes to extract a pure face sequence. After obtaining the pure face sequence, the first frame of the sampled sequence is used as the starting frame, and the remaining frames are used as peak frames for optical flow image calculation. Finally, the generated optical flow image sequence is used as the input of the MMCN model. It is worth noting that, to adapt to real-world scenarios, MMCN introduces neutral data during training, so it does not output emotion predictions for frames without significant emotional changes. Finally, MMCN returns a sequence predicting the emotion.

[0054] (3) Sensory Recognition Result Display: The sensory recognition result display mainly focuses on the emotion sequence generated by the background algorithm. The display forms are mainly divided into statistical display and temporal display. The statistical display uses pie charts and other graphics to intuitively present the percentage of each emotion category in the whole sequence, thereby reflecting the overall emotion distribution; the temporal display uses different colors to mark each emotion according to the time order of the original emotion sequence, thereby intuitively showing the dynamic changes of emotions over time.

[0055] Image preprocessing: Micro-expressions are essentially subtle dynamic changes manifested by facial movements. Two main methods are typically used for micro-expression motion recognition. One method is classification based on the complete video sequence, and the other is calculating motion information based on the starting and peak frames of the expression. However, because the amplitude of micro-expression movements is extremely small, and the image changes between adjacent frames are also very weak, this can lead to significant redundancy in the calculation process.

[0056] This embodiment employs an optical flow calculation method based on the start frame and peak frame to describe micro-expression motion. Since micro-expressions are mainly manifested in local facial regions, using Region of Interest (ROI) selection to process the micro-expression motion image helps highlight the motion information of key regions, thereby improving the effectiveness of subsequent feature extraction. Considering the above, the HTNet preprocessing method is also referenced. This embodiment combines optical flow processing with ROI selection based on facial landmark detection, increasing the density of effective features in the input data while preserving spatiotemporal information. First, MTCNN is used for face detection in the start and peak frames of the micro-expression segment to obtain facial landmark location information. Second, the TV-L1 method is used to calculate the vertical optical flow, horizontal optical flow, and optical flow strain between the start and peak frames. Subsequently, these three single-channel optical flow results are stacked to form a three-channel visual representation of micro-expression motion.

[0057] Subsequently, the micro-expression optical flow motion image is sampled using facial landmark location information (such as the left and right eyebrow regions and the left and right corners of the mouth). Similar to the method used in HTNet, these sampled regions are aggregated to form the actual input of STRNet. This region of interest (ROI) selection method can effectively focus on high-frequency micro-expression regions, providing key raw input features for subsequent classification networks. At the same time, this method can significantly filter facial identity information, reducing individual differences that may interfere with the recognition task. The complete preprocessing workflow is as follows: Figure 1 As shown.

[0058] Furthermore, micro-expression movements are localized and subtle. During data acquisition, subjects may exhibit slight head movements, which could lead to mild noise interference in the input image. Based on this consideration, this embodiment resizes the input image to 28×28×3 during data preprocessing to more effectively distinguish the subtle movement features of micro-expressions from potential facial identity information.

[0059] Data Validation Method: Current micro-expression recognition research is mainly based on small-scale datasets, and there are significant differences in facial features among individuals (i.e., the influence of facial identity information). To reduce the impact of individual identity information on data distribution and improve the generalization ability of experimental evaluation indicators on different subjects, this embodiment adopts Leave-One-Subject-Out (LOSO) as the model validation method to ensure the robustness and reliability of the model in cross-individual recognition tasks.

[0060] Leave-one-out cross-validation (LOSO) is a common cross-validation method used for individual-independent tasks, widely applied in pattern recognition and machine learning. Its core idea is to select all data from one subject in the dataset each time, using all data from the remaining subjects as the test set, while using the remaining data from all subjects to train the model. This process iterates through all subjects in the dataset, ensuring that each subject's data is evaluated on the test set at least once. The overall model performance is then measured by averaging all test results. LOSO offers several advantages. First, it effectively assesses the model's individual independence, ensuring that training and test data come from different individuals, thus preventing overfitting due to excessive reliance on individual features. Second, with smaller datasets, LOSO maximizes the utilization of training data and is more stable in small-sample scenarios compared to K-Fold Cross Validation. Furthermore, because each test set consists of a complete single individual, LOSO more accurately evaluates the model's generalization ability on unseen individuals, making it particularly suitable for tasks with significant individual differences.

[0061] In micro-expression recognition tasks, due to significant differences in individual facial features, the expression patterns of different subjects may also exhibit marked variations. Therefore, using LOSO cross-validation can reduce the interference of individual identity information on model learning, ensuring that the model focuses on the common features of micro-expressions rather than individual-specific features. Simultaneously, this method can evaluate the model's adaptability to unseen subjects, which is of significant value for the application of micro-expression analysis in real-world scenarios (such as emotion detection).

[0062] Data Evaluation Metrics: After completing the label integration of the dataset, the research faces the problem of imbalanced class distribution. To address this issue, this embodiment employs the following two evaluation metrics: Unweighted F1 score (UF1), used to comprehensively evaluate the overall balance of the classification model; and Unweighted Average Recall (UAR), used to evaluate the model's recall performance across each class. The specific calculation formulas for these metrics are detailed in Formulas 1 to 4.

[0063]

[0064]

[0065]

[0066] In the formula, E represents the total number of categories, and k represents the number of experimental subjects. and This represents the true positive, false negative, and false positive of object β in category α.

[0067] In this embodiment, the model was trained for 800 rounds and validated using leave-one-out cross-validation. The experiment was set with an optimizer learning rate of 5e-5, a batch size of 256, and a cross-entropy loss function. All experiments were performed on an NVIDIA GeForce RTX 2080ti hardware platform.

[0068] A Spatiotemporal Facial Region Relationship Micro-expression Recognition Model: To balance effective micro-expression feature extraction with modeling the relationships between different facial regions, this embodiment proposes STRNet. STRNet mainly consists of two branches and a feature fusion model: a Spatiotemporal Feature Extraction (SEM) module, a Region Relationship Extraction (REM) module, and a Feature Fusion (FFM) module. SEM employs a multi-layered architecture to extract micro-expression features across multiple scales. The processed results are unified along the channel dimension to obtain refined micro-expression features while reducing interference from irrelevant information. REM aims to model the relationships between different facial regions at a more macroscopic scale. These two branches use different strategies to process information, thus requiring a dedicated module to integrate these two types of information and minimize feature loss. FFM aims to fuse the features extracted by SEM and REM, thereby enhancing the model's generalization ability. Compared to structures that only extract spatial information, this module can acquire richer information. The final result serves as the input to the MLP layer. The MLP layer consists of a LayerNorm (LN) layer, a Reduce layer, and a Linear layer. The overall architecture of the network is as follows: Figure 2 As shown.

[0069] Spatiotemporal Feature Extraction (SEM) Module: After image preprocessing, the SEM branch is responsible for extracting detailed motion features of micro-expressions. The backbone of this module uses a feature extractor from HTNet. HTNet has significant advantages in feature extraction because it employs a multi-level structure during the feature extraction process, as shown in Equations 5 and 6:

[0070]

[0071] k = floor(h mod 2) (6)

[0072] in, This represents micro-expression features, where k represents the number of partitions applied to the original input image, and f a (.) indicates a convolution operation used to aggregate images, t n This represents an n-layer Transformer encoder. This embodiment uses the best-performing parameter settings in HTNet. `floor(.)` represents the floor function, and `h` represents the width of the input image. In this network, the height and width of the input image are set to be equal.

[0073] The input image undergoes an embedding operation before partitioning. After image partitioning, features are extracted from smaller image patches using a Transformer encoder at each layer, and these patches are then aggregated using convolutional layers. SEM then models the optical flow features of micro-expressions from detail to overall structure. The fully aggregated features effectively model the spatiotemporal characteristics of micro-expressions.

[0074] Region Relationship Extraction Module (REM): The REM branch is primarily responsible for modeling the relative macroscopic relationships in micro-expression motion images. This network utilizes the Visual Transformer (ViT) to extract relationships between facial regions. ViT is a practical application of the Transformer architecture in image processing. As mentioned earlier, the spatial dimension of the input image has been compressed. To prevent the loss of inherent micro-expression features, this embodiment sets the image patch size to 1. The advantage is that each pixel can be used independently as a token, thus more accurately modeling the relationships between different facial regions. The patch-based method in this module includes multiple image patches within the potential facial motion region. Therefore, the output of the REM branch is coarser than that of the SEM branch, but its focus is different.

[0075] Feature Fusion Module (FFM): SEM and REM methods extract features from micro-expression optical flow images from different perspectives. The multi-level Transformer architecture in SEM effectively captures the temporal and spatial features of micro-expressions. REM models features using scaled micro-expression optical flow images, capturing the correlations between different facial regions. The architectures of these two models are not uniform, which may lead to uneven distribution of the output feature tensors. Therefore, the FFM module is proposed to address this issue. This module helps enhance the generalization ability of the micro-expression features extracted by the network, and the computation process is shown in Equations 7 and 8.

[0076] I V =f dim (f emb (I V0 (7)

[0077] I=α·IS +(1-α)·I V (8)

[0078] In the formula, α represents the weighting coefficient of the weighted summation of the fusion module mechanism, which was ultimately set to 0.7 after experiments. S and I V These represent the facial motion spatial features processed by SEM and the facial region features processed by REM, respectively. V0 This indicates the direct output of REM, while f dim and f emb They respectively refer to those applied to I V0 Embedding reshaping and dimensionality reduction operations to make it compatible with I S Feature alignment.

[0079] First, as shown in Equation 7, the output I of the REM branch V0 After tensor embedding, the method used in the SEM branch is referenced. Next, the embedded tensors are reshaped through a lossless rearrangement layer to modify the data structure. The shape, width, and height information of the SEM results are aggregated into the channel dimension. Subsequently, a 2D convolution operation with a kernel size of 1 is applied to scale the channel dimension to align it with the shape of the SEM results. Finally, as shown in Equation 8, the results from the two branches are fused through a weighted summation. The fused features are then passed through an MLP to obtain the final classification result.

[0080] Ablation experiment:

[0081] To verify the contributions of each component in STRNet, this embodiment conducted ablation experiments on the CrossDatasets Evaluation (CDE) task. The ablation experiments mainly focused on evaluating the independent contributions of the main components of the model. The experimental results are shown in Table 1.

[0082] Table 1 Ablation Experiment Results

[0083]

[0084] REM focuses solely on modeling different local facial regions, resulting in extracted features lacking detailed information. To extract overall feature information, REM further compresses the spatial features of the input image. Therefore, the features extracted by REM are more general, leading to relatively lower performance on SMIC, CASMEII, and SAMM. In qualitative analysis, REM performs similarly on SMIC and SAMM. REM performs slightly better on CASMEII, possibly because the CASMEII data is more homogeneous in ethnic classification, resulting in less variation in intra-class features. Using only the SEM branch significantly improves overall performance compared to using only REM. Specifically, on the SMIC dataset, the network improves by 20% on UF1 and UAR respectively. On the CASMEII dataset, UF1 and UAR values ​​are improved by approximately 30%. For the SAMM dataset, UF1 is improved by approximately 20%, while UAR is increased by 24%. One reason for the superior performance of using SEM independently is that its backbone network is specifically designed to extract spatial features of micro-expressions. Unlike REM, SEM does not further compress input features, thus enabling it to capture the intrinsic spatial features of micro-expressions.

[0085] When both branches and FFM are included, the model's performance is further improved on top of SEM. On the CASMEII dataset, UF1 and UAR improve by nearly 3%, reaching 0.9764 and 0.9792, respectively. On the SAMM dataset, UF1 improves by 6%, and UAR increases by 7%, reaching 0.8754 and 0.8862, respectively. The performance on the SMIC dataset is similar to that of the SEM branch. Experimental results show that the overall UF1 performance is optimal when both modules are present and (α = 0.7). The complete STRNet combines the association information of different facial regions generalized by REM with the micro-expression spatial features extracted by SEM, thereby capturing micro-expression features more comprehensively and thus improving performance.

[0086] Public dataset experiments:

[0087] (1) Experimental results of mixed datasets:

[0088] Figure 3Tables (a)-(c) present the confusion matrix analysis results for the Cross Datasets Evaluation (CDE) task. Experimental results show that the network exhibits optimal performance in negative emotion classification. Specifically, in the CASMEII dataset, the classification accuracy for each category is significantly high, with only a small number of positive samples being misclassified as surprised. However, in the SAMM and SMIC datasets, both surprised and positive emotions are misclassified as negative emotions. This misclassification may be attributed to the lower feature complexity of negative emotions and the relatively high proportion of negative emotion samples in the datasets. Furthermore, the differences in misclassification rates between different datasets may be closely related to the heterogeneity of the feature distribution of the datasets. Notably, the age distribution of volunteers in the CASMEII dataset is relatively concentrated, and the ethnic composition is relatively homogeneous; while the SAMM and SMIC datasets cover volunteers from diverse ethnicities and age groups, exhibiting a higher degree of balance in their ethnic and age distribution. This difference may further affect the classification performance of the model. To evaluate the effectiveness of the proposed method, experiments were first conducted on the cross-datasets evaluation task. The table summarizes the performance of the entire network in the CDE three-class classification task. The table also includes evaluation results of other competing and classic methods on the CDE task for comparison. Comparison methods include traditional methods such as LBP-TOP and Bi-WOOF, classic deep learning-based methods such as OFF-ApexNet and STSTNet, and current state-of-the-art methods such as BDCNN+I_w+GA and HTNet. Since most models in this field are not fully open-source, the comparison data comes directly from their respective papers.

[0089] Table 2 Performance Comparison of CDE Tasks

[0090]

[0091] Overall, as shown in Table 2, STRNet performs superiorly in both UF1 and UAR metrics. Specifically, traditional methods such as LBP-TOP and Bi-WOOF are used to capture motion features. Neither method achieves UF1 and UAR scores exceeding 0.5211 on the SAMM dataset, nor exceeding 0.5829 on the SMIC dataset. On the CASMEII dataset, both metrics do not exceed 0.8026. The overall performance of these two methods is relatively poor. One possible reason for this poor performance is the subtle features of micro-expressions, which may be interfered with by other information (such as identity-related features) during motion computation. Each micro-expression segment introduces varying degrees of noise, highlighting the limitations of these methods in generalization.

[0092] Regarding deep learning methods, OFF-ApexNet and STSTNet are classic approaches in this field. OFF-ApexNet performs relatively poorly on the SMIC dataset, achieving UF1 and UAR scores of 0.6817 and 0.6695, respectively. Its performance on the SAMM dataset remains poor, with UF1 and UAR scores of 0.5409 and 0.5392, respectively. STSTNet performs slightly better than OFF-ApexNet on the SMIC dataset, achieving UF1 and UAR scores of 0.6801 and 0.7013, respectively. This suggests that STSTNet may have successfully improved the classification accuracy of micro-expressions through certain improvements, particularly in feature extraction and noise suppression. However, this improvement remains limited, possibly due to the noise and complexity of the data in the SMIC dataset. STSTNet's performance on the SAMM and CASMEII datasets is 0.6588 and 0.6810 (UF1 and UAR), and 0.8382 and 0.8686 (UF1 and UAR), respectively. These results show a similar performance pattern to OFF-ApexNet. Although both networks employ deep learning methods and use optical flow images of micro-expressions as input, their recognition performance remains limited. This may be because the relatively simple structure of these two models results in weaker feature extraction capabilities, making it difficult to effectively capture subtle changes in micro-expressions. Therefore, despite employing advanced image input methods, the model's performance is still somewhat limited.

[0093] BDCNN achieved UF1 and UAR values ​​of 0.7859 and 0.7869 on the SMIC dataset, 0.8186 and 0.8124 on the SAMM dataset, and 0.9501 and 0.9516 on the CASMEII dataset. This demonstrates that BDCNN significantly outperforms previous classical methods across multiple datasets, particularly achieving significant improvements across various metrics. Compared to STSTNet, BDCNN improves UF1 values ​​by 10%, 12%, and 15% on the SMIC, CASMEII, and SAMM datasets, respectively. This improvement is likely attributed to BDCNN's more complex and efficient network architecture for feature extraction, enabling it to better capture subtle changes and global features in micro-expressions. HTNet is another recently effective method that employs a multi-layer Transformer architecture to extract micro-expression features. On the SMIC dataset, its UF1 and UAR values ​​are 0.8049 and 0.7905, respectively. On the CASMEII dataset, these values ​​are 0.9532 and 0.9516, respectively. On the SAMM dataset, the scores are 0.8131 and 0.8124 respectively. Overall, HTNet shows a slight improvement over BDCNN in all metrics.

[0094] The proposed STRNet in this embodiment achieves further improvements. On the CASMEII dataset, STRNet's UF1 score is nearly 3% higher than the second-best HTNet, reaching 0.9792. The UAR value is also improved by about 2%, reaching 0.9764. On the SAMM dataset, these two metrics are improved by 6% and 7%, respectively, reaching 0.8754 and 0.8862. On the SMIC dataset, the performance is comparable. The difference in numerical performance may be related to the richness of intra-class features (CASMEII and SAMM are ensembles of multiple classes). Overall, STRNet shows a significant improvement compared to the second-best method. STRNet's results vary across different datasets. Overall, STRNet outperforms the SMIC dataset on the CASMEII and SAMM datasets. Differences in the data distribution between classes in different datasets may be a potential cause of this issue. Another possible reason is that the SMIC dataset has a lower recording frame rate (100fps, compared to 200fps on other datasets), making it more susceptible to various noise factors.

[0095] Other images in the Surprise category of the SMIC dataset (such as...) Figure 4Compared to the sample shown, the differences between different samples are significant. This indicates that small inter-class differences and potentially large intra-class differences can be challenging in micro-expression recognition tasks. Facial motion interference other than facial expressions is the main reason for the large intra-class differences. However, STRNet is still able to discern some optical flow feature maps with minimal differences, demonstrating the effectiveness of the proposed network in extracting micro-expression features.

[0096] (2) CASME3 Experimental Results: To evaluate the model's performance on a larger dataset, experiments were also conducted on CASME3. Comparison methods included the classic deep learning micro-expression recognition algorithms STSTNet and HTNet. Since the experimental results of BDCNN on the CASME3 dataset were not reported in the original paper, they were not used as a comparison method. The overall results are shown in Table 3.

[0097] Table 3 Comparison of CASME3 Experimental Results

[0098]

[0099] STSTNet achieved scores of 0.3795 (UF1) and 0.3792 (UAR) on CASME3. STSTNet's relatively poor performance on the CASME3 dataset may be attributed to its structural simplicity, which fails to adequately handle the complexity of larger datasets. HTNet performed better on the CASME3 dataset, achieving scores of 0.5767 and 0.5415, respectively. STRNet further improved its performance on the CASME3 dataset, increasing by approximately 1% and 2%, achieving scores of 0.5848 and 0.5601, respectively. This indicates that the STRNet network exhibits stronger generalization ability on larger datasets. STRNet's performance on the CASME3 dataset is generally lower than its performance in cross-dataset evaluations. One possible reason is that CASME3 itself contains more subject and experimental data, making the training process more challenging. Furthermore, unlike the original CASMEII and SAMM datasets which offer five categories, the CASME3 dataset includes seven categories. This results in a more complex feature distribution in the CASME3 dataset after classification compression.

[0100] To effectively integrate spatial information and facial region relation information from micro-expression motion images, and to construct a micro-expression recognition algorithm that achieves excellent performance on both self-built video sensory datasets and publicly available datasets, this embodiment proposes a Spatiotemporal Facial Region Relationship Network (STRNet). This model aims to fully exploit the local dynamic features and global region relations of micro-expressions, enhancing the expressive power of micro-expression features to improve recognition accuracy and generalization ability.

[0101] In this network, a multi-level spatiotemporal feature extraction module is configured to progressively extract multi-scale features of micro-expressions. Simultaneously, a region relation extraction module is used to model the relationships between different local facial regions. Finally, a special feature fusion module is designed to effectively align and integrate region relation information with micro-expression spatial features, enhancing the generalization ability of the network's extracted micro-expression features. This overall architecture, through multi-branch fusion, can extract more comprehensive information.

[0102] In mixed dataset experiments, the model achieved a UF1 score of 0.9764 and a UAR score of 0.9792 on the CASMEII dataset, reaching state-of-the-art performance. On the SAMM dataset, the scores were 0.8754 and 0.8862, respectively. On the large-scale CASME3 dataset, the model achieved scores of 0.5848 and 0.5601, respectively, representing a 2% improvement in UF1 scores compared to the suboptimal method. Experimental results demonstrate that STRNet achieves significant performance.

[0103] Implementable, this embodiment also provides a micro-expression recognition system based on spatiotemporal facial region relationships, including:

[0104] The data acquisition module is used to acquire micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, and extract the region of interest including facial key point location information from the image sequence to obtain preprocessed image data.

[0105] The micro-expression recognition module is used to input preprocessed image data into the micro-expression recognition model for classification and prediction to obtain micro-expression recognition results. The micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence. The feature extraction module includes a spatiotemporal feature extraction sub-module and a regional relationship extraction sub-module set in parallel.

[0106] In this embodiment, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the micro-expression recognition method based on spatiotemporal facial region relationships.

[0107] In practice, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned micro-expression recognition method based on spatiotemporal facial region relationships.

[0108] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A micro-expression recognition method based on spatiotemporal facial region relationships, characterized in that, include: Acquire micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, extract the region of interest including facial key point location information from the image sequence to obtain preprocessed image data; The preprocessed image data is input into the micro-expression recognition model for classification and prediction to obtain the micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module and a prediction module connected in sequence, and the feature extraction module includes a spatiotemporal feature extraction sub-module and a regional relationship extraction sub-module set in parallel; The processing steps of the micro-expression recognition model specifically include: The preprocessed micro-expression video image is input into the spatiotemporal feature extraction submodule and the regional relationship extraction submodule to extract spatiotemporal features and regional relationship features. The extracted features are input into the feature fusion module for feature fusion to obtain fused features. The fused features are input into the prediction module for classification prediction to obtain the corresponding micro-expression recognition result. The processing procedure of the spatiotemporal feature extraction submodule specifically includes: The micro-expression video images are mapped to a high-dimensional feature space through the embedding operation. After the embedding operation is completed, the image partitioning operation is performed. Each layer divides the micro-expression video images into multiple small images. In each layer, the Transformer encoder is used to extract local features from the small images. The extracted local features are aggregated through the convolution operation to obtain the spatiotemporal features. The processing procedure of the region relationship extraction submodule specifically includes: In the region relationship extraction submodule, the relationship between facial regions in micro-expression video images is extracted using the visual Transformer model to obtain region relationship features.

2. The micro-expression recognition method based on spatiotemporal facial region relationships according to claim 1, characterized in that, Extracting the region of interest, including facial landmark location information, from the image sequence specifically includes: Face detection is performed on the starting frame and peak values ​​of the image sequence to obtain facial key point location information; The optical flow information between the start frame and the peak frame is calculated based on the TV-L1 method. The optical flow information includes vertical optical flow, horizontal optical flow and optical flow strain. The three single-channel optical flow information are stacked to form a three-channel visual representation of micro-expression motion, and a micro-expression optical flow motion image is obtained. In the micro-expression optical flow motion image, regions of interest including facial key point location information are extracted. The extracted regions of interest are aggregated to obtain the input data of the micro-expression recognition model. Each micro-expression video image is processed repeatedly to construct preprocessed image data.

3. The micro-expression recognition method based on spatiotemporal facial region relationships according to claim 1, characterized in that, The training process of the micro-expression recognition model specifically includes: Obtain a training dataset, which includes micro-expression video images and corresponding micro-expression recognition labels; Construct an initial micro-expression recognition model; Data validation: Select one subject's data from the training data as the test set, and use the data from all remaining subjects to train the initial micro-expression recognition model. Iterate through all the subject data in the training dataset and calculate the average of all test results to measure the overall performance of the initial micro-expression recognition model. After data verification, the class distribution balance performance of the initial micro-expression recognition model is evaluated based on preset indicators, and the model is optimized based on the evaluation results. The preset indicators include unweighted F1 score and unweighted average recall. The micro-expression recognition task is performed on the image to be recognized based on the trained micro-expression recognition model.

4. The micro-expression recognition method based on spatiotemporal facial region relationships according to claim 1, characterized in that, The processing procedure of the feature fusion module specifically includes: The output of the region relation extraction submodule is aligned with the output of the spatiotemporal feature extraction submodule through tensor embedding, reshaping, and dimensionality reduction operations. After alignment, the outputs of the two modules are fused by weighted summation and used as the input of the prediction module.

5. A micro-expression recognition system based on spatiotemporal facial region relationships, used to implement the method as described in any one of claims 1 to 4, characterized in that, include: The data acquisition module is used to acquire micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, and extract the region of interest including facial key point location information from the image sequence to obtain preprocessed image data. The micro-expression recognition module is used to input preprocessed image data into the micro-expression recognition model for classification and prediction to obtain micro-expression recognition results. The micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence. The feature extraction module includes a spatiotemporal feature extraction sub-module and a regional relationship extraction sub-module set in parallel.

6. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program and the processor runs the computer program to enable the electronic device to perform a micro-expression recognition method based on spatiotemporal facial region relationships according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements a micro-expression recognition method based on spatiotemporal facial region relationships as described in any one of claims 1-4.

Images