Micro-expression recognition method, system and device based on space-time facial region relation and medium

Abstract

The invention belongs to the technical field of expression recognition, and discloses a micro-expression recognition method, system and device based on a space-time facial region relation and a medium, and the method comprises the steps: obtaining a micro-expression video image, carrying out the frame sampling processing of the micro-expression video image, and obtaining an image sequence, extracting a region of interest including face key point position information in the image sequence to obtain preprocessed image data; inputting the preprocessed image data into a micro-expression recognition model for classification prediction to obtain a micro-expression recognition result; wherein the micro-expression recognition model comprises a feature extraction module, a feature fusion module and a prediction module which are connected in sequence, and the feature extraction module comprises a spatial-temporal feature extraction sub-module and a regional relationship extraction sub-module which are arranged in parallel. According to the technical scheme, the accuracy and generalization ability of micro-expression recognition are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of facial expression recognition, and particularly relates to a micro-expression recognition method, system, device and medium based on spatio-temporal facial region relationships. Background Art

[0002] Micro-expressions are rapid, subtle, and hardly perceptible facial expressions, usually with an extremely short duration, which can reveal the true emotions or intentions that an individual attempts to hide. The application value of micro-expressions stems from their spontaneity, and their non-actable characteristics enable them to become facial movement features for measuring a person's true emotional expression. Therefore, micro-expression recognition has important application values in fields such as emotion analysis and psychological state assessment. In recent years, with the development of computer vision and machine learning technologies, micro-expression recognition has gradually become a research hotspot.

[0003] The core challenge of micro-expression recognition lies in its extremely small movement amplitude and weak changes, which makes it very difficult to accurately extract micro-expression features from video sequences. The existing micro-expression recognition technologies mainly have the following deficiencies: First, in terms of feature extraction, the micro-expression recognition methods using videos as training inputs have problems of redundant information input and interference from face identity information; although the micro-expression recognition methods using images as inputs have achieved good results, however, the existing research mainly focuses on the extraction of micro-expression spatial features, lacking attention to the differences in the contributions of micro-expressions in different regions of the face; second, the existing technologies fail to fully model the correlations between different regions of the face, resulting in the fragmentation of local and global features and restricting the improvement of recognition accuracy; in addition, the image preprocessing method is simple, unable to effectively highlight the key regions of micro-expressions, and the feature extraction efficiency is low, making it difficult to meet the actual application requirements. Summary of the Invention

[0004] The purpose of the present invention is to provide a micro-expression recognition method, system, device and medium based on spatio-temporal facial region relationships to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above purpose, the present invention provides a micro-expression recognition method based on spatio-temporal facial region relationships, including:

[0006] Obtain micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, and extract regions of interest including the position information of face key points in the image sequence to obtain preprocessed image data;

[0007] Input the preprocessed image data into a micro-expression recognition model for classification prediction to obtain a micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module and a prediction module connected in sequence, and the feature extraction module includes a spatio-temporal feature extraction sub-module and a region relationship extraction sub-module arranged in parallel.

[0008] Optionally, extracting the region of interest including the facial key point position information in the image sequence specifically includes:

[0009] Performing face detection on the starting frame and peak of the image sequence to obtain the facial key point position information;

[0010] Calculating the optical flow information between the starting frame and the peak frame based on the TV-L1 method. The optical flow information includes vertical optical flow, horizontal optical flow, and optical flow strain. Stacking the three single-channel optical flow information to form a three-channel micro-expression motion visual representation, and obtaining the micro-expression optical flow motion image;

[0011] Extracting the region of interest including the facial key point position information from the micro-expression optical flow motion image, and aggregating the extracted regions of interest to obtain the input data of the micro-expression recognition model;

[0012] Repeatedly process each micro-expression video image to construct the preprocessed image data.

[0013] Optionally, the training process of the micro-expression recognition model specifically includes:

[0014] Obtaining a training data set, where the training data includes micro-expression video images and corresponding micro-expression recognition labels;

[0015] Constructing an initial micro-expression recognition model;

[0016] Performing data verification: Selecting the data of one subject in the training data as the test set, and using the data of all the remaining subjects to train the initial micro-expression recognition model. Traversing all the subject data in the training data set, and calculating the average value of all test results to measure the overall performance of the initial micro-expression recognition model;

[0017] After the data verification is completed, evaluating the class distribution balance performance of the initial micro-expression recognition model based on preset metrics, and optimizing the model based on the evaluation results. Among them, the preset metrics include unweighted F1 score and unweighted average recall rate;

[0018] Performing the micro-expression recognition task of the image to be recognized based on the trained micro-expression recognition model.

[0019] Optionally, the processing process of the micro-expression recognition model specifically includes:

[0020] Inputting the preprocessed micro-expression video image into the spatio-temporal feature extraction sub-module and the region relationship extraction sub-module to extract spatio-temporal features and region relationship features, inputting the extracted features into the feature fusion module for feature fusion to obtain the fused features, and inputting the fused features into the prediction module for classification prediction to obtain the corresponding micro-expression recognition result.

[0021] Optionally, the processing process of the spatio-temporal feature extraction sub-module specifically includes:

[0022] Map the micro-expression video image into a high-dimensional feature space through an embedding operation. After the embedding operation is completed, perform an image partitioning operation. Each layer divides the micro-expression video image into multiple small image blocks. Use a Transformer encoder to extract local features from the small image blocks in each layer, and aggregate the extracted local features through a convolution operation to obtain spatio-temporal features.

[0023] Optionally, the processing process of the regional relationship extraction sub-module specifically includes:

[0024] In the regional relationship extraction sub-module, use a vision Transformer model to extract the relationships between facial regions in the micro-expression video image to obtain regional relationship features.

[0025] Optionally, the processing process of the feature fusion module specifically includes:

[0026] Align the output of the regional relationship extraction sub-module with the output of the spatio-temporal feature extraction sub-module through tensor embedding, reshaping, and dimensionality reduction operations. After alignment, fuse the outputs of the two modules through weighted summation as the input of the prediction module.

[0027] A micro-expression recognition system based on spatio-temporal facial region relationships includes:

[0028] A data acquisition module, configured to obtain a micro-expression video image, perform frame sampling processing on the micro-expression video image to obtain an image sequence, and extract an interested region including the position information of facial key points in the image sequence to obtain preprocessed image data;

[0029] A micro-expression recognition module, configured to input the preprocessed image data into a micro-expression recognition model for classification prediction to obtain a micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence, and the feature extraction module includes a spatio-temporal feature extraction sub-module and a regional relationship extraction sub-module arranged in parallel.

[0030] An electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the described micro-expression recognition method based on spatio-temporal facial region relationships.

[0031] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the described micro-expression recognition method based on spatio-temporal facial region relationships.

[0032] The technical effects of the present invention are as follows:

[0033] The spatio-temporal facial region relationship network provided by the present invention fully exploits the local dynamic features and global region relationships of micro-expressions by integrating a spatio-temporal feature extraction module and a region relationship extraction module, effectively enhancing the micro-expression feature expression ability. The SEM module finely extracts the spatio-temporal features of micro-expressions using a multi-level structure, and the REM module models the correlation between facial regions through a vision Transformer. The combination of the two and the integration of information through the feature fusion module significantly improve the accuracy and generalization ability of micro-expression recognition, and perform well in dealing with small-sample datasets and tasks with significant individual differences, providing a new effective solution for the field of micro-expression recognition.

[0034] The STRNet model in this application adopts a parallel structure of a spatio-temporal feature extraction module and a region relationship extraction module, combined with a feature fusion module. It can not only extract the spatio-temporal features of micro-expressions, but also model the correlation between facial regions, enhancing the feature expression ability. Compared with the method of a single feature extraction module, the performance is significantly improved. On the CASMEII dataset, UF1 and UAR reach 0.9764 and 0.9792 respectively, which are better than other advanced methods. On the CASME3 dataset, UF1 and UAR reach 0.5848 and 0.5601 respectively, which is about 2% higher than the sub-optimal method. At the same time, the present invention is trained by the leave-one-subject validation method, which can effectively reduce the influence of individual differences and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0036] The drawings forming a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0037] Figure 1 It is a schematic diagram of the image preprocessing process in the embodiment of the present invention;

[0038] Figure 2 It is a schematic diagram of the STRNet model structure in the embodiment of the present invention;

[0039] Figure 3The analysis result of the confusion matrix for the Cross Datasets Evaluation (CDE) task in the embodiments of the present invention;

[0040] Figure 4 The different sample graphs in the surprised category in the SMIC dataset in the embodiments of the present invention;

[0041] Figure 5 The recognition flow chart in the embodiments of the present invention. Detailed implementation manners

[0042] Now, various exemplary implementation manners of the present invention will be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and implementation manners of the present invention.

[0043] It should be understood that the terms described in the present invention are only for describing specific implementation manners and are not used to limit the present invention. Additionally, for the numerical ranges in the present invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Each intermediate value within any stated value or stated range, as well as each smaller range between any other stated value or intermediate value within the stated range, is also included in the present invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0044] Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Although the present invention only describes preferred methods, any method similar or equivalent to those described herein may also be used in the implementation or testing of the present invention. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In case of conflict with any incorporated document, the content of this specification shall prevail.

[0045] Without departing from the scope or spirit of the present invention, various improvements and changes can be made to the specific implementation manners of the specification of the present invention, which are obvious to those skilled in the art. Other implementation manners obtained from the specification of the present invention are obvious to those skilled in the art. The specification and embodiments of this application are only exemplary.

[0046] Regarding the use of "comprising", "including", "having", "containing", etc. herein, they are all open-ended terms, meaning including but not limited to.

[0047] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine with the embodiments to detail this application.

[0048] Such as Figure 1 -Figure 5 As shown in Figure 5 , in this embodiment, a micro-expression recognition method based on spatiotemporal facial region relationships is provided, including: obtaining a micro-expression video image, performing frame sampling processing on the micro-expression video image to obtain an image sequence, and extracting a region of interest including the position information of facial key points in the image sequence to obtain preprocessed image data; inputting the preprocessed image data into a micro-expression recognition model for classification prediction to obtain a micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence, and the feature extraction module includes a spatiotemporal feature extraction sub-module and a region relationship extraction sub-module arranged in parallel.

[0049] Existing research mainly focuses on the extraction of spatial features of micro-expression motion images, while ignoring the contribution differences of micro-expressions in different regions of the face, which to a certain extent limits the improvement of the accuracy of micro-expression recognition. Therefore, there is an urgent need to explore a research direction that can fuse various micro-expression information and comprehensively use it for emotion classification.

[0050] To solve the above problems, this embodiment proposes a spatiotemporal and region relationship micro-expression recognition model (Spatiotemporal and Region Relationship Network for Micro-expression Recognition, STRNet). Based on the extraction of spatial features of traditional micro-expression motion images, this model introduces the association information between different regions of the face and fuses it with the existing spatial features. Experiments are not only tested on public datasets (SMIC, SAMM, CASME II, and CASME3), but also trained and verified on a self-built food sensory micro-expression dataset. The experimental results show that this model exhibits excellent performance in micro-expression recognition tasks.

[0051] To meet the above requirements, this embodiment will elaborate in detail from three aspects: input of human face data information, processing of facial motion data, and display of sensory recognition results.

[0052] (1) Input of facial data information: In the design of this embodiment, there are two ways of information input: one is to collect data in real time through a camera, and the other is non-real-time input through a video file. Currently, common ordinary cameras on the market generally collect images at a rate of 30 frames per second. However, image processing algorithms are slower in operation compared to sequence processing algorithms, mainly because a single-frame image contains multi-dimensional information such as width, height, and channels. At the same time, the change time of micro-expressions is about 0.5 seconds, and only the movement changes between every 15 frames need to be calculated in 30-frame data. In addition, when showing emotions in real time, the screen may flicker due to frame rate fluctuations, affecting the user experience, which is not as ideal as the overall display effect after recording. Therefore, in the real-time input mode of the camera, the software will record a video after starting the recording and recognition functions and perform algorithm processing at a frequency of every 15 frames; in the video file input mode, the same processing strategy is also adopted.

[0053] (2) Processing of facial motion data: The facial motion data processing module mainly processes the image data containing facial information received by the software and generates a sequence containing the algorithm inference results after processing. After receiving the micro-expression video image, the system will create a new thread to execute the algorithm inference. First, the amount of calculation is significantly reduced through frame sampling operations; the sampled image sequence is then processed according to the preprocessing process of the micro-expression recognition algorithm, that is, first using MTCNN for face detection, then performing affine transformation based on facial feature points to achieve face alignment, and then cropping the face with the inner canthus distance as the unit to extract a pure face sequence. After obtaining the pure face sequence, the first frame of the sampled sequence is used as the starting frame, and the remaining frames are used as peak frames for the calculation of optical flow images. Finally, the generated optical flow image sequence will be used as the input of the MMCN model. It should be noted that to adapt to the actual scenario, neutral data is introduced during the training of MMCN, so there will be no emotion prediction output for frames without obvious emotion changes. Finally, MMCN returns a sequence of predicted emotions.

[0054] (3) Display of sensory recognition results: The display of sensory recognition results mainly focuses on the emotion sequence generated by the background algorithm inference, and its display forms are mainly divided into statistical display and temporal display. The statistical display intuitively presents the percentage of each emotion category in the entire sequence through graphs such as pie charts, thereby reflecting the overall emotion distribution; the temporal display marks various emotions with different colors in the time order of the original emotion sequence, thereby intuitively showing the dynamic change of emotions over time.

[0055] Image preprocessing: Microexpressions are essentially subtle dynamic changes presented by facial movements. For the recognition of microexpression movements, two main methods are usually adopted. One method is to classify based on the complete video sequence, and the other method is to calculate the motion information based on the starting frame and peak frame of the expression. However, due to the extremely small amplitude of microexpression movements and the extremely weak image changes between adjacent frames, there may be a large amount of redundancy in the calculation process.

[0056] In this embodiment, an optical flow calculation method based on the starting frame and peak frame is adopted to describe microexpression movements. Since microexpressions are mainly manifested in local facial regions, the use of Region of Interest (ROI) selection to process microexpression movement images helps to highlight the motion information of key regions, thereby improving the effectiveness of subsequent feature extraction. Considering the above, the HTNet preprocessing method is also referred to. This embodiment combines optical flow processing with Region of Interest (ROI) selection based on facial key point detection, while retaining spatio-temporal information, increasing the density of effective features in the input data. First, for the starting frame and peak frame of the microexpression segment, MTCNN is used for face detection to obtain the position information of facial key points. Secondly, the TV-L1 method is used to calculate the vertical optical flow, horizontal optical flow, and optical flow strain between the starting frame and peak frame. Subsequently, the results of these three single-channel optical flows are stacked to form a three-channel visual representation of microexpression movements.

[0057] Subsequently, the microexpression optical flow movement images are sampled using the facial key point position information (such as the left and right eyebrow regions and the left and right mouth corner regions). Similar to the method adopted by HTNet, after these sampled regions are aggregated, they form the actual input of STRNet. This Region of Interest (ROI) selection method can effectively focus on the high-frequency occurrence regions of microexpressions, providing key original input features for the subsequent classification network. At the same time, this method can significantly filter facial identity information, reducing individual difference factors that may interfere with the recognition task. The complete preprocessing process is as Figure 1 shown.

[0058] In addition, due to the locality and subtlety of microexpression movements, during the data collection process, the subjects may have slight head movements, which may cause the input images to be slightly noisy. Considering this, in this embodiment, the input images are scaled to 28×28×3 during data preprocessing to more effectively distinguish the subtle movement features of microexpressions from possible facial identity information.

[0059] Data validation method: Current micro-expression recognition research is mainly based on relatively small datasets, and there are significant differences in facial features among individuals (i.e., the influence of facial identity information). To reduce the impact of individual identity information on the data distribution and improve the generalization ability of the experimental evaluation metrics for the model across different subjects, this embodiment adopts the Leave-One-Subject-Out (LOSO) method as the model validation method to ensure the robustness and reliability of the model in cross-individual recognition tasks.

[0060] Leave-One-Subject-Out cross-validation is a cross-validation method commonly used in individual-independent tasks and is widely applied in the fields of pattern recognition and machine learning. Its core idea is that in the dataset, each time all the data of one subject is selected as the test set, while the data of all the remaining subjects is used to train the model. This process iterates through all the subjects in the dataset, enabling the data of each subject to be used as the test set once to evaluate the model. Finally, the overall performance of the model is measured by calculating the average of all test results. LOSO cross-validation has several advantages. First, this method can effectively evaluate the individual independence of the model, that is, ensure that the training data and the test data come from different individuals, thus avoiding the model over-relying on individual features and resulting in overfitting. Second, in the case of a small dataset scale, LOSO can maximize the utilization rate of the training data and performs more stably in small-sample scenarios compared to K-Fold Cross Validation. In addition, since each test set consists of a complete single individual, LOSO can more accurately evaluate the generalization ability of the model on unseen individuals and is particularly suitable for tasks with significant individual differences.

[0061] In micro-expression recognition tasks, due to the significant differences in individual facial features, the expression patterns of different subjects may also show significant variations. Therefore, adopting LOSO cross-validation can reduce the interference of individual identity information on model learning, ensuring that the model focuses on the common features of micro-expressions rather than individual-specific features. At the same time, this method can evaluate the adaptability of the model on unseen subjects, which is of great value for the application of micro-expression analysis in actual scenarios (such as emotion detection, etc.).

[0062] Data evaluation metrics: After completing the label integration of the dataset, the research faces the problem of unbalanced class distribution. To solve this problem, this embodiment adopts the following two evaluation metrics: Unweighted F1-score (UF1), which is used to comprehensively evaluate the overall balance of the classification model; Unweighted Average Recall (UAR), which is used to evaluate the recall performance of the model on each class. The specific calculation formulas of the above metrics are shown in Formulas 1 to 4.

[0063]

[0064]

[0065]

[0066] In the formula, E represents the total number of classifications, and k represents the number of experimental subjects. and represent the true positive, false negative, and false positive of object β in category α.

[0067] In this embodiment, the model was generally trained for 800 rounds and verified using the leave-one-subject cross-validation method. The learning rate of the optimizer in the experiment was set to 5e-5, the batch size was 256, and the cross-entropy loss function was used. All experiments were completed on the NVIDIA GeForce RTX 2080ti hardware platform.

[0068] Spatio-temporal Facial Region Relationship Micro-expression Recognition Model: To balance effective micro-expression feature extraction and the modeling of relationships between different facial regions, this embodiment proposes STRNet. STRNet mainly consists of two branches and a feature fusion model: namely, the spatio-temporal feature extraction module (SEM), the region relationship extraction module (REM), and the feature fusion module (FFM). The SEM adopts a multi-level architecture to extract micro-expression features across multiple scales. The processed results are unified in the channel dimension to obtain refined micro-expression features while reducing the interference of irrelevant information. The REM aims to model the relationships between different facial regions from a more macroscopic scale. Since these two branches process information using different strategies, a dedicated module is needed to integrate these two types of information and minimize the loss of feature information. The FFM aims to fuse the features extracted by the SEM and REM, thereby enhancing the generalization ability of the model. Compared with structures that only extract spatial information, this module can obtain richer information. The final result is used as the input to the MLP layer. The MLP layer consists of a LayerNorm (LN) layer, a Reduce layer, and a Linear layer. The overall architecture of the network is as Figure 2 shown.

[0069] Spatio-temporal Feature Extraction Module (SEM): After the image is preprocessed, the SEM branch is responsible for extracting the detailed motion features of micro-expressions. The backbone of this module uses the feature extractor from HTNet. HTNet has significant advantages in feature extraction because it adopts a multi-level structure during the feature extraction process, as shown in Equations 5 and 6:

[0070]

[0071] k = floor(h mod 2) (6)

[0072] Among them, represents the micro-expression feature, where k represents the number of partitions applied to the original input image, and f a (.) represents the convolutional operation used to aggregate the images, and t n represents the n-layer Transformer encoder. In this embodiment, the best-performing parameter settings in HTNet are adopted. floor(.) represents the floor function, and h represents the width of the input image. In this network, the height and width of the input image are set to be equal.

[0073] The input image undergoes an embedding operation before partitioning. After performing the image partitioning operation, the Transformer encoder is used at each layer to extract features from the smaller image patches, and then the convolutional layer is used to aggregate these image patches. Then, SEM models the optical flow features of the micro-expression from details to the overall structure. The fully aggregated features effectively model the spatio-temporal features of the micro-expression.

[0074] Region Relationship Extraction Module (REM): The REM branch is mainly responsible for modeling the relative macroscopic relationships of the micro-expression motion images. This network uses the Vision Transformer (ViT) to extract the relationships between facial regions. ViT is the practical application of the Transformer architecture in image processing. As mentioned above, the spatial dimension of the input image has been compressed. To prevent the loss of the inherent features of the micro-expression, in this embodiment, the image patch size is set to 1. Its advantage is that each pixel can be independently used as a token, thus more precisely modeling the relationships between different regions of the face. The patch-based method in this module includes multiple image patches within the potential facial motion region. Therefore, the output result of the REM branch is coarser than that of the SEM branch, but their focuses are different.

[0075] Feature Fusion Module (FFM): The SEM and REM methods extract the features of the micro-expression optical flow images from different perspectives. The multi-level Transformer architecture in SEM effectively captures the temporal and spatial features of the micro-expression. REM models the features through the scaled micro-expression optical flow images and captures the correlations between different facial regions. The architectures of these two models are not unified, which may lead to uneven distributions of the output feature tensors. Therefore, the FFM module is proposed to solve this problem. This module helps to enhance the generalization ability of the micro-expression features extracted by the network, and the calculation process is shown in Equations 7 and 8.

[0076] I V = f dim (f emb (I V0 )) (7)

[0077] I = α · IS +(1 - α)·I V (8)

[0078] Where α represents the weight coefficient of weighted summation in the fusion module mechanism, and is finally set to 0.7 through experiments. I S and I V respectively represent the facial motion space features processed by SEM and the facial region features processed by REM. I V0 represents the direct output of REM, while f dim and f emb respectively refer to the embedding reshaping and dimensionality reduction operations applied to I V0 to align it with the I S features.

[0079] First, as shown in Equation 7, the output I V0 of the REM branch undergoes tensor embedding processing, referring to the method actually used in the SEM branch. Next, the embedded tensor is reshaped through a lossless rearrangement layer to modify the data structure. Referring to the shape of the SEM result, the width and height information are aggregated into the channel dimension. Subsequently, a 2D convolution operation with a kernel size of 1 is applied to scale the channel dimension to align it with the shape of the SEM result. Finally, as shown in Equation 8, the results of the two branches are fused through weighted summation. The fused features are then passed through the MLP to obtain the final classification result.

[0080] Ablation experiment:

[0081] To verify the contributions of each component in STRNet, this embodiment conducts ablation experiments on the Cross - Datasets Evaluation (CDE) task. The ablation experiments mainly focus on evaluating the independent contributions of the main components of the model. The experimental results are shown in Table 1.

[0082] Table 1 Ablation experiment results

[0083]

[0084] REM only focuses on modeling different local facial regions, resulting in the extracted features lacking detailed information. To extract overall feature information, REM further compresses the spatial features of the input image. Therefore, the features extracted by REM are more general, leading to relatively lower performance on SMIC, CASMEII, and SAMM. In qualitative analysis, REM's performance on SMIC and SAMM is similar. REM performs slightly better on CASMEII, which may be due to the fact that the CASMEII data is more homogeneous in racial classification, resulting in smaller intra-class feature variations. Using only the SEM branch compared to the structure using only REM has significantly improved the overall performance. Specifically, on the SMIC dataset, the network has increased by 20% on UF1 and UAR respectively. On the CASMEII dataset, the UF1 and UAR values have increased by approximately 30%. For the SAMM dataset, UF1 has increased by about 20%, while UAR has increased by 24%. One reason for the superior performance of using SEM independently is that its backbone network is specifically designed to extract the spatial features of micro-expressions. Different from REM, SEM does not further compress the input features, thus being able to capture the inherent spatial features of micro-expressions.

[0085] When both branches and FFM are included, the performance of the model is further improved based on SEM. On the CASMEII dataset, UF1 and UAR have increased by nearly 3% respectively, reaching 0.9764 and 0.9792. On the SAMM dataset, UF1 has increased by 6% and UAR has increased by 7%, reaching 0.8754 and 0.8862 respectively. The performance on the SMIC dataset is similar to that of the SEM branch. The experimental results show that when the two modules coexist and (α = 0.7), the overall UF1 performance reaches the best. The complete STRNet combines the association information of different facial regions generalized by REM with the spatial features of micro-expressions extracted by SEM, thus capturing micro-expression features more comprehensively and further improving the performance.

[0086] Experiments on public datasets:

[0087] (1) Experimental results of the mixed dataset:

[0088] Figure 3(a)-(c) of show the results of the confusion matrix analysis for the Cross Datasets Evaluation (CDE) task. The experimental results indicate that the network demonstrates optimal performance in negative emotion classification. Specifically, in the CASMEII dataset, the classification accuracy for each category is significantly higher, with only a small number of positive samples misjudged as surprised. However, in the SAMM and SMIC datasets, both surprised and positive emotions are misjudged as negative emotions. This misjudgment phenomenon may be attributed to the relatively low complexity of the characteristics of negative emotions and the relatively high proportion of negative emotion samples in the datasets. In addition, the difference in the misclassification rate between different datasets may be closely related to the heterogeneity of the dataset feature distributions. It is worth noting that the age distribution of volunteers in the CASMEII dataset is relatively concentrated, and the racial composition is relatively homogeneous; while the SAMM and SMIC datasets cover volunteers of diverse races and age groups, with a higher balance in their racial and age distributions. This difference may further affect the classification performance of the model. To evaluate the effectiveness of the proposed method, experiments were first conducted on the cross-dataset evaluation task. The table summarizes the performance of the entire network in the CDE three-classification task. The table also includes the evaluation results of other competitive and classical methods on the CDE task for comparison. The comparison methods include traditional methods such as LBP-TOP and Bi-WOOF, classical deep learning-based methods such as OFF-ApexNet and STSTNet, and current state-of-the-art methods such as BDCNN+I_w+GA and HTNet. Since most models in this field are not fully open-source, the comparison data are directly from their respective papers.

[0089] Table 2 Performance Comparison on CDE Task

[0090]

[0091] Overall, as shown in Table 2, STRNet performs superiorly in both the UF1 and UAR metrics. Specifically, traditional methods such as LBP-TOP and Bi-WOOF are used to capture motion features. The UF1 and UAR scores of both methods on the SAMM dataset do not exceed 0.5211, and on the SMIC dataset, they do not exceed 0.5829. On the CASMEII dataset, these two metrics do not exceed 0.8026. The overall performance of these two methods is relatively poor. One possible reason for the poor performance is that the subtle features of micro-expression movements may be interfered with by other information (such as identity-related features) during the motion calculation process. Each micro-expression segment introduces different degrees of noise, highlighting the lack of generalization ability of these methods.

[0092] Regarding deep learning methods, OFF-ApexNet and STSTNet are classical methods in this field. The performance of OFF-ApexNet on the SMIC dataset is relatively low, with UF1 and UAR reaching 0.6817 and 0.6695 respectively. Its performance on the SAMM dataset is still poor, with UF1 and UAR scores of 0.5409 and 0.5392 respectively. Regarding deep learning methods, OFF-ApexNet and STSTNet are classical methods in this field. The performance of OFF-ApexNet on the SMIC dataset is relatively low, with UF1 and UAR reaching 0.6817 and 0.6695 respectively. Its performance on the SAMM dataset is still poor, with UF1 and UAR scores of 0.5409 and 0.5392 respectively. STSTNet performs slightly better than OFF-ApexNet on the SMIC dataset, with UF1 and UAR reaching 0.6801 and 0.7013 respectively. This indicates that STSTNet may have successfully improved the classification accuracy of microexpressions through certain improvements, especially in feature extraction and noise suppression. However, this improvement still appears limited, possibly related to the data noise and complexity in the SMIC dataset. The performance of STSTNet on the SAMM and CASMEII datasets is 0.6588 and 0.6810 (UF1 and UAR), and 0.8382 and 0.8686 (UF1 and UAR) respectively. These results show a performance pattern similar to that of OFF-ApexNet. Although both of these networks adopt deep learning methods and use the optical flow images of microexpressions as inputs, their recognition performance is still limited. This may be because the structures of these two models are relatively simple, resulting in weak feature extraction capabilities and difficulty in effectively capturing the subtle changes of microexpressions. Therefore, despite the adoption of advanced image input forms, the performance of the models is still limited to a certain extent.

[0093] BDCNN achieved UF1 and UAR values of 0.7859 and 0.7869 respectively on the SMIC dataset, 0.8186 and 0.8124 on the SAMM dataset, and 0.9501 and 0.9516 on the CASMEII dataset. This indicates that BDCNN significantly outperforms previous classical methods on multiple datasets, especially achieving significant improvements in multiple metrics across different datasets. Compared with STSTNet, BDCNN increased the UF1 value by 10%, 12%, and 15% on the SMIC, CASMEII, and SAMM datasets respectively. This improvement may be attributed to BDCNN adopting a more complex and efficient network architecture for feature extraction, enabling it to better capture the detailed changes and global features in microexpressions. HTNet is another recently effective method that uses a multi-layer Transformer architecture to extract microexpression features. On the SMIC dataset, its UF1 and UAR are 0.8049 and 0.7905 respectively. On the CASMEII dataset, these two metric values are 0.9532 and 0.9516 respectively. On the SAMM dataset, they are 0.8131 and 0.8124 respectively. Overall, HTNet shows a slight improvement compared to BDCNN in all metrics.

[0094] The STRNet proposed in this embodiment achieved further improvement on this basis. On the CASMEII dataset, the UF1 score of STRNet increased by nearly 3% compared to the second-place HTNet, reaching 0.9792. The UAR value also increased by approximately 2%, reaching 0.9764. On the SAMM dataset, these two metrics increased by 6% and 7% respectively, reaching 0.8754 and 0.8862. On the SMIC dataset, the performance was comparable. The differences in numerical performance may be related to the richness of intra-class features (CASMEII and SAMM are integrations of multiple classes). Overall, compared with the second-best method, STRNet showed a significant improvement. The results of STRNet vary across different datasets. Generally speaking, STRNet performs better on the CASMEII and SAMM datasets than on the SMIC dataset. The differences in data distribution among different classes in different datasets may be the potential reason for this problem. Another possible reason is that the recording frame rate of the SMIC dataset is lower (100fps, while the other datasets are 200fps), so it is more vulnerable to various noise factors.

[0095] Other images of the surprise category in the SMIC dataset (such as Figure 4Compared with (as shown), the differences among different samples are relatively large. This indicates that smaller inter-class differences and potentially larger intra-class differences may be challenges in the micro-expression recognition task. Action interference on the face other than expressions is the main reason for the large intra-class differences. However, STRNet can still distinguish some optical flow feature maps with the smallest differences, demonstrating the effectiveness of the proposed network in extracting micro-expression features.

[0096] (2) Experimental results on CASME3: To evaluate the performance of the model on a larger dataset, the experiment was also tested on CASME3. The comparison methods include the classic deep learning micro-expression recognition algorithm STSTNet and HTNet. Since the experimental results of BDCNN on the CASME3 dataset were not reported in the original paper, it was not used as a comparison method. The overall results are shown in Table 3.

[0097] Table 3 Comparison of Experimental Results on CASME3

[0098]

[0099] The performance of STSTNet on CASME3 is 0.3795 (UF1) and 0.3792 (UAR). The relatively poor performance of STSTNet on the CASME3 dataset may be attributed to the simplicity of its structure, which cannot fully handle the complexity of a larger dataset. HTNet achieved good performance on the CASME3 dataset, reaching the metric values of 0.5767 and 0.5415 respectively. STRNet further improved the performance on the CASME3 dataset, increasing by approximately 1% and 2% respectively, reaching the performance values of 0.5848 and 0.5601. This indicates that the STRNet network shows stronger generalization ability on a larger dataset. The performance of STRNet on the CASME3 dataset is generally lower than its performance in cross-dataset evaluation. A possible reason is that CASME3 itself contains more subjects and experimental data, making the training process more challenging. In addition, different from the 5 categories provided by the original CASMEII and SAMM datasets, the CASME3 dataset includes 7 categories. This results in a more complex feature distribution in the CASME3 dataset after classification compression.

[0100] To effectively fuse the spatial information of micro-expression motion images and the facial region relationship information, and construct a micro-expression recognition algorithm that can achieve excellent performance on both the self-built video sensory dataset and the public dataset, this embodiment proposes a spatio-temporal facial region relationship network (STRNet). The model aims to fully explore the local dynamic features and global region relationships of micro-expressions, enhance the micro-expression feature expression ability, and improve the recognition accuracy and generalization ability.

[0101] In this network, a spatio-temporal feature extraction module with a multi-level structure is set up to gradually extract multi-scale features of micro-expressions. Meanwhile, a regional relationship extraction module is set up to model the relationships between different local facial regions. Finally, a special feature fusion module is designed to effectively align and integrate the regional relationship information with the micro-expression spatial features, which enhances the generalization ability of the micro-expression features extracted by the network. Through multi-branch fusion, this overall architecture can extract more comprehensive information.

[0102] In the experiments on the mixed dataset, the UF1 score of the model on the CASMEII dataset reached 0.9764, and the UAR score also reached 0.9792, achieving the state-of-the-art performance level. On the SAMM dataset, the metric scores were 0.8754 and 0.8862 respectively. On the large-scale CASME3 dataset, the model achieved scores of 0.5848 and 0.5601 respectively, with a 2% improvement in the UF1 value compared to the sub-optimal method. The experimental results show that STRNet has achieved remarkable performance.

[0103] Implementable, this embodiment also provides a micro-expression recognition system based on spatio-temporal facial region relationships, including:

[0104] A data acquisition module, which is used to obtain micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, and extract a region of interest including the position information of facial key points in the image sequence to obtain preprocessed image data;

[0105] A micro-expression recognition module, which is used to input the preprocessed image data into a micro-expression recognition model for classification prediction to obtain a micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence, and the feature extraction module includes a spatio-temporal feature extraction sub-module and a regional relationship extraction sub-module arranged in parallel.

[0106] Implementable, this embodiment also provides an electronic device, including a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the described micro-expression recognition method based on spatio-temporal facial region relationships.

[0107] Implementable, this embodiment also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the described micro-expression recognition method based on spatio-temporal facial region relationships.

[0108] As described above, the above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A micro-expression recognition method based on spatio-temporal facial region relationships, characterized in that, Including: Obtain a micro-expression video image, perform frame sampling processing on the micro-expression video image to obtain an image sequence, extract a region of interest including facial key point position information from the image sequence to obtain preprocessed image data; Input the preprocessed image data into a micro-expression recognition model for classification prediction to obtain a micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence, and the feature extraction module includes a spatio-temporal feature extraction sub-module and a regional relationship extraction sub-module arranged in parallel.

2. The micro-expression recognition method based on spatio-temporal facial region relationship according to claim 1, wherein The extracting a region of interest including facial key point position information from the image sequence specifically includes: Perform face detection on the starting frame and peak of the image sequence to obtain facial key point position information; Calculate the optical flow information between the starting frame and the peak frame based on the TV-L1 method, the optical flow information includes vertical optical flow, horizontal optical flow, and optical flow strain, stack the three single-channel optical flow information to form a three-channel micro-expression motion visual representation, and obtain a micro-expression optical flow motion image; Extract a region of interest including facial key point position information from the micro-expression optical flow motion image, and aggregate the extracted regions of interest to obtain the input data of the micro-expression recognition model; Repeat the processing of each micro-expression video image to construct preprocessed image data.

3. A micro-expression recognition method based on spatio-temporal facial region relationships according to claim 1, characterized in that, The training process of the micro-expression recognition model specifically includes: Obtain a training data set, the training data includes micro-expression video images and corresponding micro-expression recognition labels; Construct an initial micro-expression recognition model; Perform data verification: select the data of one subject in the training data as a test set, and use the data of all the remaining subjects to train the initial micro-expression recognition model. Traverse all the subject data in the training data set, and calculate the average value of all test results to measure the overall performance of the initial micro-expression recognition model; After the data verification is completed, evaluate the class distribution balance performance of the initial micro-expression recognition model based on a preset index, and optimize the model based on the evaluation result, wherein the preset index includes an unweighted F1 score and an unweighted average recall rate; Perform the micro-expression recognition task of the image to be recognized based on the trained micro-expression recognition model.

4. A micro-expression recognition method based on spatio-temporal facial region relationships according to claim 1, characterized in that, The processing process of the micro-expression recognition model specifically includes: Input the preprocessed micro-expression video image into the spatio-temporal feature extraction sub-module and the regional relationship extraction sub-module to extract spatio-temporal features and regional relationship features, input the extracted features into the feature fusion module for feature fusion to obtain fused features, and input the fused features into the prediction module for classification prediction to obtain the corresponding micro-expression recognition result.

5. A micro-expression recognition method based on spatio-temporal facial region relationship according to claim 4, characterized in that, The processing process of the spatio-temporal feature extraction sub-module specifically includes: Map the micro-expression video image to a high-dimensional feature space through an embedding operation. After the embedding operation is completed, perform an image partitioning operation. Each layer divides the micro-expression video image into multiple small block images. Use a Transformer encoder to extract local features from the small block images at each layer, and aggregate the extracted local features through a convolution operation to obtain spatio-temporal features.

6. The micro-expression recognition method based on spatio-temporal facial region relationship according to claim 4, characterized in that, The processing process of the regional relationship extraction sub-module specifically includes: In the regional relationship extraction sub-module, the relationship between facial regions in the micro-expression video image is extracted by using a vision Transformer model to obtain regional relationship features.

7. A micro-expression recognition method based on spatio-temporal facial region relationships according to claim 4, characterized in that The processing process of the feature fusion module specifically includes: The output of the regional relationship extraction sub-module is aligned with the output of the spatio-temporal feature extraction sub-module through tensor embedding, reshaping, and dimensionality reduction operations. After alignment, the outputs of the two modules are fused by weighted summation as the input of the prediction module.

8. A micro-expression recognition system based on spatio-temporal facial region relationships, characterized in that, It includes: A data acquisition module, configured to obtain micro-expression video images, perform frame sampling processing on the micro-expression video images to obtain an image sequence, and extract regions of interest including face key point position information from the image sequence to obtain preprocessed image data; A micro-expression recognition module, configured to input the preprocessed image data into a micro-expression recognition model for classification prediction to obtain a micro-expression recognition result; wherein, the micro-expression recognition model includes a feature extraction module, a feature fusion module, and a prediction module connected in sequence, and the feature extraction module includes a spatio-temporal feature extraction sub-module and a regional relationship extraction sub-module arranged in parallel.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a micro-expression recognition method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by the processor, it implements a micro-expression recognition method according to any one of claims 1-7.

Images