A facial micro-expression recognition method

Abstract

The application discloses a face micro-expression recognition method, and relates to the technical field of artificial intelligence. In the spatial branch, the texture features of the face in the peak frame are extracted, the features of the key parts such as eyebrows and corners of the mouth in the texture features are enhanced, and the peak frame feature vector is obtained. In the time branch, the motion features of the face in the optical flow feature map are extracted, the motion features of the eyes and the mouth contained in the motion features are distinguished and enhanced, the optical flow feature vector is obtained, and the peak frame feature vector and the optical flow feature vector are spliced into the reference feature vector. The processing mode of the same structure and different modes realizes the complementary fusion of the multi-modal features in the frequency domain and the time domain. Then, the Mahalanobis distance between the micro-expression reference feature vectors of the target domains before and after alignment is compared, and the recognition result is output. Therefore, the multi-modal features after fusion are further subjected to the source domain and target domain modeling processing, and the micro-expression of the face can be accurately and comprehensively recognized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for facial micro-expression recognition. Background Technology

[0002] Facial microexpressions (ME) are extremely short-lived facial expressions (1 / 25 to 1 / 5 of a second) with minimal muscle movement and no voluntary control. They typically occur when an individual attempts to conceal their true emotions and are impossible to fake. Discovered by Haggard and Isaacs in the 1970s, they were further systematized by Ekman and Friesen in lie detection research, revealing their applications in psychology, criminal investigation, and security. The uncontrollability of microexpressions makes them a key clue for emotion recognition, but their subtlety and transience make them difficult to detect with the naked eye, requiring automated analysis using computer vision technology. Compared to macro-expressions, microexpression recognition constitutes a more complex task due to their fleeting nature and subtle changes in facial areas, demanding unprecedented accuracy from recognition technologies and algorithms. Because they reveal an individual's ability to conceal their true emotions, they are considered a more reliable indicator of emotion than consciously presented macro-expressions.

[0003] Early research on micro-expressions relied on hand-designed features, such as Local Binary Patterns (LBP), optical flow methods (Bi-WOOF, MDMO), etc., to describe micro-expressions through texture or motion information. However, such methods are limited by lighting sensitivity and changes in head posture, and cannot capture deep spatiotemporal correlations. With the popularization of convolutional neural networks (CNN) and recurrent neural networks (RNN), researchers have tried to extract spatiotemporal features jointly through 3D convolutional networks (3DCNN) or combine them with LSTM to model temporal dynamics. However, due to the extremely small size of micro-expression datasets, deep models (such as shallow three-flow 3DCNN) are prone to overfitting and it is difficult to achieve accurate recognition results.

[0004] To address the problems of convolutional neural networks, researchers have proposed LBP-TOP (local binary patterns from three orthogonal planes) network models, spatiotemporal Transformer-based models, and lightweight architecture models. However, these methods typically rely on extracting features from single optical flow or texture features, neglecting the fusion modeling of complementary information in the frequency and temporal domains. This results in a rather one-sided recognition of facial micro-expressions, making it difficult to accurately identify facial micro-expressions. Summary of the Invention

[0005] This invention provides a method for facial micro-expression recognition, which can solve the problem that existing methods usually rely on extracting features from a single optical flow or texture feature, ignoring the fusion modeling and processing of complementary information in the frequency domain and time domain, resulting in a one-sided recognition of facial micro-expressions and difficulty in accurately recognizing facial micro-expressions.

[0006] This invention provides a method for facial micro-expression recognition, comprising the following steps:

[0007] Obtain source domain macro-expression video sequences and target domain micro-expression video sequences of the face;

[0008] For macro-expression video sequences and micro-expression video sequences respectively, the start frame and peak frame of the video sequence are extracted, and the motion vector between the start frame and peak frame is calculated to generate optical flow feature maps;

[0009] The peak frame is spatially downsampled, and the texture features of the face within the peak frame are extracted. The detailed features contained in the texture features are enhanced to obtain a 128-dimensional peak frame feature vector. The optical flow feature map is temporally downsampled, and the motion features of the face within the optical flow feature map are extracted. The motion features of the eyes and mouth contained in the motion features are distinguished and enhanced to obtain a 128-dimensional optical flow feature vector.

[0010] The peak frame feature vector and the optical flow feature vector are concatenated to form a 256-dimensional reference feature vector, and the reference feature vectors for the source domain macro-expression and the target domain micro-expression are obtained.

[0011] Align the feature distributions of the source domain macro-expression and the target domain micro-expression baseline feature vectors to obtain the aligned target domain micro-expression baseline feature vectors.

[0012] Obtain the Mahalanobis distance between the baseline feature vectors of micro-expressions in the target domain before and after alignment, and map the Mahalanobis distance metric to a five-class probability distribution to output the recognition result.

[0013] Preferably, the acquisition of the optical flow feature map includes:

[0014] After acquiring the start and peak frames of the video sequence, the TV-L1 energy functional is used for optical flow estimation. Let u and v represent the horizontal and vertical components, respectively, as follows:

[0015] ;

[0016] Where: dx and dy represent pixel changes along the x and y dimensions, respectively; dt represents time changes;

[0017] Optical strain is expressed as the facial deformation intensity, and is obtained by calculating the derivative of the optical flow. , represented as:

[0018] ;

[0019] Among them: U=[u, v] T Represents the displacement vector;

[0020] Meanwhile, optical strain It can also be expressed as:

[0021] ;

[0022] in:( , () represents the normal strain component; , () represents the shear strain component;

[0023] The optical strain of each pixel is obtained by summing the squares of the normal strain component and the shear strain component. Size, expressed as:

[0024] ;

[0025] u, v, These three optical flow information information, containing micro-expressions, are stacked sequentially to form a three-channel template image X={u, v, ... }, where X is the optical flow feature map.

[0026] Preferably, obtaining the 128-dimensional peak frame feature vector includes:

[0027] A spatiotemporal feature fusion network model is constructed, which includes spatial and temporal branches; each of the spatial and temporal branches includes a cascaded 7×7 convolutional layer, a batch normalization layer (BN), a ReLU activation function, a single-layer MobileViT module, and a BasicBottle residual block.

[0028] The peak frame is input into the spatial branch and spatially downsampled using a 7×7 convolutional layer, a batch normalization layer (BN), and a ReLU activation function. The spatially downsampled peak frame is then input into a single-layer MobileViT module. The single-layer MobileViT module extracts facial texture features using a 3×3 depthwise separable convolution and a 4-head self-attention mechanism. The spatial attention module focuses on key areas of the face, such as eyebrows and corners of the mouth. Two layers of BasicBottle residual blocks enhance the texture features of key areas of eyebrows and corners of the mouth. The enhanced texture features are then flattened and compressed into a 128-dimensional peak frame feature vector through a fully connected layer.

[0029] Preferably, obtaining the 128-dimensional optical flow feature vector includes:

[0030] The optical flow feature map is input into the time branch and spatially downsampled using a 7×7 convolutional layer, a batch normalization layer (BN), and a ReLU activation function. The spatially downsampled optical flow feature map is then input into a single-layer MobileViT module. The single-layer MobileViT module extracts facial motion features using a 3×3 depthwise separable convolution and a 4-head self-attention mechanism. A channel attention module distinguishes between eye and mouth motion features. Two BasicBottle residual block groups enhance the eye and mouth motion features, and the enhanced motion features are flattened and compressed into a 128-dimensional peak frame feature vector through a fully connected layer.

[0031] Preferably, after obtaining the reference feature vector of the source domain macro-expression and the reference feature vector of the target domain micro-expression, the method further includes:

[0032] Based on the Mahalanobis distance metric, which measures the distance between various expression prototypes in the source domain macro-expressions and the target domain micro-expressions, the Mahalanobis distance metric is expressed as:

[0033] ;

[0034] in: x and y The feature vector representing the sample point. T Indicates the transpose operation; This represents the difference vector between two sample points x and y; Indicates the first k The inverse of the covariance matrix of the class prototype; Indicates the first k The covariance matrix of the class prototype is represented as:

[0035] ;

[0036] in: Indicates the first k Covariance matrix between images; This represents the covariance matrix among images of all categories. It is used for balance and The weighting coefficients, This indicates that support for centralized categories belongs to the category. k ; I It is the identity matrix;

[0037] Using the first in the support set k Class Dataset The embedded feature generation is represented as:

[0038] ;

[0039] in: Indicates a feature extractor; and They respectively represent the support for the centralization of the first k Class Dataset The sample points and their corresponding labels; Indicates the current task's... k Class mean vector;

[0040] By calculating the Mahalanobis distance between the query sample and each category prototype, and converting it into category prediction probabilities; using the cross-entropy loss function to model the difference between the prediction results and the true labels, and employing mathematical expectation to average over multiple tasks, the loss function in the source domain is expressed as:

[0041] ;

[0042] in: {( , )}、 These represent the support set and query set on the source domain, respectively. Represents query set medium sample Class distribution; and Representing query sets respectively Samples and labels in the data;

[0043] The training process for the target domain is the same as that for the source domain, the difference being that... Divided into labeled training sets and unlabeled test set The loss function over the target domain is expressed as:

[0044] ;

[0045] in: {( , )}、 These represent the support set and query set on the target domain, respectively. Represents query set medium sample Class distribution; and Representing query sets respectively Samples and labels in the data;

[0046] Based on the loss function in the source domain and the loss function in the target domain, feature distribution density enhancement is performed on the baseline feature vector of macro-expression in the source domain and the baseline feature vector of micro-expression in the target domain, respectively.

[0047] Preferably, aligning the feature distributions of the source domain macro-expression and target domain micro-expression reference feature vectors includes:

[0048] The Local Maximum Mean Difference (LMMD) method is used to measure the difference between the baseline feature vectors of macro-expressions in the source domain and micro-expressions in the target domain after enhancing feature distribution density. Its loss function is... Represented as:

[0049] ;

[0050] in: H It is a regenerating nucleus Hilbert space RKHS. It is the RKHS norm, used to measure distance in a high-dimensional feature space; C This represents the number of emotion categories that are shared by the source and target domains. and These represent the source and target domain datasets, respectively. Indicates the first in the source domain i Feature vectors of each sample; Indicates the first in the target domain j Feature vectors of each sample; and These represent the class weights of the source and target domains, respectively. Represents the feature mapping function;

[0051] By minimizing the loss function Align the similar feature distributions of the source domain macro-expression and target domain micro-expression baseline feature vectors.

[0052] This invention provides a facial micro-expression recognition method, which has the following advantages compared with the prior art:

[0053] In the spatial branch, this invention uses micro-expression peak frames as input to extract facial texture features within the peak frames, and enhances key features such as eyebrows and corners of the mouth within the texture features to obtain peak frame feature vectors. In the temporal branch, it uses optical flow feature maps as input to extract facial motion features within the optical flow feature maps, distinguishes and enhances the motion features of the eyes and mouth contained within the motion features to obtain optical flow feature vectors, and concatenates the peak frame feature vectors and optical flow feature vectors to form a base feature vector. This isomorphic heteromodal processing method realizes the complementary fusion of multimodal features in the frequency and temporal domains.

[0054] Then, the feature distributions of the source domain macro-expression and target domain micro-expression baseline feature vectors are aligned, the Mahalanobis distance between the target domain micro-expression baseline feature vectors before and after alignment is compared, and the recognition result is output. Thus, this process eliminates inter-domain differences by aligning the fused multimodal features and performs source and target domain modeling, which can comprehensively and accurately recognize facial micro-expressions. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall process of a facial micro-expression recognition method provided in an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the face preprocessing process and TV-L1 optical flow feature map extraction of a facial micro-expression recognition method provided in an embodiment of the present invention.

[0057] Figure 3 A schematic diagram of the recognition framework for a facial micro-expression recognition method provided in an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the spatiotemporal feature fusion network model architecture of a facial micro-expression recognition method provided in an embodiment of the present invention;

[0059] Figure 5 A schematic diagram illustrating the source and target domain alignment process of a facial micro-expression recognition method based on LMMD, provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of a prototype metric for covariance based on Mahalanobis distance optimization for a facial micro-expression recognition method provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0062] See Figure 1 and Figure 3 This invention provides a method for facial micro-expression recognition, comprising the following steps:

[0063] Step S1: Obtain macro and micro-expression video sequences, preprocess the face images, extract the start frame and peak frame from the sequence, and generate an optical flow feature map by calculating the motion vector between the two frames using the TV-L1 optical flow algorithm. Figure 2 shown. Specifically:

[0064] Step S11: The source domain uses the CASME^3 and MMEW macro expression datasets, and the target domain uses the CASME II, SAMM, and MMEW micro expression datasets. Face detection and 68-point key point localization are performed using the Face++ API. The face regions are aligned through affine transformation and uniformly cropped to a standardized size of 224×224 pixels to eliminate pose and scale differences.

[0065] Step S12: After the facial expression sequence has undergone preprocessing in step S11, the displacement between the starting frame and the peak frame is calculated; the TV-L1 energy functional is used for optical flow estimation, and u and v represent the horizontal and vertical components, respectively; the calculation can be expressed as:

[0066] .

[0067] Where dx and dy represent pixel variations along the x and y dimensions, respectively, and dt represents time variation. Optical strain can approximate the intensity of facial deformation. It can be obtained by calculating the derivative of the optical flow and can be defined as:

[0068] .

[0069] Where: U = [u, v]T is the displacement vector, therefore It can be represented as:

[0070] .

[0071] Where: in the diagonal strain component, ( , ) is the normal strain component, ( , The shear strain component is the optical strain of each pixel. The optical strain magnitude can be calculated by taking the sum of the squares of the normal strain component and the shear strain component.

[0072] .

[0073] Step S13: Stack the three optical flow information u, v, and ε, which contain rich micro-expression information, in sequence to form a three-channel template image X = {u, v, ε}, which is the required optical flow feature image.

[0074] Step S2: Input the preprocessed peak frames and optical flow maps into the spatiotemporal feature fusion network to extract the temporal and spatial features of macro / micro expressions, and then fuse the extracted features, such as... Figure 4 shown. Specifically:

[0075] Step S21: Input the preprocessed micro-expression peak frames into the spatial branch network. First, spatial downsampling is performed through 7×7 convolutional kernels, followed by batch normalization (BN) and ReLU activation function. Then, it is input into a single-layer lightweight Transformer block (MobileViT) to extract global features through 3×3 depthwise separable convolution and 4-head self-attention mechanism. Next, local features are refined through two layers of basic residual blocks. Finally, the spatial attention module focuses on key facial regions and flattens the features before compressing them into a 128-dimensional feature vector through a fully connected layer.

[0076] Step S22: Input the TV-L1 optical flow feature map into the temporal branch network. Its network structure is completely symmetrical with the spatial branch. It goes through 7×7 convolution downsampling, BN layer, BN-ReLU activation, single-layer MobileViT module, and two layers of BasicBottle residual block group in sequence. The difference is that the channel attention module is used instead of the spatial attention module. This is because the optical flow feature channel dimension carries the physical meaning of different motion modes. Channel attention can effectively distinguish the spatiotemporal synchronicity of micro-expressions of the eyes and mouth. Finally, it also outputs a 128-dimensional feature vector.

[0077] Step S23: Concatenate the 128-dimensional feature vector output from the spatial branch with the 128-dimensional feature vector output from the temporal branch to form a 256-dimensional joint feature.

[0078] Step S24: The final output of 256-dimensional fusion features will serve as the benchmark representation for cross-domain alignment. It will be used for both the calculation of class covariance prototypes based on Mahalanobis distance in meta-learning tasks and the alignment of macro-micro expression distributions achieved through LMMD loss in domain adaptation.

[0079] Step S3: Employing a meta-learning and domain adaptation collaborative optimization framework, feature optimization and cross-domain alignment are achieved through a macro-micro expression alternating dual adjustment strategy, such as... Figure 5 shown. Specifically:

[0080] Step S31: Assume the source domain dataset and the target domain dataset are respectively and , = That is, the source domain and the target domain do not contain the same data; the training process of the source domain can be regarded as several small meta-tasks (episodes); after obtaining the embedding features of the samples, the source domain model is optimized for each episode by calculating the distance between the embedding features of the unlabeled samples and the class prototype of the labeled samples.

[0081] First, from the five fixed categories in the source domain, K labeled samples are extracted from each category to form the support set. ={( , Then, from the remaining samples of the same 5 categories, T unlabeled samples are drawn from each category to form a query set. = {( , )}; medium sample Class distribution Represented as:

[0082] .

[0083] in: To support the collection The first in k Class prototype, Indicates an embedded feature extractor. and Representing query sets respectively The samples and labels in the sample, For Mahalanobis distance metric methods, a prototype metric for covariance based on Mahalanobis distance optimization is as follows: Figure 6 As shown ( (Representing various prototypes) The definition method is as follows:

[0084] .

[0085] in: x and y The feature vector representing the sample point. T Indicates the transpose operation; This represents the difference vector between two sample points x and y; Indicates the first k The inverse of the covariance matrix of the class prototype.

[0086] The covariance matrix of the k-th class is expressed as:

[0087] .

[0088] in: Indicates the first k Covariance matrix between images; This represents the covariance matrix among images of all categories. It is used for balance and The weighting coefficients, This indicates that support for centralized categories belongs to the category. k ; I It is an identity matrix.

[0089] Using the first in the support set k Class Dataset It is generated using the embedded features, and is represented as:

[0090] .

[0091] in: Indicates a feature extractor; and They respectively represent the support for the centralization of the first k Class Dataset The sample points and their corresponding labels; This is the first of the current tasks. k Class mean vector; if = 1, then It will be defined as a suitable zero-size matrix.

[0092] Represented as:

[0093] .

[0094] The distance between the query sample and each category prototype is calculated and converted into category prediction probabilities. Then, the cross-entropy loss function is used to model the difference between the prediction results and the true labels, and the expected value is used to average across multiple tasks, in the following form:

[0095] .

[0096] in: {( , )}、 These represent the support set and query set on the source domain, respectively. Represents query set medium sample Class distribution; and Representing query sets respectively The samples and labels in the text.

[0097] The training process for the target domain is similar to that for the source domain, with the main difference being... Divided into labeled training sets and unlabeled test set The loss function over the target domain is expressed as:

[0098] .

[0099] in: {( , )}、 These represent the support set and query set on the target domain, respectively. Represents query set medium sample Class distribution; and Representing query sets respectively The samples and labels in the text.

[0100] Step S32: Due to significant differences between macro-expressions and micro-expressions in terms of amplitude, duration, and muscle motor unit activation patterns (macro-expressions have large amplitude and duration of 1-5 seconds, while micro-expressions have small amplitude and duration of 0.25-0.5 seconds), models trained on macro-expression datasets are difficult to directly transfer to the micro-expression domain. Therefore, this invention proposes a cross-domain adaptive strategy based on Improved Local Maximum Mean Difference (LMMD). Features of the source and target domain datasets are obtained through a spatiotemporal feature fusion network, and the LMMD is used to measure the inter-domain distribution difference, expressed as:

[0101] .

[0102] in: H It is a regenerating nucleus Hilbert space RKHS. It is the RKHS norm, used to measure distance in a high-dimensional feature space; C This represents the number of emotion categories jointly contained in the source and target domains. and Data sets for the source and target domains; For the first in the source domain i Feature vectors of each sample; For the first in the target domain j Feature vectors of each sample; and Let represent the class weights of the source domain and the target domain, respectively, indicating whether the sample belongs to the _th ... c The confidence score of a class is calculated from the probability output by the classifier; The feature mapping function maps the original features to the reproducing kernel Hilbert space (RKHS); LMMD incorporates the mathematical expectation of each class as a whole, minimizing the class distance between the source and target domains.

[0103] Step S33: The total loss function of the source domain is expressed as:

[0104] .

[0105] Similarly, the total loss function for the target domain is expressed as:

[0106] .

[0107] Step S34: Construct a two-stage task of meta-training and meta-testing. Randomly sample 1000 meta-tasks (episodes) from the CASME^3 and MMEW macro-expression datasets according to the 5-way 5-shot rule. Each episode contains a support set of 5 classes × 5 samples = 25 labeled samples and a query set of 5 classes × 2 samples = 10 unlabeled samples. For the target domain, design an independent experimental verification system for the CASME II, SAMM, and MMEW datasets, construct the target domain task pool in three stages, and generate 1000 cross-domain meta-tasks according to the same rules. During target domain training, reserve all samples of one subject as the meta-test set each time.

[0108] A cross-domain episode hybrid strategy is adopted, with each training batch containing one source domain episode and one target domain episode, forcing the model to learn domain-invariant features. Since the training of the source domain and the target domain is carried out alternately and the training methods are the same, the training method of the source domain is used here to illustrate how class covariance measurement and domain adaptation work. After the support set and the query set are respectively processed by the feature fusion network in step S2, a 256-dimensional feature vector is obtained.

[0109] Step S4: In the cross-domain micro-expression recognition framework, leave-one-out cross-validation (LOSO) is used to perform independent testing in the target domain, completing micro-expression recognition for all target domain datasets. Specifically:

[0110] Step S41: The target domain independent test data uses Leave-One-Subject-OutCross-Validation (LOSO). Each time, all micro-expression samples of one subject are reserved as the test set, and the data of the remaining subjects are used for model training and tuning. This process is repeated until all subjects are validated.

[0111] Step S42: After the target domain test data is aligned and cropped by Face++ API, the peak frame and TV-L1 optical flow features are extracted (consistent with the S1 process) to generate a standardized input sequence.

[0112] Step S43: Load the trained spatiotemporal feature fusion network weights into the testing framework to process the target domain test samples; input the test data into the dual-branch network to extract 128-dimensional fused features, and calculate the Mahalanobis distance between the fused features and the target domain support set prototype based on the class covariance matrix. .

[0113] Step S44: Map the distance metric to five probability distributions using temperature scaling Softmax.

[0114] This invention employs a dual-branch network architecture to achieve spatiotemporal feature fusion, where the temporal and spatial branches share the same network structure. The spatial branch takes micro-expression peak frames as input and captures subtle facial texture changes through the MobileViT module. The temporal branch takes TV-L1 optical flow features as input and models transient dynamic characteristics through the same MobileViT module. The features output from the two branches are dynamically weighted and fused through a channel attention gating mechanism. This isomorphic heteromodal processing method achieves complementary fusion of multimodal features, significantly improving the robustness of feature representation.

[0115] This invention introduces a cross-domain adaptive strategy based on improved maximum mean difference (LMMD). By alternating training in the source domain (macro-expressions) and the target domain (micro-expressions), the feature distributions of the two types of expressions are aligned. Combined with dynamic covariance optimization under the meta-learning framework, the inter-class covariance matrix of the support set and query set is measured by Mahalanobis distance to enhance the clarity of the classification boundary.

[0116] This invention proposes a dual-domain collaborative data partitioning strategy and a LOSO cross-validation mechanism, which can complete cross-domain transfer with only a 5-way 5-shot support set, significantly reducing the dependence on large-scale labeled data. Furthermore, it alleviates the problem of differences in muscle movement amplitude between macro and micro expressions through a dynamic meta-prototype alignment strategy, enabling the model to maintain stability in cross-cultural and cross-subject scenarios.

[0117] This invention efficiently fuses peak frame features and optical flow features through a spatiotemporal feature fusion network, and combines a cross-domain adaptive strategy with dynamic covariance optimization. While reducing the number of parameters, it solves the cross-scene performance bottleneck caused by the static prototype defect of single feature modeling in existing methods, and provides an efficient solution for real-time edge analysis of micro-expression recognition.

[0118] This invention addresses the technical challenges of weak amplitude and poor cross-domain generalization ability in micro-expression recognition through innovative network architecture and optimization strategies. Specifically, it employs a spatial-temporal dual-stream architecture. The spatial branch processes peak frames of micro-expressions, using 7×7 convolutional layers to extract facial texture features. Multi-scale long-range dependencies are modeled using the MobileViT module, and local details are enhanced using BasicBottle residual blocks. A spatial attention mechanism is then used to focus on key facial regions. The temporal branch processes TV-L1 optical flow features, using the same network structure in conjunction with a channel attention module to enhance motion feature representation. For feature optimization, a Class Covariance Prototype (CCM) is constructed using Mahalanobis distance to optimize feature distribution density, while Local Maximum Mean Difference Loss (LMMD) is introduced to align the class-conditional distributions of the source and target domains. Within the meta-learning framework, a few-sample training task is constructed to improve the model's generalization performance under data scarcity conditions. This achieves effective cross-domain adaptive learning, providing an innovative solution for micro-expression recognition.

[0119] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A facial micro-expression recognition method, characterized in that, The method comprises the following steps: obtaining a source domain macro-expression video sequence and a target domain micro-expression video sequence of a face; extracting a starting frame and a peak frame of the video sequence, and calculating a motion vector between the starting frame and the peak frame to generate an optical flow feature map for the macro-expression video sequence and the micro-expression video sequence respectively; performing spatial down-sampling on the peak frame, extracting texture features of the face in the peak frame, performing feature enhancement on the detail features contained in the texture features, and obtaining a 128-dimensional peak frame feature vector; performing temporal down-sampling on the optical flow feature map, extracting motion features of the face in the optical flow feature map, distinguishing and performing feature enhancement on the eye and mouth motion features contained in the motion features, and obtaining a 128-dimensional optical flow feature vector; splicing the peak frame feature vector and the optical flow feature vector into a 256-dimensional reference feature vector to obtain a reference feature vector of the source domain macro-expression and a reference feature vector of the target domain micro-expression; aligning the feature distributions of the reference feature vectors of the source domain macro-expression and the target domain micro-expression to obtain an aligned target domain micro-expression reference feature vector; obtaining a Mahalanobis distance between the target domain micro-expression reference feature vectors before and after alignment, and mapping the Mahalanobis distance metric into a five-class probability distribution to output a recognition result.

2. The facial micro-expression recognition method of claim 1, wherein, The optical flow feature map is obtained by: After obtaining the starting frame and the peak frame of the video sequence, TV-L1 energy functional is selected for optical flow estimation, where u and v represent horizontal and vertical components respectively, and is expressed as: ； where dx and dy represent pixel changes along x and y dimensions respectively, and dt represents a time change; The optical strain is expressed as the intensity of the facial deformation by computing the derivative of the optical flow is expressed as: ； where: U = [u, v] T represents a displacement vector; At the same time, optical strain is also expressed as: ； wherein: , ) denotes the normal strain component; , ) denotes the shear strain component; The optical strain of each pixel is obtained by taking the square sum of the normal strain component and the shear strain component The size is expressed as: ； u, v, The three optical flow information containing micro-expression are stacked in turn to form a three-channel template image X = {u, v, }, X is an optical flow feature map.

3. The facial micro-expression recognition method of claim 1, wherein, The 128-dimensional peak frame feature vector is obtained by: a spatio-temporal feature fusion network model including a spatial branch and a temporal branch is constructed; the spatial branch and the temporal branch both include 7×7 convolution layers, batch normalization layers BN, ReLU activation functions, single-layer MobileViT modules and BasicBottle residual blocks which are sequentially cascaded; the peak frame is input into the spatial branch, and the peak frame is spatially down-sampled through the 7×7 convolution layers, the batch normalization layers BN and the ReLU activation functions; the spatially down-sampled peak frame is input into the single-layer MobileViT module; the single-layer MobileViT module extracts texture features of the face through 3×3 deep separable convolution and 4 heads of self-attention mechanism; the spatial attention module focuses on key areas of eyebrows and corners of the mouth; two BasicBottle residual block groups enhance the texture features of the key areas of the eyebrows and the corners of the mouth, and the enhanced texture features are flattened and compressed into a 128-dimensional peak frame feature vector through a full connection layer.

4. The facial micro-expression recognition method of claim 3, wherein, The 128-dimensional optical flow feature vector is obtained by: The optical flow feature map is input into the time branch, spatial down-sampling of the optical flow feature map is performed through a 7*7 convolution layer, a batch normalization layer BN and a ReLU activation function, the spatial down-sampled optical flow feature map is input into a single-layer MobileViT module, the single-layer MobileViT module extracts the motion features of the face through a 3*3 deep separable convolution and four heads of self-attention mechanism, a channel attention module distinguishes the motion features of the eyes and the mouth, two BasicBottle residual block groups enhance the motion features of the eyes and the mouth, and the enhanced motion features are flattened and compressed into a 128-dimensional peak frame feature vector through a full connection layer.

5. The facial micro-expression recognition method of claim 4, wherein, After the reference feature vectors of the source domain macro-expression and the reference feature vectors of the target domain micro-expression are obtained, the method further comprises: The Mahalanobis distance is used to measure the distance between each type of expression prototype in the source domain macro-expression and the target domain micro-expression, and the Mahalanobis distance measurement is represented as: ； wherein: x and y denotes a feature vector of a sample point, T denotes a transposition operation; denotes a difference vector between two sample points x and y; denotes the inverse of the covariance matrix of the k class prototype; denotes the inverse of the covariance matrix of the k class prototype, denoted as: ； wherein: denotes the covariance matrix between images of class k denotes the covariance matrix between images of class denotes the covariance matrix between images of all classes; is a weight coefficient used to balance and denotes the samples in the support set belonging to class k I is the identity matrix;​​ using the support concentrated first k class data set embedding feature generation, denoted as: ； wherein: represents a feature extractor; and respectively represent sample points and corresponding labels of a support set of a k class data set ; and represents a class mean vector of a current task of a k class. The Mahalanobis distance between the query sample and each category prototype is calculated, and the Mahalanobis distance is converted into a category prediction probability; a cross-entropy loss function is used to model the difference between the prediction result and the true label, and the mathematical expectation is used to average multiple tasks, and the loss function on the source domain is represented as: ； wherein: {( , )}、 denote the support set and the query set on the source domain, respectively; denote the query set ; denote the class distribution of samples in the query set and denote the samples and labels in the query set , respectively. The training process for the target domain is the same as that for the source domain, the difference being the target domain dataset. Divided into labeled training sets and unlabeled test set The loss function over the target domain is expressed as: ； in: {( , )}、 These represent the support set and query set on the target domain, respectively. Represents query set medium sample Class distribution; and Representing query sets respectively Samples and labels in the data; Based on the loss function on the source domain and the loss function on the target domain, the reference feature vectors of the source domain macro-expression and the reference feature vectors of the target domain micro-expression are respectively subjected to feature distribution tightness enhancement.

6. The facial micro-expression recognition method of claim 5, wherein, The feature distribution alignment of the reference feature vectors of the source domain macro-expression and the target domain micro-expression comprises: The difference between the macro-expression and micro-expression benchmark feature vectors of the source domain and the target domain after measuring the tightness of the enhanced feature distribution is measured by using local maximum mean difference (LMMD), and the loss function of LMMD is is represented as: ； wherein: H is a Reproducing Kernel Hilbert Space RKHS, is the RKHS norm, which measures the distance in the high-dimensional feature space; C denotes the number of emotion categories commonly contained in the source domain and the target domain; and denote the source domain and the target domain data set, respectively; denotes the feature vector of the i th sample in the source domain; denotes the feature vector of the j th sample in the target domain; and denote the class weights of the source domain and the target domain, respectively; denotes the feature mapping function; by minimizing a loss function aligning the same class feature distributions of the source domain macro-expression and the target domain micro-expression reference feature vectors.

Images