Facial micro-expression recognition method

Abstract

The invention discloses a face micro-expression recognition method, and relates to the technical field of artificial intelligence, and the method comprises the steps: extracting the texture features of a face in a peak frame on a spatial branch, carrying out the feature enhancement of the features of key parts, such as eyebrows, mouth corners and the like, in the texture features, and obtaining a peak frame feature vector; on the time branch, the motion features of the face in the optical flow feature map are extracted, the eye and mouth motion features contained in the motion features are distinguished and subjected to feature enhancement, an optical flow feature vector is obtained, the peak frame feature vector and the optical flow feature vector are spliced into a reference feature vector, and the isomorphic and different-mode processing mode is adopted, so that the accuracy of the processing is improved. Complementary fusion of multi-modal features in a frequency domain and a time domain is realized; and then the mahalanobis distances between the target domain micro-expression reference feature vectors before and after alignment are compared, and the recognition result is output, so that source domain and target domain modeling processing is further performed on the fused multi-modal features in the process, and the micro-expression of the face can be comprehensively and accurately recognized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a facial micro-expression recognition method. BACKGROUND

[0002] Micro-expression (ME) of face is a facial expression with extremely short duration (1 / 25 to 1 / 5 seconds), small muscle movement amplitude and uncontrollable, which usually occurs when an individual tries to hide the true emotion, and has the characteristic of being unforgeable. After being discovered by Haggard and Isaacs in the 1970s, micro-expression was further systematized by Ekman and Friesen in lie detection research, and its application value in the fields of psychology, criminal investigation and security was revealed. The uncontrollability of micro-expression makes it a key clue for emotion recognition, but its weakness and shortness make it difficult to capture with the naked eye, and it needs to rely on computer vision technology for automatic analysis. Compared with macro-expression, the recognition of micro-expression constitutes a more complex task due to its fleeting nature and weak facial changes, which requires unprecedented accuracy of recognition technology and algorithm. Due to its ability to reveal the individual's true emotions, micro-expression is considered a more reliable emotional indicator than consciously presented macro-expression.

[0003] Early research on micro-expression relied on hand-designed features such as local binary patterns (LBP), optical flow (Bi-WOOF, MDMO), which described micro-expression through texture or motion information. However, such methods are limited by light sensitivity and head pose changes, and cannot capture deep temporal and spatial correlations. With the popularity of convolutional neural networks (CNN) and recurrent neural networks (RNN), researchers have tried to jointly extract spatio-temporal features through 3D convolutional networks (3DCNN) or combine LSTM to model temporal dynamics. However, due to the extremely small size of the micro-expression dataset, deep models such as shallow three-stream 3DCNN are prone to overfitting, making it difficult to achieve accurate recognition results.

[0004] To address the problems of convolutional neural networks, researchers at this stage have proposed LBP-TOP (local binary patterns from three orthogonal planes) network models, models based on spatio-temporal Transformer, and lightweight architecture models. However, such methods usually rely on feature extraction based on a single optical flow or texture feature, ignoring the fusion modeling of complementary information in the frequency domain and time domain, resulting in a more one-sided recognition of facial micro-expression and difficulty in accurately recognizing facial micro-expression. SUMMARY

[0005] The embodiment of the present application provides a facial micro-expression recognition method, which can solve the problem that the existing method usually depends on the extraction of features under single optical flow or texture features, ignores the fusion modeling processing of complementary information of features in the frequency domain and the time domain, and thus cannot accurately recognize facial micro-expressions.

[0006] The embodiment of the present application provides a facial micro-expression recognition method, which comprises the following steps: Obtaining a source domain macro-expression video sequence and a target domain micro-expression video sequence of a face; Respectively for the macro-expression video sequence and the micro-expression video sequence, extracting a starting frame and a peak value frame of the video sequence, and calculating a motion vector between the starting frame and the peak value frame to generate an optical flow feature map; Performing spatial down-sampling on the peak value frame, extracting texture features of the face in the peak value frame, performing feature enhancement on detailed features contained in the texture features, and obtaining a 128-dimensional peak value frame feature vector; performing time down-sampling on the optical flow feature map, extracting motion features of the face in the optical flow feature map, distinguishing and performing feature enhancement on eye and mouth motion features contained in the motion features, and obtaining a 128-dimensional optical flow feature vector; Splicing the peak value frame feature vector and the optical flow feature vector into a 256-dimensional reference feature vector, and obtaining a reference feature vector of the source domain macro-expression and a reference feature vector of the target domain micro-expression; Aligning the feature distribution of the reference feature vectors of the source domain macro-expression and the target domain micro-expression, and obtaining an aligned target domain micro-expression reference feature vector; Obtaining the Mahalanobis distance between the target domain micro-expression reference feature vectors before and after alignment, mapping the Mahalanobis distance measurement into a five-class probability distribution, and outputting a recognition result.

[0007] Preferably, the optical flow feature map is obtained by: After the starting frame and the peak value frame of the video sequence are obtained, TV-L1 energy functional is selected for optical flow estimation, wherein u and v represent horizontal and vertical components respectively, and is expressed as: ; Wherein: dx and dy represent pixel changes along x and y dimensions respectively; dt represents time change; The optical strain is represented as the deformation intensity of the face, and the optical strain is obtained by calculating the derivative of the optical flow , which is expressed as: ; Wherein: U=[u,v] T represents a displacement vector; Meanwhile, the optical strain is also expressed as: ; wherein: (1) represents the normal strain component; , ) represents the shear strain component; , ; The optical strain of each pixel is obtained by taking the square sum of the normal strain component and the shear strain component The size is represented as: ; The u, v, The three optical flow information containing micro-expression are stacked in sequence to form a three-channel template image X = {u, v, }, and X is an optical flow feature map.

[0008] Preferably, the 128-dimensional peak frame feature vector is obtained, including: The spatial-temporal feature fusion network model including a spatial branch and a temporal branch is constructed; the spatial branch and the temporal branch both include a 7*7 convolution layer, a batch normalization layer BN, a ReLU activation function, a single-layer MobileViT module and a BasicBottle residual block which are sequentially cascaded; The peak frame is input into the spatial branch, and the peak frame is spatially down-sampled through the 7*7 convolution layer, the batch normalization layer BN and the ReLU activation function; the spatially down-sampled peak frame is input into the single-layer MobileViT module; the single-layer MobileViT module extracts the texture features of the face through a 3*3 depth separable convolution and a 4-head self-attention mechanism; the spatial attention module focuses on the key areas of the eyebrows and the corners of the mouth; two BasicBottle residual block groups enhance the texture features of the key areas of the eyebrows and the corners of the mouth, and the enhanced texture features are flattened and compressed into a 128-dimensional peak frame feature vector through a full connection layer.

[0009] Preferably, the 128-dimensional peak frame feature vector is obtained, including: The optical flow feature map is input into the temporal branch, and the optical flow feature map is spatially down-sampled through the 7*7 convolution layer, the batch normalization layer BN and the ReLU activation function; the spatially down-sampled optical flow feature map is input into the single-layer MobileViT module; the single-layer MobileViT module extracts the motion features of the face through a 3*3 depth separable convolution and a 4-head self-attention mechanism; the channel attention module distinguishes the motion features of the eyes and the mouth; two BasicBottle residual block groups enhance the motion features of the eyes and the mouth, and the enhanced motion features are flattened and compressed into a 128-dimensional peak frame feature vector through a full connection layer.

[0010] Preferably, after obtaining the reference feature vector of the source domain macro expression and the reference feature vector of the target domain micro expression, the method further includes: The distance between the source domain macro-expressions and the target domain micro-expressions is measured based on the Mahalanobis distance. The Mahalanobis distance metric is expressed as: ; in: x and y represents the feature vector of the sample point, T Represents a transpose operation; Represents the difference vector between two sample points x and y; Indicates the k The inverse matrix of the covariance matrix of the class prototype; Indicates the k The covariance matrix of the class prototype is expressed as: ; in: Indicates the k Covariance matrix between images; Represents the covariance matrix between images of all categories; It is used to balance and The weight coefficient of Indicates that the support concentration belongs to the category k samples; I is the identity matrix; Use the first k Class dataset The embedding feature generation is expressed as: ; in: represents a feature extractor; and Respectively support the concentration k Class dataset Sample points and corresponding labels; Indicates the current task k class mean vector; By calculating the Mahalanobis distance between the query sample and each category prototype and converting it into category prediction probability, the cross entropy loss function is used to model the difference between the prediction result and the true label, and mathematical expectation is used to average multiple tasks. The loss function on the source domain is expressed as: ; in: {( , )}、 denote the support set and query set on the source domain respectively; Represents a query set Medium sample Class distribution of and Represents query sets respectively Samples and labels in ; The training process of the target domain is the same as that of the source domain, except that Divide into labeled training sets and an unlabeled test set ; The loss function on the target domain is expressed as: ; in: {( , )}、 denote the support set and query set on the target domain respectively; Represents a query set Medium sample Class distribution of and Represents query sets respectively Samples and labels in ; Based on the loss function on the source domain and the loss function on the target domain, the feature distribution density of the baseline feature vector of the source domain macro expression and the baseline feature vector of the target domain micro expression are enhanced respectively.

[0011] Preferably, the step of aligning the feature distributions of the source domain macro-expression and the target domain micro-expression baseline feature vectors includes: The local maximum mean difference (LMMD) is used to measure the difference between the source domain macro expression and the target domain micro expression baseline feature vector after enhancing the density of feature distribution. The loss function Expressed as: ; in: H is a reproducing kernel Hilbert space RKHS, is the RKHS norm, which is used to measure the distance in high-dimensional feature space; C Indicates the number of emotion categories contained in both the source domain and the target domain; and Represent the source domain and target domain datasets respectively; Indicates the source domain i The feature vector of each sample; Indicates the target domain j The feature vector of each sample; and Represent the category weights of the source domain and target domain respectively; representing a feature mapping function; by minimizing the loss function aligning the same feature distribution of the source domain macro-expression and the target domain micro-expression reference feature vector.

[0012] The face micro-expression recognition method provided by the embodiment of the present application has the following beneficial effects compared with the prior art: The present application takes the micro-expression peak frame as input on the spatial branch, extracts the texture features of the face in the peak frame, performs feature enhancement on the key part features such as eyebrows and corners of the mouth in the texture features, and obtains a peak frame feature vector; takes the optical flow feature map as input on the time branch, extracts the motion features of the face in the optical flow feature map, distinguishes and performs feature enhancement on the eye and mouth motion features contained in the motion features, and obtains an optical flow feature vector; and splices the peak frame feature vector and the optical flow feature vector into a reference feature vector. This kind of processing mode of isomorphism and different modalities realizes the complementary fusion of multi-modal features in the frequency domain and the time domain. Then, the feature distribution of the source domain macro-expression and the target domain micro-expression reference feature vector is aligned, the Mahalanobis distance between the target domain micro-expression reference feature vectors before and after alignment is compared, and the recognition result is output, so that the domain difference is eliminated by aligning the fused multi-modal features in this process, and the source domain and the target domain are modeled, and the micro-expression of the face can be accurately recognized. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The overall flowchart of the face micro-expression recognition method provided by the embodiment of the present application is shown in the figure. Figure 2 The face preprocessing process and the TV-L1 optical flow feature map extraction of the face micro-expression recognition method provided by the embodiment of the present application are shown in the figure. Figure 3 The recognition framework of the face micro-expression recognition method provided by the embodiment of the present application is shown in the figure. Figure 4 The space-time feature fusion network model architecture of the face micro-expression recognition method provided by the embodiment of the present application is shown in the figure. Figure 5 The source domain and target domain alignment process based on LMMD of the face micro-expression recognition method provided by the embodiment of the present application is shown in the figure. Figure 6 The class covariance prototype metric based on Mahalanobis distance optimization of the face micro-expression recognition method provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0014] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the drawings. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the concept of the present application, so the present application is not limited to the specific embodiments disclosed below.

[0015] Referring to Figure 1 and Figure 3 , the embodiment of the present application provides a facial micro-expression recognition method, comprising the following steps: Step S1: Obtain macro and micro-expression video sequences, pre-process the facial image, extract the starting frame and peak frame in the sequence, calculate the motion vector between the two frames by TV-L1 optical flow algorithm to generate an optical flow feature map, as shown in Figure 2 Specifically: Step S11: The source domain uses CASME^3 and MMEW macro-expression dataset, and the target domain selects CASME II, SAMM and MMEW micro-expression dataset. Face++ API is used for face detection and 68-point key point positioning. The face area is aligned by affine transformation, and is uniformly cropped to 224x224 pixel standard size to eliminate posture and scale differences.

[0016] Step S12: After the expression sequence is pre-processed in step S11, the displacement between the starting frame and the peak frame is calculated; TV-L1 energy functional is selected for optical flow estimation, and u and v represent the horizontal component and the vertical component, respectively; the calculation can be represented as: .

[0017] Wherein: dx and dy represent the pixel change along the x and y dimensions, respectively, and dt represents the time change. The optical strain can approximate the facial deformation intensity, and the optical strain can be obtained by calculating the derivative of the optical flow, which can be defined as: .

[0018] Wherein: U=[u, v]T is the displacement vector, so can be represented as: .

[0019] Wherein: In the diagonal strain component, ( , ) is the normal strain component, ( , ) is the shear strain component; the optical strain size of each pixel can be calculated by taking the square sum of the normal strain component and the shear strain component, and the optical strain size can be obtained as: .

[0020] Step S13: Stack the three optical flow information containing rich micro-expression, u, v, and ε, in turn to form a three-channel template image X = {u, v, ε}, which is the required optical flow feature image.

[0021] Step S2: Input the pre-processed peak frame and the optical flow map into the spatio-temporal feature fusion network to extract the temporal and spatial features of macro / micro-expression, respectively, and then fuse the extracted features, as shown in Figure 4 . Specifically: Step S21: Input the pre-processed micro-expression peak frame into the spatial branch network, first perform spatial down-sampling through a 7x7 convolution kernel and pass through a batch normalization layer (BN) and a ReLU activation function, then input a single-layer lightweight Transformer block (MobileViT) to extract global features through a 3x3 depth separable convolution and a 4-head self-attention mechanism, followed by two layers of BasicBlock group for local feature refinement, and finally focus on the key facial regions through a spatial attention module and flatten the features through a fully connected layer to compress them into a 128-dimensional feature vector.

[0022] Step S22: Input the TV-L1 optical flow feature map into the temporal branch network, which has a completely symmetrical network structure with the spatial branch, sequentially passing through 7x7 convolution down-sampling, BN layer, BN-ReLU activation, single-layer MobileViT module, and two layers of BasicBottle residual block group, except that the final channel attention module is used instead of the spatial attention module, because the channel dimension of the optical flow feature carries the physical meaning of different motion patterns, and the channel attention can effectively distinguish the space-time asynchrony of eye and mouth micro-expression, and finally output a 128-dimensional feature vector.

[0023] Step S23: Concatenate the 128-dimensional feature vector output by the spatial branch and the 128-dimensional feature vector output by the temporal branch into a 256-dimensional joint feature.

[0024] Step S24: The final output of the 256-dimensional fusion feature will serve as the reference representation for cross-domain alignment, which is used for calculating the class covariance prototype based on Mahalanobis distance in the meta-learning task, and for realizing the distribution alignment of macro-micro-expression through the LMMD loss in the domain adaptation.

[0025] Step S3: Use the meta-learning and domain adaptation collaborative optimization framework to realize feature optimization and cross-domain alignment through the alternating double adjustment strategy of macro-micro-expression, as shown in Figure 5 . Specifically: Step S31: Assume that the source domain dataset and the target domain dataset are and , = , that is, the source domain and target domain do not contain the same data; the training process of the source domain can be regarded as several small meta-tasks (episodes); after obtaining the embedded features of the samples, the source domain model is optimized for each episode by calculating the distance between the embedded features of the unlabeled samples and the class prototype of the labeled samples; First, K labeled samples are extracted from each of the five fixed categories in the source domain to form a support set = {( , )}; Then, from the remaining samples of the same 5 categories, T unlabeled samples are extracted from each category to form a query set = {( , )}; Medium sample Class distribution Expressed as: .

[0026] in: To support the set The k class prototype, represents the embedding feature extractor, and Represents query sets respectively The samples and labels in The Mahalanobis distance metric method is used. The covariance prototype metric based on the Mahalanobis distance optimization is as follows: Figure 6 As shown ( Representing various prototypes), The definition method is: .

[0027] in: x and y represents the feature vector of the sample point, T Represents a transpose operation; Represents the difference vector between two sample points x and y; Indicates the k The inverse of the covariance matrix of the class prototype.

[0028] is the covariance matrix of the kth class, expressed as: .

[0029] in: Indicates the k Covariance matrix between images; denotes the covariance matrix between all classes of images; is used to balance the weight coefficients of k I denotes the samples in the support set belonging to class

[0030] is generated using the embedding features of the first k class dataset in the support set, denoted as: .

[0031] where: denotes the feature extractor; k denotes the sample points and corresponding labels of the first k class dataset in the support set; is the first class mean vector of the current task; if = 1, will be defined as a zero-sized matrix of appropriate dimensions.

[0032] .

[0033] by computing the distance between the query sample and the prototypes of each class and converting it into a class prediction probability; subsequently, the difference between the prediction and the true label is modeled using a cross-entropy loss function, and the mathematical expectation is used to average multiple tasks, which is in the form of: .

[0034] where: {( , )}、 denote the support set and query set on the source domain, respectively; denotes the class distribution of the samples in the query set denotes the samples and labels in the query set , respectively.

[0035] The training process of the target domain is similar to that of the source domain, with the main difference being that is divided into a labeled training set and an unlabeled test set ​​​​​​​; The loss function on the target domain is expressed as: .

[0036] in: {( , )}、 denote the support set and query set on the target domain respectively; Represents a query set Medium sample Class distribution of and Represents query sets respectively The samples and labels in .

[0037] Step S32: Since there are significant differences between macro-expressions and micro-expressions in terms of movement amplitude, duration, and muscle motor unit activation patterns (macro-expressions have large movement amplitude and duration of 1-5 seconds, while micro-expressions have small movement amplitude and duration of 0.25-0.5 seconds), it is difficult to directly transfer the model trained on the macro-expression dataset to the micro-expression field. To this end, the present invention proposes a cross-domain adaptation strategy based on modified maximum mean difference (LMMD). The features of the source domain dataset and the target domain dataset are obtained through a spatiotemporal feature fusion network, and the local maximum mean difference is used to measure the inter-domain distribution difference, which is expressed as: .

[0038] in: H is a reproducing kernel Hilbert space RKHS, is the RKHS norm, which is used to measure the distance in high-dimensional feature space; C is the number of emotion categories contained in both the source domain and the target domain; and are the source domain and target domain datasets; The source domain i The feature vector of each sample; The target domain j The feature vector of each sample; and Represent the category weights of the source domain and the target domain respectively, indicating that the sample belongs to c The confidence of the class is calculated by the classifier output probability; Represents the feature mapping function, which maps the original features to the reproducing kernel Hilbert space (RKHS); LMMD adds various mathematical expectations as a whole to minimize the corresponding class distance between the source domain and the target domain.

[0039] Step S33: The total loss function of the source domain is expressed as: .

[0040] Similarly, the total loss function of the target domain is represented as: .

[0041] Step S34: Constructing meta-training and meta-test two-stage tasks, 1000 meta-tasks (episodes) are randomly sampled from CASME^3, MMEW macro-expression datasets according to the 5-way 5-shot rule, each episode contains a support set (Support Set) 5 classes x 5 samples = 25 labeled samples; a query set (Query Set) 5 classes x 2 samples = 10 unlabeled samples; for the target domain, CASME II, SAMM, MMEW datasets design independent experimental verification system, divided into three stages to construct the target domain task pool, generate 1000 cross-domain meta-tasks according to the same rule; during target domain training, leave out all the samples of a subject as the meta-test set each time.

[0042] The cross-domain episode mixing strategy is adopted, each training batch contains 1 source domain episode and 1 target domain episode, forcing the model to learn domain-invariant features; since the source domain and the target domain are trained alternately, their training methods are the same, here the training method of the source domain is described to explain how class covariance measurement and domain adaptation work; after the support set and the query set pass through the feature fusion network of step S2, a 256-dimensional feature vector is obtained.

[0043] Step S4: In the cross-domain micro-expression recognition framework, leave-one-out cross-validation (LOSO) is used for target domain independent testing, and micro-expression recognition of all target domain datasets is completed. Specifically: Step S41: The target domain independent test data uses leave-one-subject-out cross-validation (LOSO), each time leaving out all the micro-expression samples of a subject as the test set (Test Set), and the rest of the subject data is used for model training and optimization, and iteration is performed until all subjects are verified.

[0044] Step S42: After the target domain test data is aligned and cropped by Face++ API, the peak frame and TV-L1 optical flow features are extracted (consistent with S1 process), and the standardized input sequence is generated.

[0045] Step S43: Load the trained spatio-temporal feature fusion network weights into the test framework for processing target domain test samples; test data input dual-branch network to extract 128-dimensional fusion features, and calculate Mahalanobis distance based on class covariance matrix .

[0046] Step S44: mapping the distance metric to a five-class probability distribution by temperature scaling Softmax.

[0047] The application adopts a dual-branch network architecture to realize spatio-temporal feature fusion, wherein a time branch and a space branch share the same network structure; the space branch takes a micro-expression peak frame as input, and captures facial subtle texture changes through a MobileViT module; the time branch takes a TV-L1 optical flow feature as input, and models transient dynamic characteristics through the same MobileViT module; the features output by the two branches are dynamically weighted and fused through a channel attention gating mechanism, and through this isomorphic and different modal processing mode, complementary fusion of multi-modal features is realized, and the robustness of feature representation is significantly improved.

[0048] The application introduces a cross-domain adaptive strategy based on improved maximum mean difference (LMMD), aligns the feature distribution of the two types of expressions through the alternating training of the source domain (macro-expression) and the target domain (micro-expression), combines dynamic covariance optimization under the meta-learning framework, uses Mahalanobis distance to measure the inter-class covariance matrix of the support set and the query set, and enhances the clarity of the classification boundary.

[0049] The application proposes a dual-domain collaborative data division strategy and a LOSO cross-validation mechanism, completes cross-domain migration under the condition of only 5-way 5-shot support set, significantly reduces the dependence on large-scale labeled data, and alleviates the muscle movement amplitude difference problem of macro-micro expressions through a dynamic meta-prototype alignment strategy, so that the model remains stable in cross-cultural and cross-subject scenarios.

[0050] The application efficiently fuses peak frame features and optical flow features through a spatio-temporal feature fusion network, and combines a cross-domain adaptive strategy of dynamic covariance optimization, thereby reducing the parameter amount while solving the cross-scene performance bottleneck caused by the single feature modeling and static prototype defects of existing methods, and providing an efficient solution for edge real-time analysis of micro-expression recognition.

[0051] The present application solves the technical problems of weak micro-expression amplitude and poor cross-domain generalization ability through innovative network architecture and optimization strategy; specifically, a space-time dual-flow architecture is adopted, the spatial branch processes the micro-expression peak frame, a 7*7 convolution layer is used to extract facial texture features, a MobileViT module is used to model multi-scale long-range dependency relationship, a BasicBottle residual block is used to enhance local details, and then a spatial attention mechanism is used to focus on the key areas of the face; the time branch processes the TV-L1 optical flow features, and the same network structure is used to cooperate with the channel attention module to enhance the motion feature representation ability; in the feature optimization aspect, the class covariance prototype (CCM) is constructed by means of Mahalanobis distance measurement to optimize the feature distribution tightness, and the local maximum mean difference loss (LMMD) is introduced to align the class conditional distribution of the source domain and the target domain; under the meta-learning framework, the small sample training task is constructed to improve the generalization performance of the model under the condition of data scarcity; effective cross-domain adaptive learning is realized, and an innovative solution is provided for micro-expression recognition.

[0052] The above-mentioned embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which belong to the protection scope of the present application. Therefore, the protection scope of the present application patent should be subject to the appended claims.

Claims

1. A facial micro-expression recognition method, characterized in that: The following steps are involved: Obtain a source domain macro-expression video sequence and a target domain micro-expression video sequence of the face; For macro-expression video sequences and micro-expression video sequences, the starting frame and peak frame of the video sequence are extracted, and the motion vector between the starting frame and the peak frame is calculated to generate an optical flow feature map; The peak frame is spatially downsampled, and the texture features of the face in the peak frame are extracted. The detail features contained in the texture features are enhanced to obtain a 128-dimensional peak frame feature vector. The optical flow feature map is temporally downsampled, and the motion features of the face in the optical flow feature map are extracted. The eye and mouth motion features contained in the motion features are distinguished and enhanced to obtain a 128-dimensional optical flow feature vector. The peak frame feature vector and the optical flow feature vector are concatenated into a 256-dimensional baseline feature vector, and the baseline feature vector of the source domain macro expression and the baseline feature vector of the target domain micro expression are obtained; Align the feature distributions of the source domain macro-expression and the target domain micro-expression baseline feature vectors to obtain the aligned target domain micro-expression baseline feature vector; The Mahalanobis distance between the target domain micro-expression baseline feature vectors before and after alignment is obtained, and the Mahalanobis distance metric is mapped into five categories of probability distribution to output the recognition results.

2. A facial micro-expression recognition method according to claim 1, characterized in that: The acquisition of the optical flow feature map includes: After obtaining the starting frame and peak frame of the video sequence, the TV-L1 energy functional is used for optical flow estimation. Let u and v represent the horizontal component and vertical component respectively, which can be expressed as: ； Where: dx and dy represent the pixel changes along the x and y dimensions respectively; dt represents the time change; The optical strain is expressed as the facial deformation intensity and obtained by calculating the derivative of the optical flow , expressed as: ； Among them: U=[u, v] T represents the displacement vector; At the same time, the optical strain Also expressed as: ； in:( , ) represents the normal strain component; ( , ) represents the shear strain component; The optical strain of each pixel is obtained by taking the square sum of the normal strain component and the shear strain component Size, expressed as: ； u, v, These three optical flow information containing micro-expressions are stacked in sequence to form a three-channel template image X = {u, v, }, X is the optical flow feature map.

3. A facial micro-expression recognition method according to claim 1, characterized in that: The 128-dimensional peak frame feature vector is obtained, including: Construct a spatiotemporal feature fusion network model including a spatial branch and a temporal branch; the spatial branch and the temporal branch each include a cascaded 7×7 convolutional layer, a batch normalization layer BN, a ReLU activation function, a single-layer MobileViT module, and a BasicBottle residual block; The peak frame is input into the spatial branch and spatially downsampled through a 7×7 convolutional layer, a batch normalization layer (BN) and a ReLU activation function. The downsampled peak frame is input into a single-layer MobileViT module. The single-layer MobileViT module extracts facial texture features through a 3×3 depthwise separable convolution and a 4-head self-attention mechanism. The spatial attention module focuses on the key areas of the eyebrows and corners of the mouth. The two-layer BasicBottle residual block group enhances the texture features of the key areas of the eyebrows and corners of the mouth. The enhanced texture features are flattened and compressed into a 128-dimensional peak frame feature vector through a fully connected layer.

4. A facial micro-expression recognition method according to claim 3, characterized in that: The 128-dimensional optical flow feature vector is obtained, including: The optical flow feature map is input into the time branch, and the optical flow feature map is spatially downsampled through a 7×7 convolution layer, a batch normalization layer BN and a ReLU activation function. The spatially downsampled optical flow feature map is input into a single-layer MobileViT module. The single-layer MobileViT module extracts facial motion features through a 3×3 depthwise separable convolution and a 4-head self-attention mechanism. The channel attention module distinguishes the eye and mouth motion features. The two-layer BasicBottle residual block group enhances the eye and mouth motion features. The enhanced motion features are flattened and compressed into a 128-dimensional peak frame feature vector through a fully connected layer.

5. A facial micro-expression recognition method according to claim 4, characterized in that, After obtaining the reference feature vector of the source domain macro expression and the reference feature vector of the target domain micro expression, the method further includes: The distance between the source domain macro-expressions and the target domain micro-expressions is measured based on the Mahalanobis distance. The Mahalanobis distance metric is expressed as: ； in: x and y represents the feature vector of the sample point, T Represents a transpose operation; Represents the difference vector between two sample points x and y; Indicates the k The inverse matrix of the covariance matrix of the class prototype; Indicates the k The covariance matrix of the class prototype is expressed as: ； in: Indicates the k Covariance matrix between images; Represents the covariance matrix between images of all categories; It is used to balance and The weight coefficient of Indicates that the support set belongs to the category k samples; I is the identity matrix; Use the first k Class dataset The embedding feature generation is expressed as: ； in: represents a feature extractor; and Respectively support the concentration k Class dataset Sample points and corresponding labels; Indicates the current task k class mean vector; By calculating the Mahalanobis distance between the query sample and each category prototype and converting it into category prediction probability, the cross entropy loss function is used to model the difference between the prediction result and the true label, and mathematical expectation is used to average multiple tasks. The loss function on the source domain is expressed as: ； in: {( , )}、 denote the support set and query set on the source domain respectively; Represents a query set Medium sample Class distribution of and Represents query sets respectively Samples and labels in ; The training process of the target domain is the same as that of the source domain, except that Divide into labeled training sets and an unlabeled test set ; The loss function on the target domain is expressed as: ； in: {( , )}、 denote the support set and query set on the target domain respectively; Represents a query set Medium sample Class distribution of and Represents query sets respectively Samples and labels in ; Based on the loss function on the source domain and the loss function on the target domain, the feature distribution density of the baseline feature vector of the source domain macro expression and the baseline feature vector of the target domain micro expression are enhanced respectively.

6. A facial micro-expression recognition method according to claim 5, characterized in that: The step of aligning the feature distributions of the source domain macro-expression and the target domain micro-expression baseline feature vectors includes: The local maximum mean difference (LMMD) is used to measure the difference between the source domain macro expression and the target domain micro expression baseline feature vector after enhancing the density of feature distribution. The loss function Expressed as: ； in: H is a reproducing kernel Hilbert space RKHS, is the RKHS norm, which is used to measure the distance in high-dimensional feature space; C Indicates the number of emotion categories contained in both the source domain and the target domain; and Represent the source domain and target domain datasets respectively; Indicates the source domain i The feature vector of each sample; Indicates the target domain j The feature vector of each sample; and Represent the category weights of the source domain and target domain respectively; represents the feature mapping function; By minimizing the loss function , align the similar feature distributions of the source domain macro-expression and target domain micro-expression baseline feature vectors.

Images