A micro-expression recognition method based on facial pose constraint

Abstract

The present application relates to a kind of microexpression recognition methods based on face pose constraint, it is characterized in that, including: A, microexpression video sequence pre-processing;B, the selection and pre-processing of anchor point sample frame;Including: selection anchor point sample frame and carry out cutting, the starting thumbnail frame after cutting is serialized according to two orthogonal directions;C, construct bidirectional orthogonal long short time memory model;The modeling ability of long short distance information dependence is used to bidirectional orthogonal long short time memory model, the fixed pose feature of the portrait of selected reference frame is extracted, and the fixed pose feature is as reference, eliminate the noise interference introduced in video frame acquisition or pre-processing process;D, construct space constraint feature auxiliary realization microexpression recognition.The present application is extracted by the decomposition of pose information in orthogonal direction to the microexpression calibration frame constructed, obtains the pose correlation between different face regions, auxiliary classification main network carries out microexpression recognition, improves the performance of recognition algorithm.

Description

Technical Field

[0001] This invention relates to a micro-expression recognition method based on face pose constraints, belonging to the field of deep learning and pattern recognition technology. Background Technology

[0002] Microexpressions, as a subtle, brief, and spontaneous emotional representation of human facial expressions, are very short, delicate, and involuntary facial expressions that usually occur when a person intentionally or unconsciously hides their true emotions. This provides a basis for revealing people's true psychology or emotions. Therefore, the accurate identification of microexpressions has extremely important application value in security interrogation, clinical diagnosis, business negotiations, and daily social interactions.

[0003] In the field of manual feature extraction, several common feature extraction methods are widely used. In particular, Zhao et al. introduced a feature called LBP-TOP, which extends the traditional LBP algorithm from two-dimensional space to three-dimensional space, thus incorporating information from the temporal domain. Furthermore, Liu proposed a Principal Direction Average Optical Flow (MDMO) feature, an effective method for facial motion analysis in video sequences. MDMO first divides the face into 36 regions of interest, then calculates the amplitude and angle of optical flow within each region, using these values ​​as feature representations. This method reduces the feature dimension to 72 dimensions, significantly reducing computational complexity. To improve upon the limitations of MDMO, Liu et al. further proposed a sparse MDMO feature, which can learn effective dictionary representations from micro-expression datasets. Building on this, they also proposed a novel distance metric method that can more effectively reveal the underlying manifold structure of features. Xu et al. proposed the Facial Dynamic Map (FDM) method to align pixels in adjacent frames to mitigate the impact of head motion. Building on these studies, Happy et al. proposed a novel feature extraction technique based on the Fuzzy Histogram of Optical Flow (FHOOF), focusing on analyzing the direction of facial muscle movements. This method attempts to extract more subtle motion features from micro-expression videos to improve recognition accuracy.

[0004] Handcrafted feature design typically relies on prior knowledge and specific assumptions, which limits its ability to fully capture complex and subtle facial expression changes. This is particularly true in micro-expression recognition, where the precise capture of extremely subtle changes in facial muscles is crucial, and traditional handcrafted features and their supporting machine learning frameworks often fail to adequately capture these critical details. Furthermore, machine learning algorithms relying on handcrafted feature extraction are highly sensitive to video frame preprocessing, inevitably introducing additional interference information that impacts the accuracy and reliability of micro-expression recognition. Therefore, seeking more advanced and effective feature extraction methods has become a key research focus in the field of micro-expression recognition.

[0005] Deep learning has developed rapidly in recent years and achieved remarkable results in many fields. Micro-expression recognition, as a cutting-edge topic in computer vision, has also made continuous breakthroughs with the help of deep learning. In 2016, DaeHoe Kim proposed a micro-expression recognition method that integrates convolutional neural networks (CNNs) and long short-term memory networks (LSTMs). This method can simultaneously capture spatial and temporal features in video sequences, providing a new approach to micro-expression recognition. In 2018, Wang further proposed a micro-expression recognition method using a transfer learning strategy, effectively improving the performance of micro-expression recognition by utilizing a model pre-trained on a large-scale dataset. In 2020, Yante Li proposed a micro-expression peak frame recognition algorithm based on three-dimensional Fourier transform, combined with the LGCcon model, further improving the accuracy of micro-expression recognition by fusing local and global features to train the network. In the same year, Xia used Euclidean video magnification technology to preprocess the micro-expression dataset to amplify dynamic information, extracted key facial features using a masking algorithm to reduce data dimensionality, and used a recurrent convolutional neural network (RCNN) to extract spatiotemporal features, further advancing the development of micro-expression recognition technology. Xie proposed a method using facial action units (AUs) to assist micro-expression recognition, enhancing its ability by recognizing subtle facial action units. In 2021, Ben conducted an in-depth investigation and analysis of the field of micro-expression detection and recognition based on video datasets and provided an outlook on its future prospects. Addressing the scarcity of micro-expression datasets, he proposed a new dataset, MMEW, providing valuable resources for micro-expression recognition research. In 2022, Chen proposed a micro-expression recognition algorithm using only the start frame and peak frame, fusing optical flow features extracted in four different ways and designing a block-based convolutional neural network to achieve more refined extraction of optical flow features. Meanwhile, Zhao et al. proposed a micro-expression recognition method based on the visual Transformer model, demonstrating the effectiveness of the Transformer model in micro-expression recognition tasks. Mao et al. explored micro-expression recognition tasks for situations where faces are occluded by objects in real-world environments and proposed a Region Heuristic Relational Reasoning Network (RRRN). This network can capture complementary relationships between different facial regions, effectively improving micro-expression recognition performance under occlusion conditions. Wei designed a novel Attention-Based Adaptive Magnification Network (AMAN), which can adaptively adjust the magnification level according to the dynamic changes of different micro-expression sequences, further improving the accuracy and robustness of micro-expression recognition.In 2023, Nguyen et al. proposed a BERT-based Facial Micro-Expression Recognition (Micron-BERT) framework. Micron-BERT leverages the powerful sequence modeling capabilities of the BERT framework to accurately locate patches of interest (POIs) in micro-expression frames. This method effectively reduces the impact of background noise and other interference factors on the model's recognition performance, making micro-expression recognition more accurate and reliable.

[0006] Micro-expression features are subtle, transient, and difficult to extract. Therefore, even the slightest interference can cause significant performance loss in recognition tasks. Considering that sampling or preprocessing micro-expression video samples may introduce subtle pose shifts, the same facial region may be incorrectly mapped to different pixel locations in the image. This makes it difficult for the network to accurately locate the subtle movements of micro-expressions, thus affecting model performance. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a micro-expression recognition method based on face pose constraints. Invention Overview:

[0009] This invention proposes a lightweight bidirectional orthogonal long short-term memory model as a Facial Pose Constraint (FPC) module to constrain abnormal shifts in facial poses in micro-expression samples. The FPC module extracts orthogonal directional pose information from calibration frames, thereby constraining the training of the micro-expression recognition main network. This invention designs standard frames to constrain the facial poses of the test samples, aligning the pose information of each feature during model training. A bidirectional orthogonal LSTM model is proposed to model low-resolution micro-expression information, ensuring the model's effective extraction of global facial information.

[0010] Terminology Explanation:

[0011] 1. Dlib Vision Library: Dlib is an open-source C++ toolkit containing machine learning algorithms that can be used to solve many practical problems in the field of machine learning. Currently, Dlib is widely used in both industry and academia.

[0012] 2. Facial Key Feature Point Detection: 68 key facial feature points are mainly distributed in the eyebrows, eyes, nose, mouth, and facial contours, such as... Figure 1 As shown, detection is performed using the Dlib vision library, which is an existing technology.

[0013] 3. Long Short-Term Memory (LSTM) Networks: These are a special type of recurrent neural network specifically designed to solve the vanishing and exploding gradient problems encountered by traditional recurrent neural networks when processing long sequences of data. Figure 2 As shown.

[0014] 4. Bidirectional Long Short-Term Memory Network: An extension of the Long Short-Term Memory network, containing two independent Long Short-Term Memory network layers, one responsible for processing the forward sequence of data and the other for processing the reverse sequence, such as... Figure 3 As shown, this structure can simultaneously capture the contextual information of image features, removing the prior knowledge introduced by the data order, thus providing a more comprehensive understanding of features.

[0015] 5. Multilayer Perceptron: A type of feedforward neural network consisting of multiple neuron layers, each of which is fully connected to the previous layer. It includes an input layer, hidden layers, and an output layer, and is used to solve classification problems.

[0016] 6. 3DResnet Network: Spatiotemporal 3D Convolutional Neural Network, a convolutional neural network structure that introduces three-dimensional convolutional kernels, enabling it to capture both spatial and temporal information in video simultaneously.

[0017] The technical solution of the present invention is as follows:

[0018] A micro-expression recognition method based on face region pose constraints includes:

[0019] A. Micro-expression video sequence preprocessing, including: acquiring video frame sequences, face detection and localization, and face alignment;

[0020] B. Selection and preprocessing of anchor point sample frames; including: selecting anchor point sample frames and cropping them, and serializing the cropped initial thumbnail frames according to two orthogonal directions;

[0021] C. Construct a bidirectional orthogonal long short-term memory model; utilize the bidirectional orthogonal long short-term memory model's ability to model the dependence of long and short distance information to extract the fixed pose features of the selected reference frame portrait, and use the fixed pose features as a reference to eliminate noise interference introduced during video frame acquisition or preprocessing.

[0022] D. Constructing spatial constraint features to assist in the recognition of micro-expressions.

[0023] According to a preferred embodiment of the present invention, step A involves preprocessing the micro-expression video sequence, including:

[0024] 1) Obtain the video frame sequence: Perform frame segmentation on the micro-expression video sequence to obtain the video frame sequence and store it;

[0025] 2) Obtain the start frame and peak frame: Select and store the start frame and peak frame of micro-expression in each video frame sequence;

[0026] The start frame refers to the first frame in a micro-expression video sequence where a micro-expression appears.

[0027] Peak frames refer to the frames in a micro-expression video sequence where facial muscle changes are most pronounced and contain the most micro-expression information.

[0028] 3) Face detection and localization: Using the Dlib vision library, face detection and localization are performed on the acquired start frame and peak frame to detect the number of faces in the video frame and the distance of the face from the image boundary.

[0029] 4) Face alignment: Using the Dlib vision library, 68 key facial feature points are identified to complete face segmentation and face correction.

[0030] According to a preferred embodiment of the present invention, step B, the selection and preprocessing of the anchor point sample frame, includes:

[0031] First, the starting frame F of each video sample Onset The selected starting frame is then considered as a pose sample frame for the reference plane region; subsequently, the selected starting frame is scaled down to a size equal to the scale of the last layer of the main network, forming a scaled-down starting frame F. on-ab The main network uses a 3DResnet network. Finally, the cropped initial thumbnail frames are serialized in two orthogonal directions, as shown in equation (1):

[0032]

[0033] Where B refers to batch size, W to feature width, H to feature height, C to number of feature channels, R to real number field, Orthogonalization, Serialize, and the initial abbreviated frame F... on-ab They are serialized into vertical pose calibration sequences F in two orthogonal directions respectively. W and horizontal pose calibration sequence F H Vertical pose calibration sequence F W and horizontal pose calibration sequence F H These are collectively referred to as anchor sample frames.

[0034] According to a preferred embodiment of the present invention, step C, constructing a bidirectional orthogonal long short-term memory model, includes the following steps:

[0035] A bidirectional orthogonal long short-term memory (BSM) model is used to model the pose constraint representation of the input anchor point sample frames from two directions. The BSM model is then used to model the pose constraint information sequentially between each pixel in the thumbnail sequence. The preprocessed vertical pose calibration sequence F is then modeled using both horizontal and vertical BSM models. W and horizontal pose calibration sequence F H The two abbreviated sequences are processed, namely: F W and F H By inputting the horizontal bidirectional long short-term memory model and the vertical bidirectional long short-term memory model respectively, the horizontal pose constraint features are obtained. Vertical pose constraint features

[0036] The representations from two different directions, processed by a bidirectional orthogonal long short-term memory model, are concatenated along the channel dimension, and then mixed through a multilayer perceptron to complete the pose-constrained feature F of the final part. pc Extraction; where the multilayer perceptron is implemented by two fully connected layer networks.

[0037] Further preferred, the starting frame F onset Orthogonally serialized to F H With F W Then, it is fed into a bidirectional orthogonal long short-term memory model, and after channel mixing, the feature Y constrained by the face pose is obtained. pc The entire calculation process is as follows:

[0038]

[0039]

[0040] Among them, BiLSTM H BiLSTM (Bilateral Long Short-Term Memory) is a horizontal bidirectional long short-term memory model. W This refers to a vertical bidirectional long short-term memory model, where dim = c refers to the feature dimension. Refers to horizontal pose constraint features. The vertical pose constraint feature refers to the feature of position, concatenate refers to dimensional splicing, and Fc refers to multilayer perceptron.

[0041] According to a preferred embodiment of the present invention, step D, which involves constructing spatially constrained feature-assisted micro-expression recognition, includes the following steps:

[0042] The input to the bidirectional orthogonal long short-term memory model is divided into two parts, one of which is a micro-expression video sequence. Another item is the starting frame F of the micro-expression video. onset Its characteristics are as follows:

[0043]

[0044] Where sub is the sample number, e is the expression label, and L is the frame length of the sampled video sequence, where all frames will be aligned and cropped into preprocessed frames with a width of W, a height of H, and a number of channels of C.

[0045] The main classification network, namely the 3DResnet network, completes the classification task, including:

[0046] micro expression sequence Micro-expression representations Y with classification significance are fed into the 3DResnet network for modeling. cls The obtained pose constraint features Y pc With Y cls The summed results are used to calculate cross-entropy loss to optimize model parameters. The calculation process is as follows:

[0047]

[0048] Where N is the sample size, Y gt denoted as the true value of the sample label, and C represents the classification category.

[0049] The beneficial effects of this invention are as follows:

[0050] This invention addresses the pose shift problem in micro-expression video sequence sampling and preprocessing by proposing a lightweight bidirectional orthogonal long short-term memory model to constrain abnormal facial pose shifts in micro-expression samples during data acquisition and preprocessing. By decomposing and extracting pose information in orthogonal directions from the constructed micro-expression calibration frames, pose correlations between different facial regions are obtained, assisting the classification main network in micro-expression recognition and improving the performance of the recognition algorithm. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of 68 key facial feature points of the present invention;

[0052] Figure 2 This is a schematic diagram of the classic LSTM model structure of this invention;

[0053] Figure 3 This is a schematic diagram of the bidirectional LSTM model structure of the present invention;

[0054] Figure 4 This is a schematic diagram of the anchor point sample frame selection and abbreviated orthogonalization preprocessing process of the present invention;

[0055] Figure 5 This is a schematic diagram of the orthogonal bidirectional LSTM module framework of the present invention;

[0056] Figure 6This is a schematic diagram of the auxiliary main network for spatial pose constraint features of the present invention;

[0057] Figure 7 This is a schematic diagram showing the F1 scores of the ablation effect of the model in Example 2 on all samples in the CASMEII dataset.

[0058] Figure 8 This is an ACC diagram showing the ablation effect of the model in Example 2 on all samples in the CASMEII dataset;

[0059] Figure 9 This is a schematic diagram showing the F1 scores of the ablation effect of the model in Example 2 on all samples in the SAMM dataset.

[0060] Figure 10 This is an ACC diagram showing the ablation effect of the model in Example 2 on all samples in the SAMM dataset;

[0061] Figure 11 This is a schematic diagram showing the F1 scores of the ablation effect of the model in Example 2 on all samples in the SMIC dataset.

[0062] Figure 12 This is an ACC diagram showing the ablation effect of the model in Example 2 on all samples in the SMIC dataset.

[0063] Figure 13(a) is a schematic diagram of sample frames of the Sub12 sample in the 012_7_1 video segment in Example 2;

[0064] Figure 13(b) is a schematic diagram of the sample frames of the Sub12 sample in the 012_3_2 video segment in Example 2. Detailed Implementation

[0065] To facilitate understanding of the present invention, the present invention will be further described below through embodiments and in conjunction with the accompanying drawings, but is not limited thereto.

[0066] Example 1

[0067] A micro-expression recognition method based on face region pose constraints includes:

[0068] A. Micro-expression video sequence preprocessing, including: acquiring video frame sequences, face detection and localization, and face alignment;

[0069] B. Selection and preprocessing of anchor point sample frames; including: selecting anchor point sample frames and cropping them, and serializing the cropped initial thumbnail frames according to two orthogonal directions;

[0070] C. Construct a bidirectional orthogonal long short-term memory model; utilize the bidirectional orthogonal long short-term memory model's ability to model the dependence of long and short distance information to extract fixed pose features of the selected reference frame portrait, and use the fixed pose features as a reference to eliminate noise interference introduced during video frame acquisition or preprocessing; thereby improving the performance of the micro-expression recognition model.

[0071] D. Constructing spatial constraint features to assist in the recognition of micro-expressions.

[0072] Example 2

[0073] The micro-expression recognition method based on face pose constraints described in Example 1 differs in that:

[0074] Step A involves preprocessing the micro-expression video sequence, including:

[0075] 1) Obtain the video frame sequence: Perform frame segmentation on the micro-expression video sequence to obtain the video frame sequence and store it;

[0076] 2) Obtain the start frame and peak frame: Based on the information annotated by experts in the micro-expression dataset, select and store the start frame and peak frame of micro-expression in each video frame sequence;

[0077] The start frame refers to the first frame in a micro-expression video sequence where a micro-expression appears.

[0078] Peak frames refer to the frames in a micro-expression video sequence where facial muscle changes are most pronounced and contain the most micro-expression information.

[0079] 3) Face detection and localization: Using the Dlib vision library, face detection and localization are performed on the acquired start frame and peak frame to detect the number of faces in the video frame and the distance of the face from the image boundary.

[0080] 4) Face Alignment: Based on face localization, 68 key facial feature points are determined using the Dlib vision library, such as... Figure 1 As shown, face segmentation and face correction are completed.

[0081] Face segmentation refers to using the Dlib vision library to segment faces using bounding boxes.

[0082] Face correction refers to the following: among the 68 key feature points detected on the face, the line connecting the key feature point 37 (marked at the left corner of the left eye) and the key feature point 46 (marked at the right corner of the right eye) has an angle α with the horizontal line. The corresponding rotation matrix is ​​obtained through this angle α, and the segmented face is rotated and transformed so that the line connecting the key feature point 37 (marked at the left corner of the left eye) and the key feature point 46 (marked at the right corner of the right eye) is parallel to the horizontal line, thereby correcting the face posture and scaling the face.

[0083] Step B, the selection and preprocessing of anchor point sample frames, includes:

[0084] The selection and preprocessing of anchor point sample frames are as follows: Figure 4 As shown. First, the starting frame F of each video sample. Onset The starting frame is considered as the pose sample frame for the reference plane. Using the starting frame as the standard can minimize the accumulation of positional offset errors caused by head shaking or face detection alignment in subsequent frames. Then, the selected starting frame is scaled down to a size equal to the scale of the last layer of the main network. on-ab The main network uses a 3DResnet network. This is because what is needed is the face location embedding (e.g., eyes are always located on either side of the nose; the size of a single sample face is consistent) rather than detailed features in the face (e.g., identity, texture, etc.), so overly complex high-resolution images are not required. In addition, the low-resolution image input greatly reduces the number of network parameters, which can alleviate the difficulty of network training. Finally, the cropped initial thumbnail frames are serialized in two orthogonal directions, which is expressed as Equation (1):

[0085]

[0086] Where B refers to batch size, W to feature width, H to feature height, C to number of feature channels, R to real number field, Orthogonalization, Serialize, and the initial abbreviated frame F... on-ab In two orthogonal directions (horizontal and vertical), they are respectively serialized into vertical pose calibration sequences F. W and horizontal pose calibration sequence F H Vertical pose calibration sequence F W and horizontal pose calibration sequence F H These are collectively referred to as anchor sample frames.

[0087] In step C, the bidirectional orthogonal long short-term memory model is constructed, including the following steps:

[0088] This invention constructs a bidirectional orthogonal long short-term memory model (O-BiLSTM), such as... Figure 5As shown, the bidirectional orthogonal long short-term memory (LSTM) model consists of a horizontal LSTM model and a vertical LSTM model, employing a lightweight network to constrain the pose of micro-expression video frames. The LSTM model corrects for facial pixel shifts and accumulated errors caused by these shifts during micro-expression video preprocessing. Specifically, it leverages the LSTM model's ability to model the dependence of long and short distance information to extract fixed pose features from a selected reference frame's portrait. These fixed pose features are then used as a benchmark to eliminate noise interference introduced during video frame acquisition or preprocessing, thereby improving the performance of the micro-expression recognition model.

[0089] First, the pose constraint representation of the input anchor point sample frame is modeled from two directions using a bidirectional orthogonal long short-term memory model. The bidirectional orthogonal long short-term memory model is used to model the pose constraint information between each pixel in the thumbnail sequence in sequence, as shown in equations (2) and (3). The horizontal bidirectional long short-term memory model and the vertical bidirectional long short-term memory model are used respectively to model the vertical pose calibration sequence F obtained by preprocessing in equation (1). W and horizontal pose calibration sequence F H The two abbreviated sequences are processed, namely: F W and F H By inputting the horizontal bidirectional long short-term memory model and the vertical bidirectional long short-term memory model respectively, the horizontal pose constraint features are obtained. Vertical pose constraint features The bidirectional long short-term memory model contains two independent LSTM layers: one for processing the forward sequence (from beginning to end) and the other for processing the reverse sequence (from end to beginning). This structure enables Bi-LSTM to capture the contextual information of image features simultaneously, eliminating the prior information introduced by the data order, thus providing a more comprehensive understanding of features. Therefore, it has the ability to model the pose constraint information between each pixel in the thumbnail sequence in sequence. Specifically, the thumbnail sequence refers to two sequences of length 196 obtained by scaling the original micro-expression video frames to 14×14 and then unfolding and serializing them in the horizontal and vertical directions, respectively. The pose constraint information refers to the feature information finally obtained in equation (4).

[0090] Finally, as shown in equation (4), the representations of the two different directions after being processed by the bidirectional orthogonal long short-term memory model in equations (2) and (3) are spliced ​​in the channel dimension, and then channel mixing is performed by a multi-layer perceptron (MLP) to complete the pose constraint feature F of the last part. pc Extraction; where the multilayer perceptron is implemented by two fully connected layer networks.

[0091] It is worth noting that, due to the anchor sample frame sequence F w With F H The scales of input sequences are not too long, making it easier for LSTM to form dependent features of varying lengths. Furthermore, compared to traditional CNN networks, this approach focuses more on the global information of the feature layers, i.e., the correspondence between facial features or muscles, rather than more fine-grained representations. Compared to Transformer models, LSTM is easier to train and more computationally friendly, and due to the specific scale of the input sequence, LSTM can also achieve better results than Transformer models.

[0092] Start frame F onset Orthogonally serialized to F H With F W Then, it is fed into a bidirectional orthogonal long short-term memory model, and after channel mixing, the feature Y constrained by the face pose is obtained. pc The entire calculation process is as follows:

[0093]

[0094] Among them, BiLSTM H BiLSTM (Bilateral Long Short-Term Memory) is a horizontal bidirectional long short-term memory model. W This refers to a vertical bidirectional long short-term memory model, where dim = c refers to the feature dimension. Refers to horizontal pose constraint features. The vertical pose constraint feature refers to the feature of position, concatenate refers to dimensional splicing, and Fc refers to multilayer perceptron.

[0095] In step D, spatial constraint features are constructed to assist in micro-expression recognition, such as... Figure 6 As shown, the steps are as follows:

[0096] The input to the bidirectional orthogonal long short-term memory model is divided into two parts, one of which is a micro-expression video sequence. Another item is the starting frame F of the micro-expression video. onset Its characteristics are as follows:

[0097]

[0098] Where sub is the sample number, e is the expression label, and L is the frame length of the sampled video sequence, where all frames will be aligned and cropped into preprocessed frames with a width of W, a height of H, and a number of channels of C.

[0099] The main classification network, namely the 3DResnet network, completes the classification task, including:

[0100] micro expression sequence Micro-expression representations Y with classification significance are fed into the 3DResnet network for modeling.cls The obtained pose constraint features Y pc With Y cls The summed results are used to calculate cross-entropy loss to optimize model parameters. The calculation process is as follows:

[0101]

[0102] Where N is the sample size, Y gt denoted as the true value of the sample label, and C represents the classification category.

[0103] In this embodiment, ablation experiments of the face pose calibration module were conducted in CASME II, SAMM, and SMIC (3-category) respectively. Figures 7 to 12 The results show the recognition F1 score and recognition accuracy (ACC) for each test sample in the corresponding dataset when cross-validating each sample using leave-one-out method on three datasets. Figure 7 This is a schematic diagram showing the F1 scores of the ablation effect of the model in Example 3 on all samples in the CASMEII dataset. Figure 8 This is an ACC diagram showing the ablation effect of the model in this embodiment on all samples in the CASMEII dataset; Figure 9 This is a schematic diagram showing the F1 scores of the ablation effect of the model in this embodiment on all samples in the SAMM dataset. Figure 10 This is an ACC diagram showing the ablation effect of the model in this embodiment on all samples in the SAMM dataset; Figure 11 This is a schematic diagram showing the F1 scores of the ablation effect of the model in this embodiment on all samples in the SMIC dataset. Figure 12 This is an ACC diagram showing the ablation effect of the model in this embodiment on all samples in the SMIC dataset.

[0104] The model parameters are set as follows: parameters of the bioorthogonal LSTM for the pose constraint module and parameters of the 3D ResNet for the classification main network, as shown in Table 1 below:

[0105] Table 1

[0106]

[0107] The pose constraint network and the classification main network consist of two distinct modules, where F represents the number of feature channels and N represents the number of basic modules in the corresponding layer. The Average Pooling method is used as a downsampling method between layers.

[0108] like Figures 7 to 12As shown in the table, by observing the bar chart of recognition metrics for each tested sample, it can be found that the ResNet3D model with the FPC module achieves higher F1-scores and recognition accuracy on most samples than the ResNet3D model using only. Furthermore, the data in the table also indicates that when testing on certain samples, the two models produced the same results, such as... Figures 7 to 12 The dashed box in the middle indicates this. This is because most of these samples contain a relatively small number of test videos (e.g., test sample Sub21 in the CASME II dataset contains only 2 video segments, and test samples Sub31 and Sub32 in the SAMM dataset contain only 1 video segment each), and the face alignment and cropping are relatively standard. It is worth noting that in the tests on all datasets, only sample Sub12 in CASMEII showed a negative effect on the FPC module, indicating that the FPC module may not adequately model certain features or conditions in very specific samples, but this only affected the model's F1-Score on that sample and did not affect the recognition accuracy on that sample. The network with the added FPC module achieved the highest accuracy on some samples, indicating that the proposed method has a strong feature modeling ability on specific samples.

[0109] For example, sample Sub12 from the SAMM dataset exhibits pose shifts due to preprocessing operations such as sampling, cropping, or alignment. Figure 13(a) shows a partial sample frame of the Sub12 sample in the 012_7_1 video segment in this embodiment; Figure 13(b) shows a partial sample frame of the Sub12 sample in the 012_3_2 video segment in this embodiment. The Sub12 sample shows significant head shifts or eye movements in both the 012_3_2 and 012_7_1 video segments. The model with the added FPC module achieved a recognition rate of 1.0 for the Sub12 sample, but the original ResNet3D model did not achieve accurate recognition of the Sub12 sample (F1-Score: 0.26, ACC: 0.33). This indicates that the FPC module effectively calibrates the facial pose information.

Claims

1. A micro-expression recognition method based on face position pose constraints, characterized in that, include: A. Micro-expression video sequence preprocessing, including: acquiring video frame sequences, face detection and localization, and face alignment; B. Selection and preprocessing of anchor point sample frames; including: selecting anchor point sample frames and cropping them, and serializing the cropped initial thumbnail frames according to two orthogonal directions; C. Construct a bidirectional orthogonal long short-term memory model; utilize the bidirectional orthogonal long short-term memory model to model the dependence of long and short distance information, extract the fixed pose features of the selected reference frame portrait, and use the fixed pose features as a reference to obtain pose constraint features, thereby eliminating noise interference introduced during video frame acquisition or preprocessing. D. Constructing pose-constrained features to assist in micro-expression recognition, including the following steps: The input to the bidirectional orthogonal long short-term memory model is divided into two parts, one of which is a micro-expression video sequence. The other item is the starting frame of the micro-expression video. Its characteristics are as follows: （5）； （6）； in, For sample number, For emoji tags, The sampled video sequence is of frame length, where all frames will be aligned and cropped to a width of [value missing]. The height is The number of channels is Preprocessed frames; The main classification network, namely the 3DResnet network, completes the classification task, including: micro expression sequence Micro-expression representations with classification significance are fed into the 3DResnet network. The obtained pose constraint features and The summed results are used to calculate cross-entropy loss to optimize model parameters. The calculation process is as follows: （7）； （8）； Where N is the sample size. For the sample label true value, For classification categories.

2. The micro-expression recognition method based on surface pose constraints according to claim 1, characterized in that, Step A involves preprocessing the micro-expression video sequence, including: 1) Obtain the video frame sequence: Perform frame segmentation on the micro-expression video sequence to obtain the video frame sequence and store it; 2) Obtain the start frame and peak frame: Select and store the start frame and peak frame of micro-expression in each video frame sequence; The start frame refers to the first frame in a micro-expression video sequence where a micro-expression appears. Peak frames refer to the frames in a micro-expression video sequence where facial muscle changes are most pronounced and contain the most micro-expression information. 3) Face detection and localization: Using the Dlib vision library, face detection and localization are performed on the acquired start frame and peak frame to detect the number of faces in the video frame and the distance of the face from the image boundary. 4) Face alignment: Using the Dlib vision library, 68 key facial feature points are identified to complete face segmentation and face correction.

3. The micro-expression recognition method based on face position pose constraints according to claim 1, characterized in that, Step B, the selection and preprocessing of anchor point sample frames, includes: First, the starting frame of each video sample. The selected starting frame is then regarded as a pose sample frame for the reference plane region; subsequently, the selected starting frame is scaled down to a size equal to the starting scale of the last layer of the main network. The main network uses a 3DResnet network. Finally, the cropped initial thumbnail frames are serialized in two orthogonal directions, as shown in equation (1): （1）； Where B refers to batch size, W to feature width, H to feature height, C to number of feature channels, R to real number field, Orthogonalization, Serialize, and start thumbnail frame. They are serialized into vertical pose calibration sequences in two orthogonal directions respectively. and horizontal pose calibration sequence Vertical pose calibration sequence and horizontal pose calibration sequence These are collectively referred to as anchor sample frames.

4. The micro-expression recognition method based on face position pose constraints according to claim 1, characterized in that, In step C, the bidirectional orthogonal long short-term memory model is constructed, including the following steps: A bidirectional orthogonal long short-term memory (BSM) model is used to model the pose constraint representation of the input anchor point sample frames from two directions. The BSM model is then used to model the pose constraint information sequentially between each pixel in the thumbnail sequence. Horizontal and vertical BSM models are used respectively to model the preprocessed vertical pose calibration sequence. and horizontal pose calibration sequence The two abbreviated sequences are processed, namely: and By inputting the horizontal bidirectional long short-term memory model and the vertical bidirectional long short-term memory model respectively, the horizontal pose constraint features are obtained. Vertical pose constraint features ; The representations from two different directions, processed by a bidirectional orthogonal long short-term memory model, are concatenated along the channel dimension, and then mixed through a multilayer perceptron to complete the pose-constrained features of the final part. Extraction; where the multilayer perceptron is implemented by two fully connected layer networks.

5. The micro-expression recognition method based on surface pose constraints according to claim 4, characterized in that, start frame Orthogonally serialized as and Then, the data is fed into a bidirectional orthogonal long short-term memory model, and after channel mixing, features constrained by the face pose are obtained. The entire calculation process is as follows: （2）； （3）； （4）； in, This refers to the horizontal bidirectional long short-term memory model. This refers to a vertical bidirectional long short-term memory model, where dim=c refers to the feature dimension. Refers to horizontal pose constraint features. The vertical pose constraint feature refers to the feature of position, concatenate refers to dimensional splicing, and Fc refers to multilayer perceptron.

Images