Facial expression positioning method and system based on action unit prior and micro-expression enhancement

Abstract

The application discloses an expression positioning method and system based on action unit priori and micro-expression enhancement, relates to the technical field of computer vision and emotion computing, and adopts MDMO optical flow features to capture the main motion direction of a face, combines with the modeling of a backbone network to associate space and time, and improves the effectiveness of feature extraction; introduces AU priori knowledge to construct a target AU set, forcibly makes the model learn the essential law of facial muscle movement through an AU-guided alignment mechanism, makes the features extracted by the model more discriminative and generalizable, and solves the problem that the features are not closely related to the essential movement of expressions in the prior art; and designs a micro-expression perception enhancement mechanism, explicitly expands the difference between the micro-expression vertex probability and the background probability through a special perception loss function, effectively solves the problem of low positioning accuracy of micro-expressions caused by low intensity and easy to be submerged in the background, and significantly improves the accuracy of micro-expression positioning.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and emotion computing, and in particular to a method and system for expression localization based on action unit priors and micro-expression enhancement. Background Technology

[0002] Facial expressions are an important vehicle for human emotional expression, and can be divided into two categories: macro-expressions and micro-expressions. Macro-expressions are characterized by their long duration, high intensity, and easy recognition; while micro-expressions are unconscious emotional expressions with extremely short duration and low intensity, which can truly reflect a person's inner emotions.

[0003] Facial Expression Spotting (FES) aims to identify the start and end times of macro-expressions (MaE) and micro-expressions (ME) from long video sequences. Existing techniques typically employ deep learning methods, such as using 3D-CNNs to extract spatiotemporal features, or utilizing Graph Convolutional Networks (GCNs) and Transformers to model the temporal dynamics of facial feature points. For example, the existing SpoT-GCN method utilizes the node relationships within facial regions of interest (ROIs) to capture features. However, existing techniques suffer from the following significant drawbacks:

[0004] 1. Ignoring structured facial muscle movement information: Most existing methods rely on general spatiotemporal features, often ignoring the structured muscle movements encoded by Action Units (AUs) defined by the Facial Action Coding System (FACS). Although a few studies have attempted to utilize AU information, it is usually treated only as static graph edge weights, ignoring the significantly different dynamic activation patterns of AUs between micro-expressions and macro-expressions.

[0005] 2. Low micro-expression recognition rate and susceptibility to interference: Micro-expressions are characterized by short duration (less than 0.5 seconds) and low intensity. Existing models struggle to capture these subtle movements, easily misinterpreting minute facial movements in the background (such as tremors caused by breathing or unconscious muscle twitches) as micro-expressions, or missing genuine micro-expressions. Current technologies lack effective enhancement mechanisms to address this "micro-expression-background" confusion problem.

[0006] Action units (AUs) are the smallest units of facial muscle movement. Different expressions are formed by the activation of specific AU combinations (for example, a happy expression is usually activated by AU6 (cheek lift) and AU12 (corner of the mouth lift), while a surprised expression is usually activated by AU1 (inner eyebrow lift), AU2 (outer eyebrow lift), and AU5 (upper eyelid lift). Therefore, introducing prior knowledge of AUs to guide model learning enables the model to more accurately capture the essential motion features of expressions. Simultaneously, considering the weak signal characteristics of micro-expressions, specialized perceptual enhancement mechanisms need to be designed to strengthen the difference between micro-expression features and background features, thereby improving the accuracy of micro-expression localization. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, and addressing the issues of neglecting the dynamic characteristics of the Action Unit (AU) and the easy confusion of micro-expressions with the background in existing facial expression localization methods, this invention provides an expression localization method and system based on action unit priors and micro-expression enhancement.

[0008] To achieve the above objectives, the present invention adopts the following technical solution, including:

[0009] The expression localization method based on action unit priors and micro-expression enhancement follows the model training process as follows:

[0010] Step 1: Extract optical flow features for each frame of the video sequence; input the extracted optical flow features into the backbone network to extract the spatiotemporal features of each frame;

[0011] Step 2: Construct the alignment mechanism for AU bootstrapping:

[0012] Construct a set of target AUs corresponding to the video sequences;

[0013] Using an AU classifier based on the spatiotemporal features of each frame, the activation probability of all AU classes in the target AU set and the predicted probability of the background are predicted; the AU alignment loss is calculated based on the prediction results of the AU classifier.

[0014] Step 3: Process the spatiotemporal features of each frame using an expression classifier, and output the prediction probabilities of micro-expressions and macro-expressions for each frame, including: the prediction probability of each frame being the start frame, vertex frame, end frame, and micro-expression frame of a micro-expression, and the prediction probability of each frame being the start frame, vertex frame, end frame, and macro-expression frame of a macro-expression; calculate the conventional classification loss based on the prediction results of the expression classifier.

[0015] Step 4: Enhance the perception of micro-expressions:

[0016] Within the real micro-expression vertex interval, the prediction probability of the micro-expression vertex frame is maximized while the prediction probability of the background is suppressed. The micro-expression vertex interval refers to the interval obtained by expanding a set number of frames to the left and right from the center of the micro-expression vertex frame. Within the real background interval, the prediction probability of the background is maximized while the prediction probability of the micro-expression vertex frame is suppressed. Based on the micro-expression perception enhancement mechanism, the micro-expression perception loss is calculated.

[0017] Step 5, Joint Training:

[0018] The model is jointly trained by combining the conventional classification loss from step 3, the AU alignment loss from step 2, and the micro-expression perception loss from step 4, and then used for facial expression localization.

[0019] Preferably, in step 1, the input video sequence is preprocessed, including face detection and face alignment; MDMO optical flow features are extracted from each frame of the preprocessed video sequence; and a spatiotemporal feature extraction network based on graph convolutional networks is used to extract spatiotemporal features from the MDMO optical flow feature sequence, thereby extracting the spatiotemporal features of each frame.

[0020] Preferably, in step 2, the specific method for constructing the target AU set is as follows:

[0021] First, select AUs whose activation frequency is higher than a set threshold to form a high-frequency AU set;

[0022] Then, select AUs that are spatially related to facial key points in the optical flow feature extraction process to form a spatially aligned AU set;

[0023] Finally, the intersection of the high-frequency AU set and the spatially aligned AU set is taken as the final target AU set.

[0024] Preferably, in step 2, Focal Loss is used to supervise the training of the prediction results of the AU classifier and calculate the AU alignment loss.

[0025] Preferably, in step 4, the micro-expression perception loss The calculation formula is as follows:

[0026] ;

[0027] in, Represents micro-expression vertex frames. Indicates background; The true vertex frame probability of micro-expressions; The predicted vertex frame probability for micro-expressions; The predicted background frame probability; This represents the probability of the actual background frame.

[0028] Preferably, in step 5, the total loss The formula is as follows:

[0029] ;

[0030] in, For conventional classification losses, For AU alignment loss, For micro-expression perception loss, , This represents the corresponding loss weight.

[0031] Preferably, the model inference stage process is as follows:

[0032] For each frame of the video sequence, optical flow features are extracted; the extracted optical flow features are then input into the backbone network to extract the spatiotemporal features of each frame.

[0033] The spatiotemporal features of each frame are processed using an expression classifier, and the predicted probabilities of micro-expressions and macro-expressions for each frame are output.

[0034] Based on the predicted probabilities of the output, the final expression localization range is obtained through filtering and optimization operations. The specific filtering and optimization operations are as follows:

[0035] Vertex frame filtering: Select frames with a vertex frame probability greater than a set probability as candidate vertex frames;

[0036] Interval generation: Using the candidate vertex frame as the center, search to the left and right for the frames with the highest prediction probabilities of the starting frame and the ending frame, respectively, to form candidate intervals;

[0037] Scoring and Deduplication: Calculate the score for each candidate interval, and use non-maximum suppression to remove overlapping redundant candidate intervals, outputting the final expression localization interval.

[0038] Preferably, the score is the product of the predicted probabilities of the start frame, the vertex frame, and the end frame; the overlap of two candidate intervals = the intersection length of the two candidate intervals / the union length of the two candidate intervals; when the overlap of two candidate intervals is higher than a preset overlap threshold, the candidate interval with the higher score is retained, and the redundant candidate interval with the lower score is deleted.

[0039] The present invention also provides an expression localization system based on action unit prior and micro-expression enhancement, which is applicable to the expression localization method based on action unit prior and micro-expression enhancement. The system is characterized in that it includes: a feature extraction module, an AU guided alignment module, an expression prediction module, a micro-expression perception enhancement module, and a candidate generation module.

[0040] Feature extraction module: used to receive the input video sequence, extract optical flow features and encode spatiotemporal features of the video sequence, and output the spatiotemporal features of the video sequence;

[0041] AU Guided Alignment Module: Internally contains an AU classifier, which predicts the activation probability of all AU classes in the target AU set, as well as the background probability, based on the spatiotemporal features of each frame;

[0042] Expression prediction module: used to generate the probability of each frame belonging to each stage of micro-expression or macro-expression based on spatiotemporal features;

[0043] Micro-expression perception enhancement module: The probability of micro-expression vertex frames output by the expression prediction module is compared with the background probability output by the AU guided alignment module to optimize model parameters during the training phase and enhance the response strength to micro-expression vertex frames.

[0044] Candidate generation module: Used to generate the final expression localization range based on the predicted probability, through threshold filtering and non-maximum suppression.

[0045] The present invention also provides a computer program product, which includes a computer program / instruction that, when executed by a processor, implements the aforementioned expression localization method based on action unit prior and micro-expression enhancement.

[0046] The advantages of this invention are:

[0047] This invention proposes an expression localization method based on action unit priors and micro-expression enhancement. By constructing an AU-guided alignment mechanism to strengthen the association between features and muscle movements, and by using a micro-expression perception enhancement mechanism to solve the problem that micro-expressions are easily obscured by the background, the accuracy and generalization ability of expression localization are significantly improved.

[0048] (2) Introducing AU prior knowledge to construct a target AU set, and using the alignment mechanism guided by AU to force the model to learn the essential laws of facial muscle movement, making the features extracted by the model more discriminative and generalizable, thus solving the problem that the features are not closely related to the essential movement of facial expressions in existing methods.

[0049] (3) A micro-expression perception enhancement mechanism was designed. By using a special perception loss function, the difference between the probability of micro-expression vertices and the probability of background was explicitly amplified. This effectively solved the problem of low positioning accuracy caused by low intensity of micro-expressions and easy obscuration by the background, and significantly improved the accuracy of micro-expression positioning.

[0050] (4) MDMO optical flow features are used to capture the main direction of facial movement, and the spatiotemporal correlation is modeled by backbone network (such as SpoT-GCN), which improves the effectiveness of feature extraction. At the same time, background category and Focal Loss are introduced to solve the class imbalance problem, which further improves the robustness of the model.

[0051] (5) This method is applicable to both micro-expression and macro-expression localization and has good versatility.

[0052] (6) Experiments show that the present invention is effective in SAMM-LV and CAS(ME). 2 On benchmark datasets, especially in the task of micro-expression localization, it significantly outperforms existing technologies. Attached Figure Description

[0053] Figure 1 This is a flowchart of the facial expression localization method based on action unit prior and micro-expression enhancement according to the present invention.

[0054] Figure 2 A visual comparison chart of the localization results. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The facial expression localization system based on action unit prior and micro-expression enhancement of the present invention includes: a feature extraction module, an AU-guided alignment module, an expression prediction module, a micro-expression perception enhancement module, and a candidate generation module.

[0057] The connections and functions of each module are as follows:

[0058] Feature extraction module: This module receives the original input video sequence and extracts optical flow features and encodes spatiotemporal features. Preferably, this module includes an MDMO optical flow extraction unit and a spatiotemporal graph convolutional network (Backbone, such as SpoT-GCN), outputting high-dimensional spatiotemporal features X of the video sequence.

[0059] AU-guided alignment module: Connected to the feature extraction module, it receives spatiotemporal features X. Internally, this module contains an AU classifier (fully connected layer structure). This module filters out a set of AUs that are relevant to the current facial feature space and are frequently triggered, predicts AU probabilities through a classification task, and aligns the dynamic activation patterns of the AUs with the input features.

[0060] Expression prediction module: Connected to the feature extraction module, it generates the probability of each frame belonging to each stage of micro-expression (ME) or macro-expression (MaE) based on spatiotemporal features X. This module contains a classifier, and the output includes the probabilities of the start frame (onset), vertex frame (apex), end frame (offset), and expression interval.

[0061] Micro-expression perception enhancement module: Connecting the expression prediction module and the AU guided alignment module. This module does not involve physical hardware structure, but instead uses a specific logic operation unit to compare the micro-expression vertex probabilities output by the expression prediction module with the background probabilities output by the AU guided alignment module. This is used to optimize model parameters during the training phase and enhance the response strength to micro-expression vertices.

[0062] Candidate generation module: Connected to the output of the expression prediction module, it is used to generate the final expression localization range (including start time, end time and expression category) based on the predicted probability through threshold filtering and non-maximum suppression (NMS) algorithm.

[0063] The facial expression localization method based on action unit prior and micro-expression enhancement of the present invention has the following specific steps:

[0064] Step 1: Video Preprocessing and Feature Extraction

[0065] Step 1.1: Preprocess the input video sequence V (containing T frames). During preprocessing, face detection and face alignment need to be performed on the video sequence V to ensure that the face position is consistent in each frame and avoid feature extraction errors caused by face offset.

[0066] Step 1.2: Extract MDMO (Main Directional Mean Optical flow) features for each frame in the video sequence V. Compared with traditional dense or sparse optical flow, MDMO optical flow features have the advantages of small data volume, strong noise resistance, and the ability to focus on the main motion direction, making them very suitable for the feature description of facial expression motion.

[0067] In the MDMO optical flow feature extraction process: First, the video sequence is divided into several windows using a sliding window mechanism. Then, the frontal face detector and shape predictor provided by the dlib library are used to locate 68 facial key points for each frame of the face. Next, 10 key facial key points are selected: 3 for the left eyebrow, 3 for the right eyebrow, and 4 for the mouth. Then, a rectangular region of a certain size is selected as the region of interest (ROI) centered on each selected facial key point. Then, using the OpenCV library, the optical flow of each pixel in each ROI relative to the first frame of the image within the window is calculated. Finally, using the calculation rules of the average optical flow in the principal direction, the principal direction of each region is selected, and the average optical flow in that direction is calculated as the optical flow of that ROI. The sliding window size is preferably set to 17, and the step size is set to 1.

[0068] Step 1.3: Input the extracted MDMO optical flow features into the model backbone network (such as SpoT-GCN) to extract the spatiotemporal features X of each frame.

[0069] SpoT-GCN is a spatiotemporal feature extraction network based on Graph Convolutional Networks (GCNs). Spatially, it uses GCNs to model the motion relationships between different facial regions, while temporally, it uses 1D convolutional neural networks to extract temporal features, thus capturing continuous changes in movement. The entire network is designed with a three-level structure, enabling it to focus on features at different temporal and spatial scales. Ultimately, it obtains a spatiotemporal feature X for each frame, which includes both the spatial appearance information of the face and the temporal motion information between frames. The adjacency matrix of the graph in the GCN is set according to the facial topology.

[0070] Step 2: Construct the alignment mechanism for AU bootstrapping

[0071] The core innovation of this step lies in introducing prior knowledge of AUs. By constructing a target AU set and designing auxiliary classification tasks, the model is forced to learn the dynamic activation rules of AUs, so that the spatiotemporal features X extracted by the model are closely related to the essence of facial muscle movement, thereby improving the discriminative and generalization capabilities of the features.

[0072] Step 2.1: Construct the target AU set:

[0073] (1) First, select the high-frequency AU set with high activation frequency in the training dataset. The high-frequency AU set reflects the most representative facial motion units in the dataset.

[0074] By analyzing the frequency of various AUs in the SAMMLV dataset, it was found that the activation frequency of AUs exhibits a long-tailed distribution, indicating that AUs with higher frequency of occurrence are more closely associated with facial expressions. Therefore, based on their activation frequency, AU categories with higher activation frequencies were selected to form a high-frequency AU set.

[0075] (2) Then select the spatially aligned AU set that has spatial correlation with the facial key points in the optical flow feature extraction process.

[0076] Based on the definition of each AU by the Facial Action Coding System (FACS), the muscle movements described by each AU are associated with facial key points. For example, if AU12 is defined as the corners of the mouth turning up, then AU12 can be defined as having a spatial association with two facial key points at the corners of the mouth. In this way, AU categories that have a spatial association with the extracted 10 facial key points are selected to form a spatially aligned AU set.

[0077] (3) Take the intersection of the high-frequency AU set and the spatially aligned AU set as the final target AU set.

[0078] Step 2.2: Introduce an auxiliary classification task: Use an AU classifier (fully connected layer) to predict the activation probability of each AU in the target AU set based on spatiotemporal features X. This classification task uses the AU labels in the dataset as supervision information.

[0079] A higher predicted activation probability for an AU by the classifier indicates a greater likelihood of AU activation. This auxiliary classification task forces the model to mine information related to AU activation from spatiotemporal features X by predicting AU activation probabilities, thus aligning spatiotemporal features X with AU motion patterns. Simultaneously, due to the lack of annotations for AUs activated in the background region in the dataset, a background category is added to the AU classifier to identify expressionless background segments, improving the model's robustness. In this case, the output of the auxiliary classification task includes the predicted probabilities of all AU categories in the target AU set and one background category. Using AU labels from the dataset as supervision information ensures the accuracy of model learning, enabling the model to distinguish between expressions (with specific AU activation categories) and expressionless (background) in each frame.

[0080] Step 2.3: Calculate AU alignment loss: Focal Loss is used to supervise the training of the prediction results (predicted probabilities of each AU category and the background category) for the auxiliary classification task. This step forces the model (backbone network + expression classifier) ​​to learn the dynamic activation rules of AUs. The AU alignment loss formula is as follows:

[0081] ;

[0082] Where N represents the number of predicted AU categories plus the background category, totaling N categories. For the first i The predicted probabilities of each category, These are weighting coefficients used to adjust for class imbalance problems.

[0083] Step 3: Facial Expression Probability Prediction

[0084] The spatiotemporal features X of each frame are processed using an expression classifier, and the predicted probabilities of micro-expressions (ME) and macro-expressions (MaE) for each frame are output respectively:

[0085] ;

[0086] ;

[0087] in, Indicates the first The set of micro-expression prediction probabilities for frames. Indicates the first A set of macro-expression prediction probabilities for frames. Indicates the first The frame represents the predicted probability of the starting frame for a micro-expression. Indicates the first The frame represents the predicted probability of a micro-expression vertex frame. Indicates the first The frame represents the predicted probability of the end of a micro-expression. It means the first The predicted probability of a frame being a micro-expression frame; Indicates the first The frame represents the predicted probability of the macro expression's starting frame. Indicates the first The frame represents the predicted probability of the macro-expression vertex frame. Indicates the first The frame represents the predicted probability of the macro expression ending frame. Indicates the first The frame represents the predicted probability of a macro expression frame.

[0088] Calculate the conventional classification loss based on the prediction results of the facial expression classifier. .

[0089] Step 4: Micro-expression awareness enhancement (ME-aware Enhancement)

[0090] To address the issue of low micro-expression intensity and easy obscuration by the background, a micro-expression perception loss is calculated. The core logic of this step is to explicitly amplify the difference between the micro-expression vertex probability and the background probability.

[0091] Step 4.1: Within the true micro-expression vertex (Apex) range: maximize the prediction probability of micro-expression vertex frames while suppressing the background probability.

[0092] The Apex range refers to the range obtained by extending three frames to the left and right from the Apex frame.

[0093] Step 4.2: Within the background region: maximize the predicted background probability while suppressing the predicted probability of micro-expression vertex frames.

[0094] The formula for the loss of micro-expression perception is as follows:

[0095] ;

[0096] in, Represents micro-expression vertex frames. Indicates background; This represents the true vertex frame probability of micro-expressions; This represents the predicted probability of micro-expression vertex frames; This indicates the predicted probability of the background frame. This represents the probability of the actual background frame.

[0097] Step 5: Joint Training

[0098] Combined with the conventional classification loss in step 3 Step 2 AU alignment loss and the loss of micro-expression perception in step 4 Joint training is performed to obtain a trained model, which is then used for facial expression localization.

[0099] Total loss The formula is as follows:

[0100] ;

[0101] In this embodiment, the preferred parameters for each loss weight are set as follows: =0.15, =0.01.

[0102] Step 6: Candidate generation and localization in the model inference stage

[0103] The model inference stage outputs the predicted probability based on the method in step 3. Through a series of filtering and optimization operations, the final expression localization result (i.e., the start frame, vertex frame, end frame and complete range of each expression) is obtained.

[0104] Step 6.1, Vertex Frame Filtering: Select frames with a vertex frame prediction probability greater than 0.5 as candidate vertex frames. Vertex frames are the core keyframes of facial expressions, and their positioning accuracy directly affects the positioning effect of the entire facial expression range.

[0105] Step 6.2, Interval Generation: Using the candidate vertex frame as the center, search left and right for the frame with the highest prediction probability of the starting frame and the prediction probability of the ending frame to form a candidate interval.

[0106] Step 6.3, Scoring and Deduplication: Calculate the score of each candidate interval (the product of the predicted probabilities of the start frame, vertex frame, and end frame), use non-maximum suppression (NMS) to remove redundant candidate intervals with high overlap, and output the final expression localization result.

[0107] Set a preset overlap threshold (e.g., 0.5). The overlap between two candidate intervals is calculated as the intersection length of the two candidate intervals divided by the union length of the two candidate intervals. When the overlap between two candidate intervals exceeds the preset overlap threshold, the candidate interval with the higher score is retained, and the redundant candidate interval with the lower score is deleted. The final output candidate interval is the final facial expression localization interval.

[0108] This embodiment has been validated on standard benchmark datasets, and the methods have been tested on the facial expression datasets SAMM-LV and CAS(ME). 2 The experimental results are shown in Table 1 below. The quantitative indicators of this invention are significantly better than those of the prior art:

[0109] Table 1. Results of each method on the facial expression datasets SAMM-LV and CAS(ME) 2 Experimental results

[0110]

[0111] On the SAMM-LV dataset, the F1 score for micro-expression (ME) localization using this invention reaches 0.4317, an improvement of approximately 0.84% ​​compared to the SpotFormer method, with an overall F1 score of 0.4591. (In CAS(ME)...) 2 On the dataset, the F1 score for macro-expression (MaE) localization of this invention reaches 0.5114, and the overall F1 score reaches 0.4924, which is superior to the state-of-the-art methods such as MESFusion. This proves the effectiveness of this technical solution in solving the problem of low intensity and difficulty in localizing micro-expressions.

[0112] The sources of prior art in Table 1 are as follows:

[0113] The publicly disclosed MDMD method for spotting macro- and micro-expression intervals in long video sequences.

[0114] Optical strain based macro- and micro-expression sequence spotting in long video: a publicly available Optical Strain method.

[0115] The method published by Zhang et al.: Spatiotemporal fusion for macro- and micro-expression spotting in long video sequences.

[0116] Research on micro-expression spotting method based on optical flowfeatures, according to the method published by He et al.

[0117] The method published by Zhao et al.: Rethinking optical flow methods for micro-expression spotting.

[0118] A unique m-pattern for micro-expression spotting in long videos, as described by Wang et al.

[0119] The exposed LBCNN method of Local bilinear convolutional neural network for spotting macro-andmicro-expression intervals in long video sequences.

[0120] A convolutional neural network for spotting multi-scale micro-expression intervals in long videos: The publicly disclosed MESNet method.

[0121] Shallow optical flow three-stream cnn for macro-and micro-expressionspotting from long videos published SOFTNet method.

[0122] Concat-CNN method exposed by Facial action unit-based deep learning framework for spotting macro-and micro-expressions in long video sequences.

[0123] A publicly available LSSNet method for spotting macro- and micro-expressions in long videos using a two-stream convolutional neural network.

[0124] A multi-temporal stream network for spotting facial macro-and micro-expression with hard and soft pseudo-labels disclosed MTSN method.

[0125] 3D-CNN for facial micro-and macro-expression spotting on long videosequences using temporal oriented reference frame disclosed 3D-CNN method.

[0126] Apex and boundary perception network for micro- and macro-expressionspotting: The publicly disclosed ABPN method.

[0127] The publicly available AUW-GCN method for AU-aware graph convolutional network for macro-and micro-expressionspotting.

[0128] A two-stream network for micro- and macro-expression spotting with background modeling, based on the publicly available LGSNet method.

[0129] The publicly available SpoT-GCN method for multi-scale spatio-temporal graph convolutional network for facial expression spotting.

[0130] The SpotFormer method is exposed for multi-scale spatio-temporal transformers for facial expressionspotting.

[0131] The MESFusion method exposed by Micro-expression spotting based on multi-modal hierarchical semantic-guided deep fusion model.

[0132] Furthermore, a visual comparative analysis was conducted on the localization performance of this invention (Ours) and the localization performance of the SpoT-GCN method, such as... Figure 2 As shown, Figure 2 In this context, GT represents the dataset annotation, Macro Expression (MaE) represents the macro expression segment, Micro Expression (ME) represents the micro expression segment, True Positive (TP) represents a true positive (the prediction result is consistent with the dataset annotation), and False Positive (FP) represents a false positive.

[0133] It can be seen that, compared to the baseline, this invention misidentifies fewer background samples as facial expressions and has a stronger ability to locate facial expressions from subtle movements, thus reflecting the two advantages of this invention:

[0134] 1. Enhanced Fine-Grained Feature Capture Capability: By introducing an AU-guided alignment module, this invention utilizes AU as prior knowledge, enabling the model to not only focus on pixel changes during feature extraction but also understand the structured movements of facial muscles. This significantly improves the model's sensitivity to micro-expressions.

[0135] 2. Enhanced resistance to background interference: Through the micro-expression perception enhancement module, this invention explicitly increases the distance between micro-expression vertices and background noise at the loss function level. Experiments show that this invention can effectively reduce the number of cases where background noise is misidentified as micro-expressions (false positives are reduced).

[0136] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A facial expression localization method based on action unit priors and micro-expression enhancement, characterized in that, The model training phase is as follows: Step 1: Extract optical flow features for each frame of the video sequence; input the extracted optical flow features into the backbone network to extract the spatiotemporal features of each frame; Step 2: Construct the alignment mechanism for AU bootstrapping: Construct a set of target AUs corresponding to the video sequences; Using an AU classifier based on the spatiotemporal features of each frame, the activation probability of all AU classes in the target AU set and the predicted probability of the background are predicted. Calculate the AU alignment loss based on the prediction results of the AU classifier; Step 3: Process the spatiotemporal features of each frame using an expression classifier, and output the prediction probabilities of micro-expressions and macro-expressions for each frame, including: the prediction probability of each frame being the start frame, vertex frame, end frame, and micro-expression frame of a micro-expression, and the prediction probability of each frame being the start frame, vertex frame, end frame, and macro-expression frame of a macro-expression; calculate the conventional classification loss based on the prediction results of the expression classifier. Step 4: Enhance the perception of micro-expressions: Within the real micro-expression vertex interval, the prediction probability of the micro-expression vertex frame is maximized while the prediction probability of the background is suppressed. The micro-expression vertex interval refers to the interval obtained by expanding a set number of frames to the left and right from the center of the micro-expression vertex frame. Within the real background interval, the prediction probability of the background is maximized while the prediction probability of the micro-expression vertex frame is suppressed. Based on the micro-expression perception enhancement mechanism, the micro-expression perception loss is calculated. Step 5, Joint Training: The model is jointly trained by combining the conventional classification loss from step 3, the AU alignment loss from step 2, and the micro-expression perception loss from step 4, and then used for facial expression localization.

2. The expression localization method based on action unit prior and micro-expression enhancement according to claim 1, characterized in that, In step 1, the input video sequence is preprocessed, including face detection and face alignment; For each frame of the preprocessed video sequence, MDMO optical flow features are extracted; a spatiotemporal feature extraction network based on graph convolutional network is used to extract spatiotemporal features from the MDMO optical flow feature sequence, extracting the spatiotemporal features of each frame.

3. The expression localization method based on action unit prior and micro-expression enhancement according to claim 1, characterized in that, In step 2, the specific method for constructing the target AU set is as follows: First, select AUs whose activation frequency is higher than a set threshold to form a high-frequency AU set; Then, select AUs that are spatially related to facial key points in the optical flow feature extraction process to form a spatially aligned AU set; Finally, the intersection of the high-frequency AU set and the spatially aligned AU set is taken as the final target AU set.

4. The expression localization method based on action unit prior and micro-expression enhancement according to claim 1, characterized in that, In step 2, Focal Loss is used to supervise the training of the prediction results of the AU classifier, and the AU alignment loss is calculated.

5. The expression localization method based on action unit prior and micro-expression enhancement according to claim 1, characterized in that, In step 4, the loss of micro-expression perception The calculation formula is as follows: in, Represents micro-expression vertex frames. Indicates background; The true vertex frame probability of micro-expressions; The predicted vertex frame probability for micro-expressions; The predicted background frame probability; This represents the probability of the actual background frame.

6. The expression localization method based on action unit prior and micro-expression enhancement according to claim 1, characterized in that, In step 5, the total loss The formula is as follows: in, For conventional classification losses, For AU alignment loss, For loss of micro-expression perception, , This represents the corresponding loss weight.

7. The expression localization method based on action unit prior and micro-expression enhancement according to any one of claims 1-6, characterized in that, The model inference phase process is as follows: For each frame of the video sequence, optical flow features are extracted; the extracted optical flow features are then input into the backbone network to extract the spatiotemporal features of each frame. The spatiotemporal features of each frame are processed using an expression classifier, and the predicted probabilities of micro-expressions and macro-expressions for each frame are output. Based on the predicted probabilities of the output, the final expression localization range is obtained through filtering and optimization operations. The specific filtering and optimization operations are as follows: Vertex frame filtering: Select frames with a vertex frame probability greater than a set probability as candidate vertex frames; Interval generation: Using the candidate vertex frame as the center, search to the left and right for the frames with the highest prediction probabilities of the starting frame and the ending frame, respectively, to form candidate intervals; Scoring and Deduplication: Calculate the score for each candidate interval, and use non-maximum suppression to remove overlapping redundant candidate intervals, outputting the final expression localization interval.

8. The expression localization method based on action unit prior and micro-expression enhancement according to claim 7, characterized in that, The score is the product of the predicted probabilities of the start frame, vertex frame, and end frame; the overlap of two candidate intervals = the intersection length of the two candidate intervals / the union length of the two candidate intervals. When the overlap of two candidate intervals is higher than a preset overlap threshold, the candidate interval with the higher score is retained, and the redundant candidate interval with the lower score is deleted.

9. An expression localization system based on action unit priors and micro-expression enhancement, characterized in that, The facial expression localization method based on action unit prior and micro-expression enhancement as described in any one of claims 1-8 is characterized in that the system includes: a feature extraction module, an AU-guided alignment module, an expression prediction module, a micro-expression perception enhancement module, and a candidate generation module; Feature extraction module: used to receive the input video sequence, extract optical flow features and encode spatiotemporal features of the video sequence, and output the spatiotemporal features of the video sequence; AU Guided Alignment Module: Internally contains an AU classifier, which predicts the activation probability of all AU classes in the target AU set, as well as the background probability, based on the spatiotemporal features of each frame; Expression prediction module: used to generate the probability of each frame belonging to each stage of micro-expression or macro-expression based on spatiotemporal features; Micro-expression perception enhancement module: The probability of micro-expression vertex frames output by the expression prediction module is compared with the background probability output by the AU guided alignment module to optimize model parameters during the training phase and enhance the response strength to micro-expression vertex frames. Candidate generation module: Used to generate the final expression localization range based on the predicted probability, through threshold filtering and non-maximum suppression.

10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the expression localization method based on action unit prior and micro-expression enhancement as described in any one of claims 1-8.

Images