Video facial expression recognition method based on fuzzy time refinement network

Abstract

The invention discloses a video facial expression recognition method based on a fuzzy time refinement network, is applied to the technical field of computer vision and artificial intelligence, and aims to solve the problems that in the prior art, the expression time sequence dynamic state is difficult to capture, the processing feature fuzziness is low, and the refinement category recognition precision is low. The method comprises the following steps: preprocessing video frames; extracting spatial features by adopting ResNet-18 (ResNet-18); modeling inter-frame time sequence dependence through LSTM; dynamically weighting key frame features by using a fuzzy attention module (FAN) in combination with Gaussian, triangle and Sigmoid membership functions; carrying out the optimization of feature clustering through the combination of cross entropy loss and TriCenter Loss; outputting an expression category; by adopting the method, the video facial expression recognition accuracy can be effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and specifically relates to a video facial expression recognition technology. Background Technology

[0002] Facial expression recognition (FER) is a core technology in affective computing. Traditional FER relies on handcrafted features (such as LBP and HOG) combined with SVM classification, which struggles to capture the complexity and dynamism of facial expressions. With the development of deep learning, CNNs have achieved automatic spatial feature extraction, but static CNNs cannot model the temporal evolution of expressions in videos. Existing temporal models (such as LSTM and BiLSTM) can handle sequential data, but traditional attention mechanisms struggle to handle facial ambiguity (such as the "neutral-anger" transition state) and intra-class differences, and single loss functions (such as cross-entropy) cannot effectively separate similar facial features (such as fear and surprise). Furthermore, the limited size of FER datasets easily leads to model overfitting, and existing data augmentation methods often disrupt facial geometry, affecting recognition accuracy. Current technologies mainly suffer from the following problems:

[0003] (1) Existing static models cannot capture the temporal dynamics of video expressions, resulting in low accuracy in micro-expression and transition state recognition;

[0004] (2) Traditional attention mechanisms have difficulty handling facial expression ambiguity and cannot dynamically focus on key frame features;

[0005] (3) A single loss function cannot achieve clustering of similar features and separation of dissimilar features, resulting in poor recognition of similar expressions (such as neutral-angry).

[0006] (4) Limited dataset size and overfitting issues; existing data augmentation can easily damage facial structures.

[0007] Therefore, there is an urgent need for an FER scheme that takes into account spatial-temporal characteristics, ambiguity resistance, and high robustness. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a video facial expression recognition method based on a fuzzy temporal refinement network.

[0009] The technical solution adopted in this invention is: a video facial expression recognition method based on a fuzzy temporal thinning network, comprising:

[0010] S1. Video frame preprocessing: Obtain the video sequence of the expression to be recognized, extract key frames from the sequence and perform preprocessing;

[0011] S2. Spatial Feature Extraction: The preprocessed keyframes are input into the ResNet-18 network, and frame-level spatial features are output through convolutional and pooling layers with residual connections.

[0012] S3. Temporal Feature Modeling: Input the spatial features into the LSTM network, capture the inter-frame temporal dependencies through the input gate, forget gate, and output gate, and output the temporal features.

[0013] S4. Fuzzy attention weighting: The temporal features are input into the fuzzy attention module, and the membership function parameters are generated by the fuzzy parameter predictor composed of a multilayer perception mechanism. The feature weights are calculated by combining Gaussian, triangular and Sigmoid membership functions, and the temporal features are dynamically weighted after SoftMax normalization.

[0014] S5. Facial Expression Classification: Input the temporal features weighted in step S4 into the fully connected layer to output the facial expression classification result.

[0015] The beneficial effects of this invention are as follows: First, video frames are preprocessed; then, ResNet-18 is used to extract spatial features; next, LSTM is used to model inter-frame temporal dependencies; then, a fuzzy attention module (FAN) is used to dynamically weight keyframe features using Gaussian, trigonometric, and Sigmoid membership functions; finally, cross-entropy loss and TriCenterLoss are combined to optimize feature clustering; and finally, the expression category is output. This invention has the following advantages:

[0016] (1) Strong temporal modeling capability: Through bidirectional LSTM and positional coding, the temporal evolution of facial expressions is accurately captured, and the accuracy of micro-expression and transition state recognition is improved;

[0017] (2) Excellent anti-ambiguity: FAN combines three membership functions with dynamic weighting to effectively handle facial expression ambiguity, and the recognition accuracy of transition states such as "neutral-angry" reaches over 94%;

[0018] (3) High feature separation: TriCenterLoss achieves compact clustering of similar features and dissimilar features, reducing the recognition error of similar expressions (such as fear and surprise) by 30%;

[0019] (4) Strong robustness: Data augmentation and dropout effectively suppress overfitting, and the accuracy remains above 97% even in scenarios with changes in lighting and noise interference;

[0020] (5) Leading performance: The accuracy on the OULU-CASIA, CK+, and JAFFE datasets reached 98.77%, 99.57%, and 99.71%, respectively, which is better than the existing mainstream models. Attached Figure Description

[0021] Figure 1 A diagram illustrating the overall architecture of the Fuzzy Temporal Refinement Network (FTRN);

[0022] Figure 2 This is a schematic diagram of the Fuzzy Attention Module (FAN) structure;

[0023] Figure 3 This is a schematic diagram of the normalized confusion matrix of FTRN on different datasets;

[0024] Among them, (a) is a schematic diagram of the normalized confusion matrix of FTRN on the OULU-CASIA dataset, and (b) is a schematic diagram of the normalized confusion matrix of FTRN on the CK+ dataset.

[0025] Figure 4 This is a schematic diagram of the confusion matrix of FTRN on the JAFFE dataset;

[0026] Figure 5 The confusion matrix for FTRN on the OULU-CASIA 13-class custom dataset.

[0027] Note: The facial image samples used in the accompanying drawings of this specification are all derived from publicly available facial expression recognition research datasets (such as OULU-CASIA, CK+, JAFFE). Their use complies with research ethics and is only for illustrative purposes and does not constitute an infringement of personal rights such as portrait rights and privacy rights. Detailed Implementation

[0028] To facilitate understanding of the technical content of this invention by those skilled in the art, the following description, in conjunction with the accompanying drawings, further illustrates the invention.

[0029] This invention provides a video facial expression recognition method based on a fuzzy temporal refinement network (FTRN). The overall architecture of the fuzzy temporal refinement network (FTRN) is as follows: Figure 1 As shown, the input is the preprocessed video keyframes, which are sequentially processed by ResNet-18 (spatial feature extraction), bidirectional LSTM (temporal feature modeling), and FAN (fuzzy attention weighting), and then jointly optimized by cross-entropy loss and TriCenterLoss to finally output the facial expression category; Figure 1 The location code and dropout insertion position are also marked. The identification process of this invention includes the following steps:

[0030] 1. Video frame preprocessing

[0031] Obtain the video sequence to be identified (such as the near-infrared sequence from the OULU-CASIA dataset) and process it according to the following procedure:

[0032] (1) Keyframe Extraction: For each facial expression video, firstly, 10 consecutive frames are selected from its central time region as a candidate frame set; then, according to a preset equal-interval sampling strategy, two sets of non-contiguous and non-repeating keyframe sequences are extracted from the candidate frame set. For example, in this embodiment, the fixed equal interval is 3 frames, so the two sets of non-contiguous and non-repeating keyframe sequences are: the first set includes frames 0, 4, and 8, and the second set includes frames 1, 5, and 9 (frame numbers start from 0). This method, by covering representative frames at different time points in the video, effectively preserves the dynamic transition information during the facial expression change process while reducing data redundancy, thereby improving the robustness and expressiveness of subsequent temporal feature extraction.

[0033] (2) Image preprocessing: Normalization (pixel values ​​are normalized to [0,1]) and Sobel edge enhancement (strengthening structural features such as eyes and corners of the mouth) are performed on keyframes;

[0034] (3) Data Augmentation: In the keyframes after normalization and Sobel edge enhancement, Gaussian noise with a standard deviation of 0.08 is injected to simulate image sensor noise. Specifically, a random noise matrix with the same size as the image, a mean of 0, and a standard deviation of 0.08 is generated. This noise matrix is ​​then superimposed pixel by pixel onto the original image. Finally, the pixel values ​​are cropped to the effective range of [0,1] to simulate sensor noise in the real environment and enhance the robustness of the model. At the same time, the contrast (0.3) and brightness (0.2) are adjusted using ColorJitter. Experiments have verified that this specific parameter combination can effectively improve the robustness of the model in facial expression recognition in real scenes, while avoiding damage to key facial geometric features.

[0035] 2 Spatial Feature Extraction

[0036] ResNet-18 is used as the backbone network, and the network structure includes:

[0037] (1) Input layer: Receives preprocessed 224×224×3 keyframes;

[0038] (2) Convolutional layer: The first layer is a 7×7 convolution (stride 2, output channels 64), followed by 4 groups of residual blocks (each group contains 2 3×3 convolutional layers), and the gradient vanishing problem is solved by residual connection;

[0039] (3) Output layer: After global average pooling, the output is 512-dimensional frame-level spatial features.

[0040] 3. Temporal Feature Modeling

[0041] Input spatial features into a bidirectional LSTM network:

[0042] (1) Position Encoding: To achieve effective modeling of the temporal information of video frame sequences, before inputting the 512-dimensional spatial features extracted by ResNet-18 into the Long Short-Term Memory (LSTM) network, the method of this invention fuses the position encoding generated by the sine-cosine function with it: For the frame with position index pos in the sequence, its position encoding vector is calculated in the 2i-th and 2i+1-th dimensions (where i is the dimension index) according to the formula , Calculate, where the feature dimension d model =512; Subsequently, this encoded vector is added element by element to the spatial feature vector of the corresponding frame, thereby explicitly injecting frame order information into the model, enabling LSTM to simultaneously perceive the visual content and its temporal relationship.

[0043] Those skilled in the art will know that step 2 extracts the static spatial features (e.g., a 512-dimensional vector) of each frame of the image, and these features themselves do not contain any temporal order information. Position encoding generates a vector for each frame that is related to d... model A unique, same-dimensional position vector is input to the LSTM. This position vector is directly added to the spatial feature vector of the corresponding frame. Therefore, what is actually input to the LSTM is "enhanced features" = "spatial features" + "positional encoding". This allows each frame input received by the LSTM to contain both the visual content of "what it is" and the temporal location of "when", thus enabling it to effectively learn the dynamic evolution patterns of facial expressions in video sequences.

[0044] (2) Bidirectional LSTM: The hidden layer dimension is set to 512. The forward LSTM captures the historical frame dependency, and the backward LSTM captures the future frame dependency, outputting 512-dimensional temporal features.

[0045] 4. Fuzzy Attention Weighting

[0046] By dynamically weighting temporal features using fuzzy attention modules (FANs), the structure of a fuzzy attention module (FAN) is as follows: Figure 2 As shown, the input is the temporal features output by the LSTM. Initial scores are first generated through multi-head attention, then membership function parameters (μ, σ, w) are output through a fuzzy parameter predictor. Gaussian, triangular, and Sigmoid membership functions are then input to calculate weights. After weighted fusion, the results are normalized using SoftMax to output the final feature weights. Specifically, the process includes the following steps:

[0047] (1) Multi-head attention: The temporal features output by the LSTM (shape [sequence length, feature dimension], e.g., 6 frames × 512 dimensions) are input into a multi-head attention layer containing 8 parallel heads. Specifically, the 512-dimensional feature vector of each frame is evenly divided into 8 parts, each 64-dimensional, and assigned to 8 independent attention heads. Therefore, each attention head receives a [6, 64] matrix as input, which contains the representations of all frames in a specific feature subspace. Within its independent 64-dimensional subspace, each attention head generates a [6, 6] attention weight matrix by calculating query, key, and value vectors. This matrix quantifies the correlation strength between frames within this feature subspace. The original input is then weighted and fused using this weight matrix, and each head outputs a recalibrated [6, 64] weighted feature matrix. Meanwhile, by averaging the columns of the attention weight matrix, a vector of [1, 6] can be obtained, which serves as the initial importance score calculated by the attention head for each frame.

[0048] (2) Fuzzy parameter prediction: The original frame importance score generated by each attention head is used as input and fed into a fuzzy parameter predictor composed of two multilayer perceptrons. The predictor dynamically generates a set of three parameters for each attention head: center value (μ), width value (σ), and scaling weight (w). This same set of parameters (μ, σ, w) will be shared by the subsequent Gaussian, triangular, and sigmoid membership functions.

[0049] (3) Membership function calculation: The initial importance score (x) generated by each attention head, along with the parameter center value (μ) and width value (σ) generated by the fuzzy parameter predictor, are input into three parallel membership functions for calculation: Gaussian function Generates weights for smooth transitions, suitable for stabilizing facial expressions; trigonometric functions It generates linearly decreasing weights, excelling at capturing rapid facial expression changes; S-shaped function. The system generates progressively distinct weights to effectively handle fuzzy transition states; the three functions provide a foundation for subsequent fusion from three complementary perspectives: smooth, sharp, and progressive.

[0050] (4) Weight fusion: according to the formula The fusion weights are normalized using SoftMax and then weighted to highlight key frames (such as expression peak frames).

[0051] 5. Loss Optimization and Training

[0052] (1) Joint loss function: The cross-entropy loss (classification loss) and TriCenterLoss (feature metric loss) are used for joint optimization. TriCenterLoss is calculated according to the formula. Calculate, where α is set to 0.5 and ε is set to 1e-6;

[0053] (2) Regularization: Dropout (probability 0.2) is added between the ResNet-18 and LSTM layers to prevent overfitting;

[0054] (3) Training strategy: During the training phase, the Adam optimizer is used, and the initial learning rate is set to 1×10⁻ 4 To evaluate model performance, we implemented 10-fold cross-validation: the samples in each dataset were randomly divided into 10 parts, with 9 parts used as the training set and the remaining part as the validation set. This process was repeated 10 times to ensure unbiased evaluation. Each training iteration continued until the loss function converged or reached a preset maximum number of iterations (approximately 100 iterations). The final performance metric of the model was the average of all 10-fold results. The datasets used include, but are not limited to: the OULU-CASIA dataset (proposed in 2011 by Zhao, Huang, et al. from the University of Oulu and the Institute of Automation, Chinese Academy of Sciences), which contains facial expression video sequences in near-infrared and visible light (NIR & VIS); the CK+ dataset (Extended Cohn-Kanade Dataset, proposed in 2010 by Lucey, Cohn, Kanade, et al.), which is a widely used benchmark for dynamic facial expression videos; and the JAFFE dataset (The Japanese Female Facial Expression Database, proposed in 1998 by Lyons, Kamachi, and Gyoba), which is a classic collection of static facial expression images.

[0055] 6. Facial Expression Classification

[0056] The weighted temporal features are input into a fully connected layer (hidden unit 512), and the probability of 7 types of expressions (anger, disgust, fear, happiness, sadness, surprise, and neutral) is output through the SoftMax function. The one with the highest probability is taken as the recognition result.

[0057] The invention was verified based on the following experimental environment:

[0058] Hardware: CPU is Intel Xeon Gold 6248, GPU is NVIDIA Tesla V100 (32GB);

[0059] Software: Python 3.8, PyTorch 1.10, OpenCV 4.5, datasets: OULU-CASIA (960 near-infrared sequences), CK+ (500 sequences), JAFFE (213 images).

[0060] Implementation steps

[0061] (1) Data preparation: Download the OULU-CASIA, CK+, and JAFFE datasets, organize the files according to "expression category-subject-video frame", and select the near-infrared sequence under strong light for the OULU-CASIA dataset;

[0062] (2) Preprocessing execution: Sobel edge enhancement (horizontal and vertical operator convolution), Gaussian noise injection, and ColorJitter are implemented using OpenCV;

[0063] (3) Model construction: Based on PyTorch, ResNet-18 (load pre-trained weights and fine-tune), bidirectional LSTM (nn.LSTM, bidirectional=True), FAN (custom MLP and membership function layer), and TriCenterLoss (custom loss class) were constructed.

[0064] (4) Training process: The batch size is set to 32, the learning rate is decayed according to the cosine annealing strategy, the optimal model is saved after each round of validation, and the average accuracy is taken by 10-fold cross-validation.

[0065] (5) Testing and Validation: Input the test set into the optimal model, output the expression category, and calculate the accuracy and confusion matrix (e.g., Figures 3 to 5 As shown in the figure, the generalization ability of the model is verified.

[0066] Experimental results

[0067] Experiments show that the performance of this invention on various datasets is shown in Table 1:

[0068] Table 1 Performance of the Invention on Various Datasets

[0069]

[0070] As shown in Table 1, this invention achieved excellent performance on three mainstream facial expression recognition datasets, with accuracies of 98.77% (OULU-CASIA), 99.57% (CK+), and 99.71% (JAFFE), respectively. Particularly noteworthy are the model accuracies exceeding 99.5% on the CK+ and JAFFE datasets, approaching perfect classification levels. This fully validates the effectiveness of the model in spatial-temporal feature fusion, fuzzy attention mechanism, and joint loss function design.

[0071] The confusion matrix shows that the present invention achieves an accuracy rate of over 99.5% in recognizing "happiness" and "surprise," and an accuracy rate of 94.86% in recognizing the "neutral-anger" transition state, demonstrating the model's ability to handle ambiguous expressions and transition states.

[0072] like Figure 3 (a) shows the confusion matrix of the OULU-CASIA dataset (6 facial expressions), indicating that the accuracy rate for "anger" is 96.58% and for "fear" is 99.50%. Figure 3 (b) shows the confusion matrix of the CK+ dataset (7 types of expressions), which shows 100% for "contempt" and 99.95% for "surprise", demonstrating the high recognition accuracy of the model for most expressions.

[0073] like Figure 4 The image shows the confusion matrix of the JAFFE dataset (7 types of facial expressions), which shows that the accuracy rate for recognizing "happy" is 99.88%, "sad" is 99.69%, and the overall accuracy rate is 99.71%, validating the model's generalization ability on small sample datasets.

[0074] like Figure 5 The image shows the confusion matrix for 13 extended facial expressions (including transitional states such as "neutral-anger" and "neutral-disgust"). It shows that the recognition accuracy for "neutral-fear" is 97.50% and for "surprise" it is 99.16%, demonstrating the model's ability to handle transitional facial expressions.

[0075] Those skilled in the art will recognize that the embodiments described herein are for the purpose of helping to understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A video facial expression recognition method based on a fuzzy temporal thinning network, characterized in that, include: S1. Video frame preprocessing: Obtain the video sequence of the expression to be recognized, extract key frames from the sequence and perform preprocessing; S2. Spatial Feature Extraction: The preprocessed keyframes are input into the ResNet-18 network, and frame-level spatial features are output through convolutional and pooling layers with residual connections. S3. Temporal Feature Modeling: Input the spatial features into the LSTM network, capture the inter-frame temporal dependencies through the input gate, forget gate, and output gate, and output the temporal features. S4. Fuzzy attention weighting: The temporal features are input into the fuzzy attention module, and the membership function parameters are generated by the fuzzy parameter predictor composed of a multilayer perception mechanism. The feature weights are calculated by combining Gaussian, triangular and Sigmoid membership functions, and the temporal features are dynamically weighted after SoftMax normalization. S5. Facial Expression Classification: Input the temporal features weighted in step S4 into the fully connected layer to output the facial expression classification result.

2. The video facial expression recognition method based on a fuzzy temporal thinning network according to claim 1, characterized in that, The keyframe acquisition process in step S1 is as follows: select several frames from the time center region of the current facial expression video sequence, and select two sets of non-continuous and non-repeating keyframe sequences from these frames at a set interval.

3. The video facial expression recognition method based on a fuzzy temporal thinning network according to claim 2, characterized in that, Preprocessing in step S1 It includes the following steps: A1. Perform normalization and Sobel edge enhancement on keyframes; A2. Perform data augmentation on the keyframes after normalization and Sobel edge enhancement; specifically, generate a random noise matrix with the same size as the keyframe image and following a Gaussian distribution, and then superimpose the random noise matrix onto the keyframe.

4. The video facial expression recognition method based on a fuzzy temporal thinning network according to claim 3, characterized in that, Step S3 also includes position encoding of the spatial features and inputting the position-encoded spatial features into the LSTM network.

5. The video facial expression recognition method based on a fuzzy temporal thinning network according to claim 4, characterized in that, Step S4 includes the following sub-steps: S41. Arrange the time-series features output in step S3 according to their feature dimensions. The data is divided into 8 equal parts, each serving as the input to one of 8 independent attention heads; S42. Each attention head generates its corresponding attention weight matrix by calculating the query, key, and value vectors in its independent dimensional space. By calibrating the attention weight matrix and averaging it column by column, the initial importance score of the input partial temporal features calculated by that attention head is obtained. S43. The initial importance score output by each attention head is used as the input to the fuzzy parameter predictor. The fuzzy parameter predictor generates a set of parameters for each attention head, including the center value μ, the width value σ, and the scaling weight w. S44. Input the initial importance scores of each attention head output together with the corresponding set of parameters output by the fuzzy parameter predictor into three parallel membership functions for calculation to obtain the weights for smooth transition, linearly decreasing weights, and progressively distinguishing weights. S45. The weights for smooth transition, linear decrease, and gradual differentiation of each attention head are fused together, and the fused weights are used as the weights of the temporal features input by that attention head. The temporal features of the eight parts are weighted.

6. The video facial expression recognition method based on a fuzzy temporal thinning network according to claim 5, characterized in that, The three parallel membership functions include: Gaussian function, trigonometric function, and sigmoid function.

7. The video facial expression recognition method based on a fuzzy temporal thinning network according to claim 6, characterized in that, It also includes loss optimization and training processes: using several known expression sequences as training datasets, steps S2-S6 are iteratively trained. Specifically, cross-entropy loss and TriCenterLoss are used to jointly optimize the process of steps S2-S6. The TriCenterLoss achieves clustering of similar features and separation of dissimilar features by normalizing the Euclidean distance between features and class centers and marginal penalties.

8. The video facial expression recognition method based on a fuzzy temporal thinning network according to claim 7, characterized in that, It also includes adding a certain probability of dropout before the spatial features are input into the LSTM network.

Images