Dynamic facial expression recognition method based on multiple modes

Abstract

The invention relates to a multi-mode-based dynamic facial expression recognition method, which comprises the following steps of: acquiring an initial video sample, and acquiring an image sample sequence according to the initial video sample; inputting the image sample sequence into a dynamic facial expression recognition model, extracting a facial expression feature sequence from the image sample sequence, obtaining a feature vector embedded in a facial expression class, and matching text description according to the embedded feature vector of the facial expression class to obtain a text feature vector; similarity calculation is carried out on the facial expression feature sequence and the text feature vector, a facial expression recognition result of the initial video sample is obtained, a dynamic facial expression recognition model is constructed based on a CLIP model and is obtained based on training of a training set, and the training set comprises a historical image sample sequence and corresponding facial expressions. According to the method, the context relation between the visual class and the text description is analyzed, and finally, the dynamic facial expression recognition is completed through similarity calculation, so that the training speed and the facial expression prediction precision are improved.

Description

Technical Field

[0001] The invention relates to the technical field of dynamic facial expression classification and recognition, and in particular to a dynamic facial expression recognition method based on multimodality. Background Art

[0002] Facial expression is one of the important ways of daily conversation and communication. Facial expression recognition (FER) is a prerequisite for many computer vision (CV) tasks, such as human-computer interaction (HCI), driver safety monitoring, medical assistance, and entertainment and advertising. The development and application of facial expression recognition technology has not only promoted the progress of artificial intelligence technology, but also achieved practical value and social impact in many fields. Traditional static facial expression recognition (SFER) methods mainly focus on static images or video frames, which cannot capture the temporal dynamics of facial expressions. However, the recognition of dynamic facial expression (DFER) involves capturing the changes in facial motion over time, which can provide more information for accurate emotion recognition. Early studies on dynamic facial expression recognition (DFER) mainly focused on laboratory controlled conditions, where faces were all frontal and unoccluded. However, in the real world, facial expressions are always accompanied by changes in lighting, non-frontal postures, and occlusions. These methods can be roughly divided into 3DCNN-based, CNN-RNN-based, Transformer-based, and Transformer-based deformations.

[0003] Wang et al. redefined the previous study's practice of treating non-target frames as noise frames as a weakly supervised problem. To address the imbalance between short-term and long-term temporal relationships in DFER, a multi-dimensional three-dimensional dynamic facial expression learning (M3DFEL) framework was introduced, which uses multiple instance learning (MIL) to handle imprecise labels. M3DFEL generates three-dimensional instances to model strong short-term temporal relationships and uses 3DCNNs for feature extraction. The Dynamic Long-term Instance Aggregation Module (DLIAM) is used to learn long-term temporal relationships and dynamically aggregate instances to obtain the final prediction results.

[0004] Sun et al. found that previous research in the field of DFER mainly adopted supervised learning methods, but this method is limited by the limited labeled data in existing datasets. They used large-scale self-supervised pre-training and abundant unlabeled data to significantly promote the development of DFER. MAE-DFER developed an efficient local-global interaction transformer (LGI-Former) as an encoder, which not only reconstructed independent appearance content, but also introduced explicit temporal facial action modeling to facilitate LGI-Former to mine static appearance and dynamic motion information. Extensive experiments on six datasets show that MAE-DFER consistently outperforms existing state-of-the-art supervised methods (e.g., 6.30% UAR improvement on DFEW and 8.34% UAR improvement on MAFW), demonstrating its powerful ability to learn dynamic facial expression representations through large-scale self-supervised pre-training.

[0005] Dynamic facial expression recognition (DFER), despite its promising prospects, faces multiple challenges. First, existing datasets are usually limited in size and biased, limiting the generalization ability of the model. Second, the unbalanced distribution of data makes the model tend to recognize frequently occurring expressions. The temporal dynamics of expressions are complex, and the need for real-time processing also requires the model to be both fast and accurate. The adaptability and generalization ability of the model in different environments or for different populations is also a major challenge. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal dynamic facial expression recognition method, which adopts a multimodal method, combines visual and text information, analyzes the contextual connection between visual "classes" and text descriptions through comparative learning, and finally completes the recognition of dynamic facial expressions through similarity calculation.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A multimodal dynamic facial expression recognition method, comprising:

[0009] Acquire an initial video sample, and acquire an image sample sequence according to the initial video sample;

[0010] The image sample sequence is input into a dynamic facial expression recognition model, a facial expression feature sequence is extracted from the image sample sequence, a feature vector embedded in a facial expression class is obtained, a text description is matched according to the embedded feature vector of the facial expression class, and a text feature vector is obtained; similarity is calculated between the facial expression feature sequence and the text feature vector to obtain a facial expression recognition result of the initial video sample, wherein the dynamic facial expression recognition model is constructed based on a CLIP model and is obtained based on training of a training set, and the training set includes a historical image sample sequence and corresponding facial expressions.

[0011] Optionally, matching the text description according to the embedded feature vector of the facial expression class, obtaining the text feature vector includes: inputting the embedded feature vector of the facial expression class into a large language model to generate the text description, inputting the text description into a text encoder, and obtaining the text feature vector.

[0012] Optionally, extracting the facial expression feature sequence from the image sample sequence includes: inputting the image sample sequence into a pre-trained video encoder to extract the facial expression feature sequence.

[0013] Optionally, obtaining a feature vector embedded in a facial expression class includes: projecting the facial expression feature sequence features to an embedding space of an expression class through a fully connected layer, and obtaining a feature vector embedded in the facial expression class, wherein the embedding space of the expression class is obtained by training the dynamic facial expression recognition model.

[0014] Optionally, obtaining the embedding space of the expression category includes:

[0015] After inputting the historical image sample sequence into the dynamic facial expression recognition model to extract the historical facial expression feature sequence, adding classification tags to the historical image samples;

[0016] The historical image samples with classification marks added are input into the Transformer to obtain a processed sequence, and the classification marks are obtained through slicing to obtain the embedding space of the expression category.

[0017] Optionally, before calculating the similarity between the facial expression feature sequence and the text feature vector, the method includes: processing the facial expression feature sequence and the text feature vector using average pooling, a recursive neural network LSTM, or a temporal visual transformer encoder Temporal Transformer to obtain aligned facial expression feature sequence and text feature vector.

[0018] Optionally, the loss function of the dynamic facial expression recognition model is:

[0019]

[0020] Among them, L is the loss function, y n is the sample label, p n is the classification score of each expression, and n is the true distribution of the random variable.

[0021] The beneficial effects of the present invention are as follows: the present invention discloses a multimodal-based dynamic facial expression recognition method, which adopts a new multimodal image-text to predict dynamic facial expression recognition, and uses contrastive learning of text and image and multimodal feature fusion to predict dynamic facial expression recognition. In the visual part, firstly, an image sample sequence is extracted from the initial video, and a pre-trained video encoder is used to obtain a facial expression feature sequence of the video sample, thereby generating a learnable facial expression "class" embedding vector. For the text part, a large language model is used to generate text descriptions related to the facial expression "class", and the descriptions are input into the text encoder to generate text feature vectors. Through similarity calculation, the feature sequence vectors of facial expressions are compared with the text features. The network is more fully trained through the promotion of multimodal text-image, which improves the training speed and the prediction accuracy of facial expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A framework diagram of a multi-modal dynamic facial expression recognition method according to an embodiment of the present invention;

[0024] Figure 2 Schematic diagram of the similarity calculation structure of an embodiment of the present invention, wherein (a) average pooling, (b) time series prediction, and (c) Temporal Transformer;

[0025] Figure 3 A schematic diagram of training and validation curves of an experiment in a DFEW dataset according to an embodiment of the present invention;

[0026] Figure 4 Schematic diagram of the confusion matrix (Fold 1-Fold 5) of the experimental results of an embodiment of the present invention in the DFEW dataset. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] This embodiment provides a multi-modal dynamic facial expression recognition method, comprising:

[0030] Obtain an initial video sample, and obtain an image sample sequence according to the initial video sample;

[0031] The image sample sequence is input into the dynamic facial expression recognition model, the facial expression feature sequence is extracted from the image sample sequence, the feature vector embedded in the facial expression class is obtained, the text description is matched according to the embedded feature vector of the facial expression class, and the text feature vector is obtained; the facial expression feature sequence and the text feature vector are calculated for similarity to obtain the facial expression recognition result of the initial video sample, wherein the dynamic facial expression recognition model is constructed based on the CLIP model and is obtained through training based on the training set, and the training set includes the historical image sample sequence and the corresponding facial expressions.

[0032] Furthermore, obtaining the image sample sequence includes:

[0033] Decomposing the initial video sample into single frame images to form an image sequence;

[0034] Sort the image sequence according to time order to obtain an image sample sequence.

[0035] Specifically, in this embodiment, the initial video sample is decomposed into single frame images to form an image sequence, and a fixed frame rate is set to ensure time continuity and sufficient frame time variation. The face images are arranged in time order to construct an image sample sequence in the form of a time series.

[0036] Furthermore, the text description is matched according to the embedded feature vector of the facial expression class, and obtaining the text feature vector includes: inputting the embedded feature vector of the facial expression class into a large language model to generate a text description, and inputting the text description into a text encoder to obtain the text feature vector.

[0037] In this embodiment, based on the feature vector embedded in the learnable facial expression "class", obtaining the matching text description and text feature vector specifically includes: using a large language model related to the "class" (facial expression) to generate a text description, and inputting these text descriptions into a text encoder to generate a text feature vector.

[0038] The matching text descriptions include:

[0039] a) Happy: smiling mouth, raised cheeks, wrinkled eyes, and curved eyebrows.

[0040] b) Sadness: tears, drooping mouth corners, drooping upper eyelids, and wrinkled forehead.

[0041] c) Neutral: relaxed facial muscles, straight mouth, smooth forehead, and inconspicuous eyebrows.

[0042] d) Angry: furrowed brows, narrowed eyes, tightly closed lips, and flared nostrils.

[0043] e) Surprise: wide eyes, open mouth, raised eyebrows, and frozen expression.

[0044] f) Disgust: wrinkled nose, lowered eyebrows, tightly closed mouth, and narrowed eyes.

[0045] g) Fear: raised eyebrows, slightly parted lips, furrowed brow, and retracted chin.

[0046] Furthermore, extracting the facial expression feature sequence from the image sample sequence includes: inputting the image sample sequence into a pre-trained video encoder to extract the facial expression feature sequence.

[0047] Furthermore, obtaining the embedded feature vector of the facial expression class includes: projecting the facial expression feature sequence features to the embedding space of the expression class through a fully connected layer, and obtaining the embedded feature vector of the facial expression class, wherein the embedding space of the expression class is obtained by training a dynamic facial expression recognition model.

[0048] Furthermore, the embedding space of expression categories is obtained by:

[0049] After inputting the historical image sample sequence into the dynamic facial expression recognition model to extract the historical facial expression feature sequence, a classification mark is added to the historical image sample;

[0050] The historical image samples with classification marks added are input into Transformer to obtain the processed sequence, and the classification marks are obtained through slicing to obtain the embedding space of the expression category.

[0051] Furthermore, the score calculation method for similarity calculation is:

[0052]

[0053] Among them, p is the probability score of expression prediction in k “classes”, f V is the facial feature sequence, is the text vector feature corresponding to k “classes” of expressions, k′∈(1,2,3LC), and C is the number of “classes”.

[0054] Furthermore, the loss function of the dynamic facial expression recognition model is calculated as:

[0055]

[0056] Among them, L is the loss function, y n is the sample label, p n is the classification score of each expression, and x is the true distribution of the random variable.

[0057] The method of the present invention is further described below in conjunction with the accompanying drawings:

[0058] like Figure 1 As shown, in this embodiment, a multimodal dynamic facial expression recognition method is provided, comprising the following steps:

[0059] An initial video sample is obtained, and an image sample sequence is obtained based on the initial video sample; the image sample sequence is input into a dynamic facial expression recognition model to obtain a facial expression recognition result of the initial video sample.

[0060] Dynamic facial expression recognition model, obtains the facial expression recognition results of the initial video sample including:

[0061] Obtaining a facial expression feature sequence of the video sample through a pre-trained video encoder according to the initial image sample sequence, thereby obtaining a facial expression feature sequence of the video sample;

[0062] Obtain a learnable feature vector of facial expression "class" embedding based on the facial expression sequence characteristics of the video sample;

[0063] Match the text description according to the feature vector embedded in the facial expression "class" and obtain the text feature vector through the text encoder;

[0064] Based on the similarity calculation of facial expression feature sequence and text feature vector, the accurate score of dynamic facial expression of initial image sequence samples is obtained to complete the recognition of dynamic facial expression.

[0065] In this embodiment, the initial video sample is decomposed into single frame images to form an image sequence, and a fixed frame rate is set to ensure time continuity and sufficient frame time variation. The face images are arranged in time order to construct an image sample sequence in the form of a time series.

[0066] In this embodiment, the dynamic facial expression recognition model is trained, and the dynamic facial expression recognition model generates "class" embeddings of different expression categories, so that the expression features of the same category are close in the feature space, while the features of different categories are far apart. Among them, the generation of the "class" embedding is to add a classification mark at the front of the input data after the image feature sequence is extracted, and it is input into the Transformer to obtain the processed sequence, and then the classification mark is obtained through the slicing operation. The final output can be classified by this mark. After training, the sequence features are projected to the embedding space of the expression category through the fully connected layer to obtain the expression "class" embedding feature vector of each video. This vector can be used as a feature representation of the main expression category in the video sample.

[0067] According to the feature vector embedded in the learnable facial expression "class", the matching text description and text feature vector are obtained, specifically including: using the text description generated by a large language model related to the "class" (facial expression), and inputting these text descriptions into a text encoder to generate a text feature vector.

[0068] Based on the learnable feature vector of the facial expression "class" embedding, the matching text description is obtained, including:

[0069] a) Happy: smiling mouth, raised cheeks, wrinkled eyes, and curved eyebrows.

[0070] b) Sadness: tears, drooping mouth corners, drooping upper eyelids, and wrinkled forehead.

[0071] c) Neutral: relaxed facial muscles, straight mouth, smooth forehead, and inconspicuous eyebrows.

[0072] d) Angry: furrowed brows, narrowed eyes, tightly closed lips, and flared nostrils.

[0073] e) Surprise: wide eyes, open mouth, raised eyebrows, and frozen expression.

[0074] f) Disgust: wrinkled nose, lowered eyebrows, tightly closed mouth, and narrowed eyes.

[0075] g) Fear: raised eyebrows, slightly parted lips, furrowed brow, and retracted chin.

[0076] By calculating the similarity of image and text embeddings, the distribution of different types of expressions in the feature space is evaluated. CLIP usually performs well in cross-modal retrieval and expression classification, so its ability to distinguish expression embeddings can be further evaluated by cosine similarity. The method based on facial expression feature sequence vector and text feature vector processing specifically includes: Mean Pooling, LSTM and Temporal Transformer are set according to task requirements, as shown in the following example. Figure 2 (ac) as shown. Mean Pooling provides an overall semantic representation, allowing the model to quickly capture the average information of the entire input. This method does not introduce additional parameters, is easy to calculate, is suitable for scenarios that require fast processing, and performs well in some tasks that do not require complex dependencies. The global features provided by average pooling can reduce noise interference, making the model more robust when matching facial expressions and text features. LSTM can capture the sequential information and temporal dependencies in the sequence, and has a good modeling ability for time-sensitive features. LSTM can capture the changing trend in the sequence. For example, the change of expression over time (such as the gradual change of micro-expressions), LSTM will capture this progressive information, making the similarity calculation more accurate. For text descriptions, word-by-word dependencies can be extracted to capture the progressive relationship of emotions. Temporal Transformer self-attention mechanism models the sequence and generates a global feature representation. It is very effective when processing long sequences, and can simultaneously focus on different positions in the sequence and model complex sequence dependencies. For example, it can capture the relationship between a set of continuous expression changes and emotional states, and can also identify the association between multiple related sentences in the text. This globally aware property enables Temporal Transformer to provide a richer representation that better aligns the semantic features of facial expressions and text.

[0077] Obtaining the dynamic prediction of the initial video sample specifically includes: iteratively training the highest score of similarity calculation based on the dynamic facial expression recognition backbone network model to obtain the loss function;

[0078] The score for the similarity calculation is calculated as follows:

[0079]

[0080] Among them, p is the probability score of expression prediction in k “classes”, f V is the facial feature sequence, is the text vector feature corresponding to the k-type expression. k′∈(1,2,3LC) where C is the number of “types”, τ is the temperature parameter of CLIP, y represents the classification label or category, and x is the input data.

[0081] The loss function calculation used includes:

[0082]

[0083] Among them, L is the loss function, y n is the sample label, p n is the classification score of each expression, and n is the true distribution of the label.

[0084] The method of this embodiment is evaluated on the DFEW and FERV39K datasets, which are large-scale public datasets of facial expressions under uncontrolled conditions. DFEW is a large-scale wild dataset introduced in 2020, containing more than 16,000 video clips with dynamic facial expressions. These videos are collected from more than 1,500 movies worldwide and contain various challenging interferences such as extreme lighting, self-occlusion, and unpredictable posture changes. FERV39K is currently the largest wild dynamic facial expression recognition (DFER) dataset, containing 38,935 video clips from 4 scenarios, which are further subdivided into 22 fine scenes. It is the first DFER dataset with large-scale 39K video clips, scenario-scene division, and cross-domain support capabilities. In this embodiment, two verification indicators are used, namely the unweighted average recall rate (UAR) and the weighted average recall rate (WAR). UAR is calculated as the average of the accuracy of each category, regardless of the sample size of the category; while WAR directly reflects the overall accuracy.

[0085] The method of this embodiment was evaluated using the uncontrolled DFEW dataset and the FERV39K dataset and compared with the top methods. The cross-validation results are detailed in Table 1. It outperformed the second-best method DFER-CLIP, improving by 2.82% in WAR and 1.86% in UAR. The unbalanced distribution of the DFEW dataset may cause the model to over-favor categories with large sample sizes and may ignore expressions with insufficient samples, such as "disgust" (1.2%) and "fear" (8.14%). This may significantly affect the model's ability to generalize uncommon facial expressions. The method of this embodiment showed a significant improvement in performance, with an increase of 10.09% in the accuracy of recognizing the "disgust" expression and 6.0% in the accuracy of recognizing the "fear" expression. The detailed performance indicators of each expression of the DFEW dataset and the FERV39K dataset are provided in Tables 1 and 2. Table 1 is a comparison between the DFEW dataset and existing methods (the bold ones are the best performers, and the underlined ones are the second best performers), and Table 2 is a comparison between the FERV39K dataset and existing methods (the bold ones are the best performers, and the underlined ones are the second best performers). Figure 3 and Figure 4 Visualize the results of the experiment.

[0086] Table 1

[0087]

[0088]

[0089] Table 2

[0090]

[0091] In order to evaluate the effects of each component of the model, the performance of the model under different configurations was recorded. The relevant results and analysis are as follows:

[0092] Tables 1 and 2 show the performance comparison between the proposed method and the baseline method. The proposed method is significantly better than the baseline, especially on the DFEW dataset, where WAR is improved by 2.82% and UAR is improved by 1.86% compared with the best DFER-CLIP method in the baseline; on the FERV39K dataset, WAR is improved by 2.8% and UAR is improved by 2.82% compared with the best AEN method in the baseline. These results confirm the effectiveness of the network model structure of this embodiment.

[0093] like Figure 1 As shown in Figure 1, this example consists of two parts. In the visual part, we first obtain a sequence of image samples from the initial video sample, and extract the facial expression feature sequence of the video sample through a pre-trained video encoder to generate a feature vector for the learnable facial expression "class" embedding. In the text part, we use the text description generated by the large language model related to the "class" (facial expression), and input these text descriptions into the text encoder to generate a text feature vector. By similarity calculation (such as Figure 2 As shown in the figure), the facial expression feature sequence vector is matched with the text feature vector generated by the text encoder, and finally the highest score for facial expression prediction is obtained.

[0094] Table 3

[0095]

[0096] For similarity calculation, Figure 2 (a)-(c) are shown. It should be noted that average pooling is the simplest way of temporal fusion, which is compared with other methods as shown in Table 3. Specifically, the decrease in UAR caused by LSTM is negligible, but they all improve the accuracy of WAR. Temporal-Transformer is selected as the final visual cue because it has the best results. This design is very important because appropriate temporal sequence learning can avoid catastrophic forgetting and maintain the representation ability of existing pre-trained models, providing a shortcut for using large amounts of network data.

[0097] Model parameter analysis: In general, deeper models achieve better performance, but in this example, the best performance is obtained with a single-layer temporal model. This is because the temporal model is trained from scratch, and overfitting may occur if the model is too complex. This is also an issue that needs to be considered in learnable contexts, where increasing the number of learnable vectors does not improve the results. Increasing the number of learnable contexts or the depth of the temporal model will lead to overfitting on the training data, resulting in poor generalization performance on the test data. Table 4 compares the Temporal Transformer with different parameter settings, and then obtains the optimal parameter settings.

[0098] Table 4

[0099]

[0100] Based on the CLIP model, this paper designs a dynamic facial expression recognition (DFER) model. In the visual part, a temporal model composed of multiple Transformer encoders is introduced to extract temporal facial expression features, and the final feature embedding is obtained as a learnable "class" token. In the text part, text descriptions of facial behaviors related to the recognition category of interest (facial expressions) are used as input, and these descriptions are generated using large language models (such as ChatGPT). This is in contrast to the work that only uses class names, which can more accurately capture the relationship between them. A learnable token is also introduced to help the model learn relevant contextual information for each expression during training.

[0101] The present invention discloses a multimodal dynamic facial expression recognition method, which adopts a new multimodal image-text to predict dynamic facial expression recognition, and uses contrastive learning of text and image and multimodal feature fusion to predict dynamic facial expression recognition. In the visual part, firstly, an image sample sequence is extracted from the initial video, and a pre-trained video encoder is used to obtain the facial expression feature sequence of the video sample, thereby generating a learnable facial expression "class" embedding vector. For the text part, a large language model is used to generate a text description related to the facial expression "class", and it is input into the text encoder to generate a text feature vector. Through similarity calculation, the feature sequence vector of the facial expression is compared with the text feature. The network is more fully trained through the promotion of multimodal text-image, which improves the training speed and the prediction accuracy of facial expressions.

[0102] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multimodal dynamic facial expression recognition method, characterized in that: include: Acquire an initial video sample, and acquire an image sample sequence according to the initial video sample; The image sample sequence is input into a dynamic facial expression recognition model, a facial expression feature sequence is extracted from the image sample sequence, a feature vector embedded in a facial expression class is obtained, a text description is matched according to the embedded feature vector of the facial expression class, and a text feature vector is obtained; similarity is calculated between the facial expression feature sequence and the text feature vector to obtain a facial expression recognition result of the initial video sample, wherein the dynamic facial expression recognition model is constructed based on a CLIP model and is obtained based on training of a training set, and the training set includes a historical image sample sequence and corresponding facial expressions.

2. The multimodal dynamic facial expression recognition method according to claim 1, characterized in that: Matching the text description according to the embedded feature vector of the facial expression class, obtaining the text feature vector includes: inputting the embedded feature vector of the facial expression class into a large language model to generate the text description, inputting the text description into a text encoder, and obtaining the text feature vector.

3. The multimodal dynamic facial expression recognition method according to claim 1, characterized in that: Extracting a facial expression feature sequence from the image sample sequence includes: inputting the image sample sequence into a pre-trained video encoder to extract the facial expression feature sequence.

4. The multimodal dynamic facial expression recognition method according to claim 1, characterized in that: Obtaining a facial expression class embedded feature vector includes: projecting the facial expression feature sequence features to the embedding space of the expression category through a fully connected layer, and obtaining a facial expression class embedded feature vector, wherein the embedding space of the expression category is obtained by training the dynamic facial expression recognition model.

5. The multimodal dynamic facial expression recognition method according to claim 1, characterized in that: Obtaining the embedding space of the expression category includes: After inputting the historical image sample sequence into the dynamic facial expression recognition model to extract the historical facial expression feature sequence, adding classification tags to the historical image samples; The historical image samples with classification marks added are input into the Transformer to obtain a processed sequence, and the classification marks are obtained through slicing to obtain the embedding space of the expression category.

6. The multimodal dynamic facial expression recognition method according to claim 1, characterized in that: Before calculating the similarity between the facial expression feature sequence and the text feature vector, the method includes: using average pooling, a recursive neural network LSTM, or a temporal visual converter encoder Temporal Transformer to process the facial expression feature sequence and the text feature vector to obtain aligned facial expression feature sequence and text feature vector.

7. The multimodal dynamic facial expression recognition method according to claim 1, characterized in that: The loss function of the dynamic facial expression recognition model is: Among them, L is the loss function, y n is the sample label, p n is the classification score of each expression, and n is the true distribution of the random variable.

Images