Expression recognition method and system based on three-stage multi-mode visual language prompt

Abstract

The invention provides an expression recognition method and system based on three-stage multi-mode visual language prompt. Relates to the technical field of computer vision and natural language processing, and aims to solve the problems that an existing method cannot recognize subtle changes of facial expressions, only pays attention to video related information, neglects intra-frame information and text information, and is inaccurate in model recognition. The method comprises the steps of obtaining a dynamic facial expression video data set; extracting a facial feature vector of each frame through an image encoder, performing optimization learning by adopting a three-stage optimizable prompt learning method, and inputting fused prompts obtained by optimization learning into the time model to obtain facial video features; extracting text features related to facial expression categories through a text encoder; and performing cosine similarity calculation on the face video features and the text features to obtain a prediction probability, and determining an identification result. According to the model, the recognition accuracy is remarkably improved while the learning cost is not increased.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and natural language processing, and in particular relates to an expression recognition method and system based on three-stage multimodal visual language prompts. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Facial expressions play an important role in daily conversations. Their recognition technology is widely used in mental health, human-computer interaction, driving assistance and other fields, and is gradually attracting the attention of researchers in many fields such as linguistics, computer science, psychology, and neuroscience. In the framework of discrete emotion models, facial expression recognition (FER) is committed to classifying video sequences or images into seven basic emotion categories: neutral, sad, happy, surprised, disgusted, fearful, and angry. Traditional FER methods focus on static images or single video frames, ignoring the dynamic characteristics of facial expressions evolving over time. In contrast, dynamic facial expression recognition technology (DFER) emphasizes the capture and analysis of continuous changes in facial movements. The inclusion of dynamic information provides strong support for the accuracy of emotion recognition. Therefore, DFER has gradually become a crucial research direction in the field of affective computing and computer vision.

[0004] In the field of dynamic facial expression recognition technology DFER, early explorations focused on laboratory-controlled scenarios where faces are frontal and unobstructed, which is convenient for basic research. However, facial expressions are always accompanied by changes in non-frontal posture, lighting, and occlusion. In recent years, research on DFER in the wild has increased significantly, and its core lies in designing methods that can effectively extract spatiotemporal features to enhance the discriminability and robustness of the model. These methods can be divided into several categories: 3dCNN-based, Transformer-based, and CNN-RNN-based. Among them, the Transformer-based method has reached the forefront of the field in the DFER task.

[0005] Recently, the visual language pre-training (VLP) model, especially CLIP, has shown great potential as an emerging force in the field of visual representation learning. Its core strategy is to process image and text data separately through two independent encoders, align the two on a large-scale dataset, and then learn and understand the rich semantic associations between vision and text.

[0006] Considering that videos have temporal relationships, for dynamic facial expression recognition tasks, most of the previous approaches were to fine-tune the image encoder part of the VLP model and add a Transformer layer that introduces temporal position encoding to model the temporal model to handle dynamic facial expression recognition (DFER) tasks. However, this approach faces two major challenges: First, the image domain is significantly different from the video domain. Although the VLP model is mainly optimized for image-text pairs, in video processing tasks, videos are composed of a series of continuous frames. This means that VLP can be effectively applied to video frame-text pairs. However, traditional VLP models are not designed to process frame sequences, but focus on the analysis of static images. If the model is only fine-tuned with the initial frame data of the video, it may lead to the loss of video-related information, such as the relationship between each frame, facial movement and dynamic features (such as whether the eyebrows are raised, etc.), video context information, and how lacking it is in adapting to the video domain. Secondly, VLP is mainly used to recognize scenes and objects in images and is good at capturing the overall features of images. But they cannot recognize subtle aspects of facial expressions. When exploring the DFER task with VLP, many methods only focus on video-related information and ignore the information within each frame, such as subtle changes in key facial locations. Summary of the invention

[0007] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a method and system for facial expression recognition based on three-stage multimodal visual language prompts. Through multimodal visual-language prompt learning and key point coordination network, the dual information of video text and visual content can be fully integrated and effectively utilized, thereby achieving more accurate and efficient video understanding and processing capabilities, and improving the accuracy and adaptability of model recognition.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of the present invention discloses a method for facial expression recognition based on three-stage multimodal visual language prompts, comprising:

[0010] Obtain dynamic facial expression video dataset and facial action description text related to each expression;

[0011] The dynamic facial expression video dataset at least includes video frames and expression categories;

[0012] Obtaining an optimized prompt vector for each expression category based on the expression category;

[0013] An image encoder is used to extract facial feature vectors of each frame from video frames;

[0014] A three-stage optimizable hint learning method is used to optimize the facial feature vector and obtain the fused hint.

[0015] Input the fused prompts into the temporal model to obtain facial video features;

[0016] The facial action description text related to each expression and the optimizable hint vector for each expression category are integrated into the text encoder to extract the text features related to the facial expression category;

[0017] Calculate the cosine similarity between facial video features and text features related to facial expression categories to obtain the prediction probability;

[0018] According to the predicted probability, the recognition result is obtained;

[0019] Among them, the three-stage optimizable hints include the first-stage intra-frame hints, the second-stage inter-frame hints, and the third-stage global-level hints, specifically:

[0020] Perform the first stage intra-frame level hinting on the facial feature vector of each frame to obtain the feature embedding of each frame;

[0021] Perform the second stage of inter-frame level hints on the feature embedding of each frame to obtain inter-frame level hints;

[0022] The facial feature vector of each frame is subjected to the third stage of global-level hinting to obtain the global-level hint.

[0023] As a further technical solution, the fused prompts are input into the temporal model to obtain facial video features. Specifically, feature embedding, inter-frame level prompts and global level prompts are fused and input into the temporal model to extract facial video features.

[0024] As a further technical solution, after obtaining the dynamic facial expression video dataset, the facial key point features of the video frame are extracted through the facial key point detection model;

[0025] As a further technical solution, by constructing a consistency loss function, the consistency of facial video features and facial key point features of video frames is calculated.

[0026] As a further technical solution, text features related to facial expression categories are extracted, specifically:

[0027] Obtaining each expression category can optimize the hint vector;

[0028] The optimized hint vector for each expression category and the facial action description text related to each expression are input into the text encoder to obtain text features related to the facial expression category.

[0029] As a further technical solution, the prediction probability is optimized by constructing a cross entropy loss function.

[0030] As a further technical solution, the total loss function is constructed through the consistency loss function and the cross entropy loss function to optimize the model.

[0031] The second aspect discloses an expression recognition system based on three-stage multimodal visual language prompts, including:

[0032] A data acquisition module is used to acquire a dynamic facial expression video data set and facial action description texts related to each expression; the dynamic facial expression video data set at least includes video frames and expression categories; based on the expression category, an optimizable prompt vector is obtained for each expression category;

[0033] The facial video feature extraction module is used to extract the facial feature vector of each frame from the video frame using an image encoder; optimize the facial feature vector using a three-stage optimizable hint learning method to obtain fused hints; input the fused hints into the temporal model to obtain facial video features; wherein the three-stage optimizable hints include the first stage intra-frame hints, the second stage inter-frame hints and the third stage global hints, specifically:

[0034] Perform the first stage intra-frame level hinting on the facial feature vector of each frame to obtain the feature embedding of each frame;

[0035] Perform the second stage of inter-frame level hints on the feature embedding of each frame to obtain inter-frame level hints;

[0036] Perform the third stage global level hint on the facial feature vector of each frame to obtain the global level hint;

[0037] A text feature extraction module is used to integrate the facial action description text related to each expression and the optimizable prompt vector of each expression category into the text encoder to extract text features related to the facial expression category;

[0038] The prediction and recognition module is used to calculate the cosine similarity between the facial video features and the text features related to the facial expression category to obtain the prediction probability;

[0039] According to the predicted probability, the recognition result is obtained.

[0040] A third aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0042] One or more of the above technical solutions have the following beneficial effects:

[0043] (1) The three-stage optimizable cue learning method designed for the visual cue in the multimodal visual-language cue learning method proposed in this embodiment includes intra-frame level cues, inter-frame level cues, and global level cues, which achieves a comprehensive understanding of the video data. This method not only enhances the model's ability to learn features within each video frame, but also improves the ability to capture dynamic relationships between different frames, learns the relationship between each frame, effectively models the distribution of video data, and learns the context information of the entire video.

[0044] (2) The text prompt learning method guided by AU prior knowledge designed by the language prompt in the multimodal visual-language prompt learning method proposed in this embodiment uses facial action units (AU) as descriptive sentences and introduces text-optimizable prompt vectors to enable the model to autonomously learn contextual information related to expression categories. This not only improves the adaptability of the model, but also enhances the correspondence between video features and text features.

[0045] (3) The key point coordination network proposed in this embodiment uses a pre-trained key point detection model. By calculating the consistency loss between the extracted facial features and the key point features, the model can effectively capture subtle changes in facial expressions, especially in key facial areas (such as eyes, mouth, and eyebrows), so that the model significantly improves the recognition accuracy without increasing the learning cost.

[0046] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0048] Figure 1 A schematic diagram of a network structure of an expression recognition method based on three-stage multimodal visual language prompts according to a first embodiment of the present invention;

[0049] Figure 2 This is a flow chart of a method for facial expression recognition based on three-stage multimodal visual language prompts according to Embodiment 1 of the present invention;

[0050] Figure 3 This is a comparison chart of the attention heat map results generated by the dynamic facial header recognition model of Example 1 of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0053] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0054] Embodiment 1

[0055] In order to more clearly illustrate this embodiment, a method for facial expression recognition based on three-stage multimodal visual language prompts is provided, comprising:

[0056] S1, obtain dynamic facial expression video dataset and facial action description text related to each expression;

[0057] The dynamic facial expression video dataset at least includes video frames and expression categories;

[0058] Obtaining an optimized prompt vector for each expression category based on the expression category;

[0059] S2, extracting facial feature vectors of each frame from the video frame using an image encoder;

[0060] A three-stage optimizable hint learning method is used to optimize the facial feature vector and obtain the fused hint.

[0061] Input the fused prompts into the temporal model to obtain facial video features;

[0062] S3, integrating the facial action description text related to each expression and the optimizable prompt vector for each expression category into the text encoder to extract text features related to the facial expression category;

[0063] S4, calculating the cosine similarity between the facial video features and the text features related to the facial expression category to obtain the prediction probability;

[0064] According to the predicted probability, the recognition result is obtained.

[0065] like Figure 1 , Figure 2 As shown, in step S1, a dynamic facial expression video data set and facial action description texts related to each expression are obtained;

[0066] The dynamic facial expression video dataset at least includes video frames and expression categories;

[0067] Obtaining a cue vector for each expression category based on the expression category can optimize the cue vector.

[0068] S1-1. Obtain a dynamic facial expression video dataset.

[0069] In this embodiment, three standard datasets are used: DFEW, FERV39k and MAFW. The video dataset at least includes dynamic facial expression video frames and expression categories.

[0070] The DFEW dataset contains 11,697 outdoor scene video clips, each of which is equally divided into 5 segments without overlap. These video clips are independently annotated by 10 professionally trained annotators under the supervision of experts, and they are classified into seven basic emotion categories: happiness, anger, neutral, surprise, sadness, fear and disgust. These clips are selected from more than 1,500 different movies around the world, including a variety of environmental challenges, such as extreme lighting, occlusions, and various head postures. The method of this embodiment is evaluated by implementing a five-fold cross validation.

[0071] The FERV39k dataset, as the largest field DFER dataset in the current field, brings together a total of 38,935 video clips. These videos cover four different natural scenes, which are further divided into 22 detailed scenes including crime scenes, daily life clips, public speeches and war documentaries. In order to ensure the accuracy and diversity of the annotations, each video clip has been carefully annotated by 30 independent annotators and classified into one of the seven basic facial expression categories. In terms of data processing, the dataset adopts a strict random shuffling strategy to ensure that there is no overlap between the training set and the test set. Specifically, the dataset is divided into a training set (accounting for 80%) and a test set (accounting for 20%), providing a solid data support and verification platform for subsequent research.

[0072] The MAFW dataset collects 10,045 video clips in natural scenes. It is not only the first database to achieve large-scale multimodal, multi-label emotion annotation, but also significantly broadens the boundaries of sentiment analysis research. The dataset meticulously covers 11 basic single expressions (anger, fear, disgust, happiness, surprise, sadness, contempt, helplessness, anxiety, disappointment, and neutrality) to 32 complex multiple expression categories, fully demonstrating the diversity and complexity of human emotions. In addition, the MAFW dataset also comes with emotion description text, providing researchers with language modality data in addition to visual information.

[0073] S1-2. Obtain facial action description text related to each expression.

[0074] This example designs a text prompt learning method guided by AU prior knowledge. It abandons the direct use of category labels and instead adopts the AU sets associated with each expression category as the facial action description text related to each expression.

[0075] S1-3. Obtain an optimizable prompt vector for each expression category based on the expression category.

[0076] In this embodiment, a text-optimizable prompt vector is obtained according to each expression category to learn the unique context information of each expression category. The formula is:

[0077]

[0078] Among them, M c It defines the number of prompts contained in each category context, C is the total number of given facial expression categories, for each category n (where n is an integer ranging from 1 to C), is a set of vectors that match the dimensions of the text description embedding vector, m∈{1,2,…,M c}, these vectors are category-specific context representations and are independent of any descriptive text, ensuring the category-specificity and independence of the contextual information.

[0079] The carefully designed visual and text processing parts in this embodiment introduce a multimodal visual-language cue learning scheme and a key point coordination network. In terms of vision, it not only has the powerful image representation ability of the CLIP image encoder, but also adds a temporal model based on the Transformer encoder and a three-stage optimizable cue learning method, which is used for the model to learn the expression features in each frame, the dynamic relationship between different frames, and the contextual information of the entire video data, model the temporal features of the face, and thus generate a multi-dimensional facial feature representation including the intra-frame level, the inter-frame level, and the global level. Also in terms of vision, by adding the consistency loss calculation between the key point features extracted by the pre-trained facial key point detection model and the facial video features, the model learns more subtle changes in the key positions of the face during the training process.

[0080] In terms of text, a subset of AUs (Facial Action Units) associated with expression categories is used as text descriptors, and a text-optimizable hint vector is introduced for each expression category, so that the model can autonomously learn contextual information related to each expression category. The core goal of this embodiment is to fully integrate and effectively utilize the dual information of text and visual content in the process of facial expression video processing tasks, so as to achieve more accurate and efficient video understanding and processing capabilities.

[0081] In this embodiment, in the visual processing part, in order to model the temporal relationship and learn the distribution of video data, the CLIP image encoder is expanded and a temporal model composed of Vision Transformer (ViT) is introduced. The model is trained from the very beginning, and the encoder contains a multi-head self-attention mechanism and a feedforward neural network. The designed multimodal visual-language cue learning scheme introduces a three-stage optimizable cue learning method in the visual aspect, namely intra-frame level cue, inter-frame level cue and global level cue, and each stage is trained from scratch.

[0082] In this embodiment, the dynamic facial expression recognition model includes an image encoder, a text encoder, a three-stage optimizable prompt learning module, a time model and a facial key point detection model. Three standard data sets are input into the dynamic facial expression recognition model for training.

[0083] Among them, in this embodiment, the dynamic expression recognition model training process executes an end-to-end training process. The CLIP image encoder, text encoder and facial key point detection model are all existing and trained, and do not need to be trained separately. The text encoder of CLIP is fixed and does not participate in the update; the image encoder of CLIP is adjusted to adapt to the specific DFER task; the pre-trained facial key point detection model is in a frozen state during the training process and does not participate in the subsequent training process, thereby effectively controlling the increase of model parameters.

[0084] In this embodiment, the three-stage optimizable prompt learning module and the time model are trained from the initial stage.

[0085] During the training process, the video frames in the dynamic facial expression dataset are input into the image encoder to extract the facial feature vectors of the video frames. Then, the facial feature vectors are subjected to three-stage optimizable prompt learning training. In the first stage of intra-frame level prompt training, feature embedding is obtained. The feature embedding is subjected to the second stage of inter-frame level prompt training to obtain inter-frame level prompts. The facial feature vectors are used to train the third stage global level prompts to obtain a trained three-stage optimizable prompt learning module.

[0086] The feature embedding, inter-frame level prompts and global level prompts obtained in the three-stage optimizable prompt learning and training process are fused to get fused prompts. The fused prompts are input into the temporal model for iterative training to get a trained temporal model, which effectively integrates the video-specific information and extracts facial video features through the trained temporal model.

[0087] like Figure 1 , Figure 2 As shown, in step S2, an image encoder is used to extract a facial feature vector of each frame from the video frame;

[0088] A three-stage optimizable hint learning method is used to optimize the facial feature vector and obtain the fused hint.

[0089] The fused cues are input into the temporal model to obtain facial video features.

[0090] First, the CLIP visual encoder extracts features of all video frames. Then these features are added with learnable position encodings and input into the temporal model together with three-stage optimizable cues to help the model learn relevant information about the video data.

[0091] S2-1. Use an image encoder to extract the facial feature vector of each frame from the video frame.

[0092] The facial feature vector of each frame is extracted from the input video frames through the feature extraction part of the CLIP image encoder.

[0093] In this embodiment, a face video is given from the data set, and T frames of images are extracted from it. The size of each frame is H*W, forming a four-dimensional input tensor x∈R T×3×H×W . Use the pre-trained CLIP image encoder to encode all frames x in the video. t Extract facial feature vector where L is the length of the feature vector, t∈{1,2,…,T}.

[0094] S2-2. A three-stage optimizable hint learning method is used to optimize the learning of facial feature vectors to obtain fused hints.

[0095] The model of this embodiment designs a three-stage optimizable prompt learning strategy for video, and introduces the time model g(·), both of which are trained from the very initial stage. The designed three-stage optimizable prompt learning strategy for video is iteratively trained and learned through the time model during the training process, effectively integrating the video-specific information, so that the model can adapt to the video domain. In the prompt learning designed by this module, there are three goals: First, by summarizing the information of the corresponding frames, learn the expression features of the current frame. Second, by promoting the information interaction between frames, fully learn the time information in the video and capture the dynamic relationship between different frames. Third, introduce optimizable parameters to adjust the visual-language model to learn the distribution characteristics of the video data set.

[0096] Three additional prompts are added. First, learnable intra-frame prompts are introduced for the features of each extracted frame to learn the expression features in each frame. Second, inter-frame prompts are introduced to assist in modeling the discriminative information of each frame through intra-frame prompts to capture the dynamic relationship between different frames. Finally, M vThe global-level hints help the model learn the contextual information of the entire video data and model the distribution of video data, thereby overcoming the efficiency problem of Transformer in processing long sequence video data.

[0097] Intra-frame level hints: Intra-frame level hints are a summary of the information of the current frame, helping the model learn the expression features in the current frame. Specifically, for the facial feature vectors of all frames extracted by the CLIP image encoder, An additional intra-level hint i is added to the facial feature vector of each frame t ∈R D . Where D is the dimension of the feature embedding of the frame. The final feature embedding z of each frame is t , the formula is:

[0098]

[0099] where e is the added learnable position embedding, given by e sp With e tm It consists of two parts, sp represents the spatial position encoding, e tm represents the time position code; represents the facial feature vector of each frame, S is the length of the feature vector, t∈{1,2,…,T}; i t Intra-frame level hints representing the facial feature vector of each frame, i t ∈R D , D is the dimension of the feature embedding of the frame.

[0100] Inter-frame level hint: This inter-frame level hint is a learnable vector of the number of video frames T, which is randomly initialized and labeled as L = [l1,…,l T ], each cue is connected to the intra-level cue of the corresponding frame, and these intra-level cues assist the learning of inter-level cues by providing them with features extracted by the CLIP visual encoder The discriminative information of all frames is gathered together, where t∈{1,2,…,T}, which can model the relationship between frames and learn the temporal information of video data and the dynamic relationship between different frames. The inter-frame level prompt is:

[0101]

[0102] Among them, z t,0 is the intra-level cue obtained for each extracted frame, l t represents the corresponding t-th learnable vector in the inter-frame level cue.

[0103] Global level hint: Global level hint is also randomly initialized Mv learnable vectors, labeled It mainly helps the model learn the distribution of video data, capture the overall characteristics of video data, and learn the contextual information of the entire video data to overcome the efficiency problems of Transformer in processing long sequence video data.

[0104] S2-3. Input the fused prompts into the temporal model to obtain facial video features.

[0105] Inter-level hints and global level prompts And the obtained feature embedding Z = [z1,…,z T ] are fused together and input into the temporal model g(·) to obtain the facial video features. The formula is:

[0106]

[0107] Where Z represents feature embedding, represents the inter-frame level cue, and G represents the global level cue.

[0108] like Figure 1 , Figure 2 As shown in the figure, after the dynamic facial expression video dataset, the facial key point features of the video frame are extracted through the facial key point detection model.

[0109] Conventional FER tasks usually rely on extracting features directly from facial images, which can capture the overall changes in facial expressions, but may not be sensitive enough to subtle changes in the locations of facial key points. Facial key point detection can accurately locate key areas of the face (such as eyes, mouth, nose, etc.), which are particularly sensitive to changes in expression. Therefore, a key point coordination network is designed. This module calculates the consistency of facial key point features and facial image features, and uses facial key point detection to assist the model in learning the features of facial key point locations. By combining key point information, the accuracy of feature extraction can be improved.

[0110] (1) Extract facial key point features from video frames.

[0111] Specifically, the extracted face video of T frames, each frame of which has an image size of H*W, forms a four-dimensional input tensor x∈R T×3×H×W Input it into the frozen pre-trained facial key point detection model, extract the key point position features of each frame and fuse them into the facial key point feature f lm .

[0112] (2) By constructing a consistency loss function, the consistency between facial video features and facial key point features of video frames is calculated.

[0113] Final measurement f lm , and the facial video features f input by the temporal model V The consistency formula is

[0114]

[0115] Among them, (f V ) ij represents the facial features of the jth category in the i-th video sample, fused from all visual features, (f lm ) ij represents the facial key point features of the jth category in the ith video sample, fused from all frames, N represents the number of video samples, and K represents the number of categories.

[0116] When l con When it is minimum and close to 0, the extracted facial features are consistent with the facial key point features. The facial key point detection auxiliary model is used to learn the characteristics of the facial key point positions. By combining the key point information, the accuracy of feature extraction can be improved.

[0117] After the above steps, the consistency between the fused facial features and the key point features is calculated, which enhances the model's attention to important local features, promotes the collaborative extraction of features, and helps to improve the performance and generalization ability of the model.

[0118] like Figure 1 , Figure 2 As shown, in step S3, the facial action description text related to each expression and the optimizable prompt vector for each expression category are integrated into the text encoder to extract text features related to the facial expression category.

[0119] In the text processing part, for the proposed multimodal visual-language cue learning scheme, this example designed a text cue learning method guided by AU prior knowledge, abandoned the direct use of category labels, and instead used AU sets associated with each expression category as description sentences. In the subsequent experiments, the feasibility of other description schemes was discussed, such as trying a simple happy photo, a happy expression, etc., and even trying to use the expression phrases generated by ChatGPT related to a certain expression. The experimental results show that using AU sets associated with each expression category as description sentences works best.

[0120] Past studies have revealed that human facial expressions contain both universal features and unique and obvious marks. This reflects that in different emotional expressions, the activities of specific facial muscles have certain commonalities. For example, whether it is happiness or surprise, it is usually accompanied by the rise of eyebrows; while anger and sadness are usually manifested as the descent of eyebrows and the wrinkling of forehead. Given that the CLIP text encoder can extract semantic content from natural language, and experts usually use AU code to identify basic facial expressions (see Table 1), each expression corresponds to a set of specific action units. Therefore, the model of this embodiment uses the set of facial action units of each expression category as the text description on the text side to improve the accuracy of recognition.

[0121] Table 1

[0122]

[0123] S3-1. Obtain an optimizeable prompt vector for each expression category.

[0124] The text-optimizable hint vectors introduced for each expression category through formula (1) are designed to learn the unique contextual environment information of each expression category. These vectors are category-specific context representations and are independent of any descriptive text, ensuring the category specificity and independence of the contextual information.

[0125] S3-2, inputting the optimizable prompt vector for each expression category and the facial action description text related to each expression into a text encoder to obtain text features related to the facial expression category.

[0126] These contextual prompts n Passed as input to the text encoder h(·), the text encoder is able to generate a classification weight vector for each facial expression category n To represent the visual concept, where E represents the feature vector dimension, that is, the text features related to facial expressions, the formula is:

[0127]

[0128] like Figure 1 , Figure 2 As shown, in step S4, the cosine similarity calculation is performed on the facial video features and the text features related to the facial expression category to obtain the predicted probability, and the recognition result is obtained based on the predicted probability.

[0129] S4-1. Calculate the cosine similarity between the facial video features and the text features related to the facial expression category to obtain the prediction probability.

[0130] The temperature parameter learned by the CLIP model and the cosine similarity function are used together in the similarity calculation. The formula is:

[0131]

[0132] Where τ represents the temperature parameter learned by the visual-language model, cos(·,·) is the cosine similarity function, and f V represents facial video features, Represents textual features related to facial expressions.

[0133] S4-2. Obtain recognition results based on the predicted probability.

[0134] The predicted probability is optimized by constructing a cross entropy loss function.

[0135] Real data n The cross entropy loss of the predicted value is:

[0136]

[0137] When l cls When the absolute value of is the smallest, the predicted value is consistent with the actual data y n The closest, the difference between the predicted value and the actual data is the smallest. Specifically, during the training process, the model parameters are adjusted to minimize the cross entropy loss, thereby reducing the distance between the predicted value and the true value and improving the recognition accuracy. When processing new samples, the model outputs the predicted probability of each category and selects the category with the highest probability as the recognition result.

[0138] The total loss function is constructed through the consistency loss function and the cross entropy loss function to optimize the model.

[0139] The total loss function is:

[0140] L total = l con +l cls (9)

[0141] Among them, l con represents the consistency loss function, l cls represents the cross entropy loss function.

[0142] Three standard datasets are input into CLIP. During the training process, the text encoder of CLIP is fixed and does not participate in the update, while the image encoder of CLIP is adjusted to adapt to the specific DFER task. At the same time, the temporal model and all lightweight optimizable prompt vectors are trained from scratch. The facial key point detection model uses the pre-trained model and keeps its parameters unchanged. This model framework performs an end-to-end training process and finally uses the cross entropy loss function to evaluate the difference between the predicted results and the true labels. This model determines the final recognition result by comparing the similarity between visual features and various text embeddings. This strategy is different from the traditional classifier training method and shows better performance.

[0143] The weighted average recall (WAR, equivalent to accuracy) and unweighted average recall (UAR, the average of the accuracy of each category without considering the number of samples in each category) are used as indicators to measure the effectiveness of the method.

[0144] (1) Comparison of the model of this embodiment with various latest methods

[0145] In this study, the performance of the model of this embodiment is compared with a variety of state-of-the-art methods, using three standard datasets: DFEW, FERV39k, and MAFW. Similar to previous studies, five-fold cross validation was used for DFEW and MAFW, while FERV39k used its predefined training set and test set. To ensure the stability and reliability of the results, random seeds were used during the training process. Table 2 shows that the model of this embodiment outperforms other comparison methods in both the unweighted average recall rate (UAR) and the weighted average recall rate (WAR). Specifically, compared with the previous best results, the UAR on the DFEW, FERV39k, and MAFW datasets increased by 1.98%, 1.69%, and 0.7%, respectively, while the WAR increased by 0.9%, 0.61%, and 1.38%, respectively.

[0146] Table 2

[0147]

[0148] In order to more vividly demonstrate the model effect of this embodiment, Figure 3As shown, the visualization results of the attention heat map of DFER-CLIP and the model in this embodiment are shown. The first row is the original video frame, the second row is the facial key point map generated by the key point network on the original video frame, the third row is the attention map generated by the DFER-CLIP model, and the fourth row is the attention map generated by the model of this embodiment. By comparison, it is found that since the model of this embodiment guides the image extraction features by facial key points, and uses the AU unit set corresponding to the expression as the text description in terms of text, the model of this embodiment performs better in paying attention to the key positions of the face. For example, the model pays more attention to key areas such as eyebrows, eyes, nose, and mouth, so that the model of this embodiment can more accurately capture important information related to the emotional state, thereby achieving better results in the FER task.

[0149] The model of this embodiment is evaluated on five folds of the DFEW dataset and a visual analysis is performed, and the results are further explored through the confusion matrix. Figure 3 As shown in the figure, the visualization result of the confusion matrix is ​​comparable to the test accuracy of the model in the previous embodiment, demonstrating the superior performance of this model. However, it is noted that the model is more inclined to predict the video emotion as neutral. This may be because neutral is selected as a relatively safe prediction option compared to the risk of classifying it as other specific emotions, which may result in higher errors.

[0150] (2) Ablation experiment

[0151] When conducting the ablation experiment of the multimodal visual-language prompt method (see Table 3 for details), based on the model of this embodiment without any prompts, the text-optimizable prompt vector is first introduced, and a fixed setting is maintained here: the text-side prompts of the model of this embodiment are unified into 8 specific context prompts for each class (M c =8). The experimental data shows that the optimizable text prompt improves the recognition accuracy of the model by learning the text context information. Then, the ablation experiment of the three-stage optimizable prompt learning method on the visual side is gradually introduced. First, the intra-frame level prompt i is introduced. t , the laboratory data showed that the introduction of this optimizable vector improved the accuracy but not significantly. This may be because the intra-frame level prompts learn the expression information within each frame and mainly use it to assist the learning of inter-frame level prompts. Secondly, the inter-frame level prompt (L) was introduced. The experimental data shows that this method can help the model learn the discriminant line information of each frame, thereby learning the dependency relationship between each frame and improving the recognition accuracy. Subsequently, the global level prompt G (set M v =8), which is beneficial to the effectiveness of the model.

[0152] Table 3

[0153]

[0154]

[0155] Considering that ordinary facial expression feature extraction is not sensitive to subtle changes in the key positions of the face, a key point coordination network is introduced to enable the model to learn subtle changes in the key point positions by calculating the consistency between facial features and facial key point features. In order to verify the effectiveness of the method of this embodiment, as shown in Table 4, ablation experiments with and without the network were also performed on the DFEW, FERV39k and MAFW datasets. Compared with the method without the network, UAR increased by 0.28%, 0.31%, and 0.24%, respectively, and WAR increased by 0.23%, 0.32%, and 0.06%. The experiment shows that the key point coordination network can improve the accuracy of the model's prediction of emotion classification by combining facial expression features with key point features.

[0156] Considering that the design of the key point coordination network may cause concerns about a significant increase in model parameters, which may in turn lead to an increase in training costs. However, a pre-trained facial key point detection model was taken and frozen, and it was not involved in the subsequent training process, thereby effectively controlling the increase in model parameters. The impact of introducing the key point coordination network on the size of model parameters was evaluated. The experimental results show that after adding facial key point features and facial feature consistency loss, the increase in model parameters accounts for only 2.19%. This ratio shows that the cost of introducing the key point coordination network is relatively small and does not significantly affect the overall complexity of the model.

[0157] Table 4

[0158]

[0159]

[0160] The efficiency of the number of global-level prompts is analyzed in depth and verified through a series of experiments. The results are shown in Table 5. The experiments show that when the number of global-level prompts (M v ) reaches 8, the accuracy of the model tends to stabilize and approaches the optimal performance. Based on this finding, M v = 8 is set as the default number of global-level prompts to ensure stable and efficient model performance.

[0161] Table 5

[0162]

[0163] When studying the impact of text prompts on model performance, we focused on adjusting the number of text prompts. As shown in Table 6, the ablation experiment results on the three datasets show that increasing the number of contexts to more than 8 did not significantly improve model performance. It is inferred that too many learnable text prompts may cause the model to overfit the training set, thereby reducing the generalization ability on the test set. Based on this finding, the number of prompts on the text side is set to M. c =8.

[0164] Table 6

[0165]

[0166] The model of this embodiment is compared with three different manually designed text descriptions: "a photo of [class]", "an expression of [class]", and expression-related descriptions generated using ChatGPT (e.g., happy-related descriptions: raised cheeks, smiling mouth...). The results in Table 7 show that using the AU subsets corresponding to basic expressions as text descriptions of different emotion categories, the visual and text features are better aligned, and the recognition accuracy of the model is higher.

[0167] Table 7

[0168]

[0169] Embodiment 2

[0170] The purpose of this embodiment is to provide an expression recognition system based on three-stage multimodal visual language prompts, including:

[0171] A data acquisition module is used to acquire a dynamic facial expression video data set and facial action description texts related to each expression; the dynamic facial expression video data set at least includes video frames and expression categories; based on the expression category, an optimizable prompt vector is obtained for each expression category;

[0172] The facial video feature extraction module is used to extract the facial feature vector of each frame from the video frame using an image encoder; optimize the facial feature vector using a three-stage optimizable hint learning method to obtain fused hints; input the fused hints into the temporal model to obtain facial video features; wherein the three-stage optimizable hints include the first stage intra-frame hints, the second stage inter-frame hints and the third stage global hints, specifically:

[0173] Perform the first stage intra-frame level hinting on the facial feature vector of each frame to obtain the feature embedding of each frame;

[0174] Perform the second stage of inter-frame level hints on the feature embedding of each frame to obtain inter-frame level hints;

[0175] Perform the third stage global level hint on the facial feature vector of each frame to obtain the global level hint;

[0176] The text feature extraction module integrates the facial action description text related to each expression and the optimizable hint vector of each expression category into the text encoder to extract the text features related to the facial expression category;

[0177] The prediction recognition module is used to calculate the cosine similarity between the facial video features and the text features related to the facial expression category to obtain the prediction probability; and obtain the recognition result based on the prediction probability.

[0178] The facial expression recognition system based on three-stage multimodal visual language prompts implements the facial expression recognition method based on three-stage multimodal visual language prompts in the first embodiment.

[0179] Embodiment 3

[0180] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0181] Embodiment 4

[0182] The purpose of this embodiment is to provide a computer-readable storage medium.

[0183] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0184] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0185] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0186] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for facial expression recognition based on three-stage multimodal visual language cues, characterized in that: include: Obtain dynamic facial expression video dataset and facial action description text related to each expression; The dynamic facial expression video dataset at least includes video frames and expression categories; Obtaining an optimized prompt vector for each expression category based on the expression category; An image encoder is used to extract facial feature vectors of each frame from video frames; A three-stage optimizable hint learning method is used to optimize the facial feature vector and obtain the fused hint. Input the fused prompts into the temporal model to obtain facial video features; The facial action description text related to each expression and the optimizable hint vector for each expression category are integrated into the text encoder to extract the text features related to the facial expression category; Calculate the cosine similarity between facial video features and text features related to facial expression categories to obtain the prediction probability; According to the predicted probability, the recognition result is obtained; Among them, the three-stage optimizable hints include the first-stage intra-frame hints, the second-stage inter-frame hints, and the third-stage global-level hints, specifically: Perform the first stage intra-frame level hinting on the facial feature vector of each frame to obtain the feature embedding of each frame; Perform the second stage of inter-frame level hints on the feature embedding of each frame to obtain inter-frame level hints; The facial feature vector of each frame is subjected to the third stage of global-level hinting to obtain the global-level hint.

2. The method for facial expression recognition based on three-stage multimodal visual language prompts as claimed in claim 1, characterized in that: The fused prompts are input into the temporal model to obtain facial video features. Specifically, feature embedding, inter-frame level prompts and global level prompts are fused and input into the temporal model to extract facial video features.

3. The method for facial expression recognition based on three-stage multimodal visual language prompts as claimed in claim 1, characterized in that: After obtaining the dynamic facial expression video dataset, the facial key point features of the video frames are extracted through the facial key point detection model.

4. The method for facial expression recognition based on three-stage multimodal visual language prompts as claimed in claim 1, characterized in that: By constructing a consistency loss function, the consistency between facial video features and facial key point features of video frames is calculated.

5. The method for facial expression recognition based on three-stage multimodal visual language prompts as claimed in claim 1, characterized in that: Extract text features related to facial expression categories, specifically: Obtaining each expression category can optimize the hint vector; The optimized hint vector for each expression category and the facial action description text related to each expression are input into the text encoder to obtain text features related to the facial expression category.

6. The method for facial expression recognition based on three-stage multimodal visual language prompts as claimed in claim 1, characterized in that: The predicted probability is optimized by constructing a cross entropy loss function.

7. The method for facial expression recognition based on three-stage multimodal visual language prompts as claimed in claim 1, characterized in that: The total loss function is constructed through the consistency loss function and the cross entropy loss function to optimize the model.

8. An expression recognition system based on three-stage multimodal visual language cues, characterized in that: include: A data acquisition module is used to acquire a dynamic facial expression video dataset and facial action description texts related to each expression; The dynamic facial expression video dataset at least includes video frames and expression categories; Obtaining an optimized prompt vector for each expression category based on the expression category; The facial video feature extraction module is used to extract the facial feature vector of each frame from the video frame using an image encoder; the facial feature vector is optimized and learned using a three-stage optimizable prompt learning method to obtain a fused prompt; The fused prompts are input into the temporal model to obtain facial video features; among them, the three-stage optimizable prompts include the first-stage intra-frame prompts, the second-stage inter-frame prompts, and the third-stage global prompts, specifically: Perform the first stage intra-frame level hinting on the facial feature vector of each frame to obtain the feature embedding of each frame; Perform the second stage of inter-frame level hints on the feature embedding of each frame to obtain inter-frame level hints; Perform the third stage global level hint on the facial feature vector of each frame to obtain the global level hint; A text feature extraction module is used to integrate the facial action description text related to each expression and the optimizable prompt vector of each expression category into the text encoder to extract text features related to the facial expression category; The prediction and recognition module is used to calculate the cosine similarity between the facial video features and the text features related to the facial expression category to obtain the prediction probability; According to the predicted probability, the recognition result is obtained.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are performed.

Images