Model training method and device, facial expression recognition method and device and electronic equipment

Abstract

The embodiment of the invention provides a model training method, a facial expression recognition method and device and electronic equipment, and relates to the technical field of video processing. The model training method comprises the following steps: acquiring a sample video and a first sample label; extracting a time feature and a space feature of the sample video by using a time-space feature extraction network in the facial expression recognition model of the initial structure; calculating an attention weight representing the correlation between the spatial feature and the time feature by using a mapping network; performing weighted aggregation on the time features by using the attention weight to obtain fused spatio-temporal features; inputting the fused spatial-temporal features into a classification network to obtain a first recognition result; and performing model training based on the difference between the first recognition result and the first sample label to obtain a trained facial expression recognition model with higher accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a model training method, a facial expression recognition method, a device and an electronic device. BACKGROUND

[0002] In the technical field of video processing, an electronic device can use a trained facial expression recognition model to extract image features (which can be referred to as spatial features) of video frames in a to-be-recognized video, and obtain optical flow information (which can be referred to as temporal features) between adjacent frames in the to-be-recognized video; perform feature fusion on the spatial features and the temporal features of the to-be-recognized video; and then, based on the feature fusion result, recognize facial expressions in the to-be-recognized video.

[0003] In the prior art, the facial expression recognition model is used to linearly fuse the temporal features and the spatial features. For example, the temporal features and the spatial features are concatenated; or, the temporal features and the spatial features are mapped to the same dimension, and then the mapping results of the two features are element-wise summed; or, the mapping results of the two features can also be element-wise averaged.

[0004] However, in actual scenarios, the temporal features and the spatial features of a video are usually in a nonlinear relationship. For example, the facial expression in video 1 is "smiling", and the facial expression in video 2 is "crying". In these two videos, the mouth shape before the facial expression changes can be the same, and the mouth shape after the facial expression is completed can also be the same, such as both being flat mouth corners. Therefore, the spatial features of these two videos can be the same. However, the mouth corners of the face in video 1 can first be raised from the flat state and then flattened, and the mouth corners of the face in video 2 can first be lowered from the flat state and then flattened. That is, the temporal features of these two videos can be different. As can be seen, the temporal features and the spatial features of a video are not linearly related, that is, there is a complex nonlinear relationship between the temporal features and the spatial features. Linearly fusing the temporal features and the spatial features will cause the feature fusion result to lose the above nonlinear relationship, and thus the accuracy of the subsequent facial expression recognition result is not high. That is, the accuracy of the existing facial expression recognition model is not high. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a model training method, a facial expression recognition method, a device and an electronic device to obtain a facial expression recognition model with higher accuracy. The specific technical solutions are as follows:

[0006] In a first aspect of this invention, a model training method is provided, the method comprising: acquiring a sample video and a first sample label indicating the category of a sample facial expression in the sample video; extracting temporal and spatial features of the sample video using a spatiotemporal feature extraction network in an initial structure facial expression recognition model; wherein the facial expression recognition model further comprises a mapping network and a classification network; calculating attention weights representing the correlation between the spatial and temporal features using the mapping network; weighting and aggregating the temporal features using the attention weights to obtain fused spatiotemporal features; inputting the fused spatiotemporal features into the classification network to obtain a first recognition result representing the category of a sample facial expression in the sample video; adjusting the model parameters of the initial structure facial expression recognition model based on the difference between the first recognition result and the first sample label until the model converges, thereby obtaining a trained facial expression recognition model.

[0007] In a second aspect of this invention, a facial expression recognition method is also provided. The method includes: acquiring a video to be recognized; extracting temporal and spatial features of the video to be recognized using a spatiotemporal feature extraction network of a trained facial expression recognition model; wherein the trained facial expression recognition model is obtained based on any of the model training methods described in the first aspect; the facial expression recognition model further includes a mapping network and a classification network; using the mapping network, calculating attention weights characterizing the correlation between the spatial and temporal features; using the attention weights to perform weighted aggregation of the temporal features to obtain fused spatiotemporal features; and inputting the fused spatiotemporal features into the classification network to obtain a target recognition result representing the category of the facial expression to be recognized in the video to be recognized.

[0008] In a third aspect of the present invention, a model training apparatus is provided, the apparatus comprising: a sample acquisition module for acquiring sample videos and a first sample label indicating the category of sample facial expressions in the sample videos; a sample feature extraction module for extracting temporal and spatial features of the sample videos using a spatiotemporal feature extraction network in an initial structure facial expression recognition model; wherein the facial expression recognition model further comprises a mapping network and a classification network; a first weight calculation module for calculating attention weights characterizing the correlation between the spatial and temporal features using the mapping network; a sample fusion module for weighted aggregation of the temporal features using the attention weights to obtain fused spatiotemporal features; a first recognition module for inputting the fused spatiotemporal features into the classification network to obtain a first recognition result representing the category of sample facial expressions in the sample videos; and a parameter adjustment module for adjusting the model parameters of the initial structure facial expression recognition model based on the difference between the first recognition result and the first sample label, until the model converges to obtain a trained facial expression recognition model.

[0009] In a fourth aspect of this invention, a facial expression recognition device is provided. The device includes: a video acquisition module for acquiring a video to be recognized; a spatiotemporal feature extraction module for extracting temporal and spatial features of the video to be recognized using a spatiotemporal feature extraction network of a trained facial expression recognition model; wherein the trained facial expression recognition model is obtained based on any of the model training methods described in the first aspect; the facial expression recognition model further includes a mapping network and a classification network; a second weight calculation module for calculating attention weights representing the correlation between the spatial and temporal features using the mapping network; a spatiotemporal fusion module for weighted aggregation of the temporal features using the attention weights to obtain fused spatiotemporal features; and a second recognition module for inputting the fused spatiotemporal features into the classification network to obtain a target recognition result representing the category of the facial expression to be recognized in the video to be recognized.

[0010] In a fifth aspect of the present invention, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the model training method described in any of the first aspects above, or the facial expression recognition method described in any of the second aspects above, when executing the program stored in the memory.

[0011] In another aspect of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the model training method described in any of the first aspects above, or the facial expression recognition method described in any of the second aspects above.

[0012] In another aspect of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the model training method described in any of the first aspects above, or the facial expression recognition method described in any of the second aspects above.

[0013] The model training method provided in this invention utilizes a spatiotemporal feature extraction network in a facial expression recognition model to extract temporal and spatial features from sample videos. Spatial features are static facial features, while temporal features are dynamic facial muscle movement features. An attention weight representing the correlation between spatial and temporal features is calculated using a mapping network. This attention weight can characterize the non-linear relationship between spatial and temporal features. The attention weight is then used to weighted aggregate the temporal features, i.e., the weighted aggregation process of temporal features is guided by spatial features. Considering the correlation between spatial and temporal features, the spatial and temporal features are fused, obtaining fused spatiotemporal features without losing the non-linear relationship between them. Using these fused spatiotemporal features, the facial expression recognition model is parameterized, enabling it to learn the non-linear relationship between spatial and temporal features. Experimental results show that this achieves complementarity between spatial and temporal features, guiding the facial expression recognition model to interpret the meaning of dynamic muscle movements based on static image features, thus improving the accuracy of the facial expression recognition model. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0015] Figure 1 This is a first flowchart of a model training method provided in an embodiment of the present invention;

[0016] Figure 2 This is a second flowchart of the model training method provided in an embodiment of the present invention;

[0017] Figure 3 This is a third flowchart of the model training method provided in the embodiments of the present invention;

[0018] Figure 4A schematic diagram of a model training method provided in an embodiment of the present invention;

[0019] Figure 5 A flowchart of a facial expression recognition method provided in an embodiment of the present invention;

[0020] Figure 6 A structural diagram of a model training device provided in an embodiment of the present invention;

[0021] Figure 7 A structural diagram of a facial expression recognition device provided in an embodiment of the present invention;

[0022] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0024] Facial Expression Recognition (FER) is one of the core technologies in the fields of artificial intelligence and computer vision. By processing facial images or video frame sequences, it can identify facial expressions and then interpret a person's state, such as emotional state or physiological state, based on the identified facial expressions. Subsequent processing is then performed based on the interpreted state.

[0025] For example, in HCI (Human-Computer Interaction) scenarios, based on FER (Feedback Experience) technology, systems can be developed that provide intelligent feedback based on the interpreted emotional state of a person (such as their emotions). For instance, this system could be educational software, with the person being a student using the software; the software could adjust its teaching strategies based on the interpreted student's emotions. Alternatively, the system could be a virtual assistant or chatbot, with the person being a user; the system could adjust its empathy based on the interpreted user's emotions and communication style to enhance the virtual assistant's or chatbot's empathy.

[0026] For example, in the healthcare and psychology context, FER technology can assist doctors in objectively assessing patients' mental health status. By quantifying patients' emotional expressions, it can enable early screening and long-term detection of mental illnesses such as depression and anxiety.

[0027] For example, in the Automotive Safety scenario, ADAS (Advanced Driving Assistance System) can be built based on FER technology. By monitoring the driver's facial expressions in real time, it can determine whether the driver is in a dangerous driving state such as fatigued driving, distracted driving, or angry driving. When it is determined that the driver is in a dangerous driving state, it can issue a timely warning, thereby improving road safety.

[0028] For example, in the Marketing and Customer Analysis scenario, based on FER technology, companies can obtain market feedback and decision-making basis by detecting consumers' emotional state when facing products, advertisements, services, etc.

[0029] For example, in Public Safety and Law Enforcement scenarios, such as security monitoring, criminal investigation, and psychological analysis, FER technology can assist technical personnel in judging a person's psychological state and potential intentions, providing technical support for public safety.

[0030] It is evident that FER technology plays a crucial role in enhancing the naturalness of human-computer interaction and emotional intelligence analysis, and its application has permeated numerous fields. As the application of FER technology deepens, the requirements for its accuracy and robustness in real-world scenarios become increasingly stringent.

[0031] Early FER technology primarily focused on SFER (Stop Image Face Expression Recognition), which identifies facial expressions in a single image. SFER effectively extracts the spatial texture and shape features (spatial features) of faces in an image. However, in real-world scenarios, human perception of emotion depends on the dynamic changes in facial expressions; SFER loses the dynamic temporal information (temporal features) inherent in these changes, resulting in relatively low accuracy.

[0032] Therefore, with the development of FER technology, the core of FER technology has gradually shifted from SFER to DFER (Dynamic Facial Expression Recognition) based on video frame sequences. Based on the DFER method, by analyzing continuous video frame sequences, the complete process of dynamic changes in facial expressions is captured to obtain the aforementioned temporal features; these temporal features provide richer and more reliable information for the facial expression recognition process.

[0033] Early DFER methods required technicians to manually design feature descriptors (which can be called traditional handmade features), such as LBP (Local Binary Patterns), HOG (Histogram of Oriented Gradient), and Gabor (wavelet transform). These feature descriptors could extract texture, edge, and shape information from video frames to represent facial expressions. However, designing these feature descriptors required considerable expertise from technicians and was highly sensitive to factors such as lighting variations, head pose, and individual differences in the faces being identified, resulting in low generalization ability.

[0034] With the development of deep learning technology, CNN (Convolutional Neural Network) has become the mainstream method for extracting spatial features in SFER and DFER approaches due to its powerful automatic feature learning capabilities. Models built on CNN architectures, such as VGG (Visual Geometry Group) and ResNet (Residual Connections), can learn hierarchical features from low to high levels from the raw pixels of an image, thus extracting spatial features. Using hierarchical features for facial expression recognition can improve the accuracy and performance of the results.

[0035] In the DFER approach, temporal modeling networks, such as RNNs (Recurrent Neural Networks) or their variant LSTM (Long Short-Term Memory), can be used to perform temporal modeling of the spatial features of each image. The result of this temporal modeling can be referred to as temporal features. By fusing the spatial features of images in a video with the temporal features of the video, and then using the fused feature results for facial expression recognition, the accuracy of the recognition results can be further improved.

[0036] To better facilitate the feature fusion process described above, the DFER approach utilizes a two-stream network architecture with two parallel branches for feature extraction and fusion. This architecture includes a Spatial Stream for processing video frames to learn static facial appearance features, and a Temporal Stream for capturing facial motion information. Features extracted based on the Spatial Stream are the aforementioned spatial features; the Temporal Stream is typically implemented using Optical Flow, quantifying the dynamic changes in facial muscles by calculating the motion vector of each pixel between two adjacent video frames. These temporal features are the aforementioned temporal features. Finally, the temporal and spatial features obtained from the two-stream network architecture are fused.

[0037] However, existing technologies typically perform linear fusion of temporal and spatial features. But in real-world scenarios, the temporal and spatial features of a video often exhibit a non-linear relationship. Linear fusion of these features results in the loss of this non-linear relationship, leading to low accuracy in subsequent facial expression recognition. In other words, existing facial expression recognition models are not highly accurate.

[0038] To address the aforementioned issues, embodiments of the present invention provide a model training method applied to a training device. After training a more accurate facial expression recognition model using the method provided in this embodiment, the training device can send the trained model to a recognition device used for facial expression recognition. Subsequently, upon acquiring a video to be recognized, the recognition device can identify the category of facial expressions in the video based on its locally deployed facial expression recognition model. Alternatively, the training device and the recognition device can be the same device. After training a more accurate facial expression recognition model, the training device can identify the category of facial expressions in the video based on its own trained model; both of these approaches are reasonable.

[0039] See Figure 1 , Figure 1 This is a first flowchart of a model training method provided in an embodiment of the present invention. The method may include the following steps:

[0040] S101: Obtain the sample video and the first sample label indicating the category of the sample facial expression in the sample video.

[0041] S102: Utilize the spatiotemporal feature extraction network in the initial structure of the facial expression recognition model to extract the temporal and spatial features of the sample video.

[0042] The facial expression recognition model also includes a mapping network and a classification network.

[0043] S103: Calculate attention weights that represent the correlation between spatial and temporal features using a mapping network.

[0044] S104: Use attention weights to weight and aggregate temporal features to obtain fused spatiotemporal features.

[0045] S105: Input the fused spatiotemporal features into the classification network to obtain the first recognition result representing the category of the facial expression in the sample video.

[0046] S106: Based on the difference between the first recognition result and the first sample label, adjust the model parameters of the initial structure of the facial expression recognition model until the model converges, and obtain the trained facial expression recognition model.

[0047] Based on the above processing, a spatiotemporal feature extraction network in the facial expression recognition model is used to extract temporal and spatial features from the sample videos. Spatial features are static facial features, while temporal features are dynamic facial muscle movement features. An attention weight representing the correlation between spatial and temporal features is calculated using a mapping network. This attention weight can characterize the non-linear relationship between spatial and temporal features. Then, the attention weight is used to weighted aggregate the temporal features, i.e., the weighted aggregation process of temporal features is guided by spatial features. Considering the correlation between spatial and temporal features, the spatial and temporal features are fused, obtaining fused spatiotemporal features without losing the non-linear relationship between them. Using these fused spatiotemporal features, the facial expression recognition model is parameterized, enabling it to learn the non-linear relationship between spatial and temporal features. Experimental results show that this achieves complementarity between spatial and temporal features, guiding the facial expression recognition model to interpret the meaning of dynamic muscle movements based on static image features, thus improving the accuracy of the facial expression recognition model.

[0048] Regarding step S101, the shapes of the facial organs in the sample video can change dynamically, and the combination of changes in the shapes of the facial organs can represent facial expressions. For example, if the eyebrows in the sample video move down and the corners of the mouth turn up, the dynamic changes in the shapes of the eyebrows and mouth can represent the facial expression in the sample video as "smiling".

[0049] Sample videos can be in MP4 (Moving Picture Experts Group-4 Part 14) format, FLV (Flash Video) format, etc. Correspondingly, after acquiring the sample video, the training device can decode it to obtain a continuous sequence of video frames, which are then processed. Alternatively, the training device can directly acquire the decoded sequence of video frames as the sample video.

[0050] The first sample label can indicate the category of the sample facial expression. For example, the first sample label can be a vector, where each element corresponds one-to-one with a predefined category of facial expression; in the sample label, the element corresponding to the category of the sample facial expression is 1, and other elements are 0. Alternatively, the sample label can also record the category identifier corresponding to the category of the sample facial expression. Alternatively, the sample label can also be text describing the sample facial expression, such as "laughing," "crying," or "angry." All of these are reasonable.

[0051] The sample video and the first sample label can be obtained by technicians; or, they can be obtained by the user when performing facial expression recognition on the video to be recognized in the actual scene, based on the recognition result corresponding to the video to be recognized indicated by the user. Both of these are reasonable.

[0052] In step S102, a facial expression recognition model with an initial structure is locally deployed on the training device. The facial expression recognition model includes a spatiotemporal feature extraction network for extracting temporal and spatial features from sample videos. Temporal features represent the optical flow information between every two adjacent video frames in the sample video, including: the displacement vector of each pixel in the earlier video frame in the later video frame (i.e., the motion vector of each pixel), which can be denoted as a dynamic feature sequence. Spatial features represent the image features of video frames in a sample video, and can be denoted as static feature vectors. For example, spatiotemporal feature extraction networks can be C3D (Convolutional 3 Dimensions, a 3D convolutional neural network) or TimeSFormer (a spatiotemporal transformation network) built based on Transformer. Alternatively, spatiotemporal feature extraction networks can include temporal feature extraction networks (such as RNNs and LSTMs) and spatial feature extraction networks (such as CNNs).

[0053] For example, in this invention, RAFT (Recurrent All-Pairs Field Transforms) can be used as a temporal feature extraction network for optical flow estimation to obtain temporal features.

[0054] Traditional optical flow estimation networks, such as PWC-Net (Pyramidal Warping-Net) and LiteFlowNet, require downsampling operations. Based on a multi-level pyramid processing approach, they extract low-resolution feature maps from high-resolution feature maps (a coarse-to-fine feature processing method), and then perform optical flow estimation on these low-resolution feature maps. This traditional coarse-to-fine feature processing method leads to the loss of video content and the retention of errors from feature processing. Performing optical flow estimation based on feature maps that have lost video content and / or retain errors results in low accuracy of the final temporal features.

[0055] RAFT, without spatial scaling, can perform optical flow estimation on a set of unscaled, high-resolution feature maps. Specifically, when calculating optical flow information between two video frames, pixels indicating the same location in those two frames can be considered a pixel pair; the difference in pixel values ​​between the two pixels in a pixel pair can be used as the similarity of that pair. Based on RAFT, a 4D correlation volume containing the similarity of each pixel pair in the two video frames can be obtained. Using RAFT, optical flow information where all elements are zero (referred to as a zero optical flow field) can be generated first. Then, through a GRU (Gated Recurrent Unit), the zero optical flow field is iteratively optimized by querying the 4D correlation volume, resulting in increasingly accurate optical flow information. The final optical flow information is then used as a temporal feature. Experimental results demonstrate that in complex scenarios such as large displacements, non-rigid deformations, and occlusion between objects in videos, RAFT extracts more accurate temporal features compared to traditional optical flow estimation networks. Furthermore, RAFT can still extract relatively accurate temporal features even when the dataset changes, indicating that RAFT has stronger cross-dataset generalization ability.

[0056] When extracting temporal features from sample videos based on RAFT, for every two adjacent video frames in the sample video, these two adjacent video frames can be input into the temporal feature extraction network to obtain the optical flow information between the two adjacent video frames. The temporal features of the sample video include the optical flow information between every two adjacent video frames in the sample video.

[0057] For example, in this invention, VIT (Vision Transformer) can be used as the backbone network of the spatial feature extraction network to extract spatial features from sample videos.

[0058] Traditional spatial feature extraction networks (such as CNNs) use multiple convolutional layers to expand the receptive field, thereby capturing relationships between distant facial features in an image (such as the relationship between eyebrows and the corners of the mouth). However, the essence of this process is still to perceive global features of an image by processing local regions. This limits the acquisition of relationships between distant facial features, resulting in low accuracy of spatial features extracted by traditional spatial feature extraction networks.

[0059] VIT (Visual Interpretation of the Image) divides each video frame into multiple patches and, based on a self-attention mechanism, learns the relationships between all image patches in the image (which can be called establishing global contextual understanding), thereby extracting image features that contain global contextual understanding. Since determining facial expressions in real-world scenarios often requires considering the relationships between various facial features, such as the relationship between eyebrows and the corners of the mouth, extracting more accurate global image features based on VIT is beneficial for subsequent facial expression recognition, thus improving the accuracy of the resulting recognition.

[0060] When extracting spatial features from sample videos based on VIT, each video frame of the sample video can be input into VIT to obtain the image features of that video frame; the spatial features of the sample video can include the image features of each video frame. Alternatively, the training device can select a preset number of video frames from the videos included in the sample video, and then input the selected video frames into VIT to obtain the image features of the selected video frames; the spatial features of the sample video can include the image features of the selected video frames.

[0061] For steps S103 and S104, the facial expression recognition model further includes a mapping network, which may include multiple linear projection layers. Using the mapping network, temporal and spatial features can be mapped; that is, the dot product of the mapped feature and the weight matrix of the mapping network used to map the feature is calculated. Based on the obtained mapping result, attention weights representing the correlation between spatial and temporal features are calculated. The specific method for calculating the attention weights will be described in detail in subsequent embodiments and will not be repeated here.

[0062] Furthermore, the training device can be based on attention mechanisms, such as multi-head attention mechanisms and cross-attention mechanisms, to use attention weights to weighted aggregate temporal features, obtaining fused spatiotemporal features in vector form, which can be denoted as... .

[0063] Regarding step S105, the classification network can be a classification head, such as including fully connected layers and activation functions; or it can include an MLP (Multilayer Perceptron) and softmax (activation function). The classification network is used to predict the probability of sample facial expressions in the sample video belonging to each preset category. The classification network can output the predicted probability of the sample facial expression belonging to each preset category as the first recognition result; or it can output only the label of the category with the highest probability as the first recognition result, both of which are reasonable. In real-world scenarios, the preset facial expression categories typically include 7 basic expressions.

[0064] Regarding step S106, the training device can adjust the model parameters of the initial facial expression recognition model based on the difference between the first recognition result and the first sample label until the model convergence condition is met, thus obtaining a trained facial expression recognition model. For example, a loss function value representing the difference between the recognition result and the sample label can be calculated based on a preset loss function, and the calculated result can be used as the loss value to be utilized. The preset loss function can be the MSE Loss (Mean Squared Error Loss) function, the BCELoss (Binary Cross-Entropy Loss) function, etc. The model convergence condition can be that the model parameters of the facial expression recognition model have been adjusted a preset number of times, and the calculated loss value is less than a preset loss threshold, etc. This invention does not limit this to specific conditions.

[0065] It is understood that the facial expression recognition model involved in this embodiment of the invention is an end-to-end model, that is, a sample video is input into the facial expression recognition model, and the facial expression recognition model outputs a first recognition result representing the category of the sample facial expression in the sample video. Accordingly, when adjusting the model parameters of the facial expression recognition model, the parameters of each network included in the facial expression recognition model, such as the aforementioned spatiotemporal feature extraction network, mapping network, and classification network, will also be adjusted. In subsequent embodiments, if the facial expression recognition model also includes other networks (such as a face detection network and a face alignment network), the parameters of the networks involved in the subsequent embodiments will also be adjusted. In addition, during the model training process, the central features corresponding to each category involved in the subsequent embodiments can also be adjusted. That is, the networks included in the facial expression recognition model are jointly optimized to obtain a globally optimal trained facial expression recognition model.

[0066] In some embodiments, the facial expression recognition model further includes a face detection network and a face alignment network.

[0067] Accordingly, the aforementioned step S101 may include the following steps:

[0068] Step A1: Obtain the original video.

[0069] Step A2: For each video frame in the original video, use a face detection network to detect the face region in that video frame, which will be used as the region to be aligned.

[0070] Step A3: Using a face alignment network, perform an affine transformation on each region to be aligned to obtain the face alignment result corresponding to that region.

[0071] Step A4: Sort the face alignment results corresponding to each region to be aligned according to the temporal order of each region in the original video, and use the video composed of the sorted face alignment results as the sample video.

[0072] In this embodiment of the invention, the sample video is a video composed of aligned face regions (i.e., face alignment results); the original video is a video that has not undergone alignment processing. The original video can be in MP4 format, FLV format, etc.; the training device can decode the original video to obtain a continuous sequence of video frames (which can be called the original sequence), that is, obtain each video frame in the original video.

[0073] For each video frame in the original sequence, a face detection network can be used to determine the bounding rectangle of the face in that video frame. The image region within the determined bounding rectangle, i.e., the face region in that video frame, is used as the region to be aligned. The face detection network can be built based on RetinaFace or MTCNN (Multi-Task Convolutional Neural Network), and this invention does not limit it to this.

[0074] Then, using a face alignment network, affine transformations are performed on facial reference points (such as the tip of the nose, the center of the eyes, etc.) in each region to be aligned, aligning the faces in each region to the same size and pose (collectively referred to as the standard facial shape), thus obtaining the face alignment result corresponding to each region. The pose of the face represents the orientation and rotation angle of the face; for example, if the face in the standard facial shape is a frontal view, and the face in the region to be aligned is tilted, then the face in the region to be aligned is aligned to the standard facial shape, that is, the region to be aligned is rotated through an affine transformation to obtain an image of the frontal face, which is used as the face alignment result corresponding to that region.

[0075] The training device can identify video frames containing frontal faces from each video frame and then use the size and posture of the face in the identified video frames as the standard facial shape; alternatively, it can use the size and posture of the face in a specified frame of the original video as the standard facial shape, such as the first video frame, the last video frame, etc., which are all reasonable.

[0076] Then, according to the temporal order of each region to be aligned in the original video, the face alignment results corresponding to each region to be aligned are sorted, and the video composed of the sorted face alignment results is used as the sample video. In this way, a sample video is obtained in which the faces are aligned and the dynamic changes of the facial expressions are consistent with the dynamic changes of the facial expressions in the original video.

[0077] Because real-world scenarios are often in-the-wild, the posture of people in the video to be recognized, such as turning their heads, looking up, or looking down, can change. This can cause facial distortion in the video frames. People in the video may wear glasses, masks, or have bangs, which can obscure their faces. All of these factors interfere with subsequent feature extraction. During model training, the datasets available to technicians are often collected under controlled conditions, such as CK+ (Communicating by Facial Expression: a Database for ActionUnits and Emotional Intensity) or JAFFE (Japanese Female Facial Expression Database). In these datasets, people typically do not move their heads, and their faces are not obscured. Furthermore, using these datasets for model training may lead the model to learn facial structural features (such as face shape and wrinkles) that are related to identity but not to facial expression. These uncontrolled real-world scenarios can also be referred to as the real world.

[0078] Based on the above processing described in this embodiment of the invention, face alignment enhances the model's performance in uncontrolled scenarios and improves the robustness of the trained facial expression recognition model. Furthermore, through the complementary and guiding information from temporal and spatial features, the facial expression recognition model accurately interprets the meaning of dynamic facial muscle movements, reducing interference from identity-related features and enhancing the model's resilience to interference from identity-related features in uncontrolled scenarios. This allows the model to learn general facial expression-related features, improving its versatility and its ability to generalize to new individuals.

[0079] In some embodiments, the spatiotemporal feature extraction network includes a temporal feature extraction network and a spatial feature extraction network.

[0080] exist Figure 1 Based on this, see Figure 2 The aforementioned step S102 may include the following steps:

[0081] S1021: For each facial organ, a temporal feature extraction network is used to extract the optical flow information between the image regions occupied by the facial organ in every two adjacent video frames in the sample video, so as to obtain the temporal features of the sample video.

[0082] S1022: Based on the optical flow information of every two adjacent video frames in the sample video, select one video frame from all the video frames included in the sample video as the key video frame.

[0083] S1023: Utilize a spatial feature extraction network to extract the image features of the image regions occupied by each facial organ in key video frames, and use them as spatial features of the sample video.

[0084] In this embodiment of the invention, facial organs may include eyebrows, eyes, nose, mouth, etc.; depending on the needs of the actual scenario, adjacent facial organs may also be combined, such as combining eyebrows and eyes into eyebrows and eyes as a facial organ; or combining mouth and nose into mouth and nose as a facial organ, which are all reasonable.

[0085] It's understandable that the facial region includes both the actual face and the background; the background is unrelated to facial expressions. Furthermore, research in psychology and computer vision shows that emotions are primarily expressed through muscle movements in specific facial areas, which can be termed diagnostic regions. For example, sadness is mainly reflected in muscle movements in the eyebrow and brow area, expressed by frowning at the brow; similarly, happiness is mainly reflected in muscle movements in the mouth area, expressed by raising the corners of the mouth. In other words, apart from these diagnostic regions, other facial areas (such as the cheeks and forehead) have relatively little correlation with facial expressions. If temporal and spatial features (hereinafter collectively referred to as spatiotemporal features) are obtained based on complete video frames, or if spatiotemporal features are obtained based on extracted face regions, it is necessary to process image regions that contribute little to the facial expression recognition process, increasing the computational load of the spatiotemporal feature extraction process. Furthermore, spatiotemporal features obtained based on image regions that contribute little to the facial expression recognition process may also interfere with the training process of the facial expression recognition model, leading to a decrease in the accuracy of the trained facial expression recognition model.

[0086] Therefore, in this embodiment of the invention, spatiotemporal features are extracted based on facial organs. The aforementioned problems are addressed by reducing the processing of image regions that contribute less to the facial expression recognition process.

[0087] Specifically, for each video frame in the sample video, the training device can detect the facial landmarks of the face in that video frame, as well as the facial organs to which each facial landmark belongs, based on a facial landmark detection algorithm. Examples of facial landmark detection algorithms include MediaPipe 12 (motion-sensing interaction) and PFLD (Practical Facial Landmark Detector).

[0088] For each facial feature, the training device can determine the smallest bounding rectangle of the facial key points belonging to that facial feature in the video frame. The image region within this smallest bounding rectangle, that is, the image region occupied by that facial feature in the video frame, can be called a Region of Interest (ROI).

[0089] Since the image area occupied by the same facial organ may be different in different video frames in real-world scenarios, the training device can first resize the ROI corresponding to the same facial organ. This can be done by cropping, stretching, or filling the ROI to adjust each ROI to a preset size, such as 64×64 or 128×128. The preset size corresponding to the ROI of different facial organs can be the same or different, and this invention does not limit this.

[0090] Then, using a temporal feature extraction network, optical flow information between the image regions occupied by the facial organ in every two adjacent video frames of the sample video is extracted, generating a dense optical flow field sequence for the ROI corresponding to the facial organ. An optical flow field can be represented as a two-dimensional vector map (also called an optical flow map), including the motion information of each pixel in the ROI in the horizontal and vertical directions (i.e., the aforementioned motion vector). Given that the facial organs include three types (e.g., eyebrows, eyes, nose, and mouth), the preset size of the ROI corresponding to each facial organ is 64×64, and the sample video includes T+1 video frames, the training device can obtain a multi-channel feature optical flow map with a size of 64×64×6×T; 64×64 represents the size of one channel of the optical flow map, 6 represents 3×2, 3 represents the type of facial organ, and 2 represents the two directions (i.e., horizontal and vertical).

[0091] In one approach, the optical flow information of each facial organ corresponding to every two adjacent video frames in the sample video can be directly used as the temporal feature of the sample video.

[0092] In another approach, to further remove redundant features and reduce the computational load of subsequent processing, the training device can utilize a temporal encoder to extract features from the aforementioned multi-channel feature optical flow map, using the feature extraction results as the temporal features of the sample video. For example, the temporal encoder can be built based on 3D-CNN (three-dimensional convolutional neural network), or it can be built based on Transformer.

[0093] Furthermore, the training device can select one video frame from all the video frames in the sample video as the key video frame based on the optical flow information of every two adjacent video frames in the sample video. For example, a video frame in which the optical flow direction changes can be used as the key video frame.

[0094] In some embodiments, video frames where a facial expression reaches its peak can be used as key video frames. For example, if the facial expression is "smiling," the corners of the mouth may first rise and then flatten. The moment when the corners of the mouth transition from rising to flattening can be considered as the peak of the facial expression. Experimental results show that video frames where the facial expression reaches its peak provide the most information about the spatial features related to the facial expression. Video frames where the facial expression reaches its peak can be called apex frames.

[0095] The aforementioned step S1022 may include the following steps:

[0096] Step B1: For every two adjacent video frames in the sample video, calculate the average level of each element included in the optical flow information of each facial organ in the two video frames, and use it as the optical flow amplitude of the latest video frame in the two video frames for that facial organ.

[0097] Step B2: Calculate the average optical flow amplitude of the latest video frame in the two video frames for each facial organ, and use it as the optical flow amplitude to be utilized for the latest video frame in the two video frames.

[0098] Step B3: Identify the video frame with the largest optical flow amplitude in the sample video as the key video frame.

[0099] In this embodiment of the invention, for each pair of adjacent video frames in the sample video, the optical flow information for each facial organ in the two video frames includes multiple elements. The training device can calculate the average value of these multiple elements as the optical flow amplitude for the facial organ in the latest temporal frame among the two video frames. The optical flow amplitude is a one-dimensional numerical value.

[0100] Then, the average optical flow amplitude for each facial organ is calculated for the latest temporally consecutive video frame among the two video frames, and this average is used as the available optical flow amplitude for the latest temporally consecutive video frame. The larger the available optical flow amplitude of a video frame, the higher the probability that the video frame is a vertex frame. Therefore, the training device determines the video frame with the largest available optical flow amplitude in the sample videos as the key video frame.

[0101] Furthermore, the training device can divide each ROI in the key video frame into multiple non-overlapping image patches. For each image patch, a linear projection method is used to map the pixel matrix of the image patch into a patch embedding (image patch embedding vector). The training device can also generate a positional encoding indicating the location of the image patch in the key video frame. Feature concatenation is performed on the patch embedding and positional encoding of the image patch to obtain a feature concatenation result that preserves the positional information of the image patch in the key video frame. The Transformer encoder in VIT processes the feature concatenation result corresponding to each image patch based on a multi-head self-attention mechanism, which can learn the global spatiotemporal dependencies between image patches, thereby extracting the global features of the key video frame, i.e., obtaining the aforementioned static feature vector. .

[0102] Based on the above processing, vertex frames are used as key video frames, and the image features of the Regions of Interest (ROIs) in the vertex frames are extracted as spatial features of the sample videos. Obtaining spatial features based on vertex frames—that is, obtaining spatial features from the video frames with the most information related to facial expressions—reduces the probability of a decrease in the accuracy of the trained facial expression recognition model while reducing computational load and improving the efficiency of the model. Furthermore, extracting image features from the ROIs in the vertex frames reduces the introduction of image regions that contribute little to the facial expression recognition process, further reducing the probability of a decrease in the accuracy of the trained facial expression recognition model.

[0103] In some embodiments, the mapping network includes a query mapping layer, a key mapping layer, and a value mapping layer;

[0104] The aforementioned step S103 may include the following steps:

[0105] Step C1: Use the query mapping layer to perform feature mapping on spatial features to obtain the query vector.

[0106] Step C2: Use the key mapping layer to perform feature mapping on the time features to obtain the key matrix, and use the value mapping layer to perform feature mapping on the time features to obtain the value matrix.

[0107] Step C3: Calculate the dot product of the query vector and the key matrix to obtain the first relevance vector.

[0108] Step C4: Calculate the quotient of the first correlation vector and the scaling factor to obtain the second correlation vector.

[0109] The scaling factor is obtained based on the dimension of the key matrix.

[0110] Step C5: Normalize each element in the second relevance vector to obtain the attention weights.

[0111] Accordingly, the aforementioned step S104 may include: calculating the weighted sum of the attention weights and the vectors in the value matrix to obtain the fused spatiotemporal features.

[0112] In this embodiment of the invention, the weight matrix of the query mapping layer can be denoted as: Q represents Query. The dot product of the spatial features and the weight matrix of the query mapping layer is calculated, i.e., the query mapping layer is used to map the spatial features to obtain the query vector. For example, the aforementioned static feature vector can be mapped based on the following formula (1). Perform feature mapping to obtain the query vector Q:

[0113] ; (1)

[0114] Semantically, the query vector Q indicates which motion patterns of the dynamic feature sequence are noteworthy within a face with a specific appearance. The face with a specific appearance is represented by this static feature vector. The sample video contains the face; the appearance refers to the facial features such as eyebrows, eyes, mouth, and nose; the dynamic feature sequence focuses on the motion patterns, i.e., the temporal features of the video. The query vector Q guides the facial expression recognition model to focus on the meaning of facial muscle movements.

[0115] The weight matrix of the key mapping layer can be denoted as: K represents Key. The dot product of the temporal features and the weight matrix of the key mapping layer is calculated, that is, the key mapping layer is used to perform feature mapping on the temporal features to obtain the key matrix. For example, the aforementioned dynamic feature sequence can be based on the following formula (2). Perform feature mapping to obtain the key matrix K:

[0116] ; (2)

[0117] The weight matrix of the value mapping layer can be denoted as: V represents Value. The dot product of the time feature and the weight matrix of the value mapping layer is calculated, i.e., the value mapping layer is used to map the time feature to obtain the value matrix. For example, the aforementioned dynamic feature sequence can be mapped based on the following formula (3). Perform feature mapping to obtain the value matrix V:

[0118] ; (3)

[0119] In the key matrix K and value matrix V, each row corresponds to the same time step, meaning that each row is obtained by feature mapping of the temporal features acquired from the same video frame. Semantically, each row in the key matrix K indicates the "queryable index" in the dynamic feature sequence at that time step, i.e., the index provided by the video frame; the same row in the value matrix V indicates the "actual information content" in the dynamic feature sequence at that time step, i.e., the value matrix V is used to describe the temporal features provided by the video frame.

[0120] Furthermore, the training device can utilize Scaled Dot-Product Attention to first calculate the dot product between the query vector Q and the key matrix K, obtaining the first relevance vector; then, it calculates the square root of the dimension of the key matrix K as a scaling factor; finally, it calculates the quotient of the first relevance vector and the scaling factor to obtain the second relevance vector; and then, it uses softmax (a normalization function) to normalize each element in the second relevance vector, obtaining the attention weights that represent the relevance between spatial and temporal features. In other words, the key matrix K, guided by the query vector Q, determines the attention weights that indicate the importance of each temporal feature at each time step in the dynamic feature sequence. Obtaining these attention weights in this way reduces the probability of the model encountering the vanishing gradient problem.

[0121] Furthermore, the calculated attention weights are weighted sums of the vectors in the value matrix V to obtain the fused spatiotemporal features. That is, the importance of the temporal features at different time steps is determined based on the spatial features, and the temporal features are weighted and aggregated according to their importance. Specifically, the above processing can be achieved based on the following formula (4):

[0122] ; (4)

[0123] in, This represents the fusion of spatiotemporal features; Q represents the query vector, K represents the key matrix, and V represents the value matrix. This indicates matrix transpose processing. This represents the dimension of the key matrix K.

[0124] obtained in this way , which are the aforementioned vector-form fused spatiotemporal features This can be understood as a dynamically weighted aggregation result of temporal features guided by spatial features, including rich spatiotemporal features rich in contextual information.

[0125] In some embodiments, the facial expression recognition model also includes an auxiliary AU (facial action unit) detection head.

[0126] exist Figure 1Based on this, see Figure 3 Prior to step S106, the method may further include the following steps:

[0127] S107: Input the fused spatiotemporal features into the auxiliary AU detection head to obtain the second recognition result representing the facial expressions in the sample video corresponding to each preset AU label.

[0128] Each preset AU label is used to describe the deformation state of the facial organs of a human face.

[0129] Accordingly, the aforementioned step S106 may include the following steps:

[0130] S1061: Based on the difference between the second recognition result and the second sample label, and the difference between the first recognition result and the first sample label, adjust the model parameters of the initial structure of the facial expression recognition model until the model converges, and obtain the trained facial expression recognition model.

[0131] The second sample label indicates a preset AU label representing the facial expressions in the sample video.

[0132] In this embodiment of the invention, each preset AU tag is used to describe the deformation state of facial organs. Specifically, AU is a basic component of FACS (Facial Action Coding System), anatomically describing the movement of each facial muscle. For example, AU1 describes the lifting of the inner corner of the eyebrow, and AU12 describes the upward turning of the corners of the mouth. The combination of AUs constitutes the expressions that may appear in real-world scenarios. The preset AU tags include the aforementioned AU1, AU12, etc.

[0133] When training a facial expression recognition model, it is also possible to obtain instructions: a second sample label representing a preset AU label of facial expression in a sample video. The second sample label includes a preset AU label representing a facial expression in a sample video, that is, when a person makes that facial expression, the person's facial organs need to deform according to the deformation state described by the preset AU label. For example, if the facial expression is "smiling", that is, the corners of the mouth are raised, then the aforementioned AU12 used to describe "the corners of the mouth are raised" is the preset AU label representing "smiling".

[0134] For example, the second sample label may include each preset AU label. If the facial expression in the sample video matches the facial organ deformation state described by a preset AU label, then the preset AU label can represent the facial expression in the sample video. Correspondingly, the preset AU label can be in an active state (e.g., the state identifier is 1); otherwise, the preset AU label can be in an inactive state (e.g., the state identifier is 0). For example, based on the above example, when the facial expression in the sample video is "smiling", the state identifier of the preset AU label representing "smiling" (i.e., the aforementioned AU12) is 1; when the facial expression in the sample video is crying, that is, the corners of the mouth of the face in the sample video are drooping, then the state identifier of AU12 used to describe "the corners of the mouth are upturned" is 0.

[0135] Alternatively, the second sample label could simply include preset AU labels representing facial expressions in the sample video, which is also reasonable.

[0136] The auxiliary AU detection head is used to classify information based on the fused spatiotemporal features. For example, it can be a classification network based on MLP. After the output of the classification network based on MLP is processed by sigmoid (activation function), the probability of each preset AU label corresponding to the facial expression in the sample video is obtained, which is used as the second recognition result.

[0137] Furthermore, based on the differences between the second recognition result and the second sample label, as well as the differences between the first recognition result and the first sample label, the training device can adjust the model parameters of the initial structure of the facial expression recognition model. For example, the training device can calculate a loss function value representing the difference between the second recognition result and the second sample label (which can be called the auxiliary AU loss) and a loss function value representing the difference between the first recognition result and the first sample label (which can be called the main loss) based on a preset loss function. Then, it calculates the weighted sum of the auxiliary AU loss and the main loss to obtain the total loss; then, it adjusts the model parameters of the initial structure of the facial expression recognition model based on the total loss. The preset loss function used by the training device to calculate the auxiliary AU loss and the main loss can be the same or different; the preset loss function can be the aforementioned mean squared error loss function, binary cross-entropy loss function, etc.

[0138] For example, the training device can calculate the auxiliary AU loss based on the following formula (5):

[0139] ; (5)

[0140] in, This represents the auxiliary AU loss, where M represents the total number of preset AU labels. This indicates the activation status of the j-th preset AU label included in the sample labels. This represents the probability corresponding to the j-th preset AU label in the second recognition result.

[0141] In real-world scenarios, the emotions expressed by a person's facial expressions are complex and continuous. A single facial expression may contain multiple emotions (which can be called a compound expression), such as "mixed feelings of sadness and joy" or "surprise and delight." When acquiring datasets for training facial expression recognition models, different annotators may also have subjective differences in the labels they assign to the same facial expression, resulting in low accuracy of facial expression recognition models obtained using traditional methods.

[0142] Based on the above processing in the embodiments of the present invention, model training is performed based on auxiliary AU loss, which enables the facial expression recognition model to learn the features of underlying, anatomically significant muscle movements. This allows the model to discriminate facial expressions represented by underlying, objective muscle movements, guiding the model to perform facial expression recognition from the physiological field and improving the accuracy of the final facial expression recognition model.

[0143] In some embodiments, the main loss can be a weighted sum of Focal Loss and Center Loss. The classification network includes deep feature extraction layers and fully connected layers.

[0144] The aforementioned step S105 may include the following steps:

[0145] Step D1: Utilize the deep feature extraction layer to extract deep features from the fused spatiotemporal features, thereby obtaining the deep features of the sample.

[0146] Step D2: Use a fully connected layer to classify the deep features of the samples, and obtain the classification probability of each category for the facial expressions in the sample video as the first recognition result.

[0147] Accordingly, prior to the aforementioned step S1061, the method may further include the following steps:

[0148] Step D3: Calculate the focus loss using the classification probability of the sample facial expression category indicated by the first sample label.

[0149] Step D4: Calculate the center loss by using the difference between the deep features of the sample and the center features corresponding to the category indicated by the first sample label.

[0150] Step D5: Calculate the weighted sum of the focal loss and the center loss as the difference between the first identification result and the first sample label.

[0151] In this embodiment of the invention, the classification network includes multiple feature extraction layers for feature extraction. Using multiple feature extraction layers in the classification network for feature extraction can be called deep feature extraction. Correspondingly, the last feature extraction layer among these multiple feature extraction layers can be called the deep feature extraction layer. For example, if the last network layer of the classification network is a fully connected layer, the network layer above the fully connected layer can be used as the deep feature extraction layer. The deep feature extraction layer is used to extract deep features from the fused spatiotemporal features to obtain deep features of the sample. The fully connected layer is then used to classify the deep features of the sample to obtain the recognition result.

[0152] Then, according to the following formula (6), the center loss is calculated based on the difference between the deep features of the sample and the center features corresponding to each category that have been recorded:

[0153] ; (6)

[0154] in, Let N represent the center loss, and N represent the total number of sample videos used in one training cycle. This represents the deep feature of the i-th sample video; This represents the current central feature of the category indicated by the sample label corresponding to the i-th sample video; Representational normal form operations.

[0155] For each category of facial expressions, central features for that category can be randomly initialized first. Then, during the training of the facial expression recognition model, the central features for that category are adjusted based on the first recognition result of sample videos corresponding to that category. The central features for each category can be stored in the model as trainable model parameters.

[0156] The focus loss is calculated using the classification probability of the facial expression category indicated by the sample label, according to the following formula (7):

[0157] ; (7)

[0158] in, Indicates focal loss. and This is a hyperparameter, and its value range is (0,1). and They can be the same or different; Let t represent the classification probability of the sample facial expression category indicated by the sample label.

[0159] Furthermore, the aforementioned total loss can be calculated according to the following formula (8):

[0160] ; (8)

[0161] in, Indicates the total loss. Indicates the main loss. The weights representing the focus loss. The weights representing the center loss, This represents the weight of the auxiliary AU loss. , ,as well as This is a hyperparameter, and its value range is (0,1). , ,as well as They can be the same or different.

[0162] In real-world scenarios, different categories of facial expressions appear with varying frequencies. For example, expressions representing "neutrality" occur more frequently, while expressions expressing negative emotions such as "fear" and "disgust" occur less frequently. This leads to class imbalance in existing datasets. Training a model using traditional loss functions (such as cross-entropy loss) results in accurate recognition of the majority class (with a large number of samples) but poor recognition of the minority class (with a small number of samples), leading to poor overall model performance. Focus loss can reduce the contribution of easily classified (i.e., the majority class) samples to the total loss. Training the model based on focus loss allows it to learn the features of minority class samples, enabling minority class recognition. Training the model based on center loss can reduce the distance between the deep features of each video sample and the center features of its corresponding class. Combining focus loss with model training can reduce intra-class variations and increase the distance between features of different categories of facial expressions. By combining center loss, focus loss, and auxiliary AU loss in model training, the facial expression recognition model's ability to handle imbalanced data can be improved, the differences between the extracted features of different categories of facial expressions can be enhanced, and the model can be guided to perform facial expression recognition from a physiological perspective, thereby improving the accuracy of the final facial expression recognition model.

[0163] See Figure 4 , Figure 4 This is a schematic diagram of a model training method provided in an embodiment of the present invention.

[0164] like Figure 4 As shown, the model training method provided in this embodiment of the invention includes three stages:

[0165] Phase 1: Data preparation and preprocessing.

[0166] Input video. The input video is the sample video mentioned above.

[0167] Frame extraction and preprocessing. This involves decoding the sample video to obtain a continuous sequence of video frames; and preprocessing, which involves determining the face region in each video frame.

[0168] Frame-by-frame processing: Face detection and alignment. That is, the facial reference points in each face region are determined as mentioned above, and affine transformations are performed based on the facial reference points to obtain the face alignment results corresponding to each region to be aligned.

[0169] Facial landmark detection. That is, for each video frame in the sample video, the training device can detect the facial landmarks of the face in that video frame, as well as the facial organs to which each facial landmark belongs, based on the facial landmark detection algorithm.

[0170] Dynamic region of interest extraction: eyebrow and eye region, mouth region, and nose region. That is, for each facial feature, the training device determines the smallest bounding rectangle of the facial key points belonging to that facial feature in the video frame, thus obtaining the image region occupied by that facial feature in the video frame.

[0171] Vertex frames are selected based on optical flow amplitude. Vertex frames (also called ROI vertex frames) are the aforementioned key video frames. Specifically, for every two adjacent video frames in the sample video, the average value of each element in the optical flow information for each facial organ is calculated. This average value is then used as the optical flow amplitude of the latest temporally connected video frame for that facial organ. The average optical flow amplitude of the latest temporally connected video frame for each facial organ is then calculated and used as the usable optical flow amplitude for that latest temporally connected video frame. The video frame with the largest usable optical flow amplitude in the sample video is then identified as the key video frame.

[0172] The network layers that perform the above processing in Phase 1 can be collectively referred to as Module 1.

[0173] Phase 2: Feature Extraction.

[0174] RAFT (Recursive Full-Field Transform) Optical Flow Estimation: 6-channel optical flow stacking. That is, for each facial organ, a temporal feature extraction network is used to extract the optical flow information between the image regions occupied by that facial organ in every two adjacent video frames of the sample video. This information serves as the optical flow information for that facial organ in those two video frames. The 6 channels refer to the aforementioned 3×2, where 3 represents the type of facial organ and 2 represents the two directions. The optical flow stacking refers to the aforementioned multi-channel feature optical flow map. For each facial organ, the image region occupied by that facial organ in every two adjacent video frames of the sample video can be called the ROI (Region of Interest) frame sequence.

[0175] A temporal encoder generates dynamic features. These dynamic features are the aforementioned temporal features. Specifically, the temporal encoder is used to extract features from the multi-channel optical flow map, and the extracted features are used as the temporal features of the sample video.

[0176] VIT (Visual Transformer Interpreter) Feature Extraction: Patch Embedding + Self-Attention Encoding. As mentioned earlier, each Region of Interest (ROI) in the key video frame is divided into multiple non-overlapping image patches (ROI vertex frames). For each image patch, a linear projection is used to map the pixel matrix of that patch to a patch embedding. A positional encoding indicating the location of the image patch in the key video frame is generated. The patch embedding and positional encoding of the image patch are concatenated to obtain a feature concatenation result that preserves the positional information of the image patch in the key video frame. The Transformer encoder in VIT processes the feature concatenation result corresponding to each image patch based on a multi-head self-attention mechanism, thus obtaining a static feature vector. .

[0177] The network layers that perform the above processing in Phase Two can be collectively referred to as Module Two.

[0178] Phase Three: Integration, Classification, and Training.

[0179] Cross-modal attention fusion generates fused features. These fused features are the aforementioned fused spatiotemporal features. Specifically, the query mapping layer is used to map spatial features to obtain a query vector. A key mapping layer is used to map temporal features to obtain a key matrix, and a value mapping layer is used to map temporal features to obtain a value matrix. Attention weights are obtained using the query vector and the key matrix; then, the weighted sum of the attention weights and the vectors in the value matrix is ​​calculated to obtain the fused spatiotemporal features.

[0180] The fused features are processed using an auxiliary AU (Facial Action Unit) detector head to obtain AU prediction Logits, which are the probabilities corresponding to each of the aforementioned preset AU labels. Based on the real AU labels (i.e., the preset AU labels representing facial expressions in the sample video indicated by the aforementioned second sample labels) and the AU prediction Logits, the auxiliary AU loss is calculated. The fused features are processed using a main classification head (i.e., the aforementioned classification network) to obtain expression classification Logits, which are the classification probabilities of the aforementioned sample facial expressions for each category. The final prediction result can be obtained based on the classification probabilities. The focus loss is calculated using the classification probabilities of the sample facial expressions indicated by the real expression labels (i.e., the aforementioned first sample labels). The center loss is calculated using the deep features of the samples and the center features corresponding to the real expression labels.

[0181] The weighted sum of the auxiliary AU loss, center loss, and focal loss is calculated to obtain the total loss, which is the aforementioned unutilized loss. Then, based on the AdamW optimizer, the total loss is backpropagated to update the network weights (i.e., adjust the model parameters).

[0182] The network layer performing cross-modal attention fusion in Phase 3 can be referred to as Module 3; the other network layers performing Phase 3 can be collectively referred to as Module 4, which is used for classification and composite loss optimization. Classification involves obtaining the classification probability of the sample facial expression for each category through the master classifier; composite loss is the total loss mentioned above; composite loss optimization involves updating the network weights based on the total loss.

[0183] Because the information from static appearance features and motion feature sequences is not equivalent, but rather there is a guiding and being guided relationship, static appearance features (such as a furrowed brow) provide important "context" for interpreting subsequent subtle muscle movements (such as a slight downward turn of the mouth). An isolated motion feature is ambiguous on its own, but its emotional meaning becomes clear when combined with a specific facial state. Based on the cross-modal attention fusion mechanism provided by this invention, static appearance features are used as query vectors, and attention weights representing the correlation between spatial and temporal features are obtained based on the query vectors. Then, the attention weights are used to weight the dynamic feature sequence, overcoming the blindness of traditional linear fusion methods, realizing information complementarity and intelligent guidance between static appearance features and dynamic feature sequences, enabling the model to interpret the meaning of underlying muscle movements based on static image features. Furthermore, because the dynamic changes of micro-expressions are short (can be described as short duration), the changes in facial organs are small, and they are non-rigid movements, it is difficult to recognize facial micro-expressions based on existing technologies. Based on the embodiments of the present invention, since the model can interpret the meaning of underlying muscle movements, experimental results show that this can improve the facial expression recognition model's ability to understand and recognize complex and subtle expressions.

[0184] By introducing an auxiliary AU (Active Muscle Aspect) detector head and employing a multi-task learning approach, the model learns the most fundamental and interpretable muscle movement features of human facial expressions from a physiological perspective, rather than simply fitting data. This improves the model's generality and robustness. Furthermore, the predicted probabilities obtained based on the auxiliary AU detector head provide a transparent explanation for processing at different stages, facilitating subsequent model improvements.

[0185] Feature extraction is performed based on ROI, filtering out noise introduced by factors such as head rotation, and focusing the model processing on diagnostic areas such as the eyes and mouth, thereby accurately capturing weak, non-rigid movements generated by real muscle activity, improving the signal-to-noise ratio and reliability of dynamic features.

[0186] Based on VIT, we can better capture the collaborative relationship between facial organs that are far apart, extract image features that contain global contextual understanding, and improve the accuracy of facial expression recognition models for expressions that require comprehensive facial information to be recognized (such as "fear" and "surprise").

[0187] Furthermore, based on the facial expression recognition model provided by this invention, intelligent content production and analysis can be performed in real-world scenarios. For example, it can automatically analyze the facial expressions of characters in movies, TV dramas, and variety shows, identifying emotional climaxes such as "bursting into laughter," "crying," and "surprise." Technicians can then generate promotional materials based on the expression recognition results, improving content operation efficiency. For instance, analyzing the emotional flow curve of film and television works based on expression recognition results can provide objective data support for script evaluation and post-production editing, and can be used to analyze whether the plot pacing is reasonable and whether the emotional build-up is adequate. Alternatively, the aforementioned emotional flow curve can be combined and marked on the video playback progress bar, such as "High-energy highlights" or "Highlights of major flaws," to guide users' viewing of the video.

[0188] When pushing content to users, the system can analyze the user's interest in the content based on the facial expressions they make while viewing it. This analysis can then be used to optimize the content push strategy and improve the effectiveness of content delivery.

[0189] In some embodiments, the model structure of the facial expression recognition model can be optimized based on model pruning, knowledge distillation, and model quantization techniques to reduce the computational resources required when using the facial expression recognition model. Subsequently, the facial expression recognition model can be deployed on edge devices or mobile devices with limited computing power to achieve real-time facial expression recognition.

[0190] In some embodiments, the classification network can be a regression head for outputting valence and arousal, using tanh (activation function) as the activation function; correspondingly, the CCC (consistency correlation coefficient) and auxiliary AU loss are used to calculate the exploitable loss value. In this way, the facial expression recognition model provided by this invention can be extended to a continuous emotional dimension space and combined with the Valence-Arousal model to obtain more fine-grained facial expression recognition results, such as recognition results like "calm joy" or "excited tension".

[0191] In some embodiments, the facial expression recognition model may further include a feature extraction network that extracts features from other modalities, such as speech features (e.g., tone, voice, etc.) and physiological features (e.g., heart rate, skin conductance response, etc.). Then, an encoder is used to fuse the features from other modalities with the aforementioned temporal and spatial features to improve the accuracy of the obtained expression recognition results under extreme visual conditions such as occlusion and low light, based on more comprehensive emotion-related features, thereby improving the robustness of the facial expression recognition model.

[0192] This invention also provides a facial expression recognition method, applied to a facial expression recognition device. This recognition device and the aforementioned training device can be the same device or different devices; this invention does not limit this.

[0193] See Figure 5 , Figure 5 A flowchart of a facial expression recognition method provided in an embodiment of the present invention, the method may include the following steps:

[0194] S501: Obtain the video to be recognized.

[0195] S502: Utilize the spatiotemporal feature extraction network of the trained facial expression recognition model to extract the temporal and spatial features of the video to be recognized.

[0196] The trained facial expression recognition model is obtained based on any of the model training methods described in the foregoing embodiments; the facial expression recognition model also includes a mapping network and a classification network.

[0197] S503: Calculate attention weights that represent the correlation between spatial and temporal features using a mapping network.

[0198] S504: Use attention weights to weight and aggregate temporal features to obtain fused spatiotemporal features.

[0199] S505: Input the fused spatiotemporal features into the classification network to obtain the target recognition result representing the category of the facial expression to be recognized in the video to be recognized.

[0200] Based on the above processing, a spatiotemporal feature extraction network in the facial expression recognition model is used to extract temporal and spatial features from the video to be recognized. Spatial features are static facial features, while temporal features are dynamic facial muscle movement features. An attention weight representing the correlation between spatial and temporal features is calculated using a mapping network. This attention weight can characterize the non-linear relationship between spatial and temporal features. Then, the attention weight is used to perform weighted aggregation of temporal features, i.e., the weighted aggregation process of temporal features is guided by spatial features. Considering the correlation between spatial and temporal features, the spatial and temporal features are fused, obtaining fused spatiotemporal features without losing the non-linear relationship between them. The resulting fused spatiotemporal features are used to obtain target recognition results. Experimental results show that this method achieves complementarity between spatial and temporal features. By guiding the facial expression recognition model to interpret the meaning of dynamic muscle movements based on static image features, spatial features improve the accuracy of the target recognition results.

[0201] Regarding step S501, the video to be identified can be in MP4, FLV, or other formats. After acquiring the video, the identification device can decode it to obtain a continuous sequence of video frames, which are then processed. Alternatively, the identification device can directly acquire the decoded video frame sequence as the video to be identified.

[0202] Regarding step S502, the recognition device has a pre-trained facial expression recognition model deployed locally. The facial expression recognition model can be trained by the training device based on any of the aforementioned model training methods; for specific training methods, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0203] The facial expression recognition model includes a spatiotemporal feature extraction network for extracting temporal and spatial features from the video to be recognized. Temporal features represent the optical flow information between any two adjacent video frames, including the displacement vector of each pixel in the earlier video frame in the later video frame (i.e., the motion vector of each pixel). Spatial features represent the image features of the video frames in the video to be recognized. For example, the spatiotemporal feature extraction network can be C3D or TimeSFormer based on Transformer. Alternatively, the spatiotemporal feature extraction network can include both temporal feature extraction networks (such as RNNs and LSTMs) and spatial feature extraction networks (such as CNNs).

[0204] For example, in this invention, RAFT can be used as a temporal feature extraction network for optical flow estimation to obtain temporal features. RAFT does not require spatial scaling and can perform optical flow estimation on a set of unscaled high-resolution feature maps. Experimental results demonstrate that in complex scenes such as videos with large displacements, non-rigid deformations, and occlusion between objects, RAFT extracts temporal features with higher accuracy compared to traditional optical flow estimation networks; furthermore, RAFT has stronger cross-dataset generalization ability. When extracting temporal features from a video to be identified based on RAFT, for every two adjacent video frames in the video to be identified, these two adjacent video frames can be input into the temporal feature extraction network to obtain the optical flow information between the two adjacent video frames. The temporal features of the video to be identified include the optical flow information between every two adjacent video frames in the video to be identified.

[0205] For example, in this invention, VIT can be used as the backbone network of the spatial feature extraction network to extract spatial features from the video to be recognized. VIT divides each video frame into multiple patches and, based on a self-attention mechanism, learns the relationships between all image patches in the image (which can be called establishing global contextual understanding), thereby extracting image features that contain global contextual understanding. Since determining facial expressions in real-world scenarios often requires considering the relationships between various facial organs, extracting more accurate global image features based on VIT can improve the accuracy of subsequent recognition results.

[0206] When extracting spatial features from a video to be identified based on VIT, each video frame of the video to be identified can be input into VIT to obtain the image features of that video frame; the spatial features of the video to be identified can include the image features of each video frame. Alternatively, the recognition device can select a preset number of video frames from the videos included in the video to be identified, and then input the selected video frames into VIT to obtain the image features of the selected video frames; the spatial features of the video to be identified can include the image features of the selected video frames.

[0207] For steps S503 and S504, the facial expression recognition model further includes a mapping network, which may include multiple linear projection layers. Using the mapping network, temporal and spatial features can be mapped; that is, the dot product of the mapped feature and the weight matrix of the mapping network used to map the feature is calculated. Based on the obtained mapping result, attention weights representing the correlation between spatial and temporal features are calculated. The specific method for calculating the attention weights will be described in detail in subsequent embodiments and will not be repeated here.

[0208] Furthermore, the recognition device uses attention mechanisms, such as multi-head attention mechanisms and cross-attention mechanisms, to weight and aggregate temporal features using attention weights to obtain fused spatiotemporal features in vector form.

[0209] Regarding step S505, the classification network can be a classification head, such as one including fully connected layers and activation functions; or it can be an MLP. The classification network is used to predict the probability that the facial expression to be recognized in the video belongs to each preset category. The classification network can output the predicted probability of the facial expression to be recognized belonging to each preset category as the target recognition result; or, the classification network can output only the label of the category with the highest probability as the target recognition result, both of which are reasonable.

[0210] In some embodiments, the facial expression recognition model further includes a face detection network and a face alignment network.

[0211] The aforementioned step S501 may include the following steps:

[0212] Step E1: Obtain the video to be aligned.

[0213] Step E2: For each video frame in the video to be aligned, use a face detection network to detect the face region in that video frame, and use it as the region to be aligned.

[0214] Step E3: Using a face alignment network, perform an affine transformation on each region to be aligned to obtain the face alignment result corresponding to that region.

[0215] Step E4: Sort the face alignment results corresponding to each region to be aligned according to the temporal order of each region in the video to be aligned, and use the video composed of the sorted face alignment results as the video to be identified.

[0216] In this embodiment of the invention, the video to be identified can be a video composed of aligned face regions (i.e., face alignment results); the video to be aligned is a video that has not undergone alignment processing. The video to be aligned can be in MP4 format, FLV format, etc.; the recognition device can decode the video to be aligned to obtain a continuous sequence of video frames (which can be called the alignment sequence), that is, obtain each video frame in the video to be aligned.

[0217] For each video frame in the sequence to be aligned, a face detection network can be used to determine the bounding rectangle of the face in that video frame. The image region within the determined bounding rectangle, i.e., the face region in that video frame, is used as the region to be aligned. The face detection network can be based on RetinaFace or MTCNN, and this invention does not limit it to this.

[0218] Then, using a face alignment network, affine transformations are performed on facial reference points (such as the tip of the nose, the center of the eyes, etc.) in each region to be aligned, aligning the faces in each region to the same size and pose (which can be collectively referred to as standard facial morphology), thus obtaining the face alignment results for each region. The pose of the face represents the orientation and rotation angle of the face.

[0219] The recognition device can identify video frames that include frontal faces from each video frame, and then use the size and posture of the face in the identified video frame as the standard facial shape; alternatively, it can use the size and posture of the face in a specified frame of the video to be aligned as the standard facial shape, such as the first video frame, the last video frame, etc. in the video to be aligned, which are all reasonable.

[0220] Then, according to the temporal order of each region to be aligned in the video to be aligned, the face alignment results corresponding to each region to be aligned are sorted, and the video composed of the sorted face alignment results is used as the video to be identified. In this way, a video to be identified is obtained in which the faces are aligned and the dynamic change process of the facial expressions is consistent with the dynamic change process of the facial expressions in the video to be aligned.

[0221] Based on the above processing, face alignment enhances the model's performance in uncontrolled scenarios and improves the robustness of the trained facial expression recognition model. Furthermore, by complementing and guiding information from temporal and spatial features, the facial expression recognition model accurately interprets the meaning of dynamic facial muscle movements, reducing interference from identity-related features and enhancing the model's generalizability in uncontrolled scenarios.

[0222] In some embodiments, the spatiotemporal feature extraction network includes a temporal feature extraction network and a spatial feature extraction network.

[0223] The aforementioned step S502 may include the following steps:

[0224] Step F1: For each facial organ, use a temporal feature extraction network to extract the optical flow information between the image regions occupied by the facial organ in every two adjacent video frames in the video to be identified, and obtain the temporal features of the video to be identified.

[0225] Step F2: Based on the optical flow information of every two adjacent video frames in the video to be identified, select one video frame from the video frames included in the video to be identified as the key video frame.

[0226] Step F3: Use a spatial feature extraction network to extract the image features of the image regions occupied by each facial organ in the key video frames, as the spatial features of the video to be identified.

[0227] In this embodiment of the invention, facial organs may include eyebrows, eyes, nose, mouth, etc.; depending on the needs of the actual scenario, adjacent facial organs may also be combined, such as combining eyebrows and eyes into eyebrows and eyes as a facial organ; or combining mouth and nose into mouth and nose as a facial organ, which are all reasonable.

[0228] As mentioned earlier, apart from diagnostic regions, other areas of the face have relatively little correlation with facial expressions. Obtaining temporal and spatial features (hereinafter collectively referred to as spatiotemporal features) based on complete video frames, or obtaining spatiotemporal features based on extracted facial regions, requires processing image regions that contribute little to the facial expression recognition process, increasing the computational load of spatiotemporal feature extraction. Furthermore, spatiotemporal features obtained from image regions that contribute little to the facial expression recognition process may interfere with the training process of the facial expression recognition model, leading to a decrease in the accuracy of the trained model.

[0229] Therefore, in this embodiment of the invention, spatiotemporal features are extracted based on facial organs. The aforementioned problems are addressed by reducing the processing of image regions that contribute less to the facial expression recognition process.

[0230] Specifically, for each video frame in the video to be identified, the recognition device can detect the facial key points of the face in that video frame, as well as the facial organs to which each facial key point belongs, based on a facial key point detection algorithm. For example, facial key point detection algorithms could be MediaPipe 12, PFLD, etc.

[0231] For each facial feature, the recognition device can determine the smallest bounding rectangle of the facial key points belonging to that facial feature in the video frame. The image region within this smallest bounding rectangle, that is, the image region occupied by that facial feature in the video frame, can be called a Region of Interest (ROI).

[0232] Since the image area occupied by the same facial organ may be different in different video frames in real-world scenarios, the recognition device can first resize the ROI corresponding to the same facial organ. For example, the ROI can be cropped, stretched, or filled to adjust each ROI to a preset size. The preset size corresponding to the ROI of different facial organs can be the same or different, and this invention does not limit this.

[0233] Then, using a temporal feature extraction network, optical flow information between the image regions occupied by the facial organ in each of the two adjacent video frames in the video to be identified is extracted, generating a dense optical flow field sequence for the ROI corresponding to the facial organ. An optical flow field can be represented as a two-dimensional vector map (also called an optical flow map), which includes the motion information of each pixel in the ROI in the horizontal and vertical directions.

[0234] In one approach, the optical flow information of each facial organ corresponding to every two adjacent video frames in the video to be identified can be directly used as the temporal feature of the video to be identified.

[0235] In another approach, to further remove redundant features and reduce the computational load of subsequent processing, the recognition device can utilize a temporal encoder to extract features from the aforementioned multi-channel feature optical flow map, using the extracted results as the temporal features of the video to be recognized. For example, the temporal encoder can be built based on 3D-CNN, or it can be built based on Transformer.

[0236] Furthermore, the recognition device can select one video frame from all the video frames in the video to be recognized as the key video frame based on the optical flow information of every two adjacent video frames in the video to be recognized. For example, a video frame in which the optical flow direction changes can be used as the key video frame.

[0237] In some embodiments, video frames where facial expressions reach their peak can be used as key video frames; experimental results show that video frames where facial expressions reach their peak provide the most information about spatial features related to facial expressions. Video frames where facial expressions reach their peak can be called apex frames.

[0238] The aforementioned step F2 may include the following steps:

[0239] Step G1: For every two adjacent video frames in the video to be identified, calculate the average value of each element included in the optical flow information of each facial organ in the two video frames, and use it as the optical flow amplitude of the latest video frame in the two video frames for that facial organ.

[0240] Step G2: Calculate the average optical flow amplitude of the latest video frame in the two video frames for each facial organ, and use it as the available optical flow amplitude of the latest video frame in the two video frames.

[0241] Step G3: Identify the video frame with the largest optical flow amplitude in the video to be identified as the key video frame.

[0242] In this embodiment of the invention, for each pair of adjacent video frames in the video to be identified, the optical flow information for each facial organ in the two video frames includes multiple elements. The identification device can calculate the average value of these multiple elements as the optical flow amplitude for the facial organ in the latest temporal video frame among the two video frames. The optical flow amplitude is a one-dimensional numerical value.

[0243] Then, the average optical flow amplitude for each facial organ is calculated for the latest video frame in the two video frames, and this average is used as the available optical flow amplitude for the latest video frame in the two video frames. The larger the available optical flow amplitude of a video frame, the higher the probability that the video frame is a vertex frame. Therefore, the recognition device determines the video frame with the largest available optical flow amplitude in the video to be recognized as the key video frame.

[0244] Furthermore, the recognition device can divide each ROI in the key video frame into multiple non-overlapping image patches. For each image patch, a linear projection method is used to map the pixel matrix of the image patch to a patch embedding. The recognition device can also generate a positional code indicating the location of the image patch in the key video frame. Feature concatenation is performed on the patch embedding and positional code of the image patch to obtain a feature concatenation result that preserves the positional information of the image patch in the key video frame. The Transformer encoder in VIT processes the feature concatenation results corresponding to each image patch based on a multi-head self-attention mechanism, learning the global spatiotemporal dependencies between image patches, and thus extracting the global features of the key video frame, i.e., obtaining the spatial features of the aforementioned video to be recognized.

[0245] Based on the above processing, spatial features are obtained from vertex frames. This means acquiring spatial features from the video frame with the highest information content related to facial expressions. This reduces computational load and the probability of losing significant information in the obtained spatial features, thus lowering the probability of a decrease in the accuracy of the trained facial expression recognition model. Furthermore, image feature extraction is performed on the Regions of Interest (ROIs) in the vertex frames, reducing the introduction of image regions that contribute little to the facial expression recognition process, further reducing the probability of a decrease in the accuracy of the trained facial expression recognition model.

[0246] In some embodiments, the aforementioned step S503 may include the following steps:

[0247] Step H1: Use the query mapping layer to perform feature mapping on spatial features to obtain the query vector.

[0248] Step H2: Use the key mapping layer to perform feature mapping on the time features to obtain the key matrix, and use the value mapping layer to perform feature mapping on the time features to obtain the value matrix.

[0249] Step H3: Calculate the dot product of the query vector and the key matrix to obtain the first relevance vector.

[0250] Step H4: Calculate the quotient of the first correlation vector and the scaling factor to obtain the second correlation vector.

[0251] The scaling factor is obtained based on the dimension of the key matrix.

[0252] Step H5: Normalize each element in the second relevance vector to obtain the attention weights.

[0253] Accordingly, the aforementioned step S504 may include: calculating the weighted sum of the attention weights and the vectors in the value matrix to obtain the fused spatiotemporal features.

[0254] In this embodiment of the invention, the recognition device can calculate the dot product of the spatial features and the weight matrix of the query mapping layer, that is, use the query mapping layer to perform feature mapping on the spatial features to obtain a query vector. Semantically, the query vector Q can indicate which motion patterns in the dynamic feature sequence are worth noting in a face with a specific appearance. A face with a specific appearance refers to the face in the video to be recognized; appearance refers to the facial organs on the face; and the motion patterns in the dynamic feature sequence that need to be noted are the temporal features of the video to be recognized. In other words, the query vector Q is used to guide the facial expression recognition model to focus on the meaning of muscle movements on the face.

[0255] The recognition device can calculate the dot product between the temporal features and the weight matrix of the key mapping layer, that is, use the key mapping layer to perform feature mapping on the temporal features to obtain the key matrix. It can also calculate the dot product between the temporal features and the weight matrix of the value mapping layer, that is, use the value mapping layer to perform feature mapping on the temporal features to obtain the value matrix.

[0256] In the key matrix K and value matrix V, each row corresponds to the same time step, meaning that each row is obtained by feature mapping of the temporal features acquired from the same video frame. Semantically, each row in the key matrix K indicates the "queryable index" in the dynamic feature sequence at that time step, i.e., the index provided by the video frame; the same row in the value matrix V indicates the "actual information content" in the dynamic feature sequence at that time step, i.e., the temporal features provided by the video frame.

[0257] Furthermore, the recognition device can, based on a scaled dot product attention mechanism, first calculate the dot product of the query vector Q and the key matrix K to obtain the first relevance vector; calculate the square root of the dimension of the key matrix K as a scaling factor; calculate the quotient of the first relevance vector and the scaling factor to obtain the second relevance vector; and use softmax to normalize each element in the second relevance vector to obtain attention weights representing the correlation between spatial and temporal features. These attention weights indicate the importance of each temporal feature at each time step in the dynamic feature sequence. Obtaining attention weights in this way can reduce the probability of the model encountering the vanishing gradient problem. Then, the calculated attention weights are weighted and summed with the vectors in the value matrix V to obtain the fused spatiotemporal features. That is, spatial features are used to determine the importance of each temporal feature at different time steps, and the temporal features are weighted and aggregated according to their importance.

[0258] Based on the same inventive concept as the model training method described above, this embodiment of the invention also provides a model training apparatus. See also... Figure 6 , Figure 6 This is a structural diagram of a model training apparatus provided in an embodiment of the present invention. The apparatus includes: a sample acquisition module 601, used to acquire sample videos and a first sample label indicating the category of sample facial expressions in the sample videos; a sample feature extraction module 602, used to extract temporal and spatial features of the sample videos using a spatiotemporal feature extraction network in an initial structure facial expression recognition model; wherein the facial expression recognition model further includes a mapping network and a classification network; a first weight calculation module 603, used to calculate attention weights representing the correlation between the spatial features and the temporal features using the mapping network; a sample fusion module 604, used to perform weighted aggregation of the temporal features using the attention weights to obtain fused spatiotemporal features; a first recognition module 605, used to input the fused spatiotemporal features into the classification network to obtain a first recognition result representing the category of sample facial expressions in the sample videos; and a parameter adjustment module 606, used to adjust the model parameters of the initial structure facial expression recognition model based on the difference between the first recognition result and the first sample label until the model converges to obtain a trained facial expression recognition model.

[0259] Optionally, the mapping network includes a query mapping layer, a key mapping layer, and a value mapping layer; the first weight calculation module 603 is specifically used for: using the query mapping layer to perform feature mapping on the spatial features to obtain a query vector; using the key mapping layer to perform feature mapping on the temporal features to obtain a key matrix, and using the value mapping layer to perform feature mapping on the temporal features to obtain a value matrix; calculating the dot product of the query vector and the key matrix to obtain a first relevance vector; calculating the quotient of the first relevance vector and the scaling factor to obtain a second relevance vector; wherein the scaling factor is obtained based on the dimension of the key matrix; normalizing each element in the second relevance vector to obtain attention weights; the sample fusion module 604 is specifically used for calculating the weighted sum of the attention weights and each vector in the value matrix to obtain fused spatiotemporal features.

[0260] Optionally, the facial expression recognition model further includes an auxiliary facial action unit (AU) detection head; the device further includes: an AU recognition module, used to, before the parameter adjustment module 606 performs the adjustment of the model parameters of the initial structure facial expression recognition model based on the difference between the first recognition result and the first sample label, until the model converges and a trained facial expression recognition model is obtained, inputting the fused spatiotemporal features into the auxiliary AU detection head to obtain a second recognition result representing each preset AU label corresponding to the facial expression in the sample video; wherein, each preset AU label is used to describe the deformation state of the facial organs; the parameter adjustment module 606 is specifically used to: adjust the model parameters of the initial structure facial expression recognition model based on the difference between the second recognition result and the second sample label, and the difference between the first recognition result and the first sample label, until the model converges and a trained facial expression recognition model is obtained; wherein, the second sample label indicates: a preset AU label representing the facial expression in the sample video.

[0261] Optionally, the classification network includes a deep feature extraction layer and a fully connected layer; the first recognition module 605 is specifically used to: use the deep feature extraction layer to perform deep feature extraction on the fused spatiotemporal features to obtain sample deep features; use the fully connected layer to classify the sample deep features to obtain the classification probability of the sample facial expressions in the sample video as each category, as the first recognition result; the device further includes: a difference calculation module, used to: before the parameter adjustment module 606 performs the adjustment of the model parameters of the initial structure facial expression recognition model based on the difference between the second recognition result and the second sample label, and the difference between the first recognition result and the first sample label, until the model converges and a trained facial expression recognition model is obtained, perform the following: calculate the focus loss using the classification probability of the sample facial expression category indicated by the first sample label; calculate the center loss using the difference between the sample deep features and the center features currently corresponding to the category indicated by the first sample label; calculate the weighted sum of the focus loss and the center loss as the difference between the first recognition result and the first sample label.

[0262] Optionally, the spatiotemporal feature extraction network includes a temporal feature extraction network and a spatial feature extraction network; the sample feature extraction module 602 is specifically used for: for each facial organ, using the temporal feature extraction network to extract the optical flow information between the image regions occupied by the facial organ in every two adjacent video frames of the sample video, to obtain the temporal features of the sample video; based on the optical flow information of every two adjacent video frames of the sample video, selecting one video frame from the video frames included in the sample video as a key video frame; and using the spatial feature extraction network to extract the image features of the image regions occupied by each facial organ in the key video frame as the spatial features of the sample video.

[0263] Optionally, the sample feature extraction module 602 is specifically used for: for every two adjacent video frames in the sample video, calculating the average value of each element included in the optical flow information of each facial organ in the two video frames, as the optical flow amplitude of the latest video frame in the two video frames for the facial organ; calculating the average value of the optical flow amplitude of each facial organ in the latest video frame in the two video frames, as the usable optical flow amplitude of the latest video frame in the two video frames; and determining the video frame with the largest usable optical flow amplitude in the sample video as the key video frame.

[0264] Optionally, the facial expression recognition model further includes a face detection network and a face alignment network; the sample acquisition module 601 is specifically used for: acquiring the original video; for each video frame in the original video, using the face detection network to detect the face region in the video frame as the region to be aligned; using the face alignment network to perform an affine transformation on each region to be aligned to obtain the face alignment result corresponding to the region to be aligned; sorting the face alignment results corresponding to each region to be aligned according to the temporal order of each region to be aligned in the original video, and using the video composed of the sorted face alignment results as the sample video.

[0265] Based on the same inventive concept as the facial expression recognition method described above, this embodiment of the invention also provides a facial expression recognition device. See also... Figure 7 , Figure 7 This is a structural diagram of a facial expression recognition device provided in an embodiment of the present invention. The device includes: a video acquisition module 701 for acquiring a video to be recognized; a spatiotemporal feature extraction module 702 for extracting temporal and spatial features of the video to be recognized using a spatiotemporal feature extraction network of a trained facial expression recognition model; wherein the trained facial expression recognition model is obtained based on any of the model training methods described in the foregoing embodiments; the facial expression recognition model further includes a mapping network and a classification network; a second weight calculation module 703 for calculating attention weights representing the correlation between the spatial features and the temporal features using the mapping network; a spatiotemporal fusion module 704 for weighted aggregation of the temporal features using the attention weights to obtain fused spatiotemporal features; and a second recognition module 705 for inputting the fused spatiotemporal features into the classification network to obtain a target recognition result representing the category of the facial expression to be recognized in the video to be recognized.

[0266] Optionally, the mapping network includes a query mapping layer, a key mapping layer, and a value mapping layer; the second weight calculation module 703 is specifically used for: using the query mapping layer to perform feature mapping on the spatial features to obtain a query vector; using the key mapping layer to perform feature mapping on the temporal features to obtain a key matrix, and using the value mapping layer to perform feature mapping on the temporal features to obtain a value matrix; calculating the dot product of the query vector and the key matrix to obtain a first relevance vector; calculating the quotient of the first relevance vector and the scaling factor to obtain a second relevance vector; wherein the scaling factor is obtained based on the dimension of the key matrix; normalizing each element in the second relevance vector to obtain attention weights; the spatiotemporal fusion module 704 is specifically used for calculating the weighted sum of the attention weights and each vector in the value matrix to obtain fused spatiotemporal features.

[0267] Optionally, the spatiotemporal feature extraction network includes a temporal feature extraction network and a spatial feature extraction network; the spatiotemporal feature extraction module 702 is specifically used for: for each facial organ, using the temporal feature extraction network to extract the optical flow information between the image regions occupied by the facial organ in every two adjacent video frames of the video to be identified, to obtain the temporal features of the video to be identified; based on the optical flow information of every two adjacent video frames of the video to be identified, selecting one video frame from the video frames included in the video to be identified as a key video frame; using the spatial feature extraction network to extract the image features of the image regions occupied by each facial organ in the key video frame as the spatial features of the video to be identified.

[0268] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804. The memory 803 is used to store computer programs. When the processor 801 executes the program stored in the memory 803, it implements any of the model training method steps described in the foregoing embodiments, or implements any of the facial expression recognition method steps described in the foregoing embodiments.

[0269] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0270] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0271] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0272] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0273] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements any of the model training methods described in the above embodiments, or implements any of the facial expression recognition methods described in the above embodiments.

[0274] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the model training methods described in the above embodiments, or to execute any of the facial expression recognition methods described in the above embodiments.

[0275] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0276] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0277] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0278] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A model training method, characterized in that, The method includes: Acquire sample videos and first sample labels indicating the categories of sample facial expressions in the sample videos; The temporal and spatial features of the sample video are extracted using the spatiotemporal feature extraction network in the initial structure of the facial expression recognition model; wherein, the facial expression recognition model further includes a mapping network and a classification network; Using the mapping network, attention weights characterizing the correlation between the spatial features and the temporal features are calculated; The attention weights are used to weight and aggregate the temporal features to obtain fused spatiotemporal features. The fused spatiotemporal features are input into the classification network to obtain a first recognition result representing the category of the facial expressions in the sample video; Based on the difference between the first recognition result and the first sample label, the model parameters of the initial facial expression recognition model are adjusted until the model converges, thus obtaining a trained facial expression recognition model.

2. The method according to claim 1, characterized in that, The mapping network includes a query mapping layer, a key mapping layer, and a value mapping layer; Using the mapping network, attention weights characterizing the correlation between the spatial features and the temporal features are calculated, including: The query mapping layer is used to perform feature mapping on the spatial features to obtain a query vector; The key mapping layer is used to perform feature mapping on the time features to obtain a key matrix, and the value mapping layer is used to perform feature mapping on the time features to obtain a value matrix; Calculate the dot product of the query vector and the key matrix to obtain the first relevance vector; The second correlation vector is obtained by calculating the quotient of the first correlation vector and the scaling factor; wherein the scaling factor is obtained based on the dimension of the key matrix. The elements in the second correlation vector are normalized to obtain the attention weights; The attention weights are used to weight and aggregate the temporal features to obtain fused spatiotemporal features, including: The attention weights are calculated as a weighted sum of the vectors in the value matrix to obtain the fused spatiotemporal features.

3. The method according to claim 1, characterized in that, The facial expression recognition model also includes an auxiliary facial action unit (AU) detection head; Before adjusting the model parameters of the initial facial expression recognition model based on the difference between the first recognition result and the first sample label, until the model converges and a trained facial expression recognition model is obtained, the method further includes: The fused spatiotemporal features are input into the auxiliary AU detection head to obtain a second recognition result representing each preset AU label corresponding to the facial expression in the sample video; wherein, each preset AU label is used to describe the deformation state of the facial organs of the face; Based on the difference between the first recognition result and the first sample label, the model parameters of the initial facial expression recognition model are adjusted until the model converges, resulting in a trained facial expression recognition model, including: Based on the difference between the second recognition result and the second sample label, and the difference between the first recognition result and the first sample label, the model parameters of the initial structure of the facial expression recognition model are adjusted until the model converges, and a trained facial expression recognition model is obtained; wherein, the second sample label indicates: a preset AU label representing the facial expression in the sample video.

4. The method according to claim 3, characterized in that, The classification network includes a deep feature extraction layer and a fully connected layer; The fused spatiotemporal features are input into the classification network to obtain a first recognition result representing the category of facial expressions in the sample video, including: Using the deep feature extraction layer, deep feature extraction is performed on the fused spatiotemporal features to obtain the deep features of the sample; The fully connected layer is used to classify the deep features of the sample, and the classification probability of the facial expressions of the sample in the sample video for each category is obtained as the first recognition result. Before adjusting the model parameters of the initial facial expression recognition model based on the difference between the second recognition result and the second sample label, and the difference between the first recognition result and the first sample label, until the model converges and a trained facial expression recognition model is obtained, the method further includes: The focus loss is calculated using the classification probability of the facial expression category indicated by the first sample label. The center loss is calculated by using the difference between the deep features of the sample and the center features corresponding to the category indicated by the first sample label. Calculate the weighted sum of the focal loss and the center loss as the difference between the first identification result and the first sample label.

5. The method according to claim 1, characterized in that, The spatiotemporal feature extraction network includes a temporal feature extraction network and a spatial feature extraction network; The spatiotemporal feature extraction network in the facial expression recognition model with the initial structure extracts the temporal and spatial features of the sample video, including: For each facial organ, the temporal feature extraction network is used to extract the optical flow information between the image regions occupied by the facial organ in every two adjacent video frames of the sample video, thereby obtaining the temporal features of the sample video. Based on the optical flow information of every two adjacent video frames in the sample video, one video frame is selected from all the video frames included in the sample video as the key video frame. The spatial feature extraction network is used to extract the image features of the image regions occupied by each facial organ in the key video frame, which are then used as the spatial features of the sample video.

6. The method according to claim 5, characterized in that, Based on the optical flow information of every two adjacent video frames in the sample video, one video frame is selected from all the video frames included in the sample video as the key video frame, including: For each pair of adjacent video frames in the sample video, the average value of each element included in the optical flow information of each facial organ in the two video frames is calculated, which is used as the optical flow amplitude of the latest video frame in the two video frames for that facial organ. The average optical flow amplitude for each facial organ is calculated for the latest video frame in the two video frames, and this average is used as the optical flow amplitude to be utilized for the latest video frame in the two video frames. The video frame with the largest optical flow amplitude in the sample video is identified as the key video frame.

7. The method according to claim 1, characterized in that, The facial expression recognition model also includes a face detection network and a face alignment network; The acquisition of sample videos includes: Obtain the original video; For each video frame in the original video, the face detection network is used to detect the face region in that video frame, which is then used as the region to be aligned. Using the face alignment network, an affine transformation is performed on each region to be aligned to obtain the face alignment result corresponding to that region. According to the temporal order of each region to be aligned in the original video, the face alignment results corresponding to each region to be aligned are sorted, and the video composed of the sorted face alignment results is used as the sample video.

8. A method for facial expression recognition, characterized in that, The method includes: Obtain the video to be recognized; The temporal and spatial features of the video to be identified are extracted using the spatiotemporal feature extraction network of the trained facial expression recognition model; wherein the trained facial expression recognition model is obtained based on the model training method described in any one of claims 1-7; the facial expression recognition model further includes a mapping network and a classification network; Using the mapping network, attention weights characterizing the correlation between the spatial features and the temporal features are calculated; The attention weights are used to weight and aggregate the temporal features to obtain fused spatiotemporal features. The fused spatiotemporal features are input into the classification network to obtain the target recognition result representing the category of the facial expression to be recognized in the video to be recognized.

9. The method according to claim 8, characterized in that, The mapping network includes a query mapping layer, a key mapping layer, and a value mapping layer; Using the mapping network, attention weights characterizing the correlation between the spatial features and the temporal features are calculated, including: The query mapping layer is used to perform feature mapping on the spatial features to obtain a query vector; The key mapping layer is used to perform feature mapping on the time features to obtain a key matrix, and the value mapping layer is used to perform feature mapping on the time features to obtain a value matrix; Calculate the dot product of the query vector and the key matrix to obtain the first relevance vector; The second correlation vector is obtained by calculating the quotient of the first correlation vector and the scaling factor; wherein the scaling factor is obtained based on the dimension of the key matrix. The elements in the second correlation vector are normalized to obtain the attention weights; The attention weights are used to weight and aggregate the temporal features to obtain fused spatiotemporal features, including: The attention weights are calculated as a weighted sum of the vectors in the value matrix to obtain the fused spatiotemporal features.

10. The method according to claim 8, characterized in that, The spatiotemporal feature extraction network includes a temporal feature extraction network and a spatial feature extraction network; The process of extracting temporal and spatial features from the video to be recognized using a trained facial expression recognition model's spatiotemporal feature extraction network includes: For each facial organ, the temporal feature extraction network is used to extract the optical flow information between the image regions occupied by the facial organ in every two adjacent video frames of the video to be identified, so as to obtain the temporal features of the video to be identified. Based on the optical flow information of every two adjacent video frames in the video to be identified, one video frame is selected from the video frames included in the video to be identified as the key video frame. The spatial feature extraction network is used to extract the image features of the image regions occupied by each facial organ in the key video frame, which are then used as the spatial features of the video to be identified.

11. A model training device, characterized in that, The device includes: The sample acquisition module is used to acquire sample videos and first sample labels indicating the category of sample facial expressions in the sample videos; The sample feature extraction module is used to extract the temporal and spatial features of the sample video using the spatiotemporal feature extraction network in the initial structure of the facial expression recognition model; wherein, the facial expression recognition model further includes a mapping network and a classification network; The first weight calculation module is used to calculate attention weights that characterize the correlation between the spatial features and the temporal features using the mapping network; The sample fusion module is used to perform weighted aggregation of the temporal features using the attention weights to obtain fused spatiotemporal features; The first recognition module is used to input the fused spatiotemporal features into the classification network to obtain a first recognition result representing the category of the facial expression in the sample video; The parameter adjustment module is used to adjust the model parameters of the initial structure of the facial expression recognition model based on the difference between the first recognition result and the first sample label, until the model converges and a trained facial expression recognition model is obtained.

12. A facial expression recognition device, characterized in that, The device includes: The video acquisition module is used to acquire the video to be recognized. The spatiotemporal feature extraction module is used to extract the temporal and spatial features of the video to be identified using the spatiotemporal feature extraction network of the trained facial expression recognition model; wherein, the trained facial expression recognition model is obtained based on the model training method described in any one of claims 1-7; the facial expression recognition model further includes a mapping network and a classification network; The second weight calculation module is used to calculate attention weights that characterize the correlation between the spatial features and the temporal features using the mapping network. The spatiotemporal fusion module is used to perform weighted aggregation of the temporal features using the attention weights to obtain fused spatiotemporal features; The second recognition module is used to input the fused spatiotemporal features into the classification network to obtain a target recognition result representing the category of the facial expression to be recognized in the video to be recognized.

13. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of claims 1-7, or any one of claims 8-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of claims 1-7, or any one of claims 8-10.

Images