A face expression recognition method, device, equipment and storage medium

Abstract

The application discloses a face expression recognition method and device, equipment and storage medium, relates to the artificial intelligence technical field, and includes: in the process of utilizing a target expression recognition model to perform face expression recognition on video frames of a target face video, for any video frame at any time, extracting each face key point, and generating a spatial heat map based on each face key point, and generating a spatial feature vector based on the spatial heat map and the image features of the video frame; determining the spatial feature vector difference between the spatial feature vector of the video frame at the current time and the spatial feature vector of the video frame at the previous time, and determining the first hidden state corresponding to the spatial feature vector of the video frame at the current time and the second hidden state corresponding to the spatial feature vector difference; determining the fusion features based on the first hidden state and the second hidden state, and determining the face expression recognition result of the video frame at the current time based on the fusion features. The application can improve the stability and accuracy of dynamic face expression recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for facial expression recognition. Background Technology

[0002] Facial Expression Recognition (FER) is a core research area in computer vision and affective computing. Based on the modal differences in input data, FER technology can be divided into two categories: Static Facial Expression Recognition (SFER) and Dynamic Facial Expression Recognition (DFER). SFER focuses on feature analysis and classification of independent static facial images, while DFER aims to recognize facial expression changes in dynamic image sequences or video streams. Since human facial expressions are a continuous dynamic process that evolves over time, static recognition methods, by ignoring crucial temporal information, struggle to meet the demands of high-precision applications. Therefore, DFER has become a current research hotspot and core trend in the field.

[0003] Currently, the mainstream solutions in industry and academia rely on deep convolutional neural networks (CNNs) for static processing of video frames. Typically, the video stream is broken down into independent single-frame images. CNNs extract the spatial texture and geometric features of each frame, which are then normalized by fully connected layers and a softmax function. The probability values ​​of various expressions are then output, and the maximum value is taken as the recognition result. However, this approach has significant technical drawbacks: First, it severely lacks temporal correlation, treating video frames as isolated individuals and losing inter-frame temporal context information, failing to capture the dynamic patterns of expression evolution. Second, the recognition results are extremely unstable, exhibiting a "result flickering" phenomenon. Affected by factors such as lighting changes and pose shifts, the prediction results of adjacent frames can easily jump drastically, impacting user experience and system reliability. Third, it has weak ability to distinguish between ambiguous expressions and micro-expressions; the muscle deformation features at the beginning and end of expressions or micro-expressions are not significant, making it easy to confuse with "neutral" or similar expressions based solely on single-frame spatial features.

[0004] In conclusion, improving the stability and accuracy of dynamic facial expression recognition is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a facial expression recognition method, apparatus, device, and storage medium that can improve the stability and accuracy of dynamic facial expression recognition. The specific solution is as follows: Firstly, this application provides a facial expression recognition method, including: Acquire a target face video captured by a camera, and extract each video frame from the target face video; In the process of performing facial expression recognition on the video frame using the target expression recognition model, for the video frame at any time, each facial key point is extracted, and a corresponding spatial heat map is generated based on each facial key point, and a corresponding spatial feature vector is generated based on the spatial heat map and the image features corresponding to the video frame. Based on the element-wise difference between the spatial feature vector of the video frame at the current moment and the spatial feature vector of the video frame at the previous moment, the spatial feature vector difference is determined, and the first hidden state corresponding to the spatial feature vector of the video frame at the current moment is determined through the first long short-term memory network, and the second hidden state corresponding to the spatial feature vector difference is determined through the second long short-term memory network. Gating coefficients are generated based on the spatial feature vector difference, and the first hidden state and the second hidden state are fused based on the gating coefficients to obtain the fused features corresponding to the video frame at the current time. The facial expression recognition result of the video frame at the current time is determined based on the fused features.

[0006] Optionally, the extraction of each facial key point includes: Perform face detection on the video frame to determine the face region in the video frame; The face region in the video frame is cropped to obtain a corresponding initial face region image, and the initial face region image is scaled to a preset size to obtain a target face region image; The facial key points in the target face region image are located using a preset regression network, and each facial key point is extracted.

[0007] Optionally, generating corresponding spatial heatmaps based on each of the facial key points includes: Determine the coordinates of each of the facial key points in the target face region image; Using a two-dimensional Gaussian distribution, the response value of each pixel in the spatial heatmap is calculated based on the coordinates of each of the facial key points, and the spatial heatmap is generated based on the response values ​​of each pixel.

[0008] Optionally, generating the corresponding spatial feature vector based on the spatial heatmap and the image features corresponding to the video frame includes: The backbone network of the target expression recognition model extracts the image features corresponding to the target face region image corresponding to the video frame, and generates the spatial mask corresponding to the spatial heatmap through the convolutional layer of the target expression recognition model. The image features and the spatial mask are multiplied element-wise to obtain the corresponding target features, and the target features are then subjected to global average pooling to obtain the spatial feature vector.

[0009] Optionally, generating the gating coefficients based on the spatial feature vector difference includes: Based on the spatial feature vector difference, the preset activation function, the first preset learnable parameter, and the second preset learnable parameter, corresponding gating coefficients are generated so that the first hidden state and the second hidden state are fused based on the gating coefficients to obtain the fused feature corresponding to the video frame at the current time.

[0010] Optionally, determining the facial expression recognition result of the video frame at the current moment based on the fused features includes: Based on the fusion features, the target expression classification probability corresponding to the video frame at the current moment is determined, and the historical expression classification probability corresponding to several historical video frames adjacent to the video frame at the current moment is determined. The target expression classification probability and the historical expression classification probabilities are weighted and averaged, and the facial expression recognition result of the video frame at the current moment is determined based on the weighted average result.

[0011] Optionally, the facial expression recognition method further includes: Acquire face image training data, determine the first expression label corresponding to the face image training data, and generate the second expression label corresponding to the face image training data through label distribution estimation; the first expression label includes the first expression category corresponding to the face image training data, and the second expression label includes the probability distribution corresponding to the second expression category of the face image training data; The face image training data is input into the initial expression recognition model, and the predicted probability distribution corresponding to the predicted expression category output by the initial expression recognition model is obtained; The cross-entropy loss is determined based on the predicted probability distribution and the first expression label, and the relative entropy loss is determined based on the predicted probability distribution and the second expression label; The target loss is determined based on the cross-entropy loss and the relative entropy loss, and the initial expression recognition model is optimized based on the target loss to obtain the target expression recognition model, so as to use the target expression recognition model to perform facial expression recognition on the video frames in the target face video.

[0012] Secondly, this application provides a facial expression recognition device, comprising: The video frame extraction module is used to acquire a target face video captured by a camera and extract each video frame from the target face video; The spatial feature vector generation module is used to extract each facial key point for any given moment in the process of facial expression recognition of the video frame using the target expression recognition model, and to generate a corresponding spatial heat map based on each facial key point, and to generate a corresponding spatial feature vector based on the spatial heat map and the image features corresponding to the video frame. The hidden state determination module is used to determine the spatial feature vector difference based on the element-wise difference between the spatial feature vector of the video frame at the current time and the spatial feature vector of the video frame at the previous time, and to determine the first hidden state corresponding to the spatial feature vector of the video frame at the current time through a first long short-term memory network, and to determine the second hidden state corresponding to the spatial feature vector difference through a second long short-term memory network. The facial expression recognition module is used to generate gating coefficients based on the spatial feature vector difference, and fuse the first hidden state and the second hidden state based on the gating coefficients to obtain the fused features corresponding to the video frame at the current time, and determine the facial expression recognition result of the video frame at the current time based on the fused features.

[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned facial expression recognition method.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned facial expression recognition method.

[0015] In this application, firstly, a target face video captured by a camera is acquired, and each video frame in the target face video is extracted. During the process of facial expression recognition of the video frames using a target expression recognition model, for any given moment of the video frame, key facial points are extracted, and a corresponding spatial heatmap is generated based on each key facial point. A corresponding spatial feature vector is also generated based on the spatial heatmap and the image features corresponding to the video frame. Subsequently, based on the element-wise difference between the spatial feature vector of the current moment's video frame and the spatial feature vector of the previous moment's video frame, a spatial feature vector difference is determined. A first hidden state corresponding to the spatial feature vector of the current moment's video frame is determined using a first long short-term memory network, and a second hidden state corresponding to the spatial feature vector difference is determined using a second long short-term memory network. Finally, a gating coefficient is generated based on the spatial feature vector difference, and the first hidden state and the second hidden state are fused based on the gating coefficient to obtain the fused feature corresponding to the current moment's video frame. The facial expression recognition result of the current moment's video frame is then determined based on the fused feature. As can be seen from the above, in this application, the target face video is first captured by a camera and each video frame is extracted. Then, when the target expression recognition model is used to perform facial expression recognition on the video frames, for each video frame at any given time, the facial key points are first extracted and a spatial heatmap is generated. Then, the spatial feature vector is generated by combining the image features of the video frame. Subsequently, the difference between the spatial feature vectors of the current video frame and the video frame of the previous time is calculated. Then, the first hidden state corresponding to the spatial feature vector of the current video frame and the second hidden state corresponding to the difference between the spatial feature vectors are determined by the first long short-term memory network and the second long short-term memory network, respectively. Next, the gating coefficient is generated based on the difference between the spatial feature vectors. Finally, the first hidden state and the second hidden state are fused according to the gating coefficient to obtain the fused feature, and then the facial expression recognition result of the current video frame is output. In this way, this application can effectively filter background noise and focus on key areas of facial expression by guiding the extraction of spatial feature vectors through facial key points, thereby improving the accuracy of feature extraction. At the same time, by using the feature vector difference between video frames to capture dynamic changes in facial expressions, and by combining the fusion feature model of the first and second hidden states to model temporal correlation, it can keenly capture subtle changes such as micro-expressions, significantly improving the stability and accuracy of dynamic facial expression recognition. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the system structure applicable to a facial expression recognition scheme disclosed in this application; Figure 2 A flowchart of a facial expression recognition method provided in this application; Figure 3 A specific model training flowchart is provided for this application; Figure 4 This application provides a specific loss optimization flowchart; Figure 5 This application provides a specific flowchart for generating a space heat map; Figure 6 A specific flowchart for spatial feature vector extraction is provided in this application; Figure 7 A specific flowchart for spatial feature vector extraction is provided in this application; Figure 8 This application provides a specific flowchart for generating fusion features; Figure 9 This application provides a schematic diagram of the structure of a facial expression recognition device; Figure 10 This application provides a structural diagram of an electronic device. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Currently, mainstream solutions for facial expression recognition in industry and academia rely on deep convolutional neural networks (CNNs) for static processing of video frames. Typically, the video stream is broken down into independent single-frame images, and CNNs are used to extract the spatial texture and geometric features of each frame. After normalization by fully connected layers and the Softmax function, the probability values ​​of various expressions are output, and the maximum value is taken as the recognition result. However, this approach has significant technical drawbacks: First, it severely lacks temporal correlation, treating video frames as isolated individuals and losing inter-frame temporal context information, failing to capture the dynamic patterns of expression evolution. Second, the recognition results are extremely unstable, exhibiting a "result flickering" phenomenon. Affected by factors such as changes in lighting and pose shifts, the prediction results of adjacent frames can easily jump drastically, impacting user experience and system reliability. Third, it has weak ability to distinguish between ambiguous expressions and micro-expressions; the muscle deformation features at the beginning and end of expressions or micro-expressions are not significant, making it easy to confuse with "neutral" or similar expressions based solely on single-frame spatial features. Therefore, this application provides a facial expression recognition scheme that can improve the stability and accuracy of expression recognition.

[0020] It should be noted that the system architecture used in the facial expression recognition scheme of this application can be found in [reference needed]. Figure 1 As shown, it can specifically include: a video stream image preprocessing and keypoint representation module, a keypoint-guided spatial feature extraction module, a multi-scale motion-aware temporal fusion module, and a noise-resistant training module based on label distribution learning; wherein: To address the weakness of existing convolutional neural networks in extracting facial features in complex backgrounds, this application designs a video stream image preprocessing and keypoint representation module and a keypoint-guided spatial feature extraction module. First, the video stream image preprocessing and keypoint representation module uses a lightweight regression network to extract facial keypoints from video frame sequences and generate a spatial heatmap. Then, through a keypoint-guided spatial feature extraction module, a Locality-Aware Aggregation Block (LAA-Block) is designed to perform a weighted dot product between the spatial heatmap as a mask and the image features of the video frames. This mechanism acts like giving the network a "visual focus," forcing it to ignore background noise and focus on geometrically deformed areas closely related to facial expressions, such as eyebrows, eyes, and corners of the mouth, significantly improving feature purity.

[0021] To address the lag issues inherent in traditional temporal networks, such as LSTM (Long Short-Term Memory), when capturing fleeting micro-expressions, this application designs a multi-scale motion-aware temporal fusion module. This module abandons the traditional single-stream input approach, constructing a dual-path architecture with parallel "static feature stream" and "explicit motion stream." The static stream maintains the discrimination of the current facial state, while the motion stream captures the rate of motion change by calculating inter-frame feature differences. The two are dynamically fused through an adaptive gating unit, enabling the target expression recognition model to both accurately capture the initial movement of micro-expressions and maintain the continuity of emotion recognition over long time sequences.

[0022] To address the inherent subjectivity and ambiguity (label noise) prevalent in facial expression data annotation, this application introduces a noise-resistant training module based on label distribution learning during the training phase. Unlike traditional one-hot hard labels, this application utilizes label distribution estimation to generate soft labels containing probability distributions; for example, "bitter smile" may simultaneously include the probabilities of sadness and smile. Combined with a hybrid loss function weighted by cross-entropy and relative entropy (KL divergence), the target facial expression recognition model is guided to learn the transitional relationships between facial expression categories, thereby significantly improving the model's generalization ability to ambiguous expressions in uncontrolled environments.

[0023] It should be noted that among the four modules in this application, the video stream image preprocessing and keypoint representation module is the input front-end of the entire system, responsible for transforming the raw, unstructured video stream into high-quality, standardized input data. The keypoint-guided spatial feature extraction module, through an innovative dual-stream guided architecture, extracts spatial texture features rich in facial expression information from single-frame images and addresses background noise interference. The multi-scale motion-aware temporal fusion module upgrades traditional temporal networks, models the temporal dependencies between video frames, and captures the dynamic evolution of facial expressions, such as the onset, peak, and end of micro-expressions. The label distribution-based noise-resistant training module addresses the common problems of label subjectivity and label noise in training data during offline model training, effectively mitigating interference from illumination changes, pose shifts, and label noise, and improving the generalization ability of the target facial expression recognition model. Therefore, through the collaborative work of the above four modules, this application achieves keypoint-guided feature purification in the spatial dimension, dynamic and static combined temporal modeling in the temporal dimension, and noise-resistant soft label supervision in the training dimension, thereby constructing a high-performance dynamic facial expression recognition system suitable for video streams.

[0024] See Figure 2 As shown in the figure, an embodiment of the present invention discloses a facial expression recognition method, which may include: Step S11: Acquire the target face video captured by the camera, and extract each video frame from the target face video.

[0025] In this embodiment, a target face video can be captured by a camera on a mobile phone or other device, and then each video frame in the target face video can be extracted to obtain a video frame sequence.

[0026] Step S12: In the process of performing facial expression recognition on the video frame using the target expression recognition model, for the video frame at any time, extract each facial key point, generate a corresponding spatial heat map based on each facial key point, and generate a corresponding spatial feature vector based on the spatial heat map and the image features corresponding to the video frame.

[0027] In this embodiment, see Figure 3 As shown, a target expression recognition model needs to be trained first. Furthermore, to further improve the model's robustness, this embodiment employs a loss optimization strategy based on label distribution learning during the training phase. See [link to relevant documentation]. Figure 4 As shown. Addressing the problem that hard, "black-and-white" labels in expression annotation cannot accurately describe transitional expressions, such as embarrassment or a wry smile, this embodiment abandons one-hot labels and instead designs a noise-resistant training module based on label distribution learning. It utilizes Label Distribution Estimation (LDE) technology to generate a probability distribution vector for each training sample. Specifically, it first acquires face image training data and determines the first expression label corresponding to the face image training data, and generates a second expression label corresponding to the face image training data through label distribution estimation. The first expression label includes the first expression category corresponding to the face image training data, and the second expression label includes the probability distribution corresponding to the second expression category of the face image training data. Then, the face image training data is input into the initial expression recognition model, and the predicted probability distribution corresponding to the predicted expression category output by the initial expression recognition model is obtained. Subsequently, the cross-entropy loss is determined based on the predicted probability distribution and the first expression label, and the relative entropy loss is determined based on the predicted probability distribution and the second expression label. Finally, the target loss is determined based on the cross-entropy loss and the relative entropy loss, and the initial expression recognition model is optimized based on the target loss to obtain the target expression recognition model.

[0028] Specifically, assuming there are a total of C expression categories, for the face image training data... Generate the main category hard label, that is, the first emoji label is And generate a soft tag distribution, i.e., a second emoji tag. The training data of facial images is input into the initial expression recognition model to obtain the predicted probability distribution corresponding to the predicted expression category output by the model. It should be noted that the model's total loss function is a weighted combination of cross-entropy loss and relative entropy loss, i.e., KL divergence loss, as shown in the following formula: ; in, As the primary emoji tag, As the second emoji tag, and To balance the weights of the two loss terms, the cross-entropy loss term ensures the accuracy of the prediction results for the main category, while the KL divergence term constrains the consistency between the predicted distribution and the soft label distribution, forcing the model to learn the potential associations between different emotion categories. Finally, the target loss is determined based on the cross-entropy loss and the relative entropy loss, and the model parameters of the initial expression recognition model are optimized through backpropagation based on the target loss to obtain the target expression recognition model. This training method enables the model to provide reasonable predictions consistent with human cognition even when faced with ambiguous expressions in real-world scenarios.

[0029] It should be noted that in the process of using the target expression recognition model to perform facial expression recognition on video frames, see [reference needed]. Figure 5 As shown, a video stream image preprocessing and key point representation module was designed to extract key points and generate spatial heatmaps. The extraction of facial key points specifically includes the following steps: first, face detection is performed on the video frames to determine the face regions within them; then, the face regions are cropped to obtain initial face region images, and these images are scaled to a preset size to obtain target face region images; finally, a preset regression network is used to locate and extract the facial key points in the target face region images. Specifically, in the original video stream, faces may appear anywhere on the screen at varying scales. To ensure the stability of subsequent feature extraction, face detection and tracking are first performed on the video frames. For detected face regions, they are cropped and uniformly scaled to a standard size, such as 224. 224. The target face region image is obtained. To assist subsequent networks in focusing on key information, this embodiment uses a lightweight regression network, such as PFLD (Practical Facial Landmark Detector), to locate feature points in the target face region image and extract 68-98 facial key points including eyebrows, eyes, nose, mouth, and facial contours.

[0030] In this embodiment, a corresponding spatial heatmap is generated based on each facial key point. The specific process may include: first, determining the coordinates of each facial key point in the target face region image; then, using a two-dimensional Gaussian distribution, calculating the response value of each pixel in the spatial heatmap based on the coordinates of each facial key point, and generating the spatial heatmap based on the response values ​​of each pixel. Specifically, assuming the first... The coordinates of the individual facial landmarks are Then the pixels in the spatial heat map response value It is generated using a two-dimensional Gaussian distribution, and the calculation formula is shown below: ; in, To control the standard deviation hyperparameter of the spatial heatmap response range, K is the total number of facial keypoints. In the spatial heatmap, the pixel values ​​are highest at the location of facial keypoints and decrease Gaussian outwards, while the pixel values ​​in the background area approach 0. This provides a clear visual guidance signal for the subsequent spatial feature network.

[0031] Subsequently, the target face region image and spatial heatmap are simultaneously processed by the keypoint-guided spatial feature extraction module. (See below) Figure 6 As shown, the keypoint-guided spatial feature extraction module aims to address the issue that simple CNN networks are easily affected by background textures, such as hair and accessories. It should be noted that the specific process for generating corresponding spatial feature vectors based on spatial heatmaps and image features corresponding to video frames can include: firstly, extracting the image features corresponding to the target face region image of the video frame from the backbone network of the target expression recognition model, and generating a spatial mask corresponding to the spatial heatmap through the convolutional layers of the target expression recognition model; then, performing element-wise multiplication on the image features and the spatial mask to obtain the corresponding target features, and performing global average pooling on the target features to obtain the spatial feature vector.

[0032] Specifically, the keypoint-guided spatial feature extraction module adopts a dual-stream input architecture: one stream is the image stream, which takes an RGB target face region image as input and extracts texture features layer by layer through the backbone network of the target expression recognition model to obtain image features; the other stream is the guiding stream, which takes a spatial heatmap as input and generates a spatial mask corresponding to the spatial heatmap through downsampling in the convolutional layer of the target expression recognition model. In the middle layer of the network, this embodiment innovatively introduces a keypoint attention fusion unit, see [link to relevant documentation]. Figure 7 As shown. The keypoint attention fusion unit will guide the spatial mask generated by the flow. Feature maps generated from image streams Element-wise multiplication yields the corresponding target features. That is, the enhanced feature map, the formula is as follows: ; in, This represents the image features extracted by the backbone network. It is a spatial mask generated by downsampling and convolution of a spatial heatmap. These are learnable weight coefficients. This represents an element-wise multiplication operation. This operation achieves "denoising enhancement" at the feature level, suppressing the feature response of regions with low mask values ​​(i.e., the background) while enhancing the feature weights of high-response regions (i.e., facial features). Finally, the target features are processed by a Global Average Pooling (GAP) layer, outputting a compact one-dimensional feature vector. .

[0033] Step S13: Based on the element-wise difference between the spatial feature vector of the video frame at the current time and the spatial feature vector of the video frame at the previous time, determine the spatial feature vector difference value, and determine the first hidden state corresponding to the spatial feature vector of the video frame at the current time through the first long short-term memory network, and determine the second hidden state corresponding to the spatial feature vector difference value through the second long short-term memory network.

[0034] In this embodiment, after obtaining the spatial feature vectors of consecutive video frames, they are input into the multi-scale motion-aware temporal fusion module for temporal modeling. See [link to relevant documentation]. Figure 8 As shown, the core idea of ​​this module is to use explicit motion information to compensate for the lack of sensitivity to sudden actions in traditional recurrent networks, such as RNN (Recurrent Neural Network) and LSTM.

[0035] It should be noted that the multi-scale motion sensing temporal fusion module constructs a dual-path temporal network. First, it calculates the current time step. Spatial feature vectors of video frames Compared to the previous moment Spatial feature vectors of video frames The spatial feature vector difference between them serves as an explicit motion feature. The formula is shown below: ; Subsequently, a dual-stream LSTM architecture was constructed: (1) Static Stream: Input Spatial feature vector of video frame at time point After passing through the first Long Short-Term Memory network, the first hidden state is output. ; (2) Dynamic flow: The difference in feature vectors in the input space, i.e., motion features After passing through the second long short-term memory network, the second hidden state is output. .

[0036] Step S14: Generate gating coefficients based on the spatial feature vector difference, and fuse the first hidden state and the second hidden state based on the gating coefficients to obtain the fused features corresponding to the video frame at the current time, and determine the facial expression recognition result of the video frame at the current time based on the fused features.

[0037] In this embodiment, to effectively combine static and dynamic flow information, the multi-scale motion perception temporal fusion module introduces an adaptive gating fusion unit. Through a learnable gating coefficient, the weights are dynamically adjusted based on the current motion amplitude: when violent facial movement is detected, such as a moment of laughter, the dynamic flow weight is automatically increased to capture the changing trend; when the face is relatively still, the static flow weight is increased to maintain state discrimination. That is, the corresponding gating coefficient can be generated based on the spatial feature vector difference, a preset activation function, a first preset learnable parameter, and a second preset learnable parameter, as shown in the following formula: ; in, The gating coefficient, For activation function, The first preset learnable parameter, This is the second preset learnable parameter. The adaptive gating fusion unit dynamically adjusts the weights by calculating the motion amplitude. When it detects violent facial movements, i.e. When it is large, As the face becomes larger, the model automatically increases the dynamic flow weights to capture changing trends; when the face is relatively still, the static flow weights are increased to maintain state discrimination.

[0038] In this embodiment, the multi-scale motion-aware temporal fusion module can finally fuse the first and second hidden states based on the gating coefficients to obtain the fused features corresponding to the video frame at the current moment. The formula is shown below: ; in, This indicates element-wise multiplication. This is the first hidden state. This is the second hidden state. is the gating coefficient.

[0039] It should be noted that the specific process for determining the facial expression recognition result of the video frame at the current moment based on the fusion features may include: first, determining the target expression classification probability corresponding to the video frame at the current moment based on the fusion features, and determining the historical expression classification probabilities corresponding to several historical video frames adjacent to the video frame at the current moment; then, performing a weighted average of the target expression classification probability and each of the historical expression classification probabilities, and determining the facial expression recognition result of the video frame at the current moment based on the weighted average result.

[0040] Specifically, in one implementation, in order to eliminate the jitter of the single-frame prediction result on the time axis, a sliding window smoothing module can be connected to the end of the target expression recognition model network. This module is used to perform a weighted average of the historical expression classification probabilities corresponding to the historical 8 frames and the target expression classification probability corresponding to the current video frame, and output the final stable facial expression recognition result of the current video frame based on the weighted average result.

[0041] As can be seen from the above, in this embodiment, the target face video captured by the camera is first acquired, and each video frame in the target face video is extracted. During the process of facial expression recognition of the video frames using the target expression recognition model, for the video frame at any given time, each facial key point is extracted, and a corresponding spatial heatmap is generated based on each facial key point. A corresponding spatial feature vector is also generated based on the spatial heatmap and the image features corresponding to the video frame. Subsequently, based on the element-wise difference between the spatial feature vector of the current video frame and the spatial feature vector of the previous video frame, the spatial feature vector difference is determined. The first hidden state corresponding to the spatial feature vector of the current video frame is determined through a first long short-term memory network, and the second hidden state corresponding to the spatial feature vector difference is determined through a second long short-term memory network. Finally, a gating coefficient is generated based on the spatial feature vector difference, and the first hidden state and the second hidden state are fused based on the gating coefficient to obtain the fused feature corresponding to the video frame at the current time. The facial expression recognition result of the video frame at the current time is determined based on the fused feature. As can be seen from the above, in this embodiment, the target face video is first captured by the camera and each video frame is extracted. Then, when the target expression recognition model is used to perform facial expression recognition on the video frames, for each video frame at any given moment, the facial key points are first extracted and a spatial heatmap is generated. Then, the spatial feature vector is generated by combining the image features of the video frame. Subsequently, the difference between the spatial feature vectors of the current video frame and the previous video frame is calculated. Then, the first hidden state corresponding to the spatial feature vector of the current video frame and the second hidden state corresponding to the difference in spatial feature vectors are determined by the first long short-term memory network and the second long short-term memory network, respectively. Next, the gating coefficient is generated based on the difference in spatial feature vectors. Finally, the first hidden state and the second hidden state are fused according to the gating coefficient to obtain the fused feature, and then the facial expression recognition result of the current video frame is output. In this way, by guiding the extraction of spatial feature vectors through facial key points in this embodiment, background noise can be effectively filtered out, key areas of expression can be focused, and the accuracy of feature extraction can be improved. At the same time, by using the feature vector difference between video frames to capture dynamic changes in expression, and combining the fusion feature modeling of the first and second hidden states to model temporal correlation, subtle changes such as micro-expressions can be keenly captured, significantly improving the stability and accuracy of dynamic facial expression recognition.

[0042] Accordingly, see Figure 9 As shown in the illustration, this application also provides a facial expression recognition device, which may include: The video frame extraction module 11 is used to acquire a target face video captured by a camera and extract each video frame from the target face video; The spatial feature vector generation module 12 is used to extract each facial key point for any given moment of the video frame during the process of facial expression recognition using the target expression recognition model, and to generate a corresponding spatial heat map based on each facial key point, and to generate a corresponding spatial feature vector based on the spatial heat map and the image features corresponding to the video frame. The hidden state determination module 13 is used to determine the spatial feature vector difference based on the element-wise difference between the spatial feature vector of the video frame at the current time and the spatial feature vector of the video frame at the previous time, and to determine the first hidden state corresponding to the spatial feature vector of the video frame at the current time through a first long short-term memory network, and to determine the second hidden state corresponding to the spatial feature vector difference through a second long short-term memory network. The facial expression recognition module 14 is used to generate gating coefficients based on the spatial feature vector difference, and fuse the first hidden state and the second hidden state based on the gating coefficients to obtain the fused features corresponding to the video frame at the current time, and determine the facial expression recognition result of the video frame at the current time based on the fused features.

[0043] In some specific embodiments, the spatial feature vector generation module 12 may include: A face region determination unit is used to perform face detection on the video frame to determine the face region in the video frame; A face region cropping unit is used to crop the face region in the video frame to obtain a corresponding initial face region image, and to scale the initial face region image to a preset size to obtain a target face region image. A facial key point extraction unit is used to locate each of the facial key points in the target facial region image using a preset regression network, and to extract each of the facial key points.

[0044] In some specific embodiments, the spatial feature vector generation module 12 may include: A pixel determination unit is used to determine the coordinates of each of the facial key points in the target face region image; A spatial heatmap generation unit is used to calculate the response value of each pixel in the spatial heatmap based on the coordinates of each of the facial key points using a two-dimensional Gaussian distribution, and to generate the spatial heatmap based on the response value of each pixel.

[0045] In some specific embodiments, the spatial feature vector generation module 12 may include: The spatial mask generation unit is used to extract the image features corresponding to the target face region image corresponding to the video frame from the backbone network of the target expression recognition model, and generate the spatial mask corresponding to the spatial heatmap through the convolutional layer of the target expression recognition model. The spatial feature vector generation unit is used to perform element-wise multiplication of the image features and the spatial mask to obtain the corresponding target features, and to perform global average pooling on the target features to obtain the spatial feature vector.

[0046] In some specific embodiments, the facial expression recognition module 14 may include: The gating coefficient generation unit is used to generate corresponding gating coefficients based on the spatial feature vector difference, a preset activation function, a first preset learnable parameter, and a second preset learnable parameter, so as to fuse the first hidden state and the second hidden state based on the gating coefficients to obtain the fused feature corresponding to the video frame at the current time.

[0047] In some specific embodiments, the facial expression recognition module 14 may include: The historical expression classification probability determination unit is used to determine the target expression classification probability corresponding to the video frame at the current moment based on the fusion features, and to determine the historical expression classification probability corresponding to several historical video frames adjacent to the video frame at the current moment. The facial expression recognition unit is used to perform a weighted average of the target expression classification probability and the historical expression classification probabilities, and determine the facial expression recognition result of the video frame at the current moment based on the weighted average result.

[0048] In some specific embodiments, the facial expression recognition device may further include: An expression label generation module is used to acquire face image training data, determine the first expression label corresponding to the face image training data, and generate a second expression label corresponding to the face image training data through label distribution estimation; the first expression label includes the first expression category corresponding to the face image training data, and the second expression label includes the probability distribution corresponding to the second expression category of the face image training data; The prediction probability distribution output module is used to input the face image training data into the initial expression recognition model and obtain the prediction probability distribution corresponding to the predicted expression category output by the initial expression recognition model. The relative entropy loss determination module is used to determine the cross-entropy loss based on the predicted probability distribution and the first expression label, and to determine the relative entropy loss based on the predicted probability distribution and the second expression label; The target expression recognition model determination module is used to determine the target loss based on the cross-entropy loss and the relative entropy loss, and optimize the initial expression recognition model based on the target loss to obtain the target expression recognition model, so as to use the target expression recognition model to perform facial expression recognition on the video frames in the target face video.

[0049] Furthermore, embodiments of this application also disclose an electronic device, Figure 10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the facial expression recognition method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0050] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0051] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0052] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the facial expression recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0053] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed facial expression recognition method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0054] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0055] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0056] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0057] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A facial expression recognition method, characterized in that, include: Acquire a target face video captured by a camera, and extract each video frame from the target face video; In the process of performing facial expression recognition on the video frame using the target expression recognition model, for the video frame at any time, each facial key point is extracted, and a corresponding spatial heat map is generated based on each facial key point, and a corresponding spatial feature vector is generated based on the spatial heat map and the image features corresponding to the video frame. Based on the element-wise difference between the spatial feature vector of the video frame at the current moment and the spatial feature vector of the video frame at the previous moment, the spatial feature vector difference is determined, and the first hidden state corresponding to the spatial feature vector of the video frame at the current moment is determined through the first long short-term memory network, and the second hidden state corresponding to the spatial feature vector difference is determined through the second long short-term memory network. Gating coefficients are generated based on the spatial feature vector difference, and the first hidden state and the second hidden state are fused based on the gating coefficients to obtain the fused features corresponding to the video frame at the current time. The facial expression recognition result of the video frame at the current time is determined based on the fused features.

2. The facial expression recognition method according to claim 1, characterized in that, The extraction of key facial features includes: Perform face detection on the video frame to determine the face region in the video frame; The face region in the video frame is cropped to obtain a corresponding initial face region image, and the initial face region image is scaled to a preset size to obtain a target face region image; The facial key points in the target face region image are located using a preset regression network, and each facial key point is extracted.

3. The facial expression recognition method according to claim 2, characterized in that, The generation of corresponding spatial heatmaps based on each of the aforementioned facial key points includes: Determine the coordinates of each of the facial key points in the target face region image; Using a two-dimensional Gaussian distribution, the response value of each pixel in the spatial heatmap is calculated based on the coordinates of each of the facial key points, and the spatial heatmap is generated based on the response values ​​of each pixel.

4. The facial expression recognition method according to claim 2, characterized in that, The step of generating corresponding spatial feature vectors based on the spatial heatmap and the image features corresponding to the video frames includes: The backbone network of the target expression recognition model extracts the image features corresponding to the target face region image corresponding to the video frame, and generates the spatial mask corresponding to the spatial heatmap through the convolutional layer of the target expression recognition model. The image features and the spatial mask are multiplied element-wise to obtain the corresponding target features, and the target features are then subjected to global average pooling to obtain the spatial feature vector.

5. The facial expression recognition method according to claim 1, characterized in that, The generation of gating coefficients based on the spatial feature vector differences includes: Based on the spatial feature vector difference, the preset activation function, the first preset learnable parameter, and the second preset learnable parameter, corresponding gating coefficients are generated so that the first hidden state and the second hidden state are fused based on the gating coefficients to obtain the fused feature corresponding to the video frame at the current time.

6. The facial expression recognition method according to claim 1, characterized in that, The step of determining the facial expression recognition result of the video frame at the current moment based on the fused features includes: Based on the fusion features, the target expression classification probability corresponding to the video frame at the current moment is determined, and the historical expression classification probability corresponding to several historical video frames adjacent to the video frame at the current moment is determined. The target expression classification probability and the historical expression classification probabilities are weighted and averaged, and the facial expression recognition result of the video frame at the current moment is determined based on the weighted average result.

7. The facial expression recognition method according to any one of claims 1 to 6, characterized in that, Also includes: Acquire face image training data, determine the first expression label corresponding to the face image training data, and generate the second expression label corresponding to the face image training data through label distribution estimation; The first expression label includes a first expression category corresponding to the face image training data, and the second expression label includes a probability distribution corresponding to a second expression category of the face image training data; The face image training data is input into the initial expression recognition model, and the predicted probability distribution corresponding to the predicted expression category output by the initial expression recognition model is obtained; The cross-entropy loss is determined based on the predicted probability distribution and the first expression label, and the relative entropy loss is determined based on the predicted probability distribution and the second expression label; The target loss is determined based on the cross-entropy loss and the relative entropy loss, and the initial expression recognition model is optimized based on the target loss to obtain the target expression recognition model, so as to use the target expression recognition model to perform facial expression recognition on the video frames in the target face video.

8. A facial expression recognition device, characterized in that, include: The video frame extraction module is used to acquire a target face video captured by a camera and extract each video frame from the target face video; The spatial feature vector generation module is used to extract each facial key point for any given moment in the process of facial expression recognition of the video frame using the target expression recognition model, and to generate a corresponding spatial heat map based on each facial key point, and to generate a corresponding spatial feature vector based on the spatial heat map and the image features corresponding to the video frame. The hidden state determination module is used to determine the spatial feature vector difference based on the element-wise difference between the spatial feature vector of the video frame at the current time and the spatial feature vector of the video frame at the previous time, and to determine the first hidden state corresponding to the spatial feature vector of the video frame at the current time through a first long short-term memory network, and to determine the second hidden state corresponding to the spatial feature vector difference through a second long short-term memory network. The facial expression recognition module is used to generate gating coefficients based on the spatial feature vector difference, and fuse the first hidden state and the second hidden state based on the gating coefficients to obtain the fused features corresponding to the video frame at the current time, and determine the facial expression recognition result of the video frame at the current time based on the fused features.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the facial expression recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the facial expression recognition method as described in any one of claims 1 to 7.

Images