A Method for Establishing a Facial Action Unit Detection Model Based on Multi-Task Learning
By using a multi-task learning facial action unit detection model that shares a low-level feature extraction network and combines facial key points and emotion recognition tasks, the model utilizes GAT to learn AU relationships, thus solving the problem of low AU detection accuracy and achieving higher detection accuracy and robustness.
Patent Information
- Application Number
- CN202311821545.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-27
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-12-27
AI Technical Summary
In existing technologies, facial action unit detection methods ignore the relationships between action units (AUs), resulting in low detection accuracy. Furthermore, traditional methods require integration with handcrafted features, which limits performance.
A facial action unit detection model based on multi-task learning is adopted. By sharing the underlying feature extraction network and combining facial key point detection and facial emotion recognition tasks, the relationship between AUs is learned by using attention mechanism and graph structure. GAT is used for graph data learning, and the AU detection accuracy is improved by feature fusion module.
It improves the accuracy of facial AU detection, better utilizes the information between AUs, enhances the robustness and adaptability of the model, and improves the accuracy of detection.
Smart Images

Figure CN117765596B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of facial motion detection, and more particularly to a method for establishing a facial motion unit detection model based on multi-task learning. Background Technology
[0002] Facial Action Units (FAUs) are a systematic method for describing facial expressions based on the anatomy of facial muscles. By determining the occurrence of each Action Unit (AU) in a face image, facial AU detection can provide a comprehensive and objective analysis of facial expressions, making it an important task in facial attribute analysis. In recent years, with the development of deep learning, facial AU detection has made significant progress. However, most research focuses on a single task, achieving facial AU detection by training a model for a specific task. This neglects the relationship between AUs and other facial attributes, resulting in poor noise resistance and adaptability.
[0003] Multiple Action Units (AUs) can be activated simultaneously, and these AUs are interconnected and not isolated. For example, when a person displays a happy expression, AU6 (raised cheeks) and AU12 (upturned corners of the mouth) are likely to appear simultaneously, while AU15 (corners of the mouth pulled vertically downwards) is rarely seen. When displaying a sad expression, AU1 (raised inner eyebrows), AU15, and AU17 (pushed lower lip upwards) are likely to appear simultaneously, while AU12 is rarely seen. Utilizing the relevant information of AUs can undoubtedly improve detection accuracy. Traditional AU detection methods often employ Bayesian networks and Restricted Boltzmann Machines to utilize AU relationships. These probabilistic generative models always require combination with handcrafted features, limiting the performance of AU detection and failing to meet requirements. Therefore, this invention proposes a method for establishing a facial action unit detection model based on multi-task learning. Summary of the Invention
[0004] To overcome the shortcomings and deficiencies of existing technologies, this invention provides a method for establishing a facial action unit detection model based on multi-task learning.
[0005] A method for establishing a facial action unit detection model based on multi-task learning, characterized in that the method includes:
[0006] Step S1: Use a compilation tool to perform key point detection on the face images in the dataset, and align and crop the face images in the original dataset based on the key points;
[0007] Step S2: The aligned and cropped image is randomly cropped and horizontally flipped before being input into the network;
[0008] Step S3: Establish an initial neural network detection model and train it using the dataset to obtain a facial AU detection model;
[0009] Step S4: Train the initial neural network model end-to-end using face images labeled with AU tags, facial landmark tags, and emotion tags;
[0010] Step S5: Change some hyperparameters, retrain the model, optimize the neural network detection model using the loss function, and take the model with the highest facial AU detection accuracy as the final result.
[0011] Furthermore, the initial neural network model includes:
[0012] The underlying network is used to perform basic feature extraction on the images input to the model. In a multi-task model, the underlying network is shared by multiple tasks.
[0013] The auxiliary task learning module includes facial key point detection and facial emotion recognition. Both take the output of the underlying network as input. The underlying features are processed by their respective feature extraction networks and then by a fully connected layer to obtain the prediction results of facial key points and facial emotions.
[0014] The main task learning module, used for facial AU detection, consists of two parts: a local feature extraction module and an AU relationship learning module.
[0015] Furthermore, the underlying network uses the output as input to the local feature extraction module, and provides an attention map for each AU using the keypoint coordinates detected by the auxiliary task. Each AU uses its own independent branch to extract local features, and the network structure of each branch is the same as the network structure of the auxiliary task. The output of the local feature extraction part is used as input to the AU relationship learning module, and the relationship between AUs is encoded using an AU relationship graph. The relationship graph consists of nodes and edges, where each node represents a feature of an AU, and the edges represent the correlation between AUs. The correlation of AUs is obtained by calculating the conditional probability of each AU occurring in the training samples.
[0016] The final relation matrix is obtained, and GAT is used to learn the graph.
[0017] Furthermore, the initial neural network model also includes a feature fusion module for utilizing useful features from the auxiliary task to aid in the learning of the face AU detection task.
[0018] Furthermore, the feature fusion module introduces an attention mechanism to select useful information from the auxiliary task and uses CBAM to guide the flow of information from the auxiliary task to the main task.
[0019] Furthermore, the training loss function shown is expressed as follows:
[0020] L=λ1L au +λ2Llandmark +λ3L emotion
[0021] Among them, L au and L emotion The weighted cross-entropy loss functions for facial AU detection and facial emotion recognition are L and L, respectively. landmark λ1, λ2, and λ3 are weighted loss functions for facial landmark detection, and λ1, λ2, and λ3 are hyperparameters, assigning weights to each loss function.
[0022] The beneficial effects of this invention are:
[0023] This invention proposes a method for establishing a facial action unit detection model based on multi-task learning. It jointly learns three tasks: facial action unit (AU) detection, facial keypoint detection, and facial emotion recognition. These three tasks share a common underlying feature extraction network, enabling the network to extract more robust features. Furthermore, the keypoint coordinates obtained from the facial keypoint detection task are used to provide an attention map for the AU, thus avoiding interference from noise in irrelevant facial regions. In addition, the AU relationships are represented as graph-structured data, and Gaussian Attention Scale (GAT) is used for graph data learning. For each AU node, GAT is used to calculate the attention weights between that node and other nodes, and these weights are used to update the node's feature representation. Through multiple rounds of node information transmission and feature updates, the final feature representation of the AU node is obtained. This fully utilizes the relationships between AU nodes to capture richer contextual information, thereby improving the semantic expressive power of the AU features. Finally, an attention-based feature fusion module is used to fuse the features from the three tasks, further leveraging the complementarity between the tasks. This method effectively improves the detection accuracy of facial AUs. Attached Figure Description
[0024] Figure 1 This is a flowchart of the method steps of the present invention;
[0025] Figure 2 This is a diagram showing the composition of the neural network model modules of the present invention. Detailed Implementation
[0026] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. The following describes the application in further detail with reference to the accompanying drawings and specific embodiments.
[0027] like Figure 1 As shown, a method for establishing a facial action unit detection model based on multi-task learning is proposed, which aims to apply the automatic detection of facial action units (AUs) to fields such as human-computer interaction, emotion recognition, and medical diagnosis. The method includes:
[0028] Step S1: Use a compilation tool to perform key point detection on the face images in the dataset, and align and crop the face images in the original dataset based on the key points;
[0029] Specifically, the Dlib tool was used to perform keypoint detection on face images from the BP4D and DISFA datasets. These keypoints were then used to align and crop the original face images to 200×200 pixels. Before being input into the network, the preprocessed images were randomly cropped to 176×176 pixels and horizontally flipped to improve data diversity.
[0030] Step S2: The aligned and cropped image is randomly cropped and horizontally flipped before being input into the network;
[0031] Step S3: Establish an initial neural network detection model and train it using the dataset to obtain a facial AU detection model;
[0032] Step S4: Train the initial neural network model end-to-end using face images labeled with AU tags, facial landmark tags, and emotion tags;
[0033] Step S5: Change some hyperparameters, retrain the model, optimize the neural network detection model using the loss function, and take the model with the highest facial AU detection accuracy as the final result.
[0034] like Figure 2 As shown, the initial neural network model further includes:
[0035] The underlying network is used to perform basic feature extraction on the images input to the model. In a multi-task model, the underlying network is shared by multiple tasks.
[0036] The underlying network employs a hierarchical and multi-scale feature extraction network proposed by JAA-Net to extract multi-scale features from local regions of different sizes. After feature extraction, the input image outputs a 64×44×44 image, where 64 represents the number of feature map channels and 44 represents the length or width of the feature map.
[0037] The auxiliary task learning module includes facial key point detection and facial emotion recognition. Both take the output of the underlying network as input. The underlying features are processed by their respective feature extraction networks and then by a fully connected layer to obtain the prediction results of facial key points and facial emotions.
[0038] In the auxiliary task module, the facial key point and facial emotion feature extraction networks further refine the 64×44×44 feature map output by the bottom network to 160×5×5, and obtain the facial key point and facial emotion prediction results through their respective fully connected layers.
[0039] The main task learning module, used for facial AU detection, consists of two parts: a local feature extraction module and an AU relationship learning module.
[0040] In the main learning task, firstly, attention maps are defined using facial key points to extract local features, and the attention map generation process is described. Secondly, the features output by the local feature extraction module are input into the GAT for learning AU relationships. The AU relationship learning module outputs a feature map with a size of 160×5×5.
[0041] Graph Attention Networks (GAT) employs an attention mechanism to learn the weights of neighboring nodes, obtaining the representation of a node itself by weighted summation of the weights of its neighboring nodes. At its core, GAT uses a masked self-attention layer, which allows each node to be assigned different weights based on the features of its neighboring nodes. This means that when updating features, each node can weight itself according to the importance of its neighboring nodes, thus better capturing the complex relationships between nodes. Through this attention mechanism, GAT can more flexibly model graph data and is applicable to a wider range of graph structures and tasks.
[0042] GAT is suitable for inductive tasks: Traditional graph convolutional networks excel at handling transductive tasks (where training and testing use the same graph data), but cannot perform inductive tasks (where training and testing use different graph data). GAT, however, only requires information from first-order neighbor nodes, thus enabling it to handle a wider range of graph data and achieve inductive tasks.
[0043] Learning weights for different neighbor nodes: Traditional graph convolutional networks use the same weights for different neighbors of the same node during convolution operations, while GAT allows different weights to be learned for different neighbor nodes through an attention mechanism. This allows GAT to more accurately capture the important relationships between nodes, improving the model's performance.
[0044] Facial key point features and emotion features, each with a size of 160×5×5, are processed by spatial attention and channel attention mechanisms and then concatenated with AU features. The concatenated feature map has a size of 480×5×5 and is output as AU prediction results after passing through a fully connected layer.
[0045] Preferably, the underlying network uses the output as input to the local feature extraction module. It provides an attention map for each Active User Entity (AU) using the keypoint coordinates detected by the auxiliary task. Each AU uses its own independent branch to extract local features. The network structure of each branch is the same as that of the auxiliary task. The output of the local feature extraction part is used as input to the AU relationship learning module. The relationship between AUs is encoded using an AU relationship graph. The relationship graph consists of nodes and edges. Each node represents a feature of an AU, and the edges represent the correlation between AUs. The correlation of AUs is obtained by calculating the conditional probability of each AU occurring in the training samples, resulting in the final relationship matrix. The GAT is then used to learn the graph.
[0046] Preferably, the initial neural network model further includes a feature fusion module for utilizing useful features from the auxiliary task to aid in the learning of the face AU detection task.
[0047] Preferably, the feature fusion module introduces an attention mechanism to select useful information from the auxiliary task, and uses CBAM to guide the flow of information from the auxiliary task to the main task. Similar to a gating function, the combination of channel attention and spatial attention modules can control the information flow of the auxiliary task, automatically learning to focus on useful information. The initial keypoints and emotion features are processed by the channel attention and spatial attention modules to obtain corresponding optimized features. The optimized auxiliary task features are concatenated with the AU features and then input into the fully connected layer to obtain the AU prediction result.
[0048] CBAM consists of two key components: the channel attention module (C-channel) and the spatial attention module (S-channel). These two modules can be embedded into different layers of a CNN to enhance feature representation.
[0049] The goal of the channel attention module is to enhance the feature representation of each channel. The following are the steps to implement the channel attention module:
[0050] 1. Global Max Pooling and Global Average Pooling: For the input feature map, global max pooling and global average pooling operations are first performed on each channel to calculate the maximum and average eigenvalues for each channel. This generates two vectors containing the number of channels, representing the global maximum and average eigenvalues for each channel, respectively.
[0051] 2. Fully Connected Layer: The feature vectors obtained from global max pooling and average pooling are input into a shared fully connected layer. This fully connected layer learns the attention weights for each channel. Through learning, the network can adaptively determine which channels are more important for the current task. The intersection of the global maximum feature vector and the average feature vector yields the final attention weight vector.
[0052] 3. Sigmoid Activation: To ensure that the attention weights are between 0 and 1, the Sigmoid activation function is applied to generate channel attention weights. These weights will be applied to each channel of the original feature map.
[0053] 4. Attention Weighting: Using the obtained attention weights, multiply them by each channel of the original feature map to obtain an attention-weighted channel feature map. This will emphasize channels that are helpful to the current task and suppress irrelevant channels.
[0054] The goal of a spatial attention module is to emphasize the importance of different locations in an image. The following are the steps to implement a spatial attention module:
[0055] (1) Max pooling and average pooling: For the input feature map, max pooling and average pooling operations are performed along the channel dimension to generate features at different context scales.
[0056] (2) Concatenation and Convolution: The features obtained after max pooling and average pooling are concatenated along the channel dimension to obtain a feature map with contextual information at different scales. Then, this feature map is processed by a convolutional layer to generate spatial attention weights.
[0057] (3) Sigmoid activation: Similar to the channel attention module, the Sigmoid activation function is applied to the generated spatial attention weights to restrict the weights to between 0 and 1.
[0058] (4) Attention weighting: The obtained spatial attention weights are applied to the original feature map to weight the features at each spatial location. This can highlight important image regions and reduce the influence of unimportant regions.
[0059] Preferably, the training loss function is expressed as follows:
[0060] L=λ1L au +λ2L landmark +λ3L emtion
[0061] Among them, L au and L emotion The weighted cross-entropy loss functions for facial AU detection and facial emotion recognition are L and L, respectively. landmark λ1, λ2, and λ3 are weighted loss functions for facial landmark detection, and λ1, λ2, and λ3 are hyperparameters, assigning weights to each loss function.
[0062] The performance of the built model is tested using a test set. An appropriate optimizer is selected and hyperparameters are adjusted to ensure that the model achieves optimal accuracy on the test set.
[0063] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for establishing a facial action unit detection model based on multi-task learning, characterized in that... The method includes: Step S1: Use a compilation tool to perform key point detection on the face images in the dataset, and align and crop the face images in the original dataset based on the key points; Step S2: The aligned and cropped image is randomly cropped and horizontally flipped before being input into the network; Step S3: Establish an initial neural network detection model and train it using the dataset to obtain a facial AU detection model; Step S4: Train the initial neural network model end-to-end using face images labeled with AU tags, facial landmark tags, and emotion tags; Step S5: Change some hyperparameters, retrain the model, optimize the neural network detection model using the loss function, and take the model with the highest facial AU detection accuracy as the final result. The initial neural network model includes: The underlying network is used to perform basic feature extraction on the images input to the model. In a multi-task model, the underlying network is shared by multiple tasks. The auxiliary task learning module includes facial landmark detection and facial emotion recognition. Both take the output of the underlying network as input. The underlying features are processed by their respective feature extraction networks and then by a fully connected layer to obtain the prediction results of facial landmarks and facial emotions. The main task learning module, used for facial AU detection, consists of two parts: a local feature extraction module and an AU relationship learning module. The underlying network uses the output as input to the local feature extraction module. It provides an attention map for each AU using the keypoint coordinates detected by the auxiliary task. Each AU uses its own independent branch to extract local features. The network structure of each branch is the same as the network structure of the auxiliary task. The output of the local feature extraction part is used as input to the AU relationship learning module. The relationship between AUs is encoded using the AU relationship graph. The relationship graph consists of nodes and edges. Each node represents a feature of an AU, and the edges represent the correlation between AUs. The correlation of AUs is obtained by calculating the conditional probability of each AU occurring in the training samples. The final relation matrix is obtained, and GAT is used to learn the graph. The initial neural network model also includes: a feature fusion module for utilizing useful features from the auxiliary task to aid in the learning of the face AU detection task; The feature fusion module introduces an attention mechanism to select useful information from the auxiliary task and uses CBAM to guide the flow of information from the auxiliary task to the main task.
2. The method for establishing a facial action unit detection model based on multi-task learning as described in claim 1, characterized in that, The training loss function is expressed as follows: in, and These are the weighted cross-entropy loss functions for facial AU detection and facial emotion recognition, respectively. The weighted loss function for facial landmark detection is... , and These are hyperparameters that assign weights to each loss function.
Citation Information
Patent Citations
Face emotion recognition model and method and electronic device thereof
CN109583431A
Low expression symptom detection method based on graph neural network and facial action unit
CN116665267A
