A meta-learning-based facial action unit detection method

Abstract

The present invention discloses a method for detecting facial action units based on meta-learning, comprising the following steps: 1) constructing a data set, sampling data in the form of a task set. Each task set includes support set data and query set data, the support set data will be used for meta-training, and the query set data will be used for meta-testing; 2) constructing an action detection unit (AU) model, using an AU detection model based on regional relationship learning. The regional features use facial key point landmarks to extract the AU center, and then map them proportionally to the feature map to extract the AU regional features; the relationship model uses transformer to learn the relationship parameters between AUs; 3) constructing a meta-learning training strategy based on MAML, calling the model under the meta-learning training strategy a meta-learner, first initializing the model structure parameters of the meta-learner, and then combining the meta-learning data set and the AU detection model, using the gradient descent-based meta-learning method MAML to train and test the model.

Description

Technical Field

[0001] The present invention relates to the field of facial expression recognition, and in particular to a facial motion unit detection method based on meta-learning training. Background Art

[0002] Facial expressions can reveal a person's inner activities, mental state, and social behaviors. With the development of artificial intelligence, human-centered facial expression recognition has gradually attracted widespread attention from the industry and academia. Facial Action Units (AUs) are important objective descriptions of facial expressions. The detection of facial action units (AUs) is also one of the basic steps for automatically identifying emotional expressions and cognitive states. The detection and recognition of AUs is of great significance for understanding and describing facial expressions. Based on this, facial AU detection and analysis has gradually become one of the hot research topics in the fields of computer vision and affective computing.

[0003] However, manual annotation of AUs is a difficult, time-consuming, and tedious task. Moreover, objective and consistent annotation of AUs requires human experts with a comprehensive understanding of AUs, which is costly. Therefore, addressing the problem of insufficient annotation of AU detection datasets by proposing new methods is a task that needs to be addressed.

[0004] The current AU detection models mainly include the following:

[0005] 1. Models based on regional learning, such as the Enhanced and Cropped Network (EAC-Net) introduced by Wei et al., where E-net enhances the ROI of facial landmark features through attention mapping with C-net. These designs use independent CNNs to extract deeper features by cropping the facial regions around the detected landmarks.

[0006] 2. Models based on relationship learning, such as the AU-GCN model, first extract AU regional features, then regard the features as graph nodes, introduce them into the GCN model, and learn the relationship between AUs.

[0007] 3. Models based on the attention mechanism, such as JAA-Net proposed by Shao et al., no longer use face alignment as a preprocessing method, but jointly learn AU detection and face alignment, as well as the attention module formed by face alignment.

[0008] Existing AU detection models are usually trained using benchmark datasets with a large number of AU labels, such as DISFA, BP4D, CK+, and MMI. However, these models do not show good generalization performance when tested on new AU or new identity datasets.

[0009] Meta-learning is a deep learning method proposed in the past two years. It is different from traditional supervised learning, which requires a large number of data sets as input and obtains a model after training, which can be used to predict the corresponding data with the same distribution. However, the goal of meta-learning is to use tasks as input and obtain a meta-algorithm with multi-task prior knowledge by training multiple learning tasks. This meta-algorithm can solve new learning tasks using only a small number of new training samples. Therefore, meta-learning can well improve the generalization ability of the model and solve few-sample learning tasks.

[0010] Chelsea Finn et al. proposed a model-independent meta-learning method for fast adaptation of deep networks, referred to as MAML. The basic idea of ​​this method is to train the initial parameters of the model so that after updating the parameters through one or more gradient steps, the model has the best performance for the new task, and the gradient step is calculated based on a small amount of data in the new task. Since this method is mainly based on gradient updating parameters, it does not impose constraints on the model architecture, and can also be combined with various loss functions for back propagation. Therefore, it is a model-independent meta-learning method that can be combined with different models to complete different deep learning tasks.

[0011] Based on the above investigation and analysis, we plan to use the gradient descent-based meta-learning method MAML, combined with the characteristics of existing AU detection, to propose an AU detection model based on regional relationship learning, and then combine it with the existing benchmark expression datasets (BP4D, DISFA) to propose a meta-learning-based facial motion unit detection research method. Summary of the invention

[0012] The purpose of the present invention is to overcome the deficiencies in the prior art and propose an AU detection method based on meta-learning, which can solve the following problems:

[0013] 1. It is not only targeted at specific identity datasets, but can adapt to different identity datasets and has strong generalization ability, solving the problem of weak generalization ability of existing AU detection models;

[0014] 2. For small sample data sets, after only a few steps of gradient descent rapid training on a small amount of data, it can quickly adapt to new AU detection tasks and achieve good detection results, thus solving the problem of difficult AU data set annotation;

[0015] 3. Compared with the existing mainstream AU detection methods, the detection effect can achieve the best.

[0016] The objective of the present invention is achieved through the following technical solutions:

[0017] A face action unit detection method based on meta-learning comprises the following steps:

[0018] Step 1, sampling data using task set to construct a metadata set;

[0019] Among them, each task set contains support set data D s and query set data D q , the support set data will be used for meta-training, and the query set data will be used for meta-testing;

[0020] Step 2: Combine the characteristics of AU detection to build an AU detection model based on regional relationship learning;

[0021] Step 3: Use the model-independent meta-learning method MAML based on gradient descent as the training framework, take the metadata dataset as input, and train and test the AU detection model; where: the AU detection model under the meta-learning training strategy is called a meta-learner. Meta-learning training includes the following steps:

[0022] 301, initialize the meta-learning parameters, and record the initialization parameters as Θ 0 , as the initial input parameter of the inner loop update phase of meta-learning, that is, θ = Θ 0 ;

[0023] 302, inner loop training update:

[0024] Import task batch data. A task batch has K task sets, that is, Take one task set at a time Will support set data Input into the meta-learner, perform forward propagation on the current model parameters θ, and calculate the loss between the forward propagation prediction result and the training label Solving for the current loss Relative model parameters The gradient value of

[0025] Update the model parameters according to the obtained gradient value The mathematical expression is: Among them: α is the learning rate of the inner loop gradient update, the next time Each task set can repeat the inner loop update N times;

[0026] Perform the above inner loop update operation on a task batch, that is, loop K times, and finally get the parameters after inner loop training And the internal circulation loss

[0027] 303, Outer loop training update:

[0028] The query set data D qAs input, forward propagation is performed on the model parameters θ′, and the loss difference L between the forward propagation prediction result and the training label is calculated q ;

[0029] The inner loop loss L in step 302 is s With the current external circulation loss L q The weighted sum of is used as the optimization target of the meta-learner to initialize the model parameters Θ of the meta-learner 0 The optimization update is expressed mathematically as:

[0030]

[0031] Where: β is the learning rate of the outer loop gradient update, w a With w b is the loss weight;

[0032] 304, the model parameter θ updated in step 303 0 As a new initialization parameter group, resample the task batch and repeat steps 301 to 303 several times until the model converges and the training is completed;

[0033] Further, the generation process of the metadata set in step 1 is as follows:

[0034] 101. The data set is grouped into task sets, with a series of facial expression groups of different individuals at different times as tasks. Each input is sampled according to the task set, and the task set is divided into support set D s and query set D q , expressed mathematically as:

[0035]

[0036] in and They represent the support set and task set of a task respectively, and M is the total number of tasks;

[0037] 102. During training, the task batch sampling method is adopted, that is, K task sets are sampled in each task batch, and (S+Q) samples are sampled in each task set, where S is the training set sample and Q is the test set sample.

[0038] Further, the AU detection model generation process described in step 2 is as follows:

[0039] 201. Input the backbone network VGG16 to extract features and obtain feature maps;

[0040] 202. Using facial key point landmarks to locate the center of the facial AU on the feature map in a geometric mapping manner, extracting the AU region, and obtaining the AU feature after the AU region is subjected to a convolution operation;

[0041] 203. Using transformer as the relationship learning module, the input AU features are first normalized and then sent to transformer for relationship learning to obtain AU features with relationship information;

[0042] 204. Classify the AU features through a classifier to obtain the prediction probability of each AU.

[0043] Beneficial Effects

[0044] 1. Use meta-learning training to train the AU detection model, classify the data set by identity, and the learned model can adapt to different identity data sets. When encountering a new identity data set, only a small amount of samples are needed for rapid training to quickly adapt, and the model has strong generalization ability.

[0045] 2. The regional relationship network designed according to the characteristics of AU detection is suitable for AU detection tasks. The model can learn AU regional features and the dependencies between AUs, and has good detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a network model framework diagram, (a) is the model framework diagram, and (b) is the specific module diagram;

[0047] Figure 2 This is the overall framework diagram of meta-learning training. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] The implementation method of the present invention is given by taking the facial expression dataset BP4D dataset as an example. The overall framework of the method is shown in Figure 1 The entire system algorithm process includes three steps: metadata dataset construction, AU detection model construction, and MAML meta-learning framework construction.

[0050] The specific steps are as follows:

[0051] 1. Based on the original dataset BP4D, construct a metadata dataset:

[0052] First, the data in the BP4D dataset is preprocessed. The BP4D dataset includes a total of 41 different identities in a laboratory environment. Each identity has 8 time periods. Each time period includes a series of two-dimensional face images that change from neutral expressions to other expressions. At the same time, the dataset includes the state labels of motion units in some face images. There are 12 motion units used in the experiment, namely AU1, AU2, AU4, AU7, AU10, AU12, AU14, AU15, AU17, AU23, and AU24. Then the size of each image is adjusted to 224*224 using a linear interpolation algorithm.

[0053] The dataset is grouped into task sets, with a series of facial expressions of different individuals at different times as tasks. The BP4D dataset can be divided into 41*8=328 tasks. Each input is sampled in the task set, and the task set is divided into support set D s and query set D q , expressed mathematically as in and They represent the support set and task set of a certain task set respectively, and M is the total number of tasks.

[0054] During training, we use the task batch sampling method, that is, for each task batch, we sample K identity sets, and randomly sample (S+Q) samples from each identity set, where S is the training set sample and Q is the test set sample. That is, each task batch can be expressed as in There are K*S data samples, There are K*Q data samples. When sampling data, let K=5, S=5, Q=15.

[0055] 2. Combining the characteristics of AU detection, build an AU detection model based on regional relationship learning:

[0056] The specific implementation steps are as follows:

[0057] Step 1: Feature extraction. Input a 224*224*3 RGB face image to the backbone network VGG16 for feature extraction, and obtain a 512*14*14 feature map;

[0058] Step 2, AU regional feature learning. Due to the regionality of AU detection, that is, AU only exists in specific areas of the face, the AU distribution positions of different faces are similar, so only the AU regional features of the specific positions of the face need to be extracted for learning. Use landmarks to locate the AU center point of the original image, and then map it proportionally to the corresponding feature map, locate the AU center point on the feature map, and locate a total of 18 AU center points. Each AU center point takes a 3*3 rectangular area as the AU area. Each AU area is upsampled to 6*6, and then passes through two layers of kernel=3,padding=1,stride=1 convolution layers to obtain 512*6*6 AU regional features. Due to the symmetry of AU, the symmetrical AU areas are added and merged into one AU feature. Since some AU areas may coexist with several AUs, such as AU12, AU14, and AU15 coexist in one AU area, it is necessary to separate the area into separate AU areas. Through operations such as addition and separation, 12 AU areas are left, which are the final AU regional features. Each AU region feature is obtained by average pooling operation to obtain 512*1*1, and then a 150-dimensional AU region feature is obtained by flattening the full connection operation. At this time, there are 12 150-dimensional AU region features.

[0059] Step 3: AU relationship learning. AUs do not exist alone, but have symbiotic or mutually exclusive relationships. Therefore, it is necessary to add a relationship learning module to learn AU relationship features. The 12 AU region features in step 2 are used as input and sent to the transformer to learn the relationship parameters between AUs. Then, MLP is performed to obtain 512-dimensional AU relationship features.

[0060] Step 4: AU classifier. The AU features obtained in step 3 are sent to the AU classifier. After passing through two fully connected layers, a 12-dimensional vector is obtained, which is the prediction result.

[0061] 3. Construction of MAML meta-learning framework:

[0062] Using the metadata dataset from step 1 and the AU detection model from step 2, we build a meta-learning training framework based on MAML.

[0063] The AU detection model in step 2 is called a meta-learner in the training framework. The training process of the framework is as follows:

[0064] Step 1: Initialize the meta-learner parameters and denote the initialization parameters as Θ 0 , as the initial input parameter of the inner loop update phase of meta-learning, that is, θ = Θ 0 ;

[0065] Step 2: Inner loop training update, import task batch data. Each task batch can be expressed as:

[0066]

[0067] The support set data D s Input into the meta-learner, perform forward propagation on the current model parameters θ, and calculate the loss L between the forward propagation prediction result and the training label s , solve the current loss L s The gradient of the relative model parameter θ is used to update the model parameter θ according to the obtained gradient value;

[0068] Step 3: Update the outer loop training and update the query set data D q As input, forward propagation is performed on the model parameters θ, and the loss difference L between the forward propagation prediction result and the training label is calculated q , the inner loop loss L of step 2 s With the current external circulation loss L q The weighted sum of is used as the optimization target of the meta-learner to initialize the model parameters Θ of the meta-learner 0 Optimization update.

[0069] Step 4: Update the model parameters θ after step 3 0 Use this as a new initialization parameter group, resample the task batch, and repeat steps 1 to 3 several times until the model converges and the training is completed.

[0070] Step 5: When testing the test task, the test set is divided into an adaptation set and a query set, that is, testbatch = {D a ,D q}, use D a As input, repeat steps 1 to 2. The purpose is to enable the model to quickly adapt to the new task after a few steps of gradient descent, and then directly use the model parameters output from step 2 to test D q , do forward propagation to get the test results and complete the result prediction.

[0071] The entire experiment was trained in the BP4D dataset in a task batch mode, with a batch-size of 4, an inner loop learning rate of 0.1, an outer loop learning rate of 0.001, and Adam optimizer training. 100 task batches were sampled for each round of training, and a total of 100 rounds of training were performed until the model converged and then tested.

[0072] Using F1-score as the metric, the comparison between the AU detection results based on meta-learning and the existing AU detection model is shown in the following table:

[0073]

[0074]

[0075] It can be seen from the results that the results of the present invention are optimal on most AUs, including AU6, AU7, AU10, AU12, AU14, and AU23, and the average result is also the best, which is 1.3 percentage points higher than the previous best model, proving that our AU detection model based on meta-learning has stronger generalization ability.

[0076] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solution of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, a person of ordinary skill in the art can also make many forms of specific changes under the guidance of the present invention, which all fall within the scope of protection of the present invention.

Claims

1. A method for detecting facial action units based on meta-learning, comprising the following steps: Step 1: Sample data using task sets to construct a metadata set; in: The generation process of the metadata set is as follows:

101. The data set is grouped into task sets, with a series of facial expression groups of different individuals at different times as tasks. Each input is sampled according to the task set, and the task set is divided into support set D s and query set D q , expressed mathematically as: in and They represent the support set and task set of a task respectively, and M is the total number of tasks; Among them, each task set contains support set data D s and query set data D q , the support set data will be used for meta-training, and the query set data will be used for meta-testing; 102. During training, the task batch sampling method is adopted, that is, K task sets are sampled in each task batch, and (S+Q) samples are sampled in each task set, where S is the training set sample and Q is the test set sample; Step 2: Construct an AU detection model based on regional relationship learning in combination with the characteristics of AU detection; wherein: the AU detection model generation process:

201. Input the backbone network VGG16 to extract features and obtain feature maps; 202. Using facial key point landmarks to locate the center of the facial AU on the feature map in a geometric mapping manner, extracting the AU region, and obtaining the AU feature after the AU region is subjected to a convolution operation; 203. Using transformer as the relationship learning module, the input AU features are first normalized and then sent to transformer for relationship learning to obtain AU features with relationship information; 204. Classify the AU features through a classifier to obtain the prediction probability of each AU; Step 3: Use the model-independent meta-learning method MAML based on gradient descent as the training framework, take the metadata dataset as input, and train and test the AU detection model; where: the AU detection model under the meta-learning training strategy is called a meta-learner; the meta-learning training includes the following steps: 301, initialize the meta-learning parameters, and record the initialization parameters as Θ 0 , as the initial input parameter of the inner loop update phase of meta-learning, that is, θ = Θ 0 ; 302, inner loop training update: Import task batch data. A task batch has K task sets, that is, Take one task set at a time Will support set data Input into the meta-learner, perform forward propagation on the current model parameters θ, and calculate the loss between the forward propagation prediction result and the training label Solving for the current loss Relative model parameters The gradient value of Update the model parameters according to the obtained gradient value The mathematical expression is: Among them: α is the learning rate of the inner loop gradient update, the next time Each task set can repeat the inner loop update N times; Perform the above inner loop update operation on a task batch, that is, loop K times, and finally get the parameters after inner loop training And the internal circulation loss 303, Outer loop training update: The query set data D q As input, forward propagation is performed on the model parameters θ′, and the loss difference L between the forward propagation prediction result and the training label is calculated q ; The inner loop loss L in step 302 is s With the current external circulation loss L q The weighted sum of is used as the optimization target of the meta-learner to initialize the model parameters Θ of the meta-learner 0 The optimization update is expressed mathematically as: Where: β is the learning rate of the outer loop gradient update, w a With w b is the loss weight; 304, the model parameter θ updated in step 303 0 As a new initialization parameter group, resample the task batch and repeat steps 301 to 303 several times until the model converges and the training is completed.

2. The method for detecting facial action units based on meta-learning according to claim 1, It is characterized in that The specific implementation of the loss function calculation method in step 2 and step 3 is as follows: Where: (1) represents the multi-label weighted cross entropy loss, which is used to calculate the multi-label AU classification loss; p i represents the true occurrence probability of the i-th AU, p i ∈[0,1], represents the predicted probability of the i-th AU, with weight w i It was introduced to alleviate the problem of data imbalance; setting where r i is the occurrence rate of the i-th AU as a positive sample in the training set; (2) is expressed as Dice (F1-score) loss, ∈ is a smoothing term; (3) represents the loss function of the inner and outer loops, and the inner loop directly uses L cross Estimate the error between the predicted value and the true value, L q To calculate the test set error, use L cross With L dice The weighted sum of μ is the weight factor; the outer loop uses L s With L q The weighted sum of a With w b is the weight factor.

Images