A dynamic face generation method based on facial emotion analysis

Abstract

The present invention discloses a method for generating dynamic faces based on facial emotion analysis. First, data preprocessing is performed to detect and align faces in images; then a facial emotion analysis model is constructed, and the training of the facial emotion analysis model is completed through the preprocessed face images; finally, a dynamic face video is generated. The present invention guides the generation of faces through the analysis of facial emotions. In order to ensure the accuracy of facial emotion analysis in real scenes, the calculation of importance weights is introduced for each sample. By re-labeling samples with low importance weights, the problem of noisy labels in data sets is solved. An implicit semantic data enhancement algorithm is used, which cleverly utilizes the property of deep neural networks that are good at learning linear representations, completes the semantic amplification process in feature space, and hardly introduces any additional calculation or time overhead, thereby improving the performance of emotion analysis and increasing the rationality of face generation.

Description

Technical Field

[0001] The present invention belongs to the field of dynamic face generation, and relates to a dynamic face generation method based on facial emotion analysis, which is used for performing appropriate dynamic video generation after emotion analysis of an input facial image. Background Art

[0002] Due to the rise of various short video platforms (such as Douyin and Kuaishou), many apps have special effects for video beautification. Among these special effects, there are technologies that can make faces dynamic. However, the special effects provided by many platforms often convert facial images into videos with only a single dynamic effect, which may be inappropriate in some scenarios. For example, if a user uploads a photo with a sad expression, but the video special effects make the person in the image sing happily, this is obviously not reasonable enough. Therefore, when generating dynamic faces, the analysis of facial emotions is indispensable. At the same time, the accuracy of the analysis results and how to use the analysis results to guide the generation of dynamic faces are also crucial.

[0003] Facial expressions often determine people's facial emotions, and the progress of facial expression recognition depends largely on the progress of face recognition technology. In the past few decades, researchers have made significant progress in facial expression recognition. As early as the twentieth century, Ekman and Friesen defined six basic emotions based on cross-cultural research, which showed that humans perceive certain basic emotions in the same way, regardless of culture. These typical facial expressions are anger, disgust, fear, happiness, sadness and surprise. Subsequently, contempt became one of the basic emotions.

[0004] Traditional expression recognition work collects samples in a controlled laboratory environment for training facial expression recognition systems. Such datasets include CK+, Oulu-CASIA, and JAFFE. These datasets have fewer samples and labels are easier to obtain. However, models trained with small datasets cannot be well generalized to real-world scenarios. Therefore, it is particularly important to use large datasets collected from real scenes, such as AffectNet, RAF-DB, and FERPlus. However, for these large-scale datasets collected from the Internet, it is extremely difficult to obtain high-quality labels due to the subjectivity of the annotators and the uncertainty caused by the image quality on the Internet. Therefore, it is very important to deal with noisy labels during training.

[0005] Most of the treatments for noisy labels adopt two methods. One is to correct the labels through the noise transfer matrix. The other is to identify the noisy labels and suppress their influence during training. The former method generally performs poorly because it is difficult to accurately estimate the noise transfer matrix. The latter requires identifying the clean labels as low-loss labels in the early stage of training, which relies on the fact that the network must first fit the clean label samples before overfitting the noisy label samples, but in the early stage of training, the loss of samples with noisy labels is high, so the network will be disturbed during training, which is not conducive to the promotion of this method. In addition, the latter method requires prior information about the distribution of data noise, which is not desirable in real scenarios.

[0006] In addition, since most of the images in these large datasets come from web crawlers, the imbalance of the training set is also a serious problem, which can easily lead to overfitting during the training process. Data augmentation can increase the differences between classes to alleviate the overfitting problem. Common data augmentation methods include cropping, horizontal flipping, rotation, and color jittering. Compared with general classification work, faces have greater commonality, so data augmentation technology has excellent performance in facial expression recognition. Although these enhancement techniques are effective, they cannot perform semantic transformations, such as changing the background of an object or the texture of a foreground object. Recent work has shown that if semantic transformations are allowed, data augmentation can be more powerful and more effective in solving the overfitting problem in unbalanced datasets.

[0007] In summary, the dynamic face generation task based on facial emotion analysis is a direction worthy of in-depth research. This project intends to start from several key difficult problems in this task, provide new ideas for existing methods, and finally form a complete dynamic face generation system. Through the above content analysis, there are three main difficulties in the design process of this system:

[0008] (1) Noisy labels are difficult to handle simply.

[0009] (2) Overfitting caused by imbalanced datasets.

[0010] (3) How to use emotion analysis results to guide face generation. Summary of the invention

[0011] In view of the deficiencies in the prior art, the present invention provides a dynamic face generation method based on facial emotion analysis, which has solved the above three difficult problems.

[0012] 1. A facial emotion analysis model was designed, which added importance weights to each training set sample and divided the samples into high-importance group and low-importance group in a certain ratio by sorting the importance weights. During the training process, samples with possibly incorrect labels in the low-importance group were relabeled to solve the problem of noisy labels in the samples.

[0013] 2. Different from traditional data enhancement methods, the present invention uses an implicit semantic data enhancement method. It does not limit data transformation to simple geometric transformation, but presents it more at a more advanced semantic level. It cleverly utilizes the property of deep neural networks that are good at learning linear representations to complete the amplification process in the feature space. It does not require training of any auxiliary generation model (such as GAN, etc.), and almost does not introduce any additional calculation or time overhead. It directly optimizes an upper bound of the expected loss of infinitely amplified samples. The final form is only a new loss function, which is simple, easy to use, and easy to implement, thereby further solving the overfitting problem.

[0014] 3. In the face generation part, the present invention uses the results of emotion analysis to guide the generation of faces. It analyzes the output expression classification probability distribution, then selects a preset driving video, extracts the actions in the driving video, reenacts the input face photo, and generates a suitable dynamic face video to solve the problem of inappropriate face actions and expressions.

[0015] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0016] Step (1), data preprocessing, detecting and aligning faces in the image;

[0017] Preprocessing of the face dataset: For each face image in the face dataset, the five key points of the face are first detected through the MTCNN model pre-trained based on the WIDER dataset, and then the face area in the original image is expanded outward to ensure that there is no black area in the face image while the size remains unchanged. Finally, the face is aligned by performing affine transformation on the coordinates of the face key points to obtain the preprocessed face image.

[0018] Step (2), constructing a facial emotion analysis model;

[0019] The facial emotion analysis model includes a CNN backbone network and two branch structures.

[0020] For the input face image samples, we first extract deep features through a CNN backbone network, and then input the extracted features into two branch structures. The first branch structure adds an importance weight to the features of each sample through linear mapping, and divides the samples with importance weights into high-importance groups and low-importance groups in a certain proportion. The second branch performs linear classification on the deep features to obtain the probability distribution vector of the expression category.

[0021] Step (3), completing the training of the facial emotion analysis model through the preprocessed face images;

[0022] In the first branch of the facial emotion analysis model, the ranking loss is used to train the parameters of the importance weight calculation part to constrain the importance weight of uncertain samples. In the second branch of the facial emotion analysis model, the generated classification results are compared with the actual labels, and the implicit semantic data augmentation loss (ISDAloss) is used to train the model parameters while amplifying the features at the semantic level.

[0023] When the model is trained to have a certain recognition ability, dynamic relabeling is performed after each round of model parameter adjustment to correct the labels of uncertain samples in the low-importance group and reduce the interference caused by noise labels during model training. In the process of continuously adding more accurate importance weights to samples and relabeling noise labels, the model can better extract features of facial expressions and use the extracted features for classification.

[0024] Step (4), generating dynamic face video;

[0025] According to the probability distribution vector of the expression category obtained by the second branch of the facial emotion analysis model, a suitable driving video is selected, and finally the original image and the driving video are input into the existing face video generation model to generate the final dynamic face video.

[0026] Further, the specific method of step (1) is as follows:

[0027] Extract five key points from face images using the MTCNN model pre-trained on the WIDER dataset 5 represents the five key points of the face (nose, left eye, right eye, left corner of mouth, right corner of mouth), and 2 represents the horizontal and vertical coordinates of each key point in the figure. In order to make the face image have no black edges without changing the size, the detected face area is expanded outward to 100% to generate new key point coordinates. Finally, the face is affine transformed by the new coordinates and the size is adjusted to 224×224 to obtain a new face image. Among them C 1 is the number of channels.

[0028] The facial emotion analysis model described in step (2) is as follows:

[0029] The facial emotion analysis model includes a CNN backbone network and two branch structures.

[0030] For the input face image sample, a CNN backbone network is first used to extract deep features, and then the extracted features are input into two branch structures respectively. The calculation of importance weights in the first branch of the facial emotion analysis model consists of a fully connected layer and a Sigmoid activation function. Assume Representing the facial features of N images, the formula for importance weight is defined as follows:

[0031]

[0032] where α i is the importance weight of the i-th sample, W a Parameters of the fully connected layer, S is the Sigmoid activation function.

[0033] According to the importance weight of each sample, they are sorted in descending order and divided into high importance group and low importance group according to the ratio β.

[0034] The calculation of the classification result in the second branch consists of a fully connected layer and a Softmax activation function. After the sample feature x is input into the second branch, the probability distribution vector is obtained. Among them C 2 Indicates the number of categories.

[0035] Further, the specific method of step (3) is as follows:

[0036] 3-1. In order to constrain the importance of uncertain samples, Use ranking loss to ensure that low-importance samples are weighted The importance weight is not higher than that of high importance samples. The specific form is as follows:

[0037]

[0038]

[0039] where δ 2 is a fixed hyperparameter, α H and α L are the averages of the high importance group with β*N=M samples and the low importance group with NM samples, respectively.

[0040] 3-2. By calculating the intra-class covariance matrix of each category in the statistical data set, a zero-mean Gaussian distribution is constructed for each category, and then meaningful semantic transformation directions are sampled from it for data augmentation in the intrinsic feature space of each category.

[0041] For the depth feature x corresponding to the i-th sample i , the augmented eigenvector in the feature space All from distribution It is obtained by random sampling from any direction. That is:

[0042]

[0043] Where λ is a parameter used to control the strength of semantic data augmentation. Given this form, the traditional idea of ​​data augmentation is to augment the feature Sampling multiple times, optimizing its average cross entropy loss:

[0044]

[0045] Where N is the number of samples, y i is α i The corresponding label, ω, b is the parameter of the last linear classification layer of the network, C 2 is the number of categories, Θ is the network parameter. Using infinite sampling, that is, the case of M→∞, through Jensen inequality, we get an upper bound that is easy to calculate:

[0046]

[0047] The obtained results are converted into a new loss function (ISDA loss).

[0048] Finally, the obtained Ranking loss and ISDA loss are added according to the specified ratio γ, that is:

[0049]

[0050] Backpropagate the input to the network until the loss converges.

[0051] 3-3. For samples with uncertain labels, correct the possible wrong labels.

[0052] Specifically, consider whether to re-label samples with low importance, and obtain the probability distribution vector P of each sample through the second branch of the facial emotion analysis model. For each low-importance sample, when the difference between the maximum predicted probability in the probability distribution vector P and the predicted probability of the sample's given label is greater than a certain threshold, a new label is assigned to the sample. The specific formula is as follows:

[0053]

[0054] Where y' represents the new label, δ 1 is a fixed minimum threshold, P max is the maximum predicted probability, P gtInd is the predicted probability of a given label, l max and l org As the index of the maximum prediction and the original given label, a non-negative monotonically increasing function f is added to the fixed threshold, which means that the threshold of re-labeling increases with P gtInd The increase of can effectively avoid wrong re-labeling and make the convergence more stable. The re-labeled samples will participate in the next round of training with new labels. In order to make the re-labeled labels more accurate, dynamic re-labeling is performed only after the model has a certain recognition ability.

[0055] Furthermore, the specific method of step (4) is as follows:

[0056] When the model converges, the probability distribution vector P of the expression category is first obtained according to the second branch of the facial emotion analysis model. Then the expression with the highest probability is selected as the result of the emotion analysis, and finally the driving video corresponding to the expression and the original image are input into the FOM model pre-trained on the VoxCeleb dataset to complete the generation of the dynamic face video, and the final dynamic face video is obtained, where the driving video is the action video pre-set locally, and each expression corresponds to a driving video.

[0057] The beneficial effects of the present invention are as follows:

[0058] The present invention proposes a dynamic face generation method based on facial emotion analysis. At present, many face generation methods use a single driving video for action generation, ignoring the rationality of the expression in the final generated action. The present invention guides the generation of faces through the analysis of facial emotions. In order to ensure the accuracy of facial emotion analysis in real scenes, the emotion analysis module of the present invention uses real scene data sets for training. However, for large-scale data sets collected from real scenes, the subjectivity of the annotator and the noise labels caused by the image quality problem, and the overfitting caused by the imbalance of the data set are obstacles to the correct prediction of facial emotions. Therefore, the present invention introduces the calculation of importance weights for each sample, and solves the problem of noise labels in the data set by re-labeling samples with low importance weights. In order to reduce the overfitting of the data set, the present invention uses an implicit semantic data enhancement algorithm, cleverly utilizing the property that deep neural networks are good at learning linear representations, completing the semantic amplification process in the feature space, and almost does not introduce any additional calculation or time overhead. The method of the present invention improves the performance of emotion analysis and increases the rationality of face generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Flow chart of an embodiment of the method of the present invention;

[0060] Figure 2 This is a structural diagram of the facial emotion analysis model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The detailed parameters of the present invention are further described in detail below.

[0062] like Figure 1 As shown, a dynamic face generation method based on facial emotion analysis is provided, and the specific steps are as follows.

[0063] Step (1), data preprocessing, detecting and aligning faces in the image;

[0064] The RAF-DB dataset is used as training and testing data.

[0065] For each image in the RAF-DB dataset, five key points are extracted using the MTCNN model pre-trained on the WIDER dataset. In order to prevent the face image from having black edges without changing its size, the detected face area is expanded outward to 100% to generate new key point coordinates. Through the new coordinates and the customized affine transformation target point coordinates, the face image is affine transformed and the size is adjusted to 224×224 to obtain a new face image

[0066] Step (2), constructing a facial emotion analysis model;

[0067] In the facial emotion analysis model, ResNet-18 is used as the backbone network to extract features from the face images of the RAF-DB dataset. The dimension of the feature map of each sample is 512. In the first branch, the feature map of each sample is passed to the fully connected layer and the Sigmoid function calculation, and the importance weight of each sample is obtained, ranging from 0 to 1. It is sorted in descending order and divided into a 70% high importance group and a 30% low importance group according to the ratio β=0.7. In the second branch, the feature map of each sample is passed to the fully connected layer and the Softmax function calculation to obtain the probability distribution vector of the expression.

[0068] Figure 2 This is a structural diagram of the facial emotion analysis model according to an embodiment of the present invention.

[0069] Step (3), completing the training of the facial emotion analysis model through the preprocessed face images;

[0070] 3-1. For the importance weight of each sample, use the formula:

[0071]

[0072] Ensure that the average weight of samples in the high-importance group is higher than that in the low-importance group, where δ 2 =0.15.

[0073] 3-2. Construct a covariance matrix for each category in the RAF-DB dataset and a Gaussian distribution with zero mean Where λ = 0.5. By Infinite sampling, the feature x of each sample i Perform semantic data augmentation. This process is completed through the ISDA loss obtained previously:

[0074]

[0075] According to the obtained Ranking loss and ISDA loss, add them according to the specified ratio γ = 0.5, that is:

[0076]

[0077] Backpropagate the input to the network until the loss converges.

[0078] 3-3. The expression probability distribution vector P is obtained by passing the low importance group samples through the second branch of the sentiment analysis model. Then compare P by the formula max -P gtInd >δ 1 +f(P gtInd ) is true, if so, the sample label will be relabeled as the category with the highest probability, otherwise the original label is maintained, where δ 1 = 0.15, f is the Sigmoid function. Dynamic rescaling is performed after 10 rounds of model training.

[0079] Step (4): Generate dynamic face video

[0080] First, the probability distribution of facial expressions is obtained through the convergent emotion analysis model Then the expression with the highest probability is selected as the result of emotion analysis. Finally, the driving video corresponding to the expression and the original image are input into the FOM model pre-trained on the VoxCeleb dataset to complete the generation of dynamic faces and obtain the final dynamic face video. The driving video is a pre-set action video locally, and each expression corresponds to a driving video.

Claims

1. A method for generating dynamic faces based on facial emotion analysis, characterized in that: The steps include: Step (1), data preprocessing, detecting and aligning faces in the image; Preprocessing of the face dataset: For each face image in the face dataset, the five key points of the face are first detected through the MTCNN model pre-trained based on the WIDER dataset, and then the face area in the original image is expanded outward to ensure that there is no black area in the face image without changing its size. Finally, the face is aligned by performing affine transformation on the coordinates of the face key points to obtain the preprocessed face image; Step (2), constructing a facial emotion analysis model; Step (3), completing the training of the facial emotion analysis model through the preprocessed face images; Step (4), generating dynamic face video; According to the probability distribution vector of the expression category obtained by the second branch of the facial emotion analysis model, a suitable driving video is selected, and finally the original image and the driving video are input into the existing face video generation model to generate the final dynamic face video; The facial emotion analysis model includes a CNN backbone network and two branch structures; For the input face image sample, a CNN backbone network is first used to extract deep features, and then the extracted features are input into two branch structures respectively. The first branch structure adds an importance weight to the features of each sample through linear mapping, and divides the samples with importance weights into high importance group and low importance group in a certain proportion; the second branch performs linear classification on the deep features to obtain the probability distribution vector of the expression category; The specific method of step (2) is as follows: The facial emotion analysis model includes a CNN backbone network and two branch structures; For the input face image sample, a CNN backbone network is first used to extract deep features, and then the extracted features are input into two branch structures respectively. The calculation of the importance weight in the first branch of the facial emotion analysis model consists of a fully connected layer and a Sigmoid activation function. Assume Representing the facial features of N images, the formula for importance weight is defined as follows: where α i is the importance weight of the i-th sample, W a Parameters of the fully connected layer, S is the Sigmoid activation function; According to the importance weight of each sample, it is sorted in descending order and divided into a high importance group and a low importance group according to the ratio β; The calculation of the classification result in the second branch consists of a fully connected layer and a Softmax activation function. After the sample feature x is input into the second branch, the probability distribution vector is obtained. Where C2 represents the number of categories; In the first branch of the facial emotion analysis model, contrast loss is used to train the parameters of the importance weight calculation part to constrain the importance weight of uncertain samples; In the second branch of the facial emotion analysis model, the generated classification results are compared with the actual labels, and implicit semantic data augmentation loss is used to train the model parameters while amplifying the features at the semantic level; When the model is trained to have a certain recognition ability, dynamic relabeling is performed after each round of model parameter adjustment. The labels of uncertain samples in the low-importance group are corrected to reduce the interference caused by noise labels during model training. In the process of continuously adding more accurate importance weights to samples and relabeling noise labels, the model can better extract features of facial expressions and use the extracted features for classification.

2. The method for generating dynamic faces based on facial emotion analysis according to claim 1, characterized in that: The specific method of step (1) is as follows: Extract five key points from face images using the MTCNN model pre-trained on the WIDER dataset Where 5 represents the five key points of the face, which are the nose, left eye, right eye, left corner of the mouth and right corner of the mouth; 2 represents the horizontal and vertical coordinates of each key point in the figure; in order to prevent the face image from having black edges without changing its size, the detected face area is expanded outward to 100% to generate new key point coordinates; finally, the face is affine transformed by the new coordinates and the size is adjusted to 224×224 to obtain a new face image Where C1 is the number of channels.

3. The method for generating dynamic faces based on facial emotion analysis according to claim 2, characterized in that: The specific method of step (3) is as follows: 3-1. In order to constrain the importance of uncertain samples, ranking loss is used to ensure that the importance weight of low-importance samples is not higher than that of high-importance samples. The specific form is as follows: Where δ2 is a fixed hyperparameter, α H and α L are the averages of the high importance group with β*N=M samples and the low importance group with NM samples, respectively; 3-2. By collecting the intra-class covariance matrix of each category in the statistical data set, a zero-mean Gaussian distribution is constructed for each category, and then meaningful semantic transformation directions are sampled from it for data augmentation in the intrinsic feature space of each category; For the depth feature x corresponding to the i-th sample i , the augmented eigenvector in the feature space All from distribution is obtained by random sampling from any direction; that is: Where λ is a parameter used to control the strength of semantic data augmentation; given this form, the idea of ​​data augmentation is to augment the feature Sampling multiple times, optimizing its average cross entropy loss: Where N is the number of samples, y i is α i The corresponding labels, ω, b are the parameters of the last linear classification layer of the network, C2 is the number of categories, Θ is the network parameter; using infinite sampling, that is, M→∞, through Jensen inequality, we get an upper bound that is easy to calculate: The obtained results are converted into a new loss function Finally, the obtained Rankingloss and ISDAloss are added according to the specified ratio γ, that is: Back-propagate the input to the network until the loss converges; 3-3. For samples with uncertain labels, correct possible wrong labels; Specifically, consider whether to re-label samples with low importance, and obtain the probability distribution vector P of each sample through the second branch of the facial emotion analysis model; for each low-importance sample, when the difference between the maximum predicted probability in the probability distribution vector P and the predicted probability of the given label of the sample is greater than a certain threshold, a new label is assigned to the sample; the specific formula is as follows: Where y' represents the new label, δ1 is the fixed minimum threshold, and P max is the maximum predicted probability, P gtInd is the predicted probability of a given label, l max and l org As the index of the maximum prediction and the original given label, a non-negative monotonically increasing function f is added to the fixed threshold, which means that the threshold of re-labeling increases with P gtInd It increases with the increase of , which can effectively avoid erroneous re-labeling and make the convergence more stable; the re-labeled samples will participate in the next round of training with new labels; in order to make the re-labeled labels more accurate, dynamic relabeling is performed only after the model has a certain recognition ability.

4. The method for generating dynamic faces based on facial emotion analysis according to claim 3, characterized in that: The specific method of step (4) is as follows: When the model converges, the probability distribution vector P of the expression category is first obtained according to the second branch of the facial emotion analysis model; then the expression with the highest probability is selected as the result of the emotion analysis. Finally, the driving video corresponding to the expression and the original image are input into the FOM model pre-trained on the VoxCeleb dataset to complete the generation of the dynamic face video, and the final dynamic face video is obtained, where the driving video is an action video pre-set locally, and each expression corresponds to a driving video.

Images