A multi-view expression recognition method based on digital human

Abstract

The present application relates to a kind of multi-view expression recognition methods based on digital person, comprising the following steps: obtaining real world expression dataset;Based on the generation of multiple expressions of digital virtual face under multiple angles of MetaHuman tool, construct digital human expression dataset;Angle classification model is constructed for angle classification to input image;Expression classification model is constructed, and the network weight of angle classification model is fixed, based on digital human expression dataset and real world expression dataset, the classification result of angle classification model is combined to train expression classification model;Based on small sample learning method, the output of the angle classification model and expression classification model trained is used to adjust the network parameter of expression classification model, and the optimal expression classification model is obtained;Based on angle classification model and optimal expression classification model, the input multi-view expression is identified.Compared with prior art, the present application has the advantages of accurate identification in real scene.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a multi-view facial expression recognition method based on digital humans. Background Technology

[0002] Facial expressions are an important means of conveying human emotions. Studies have shown that in daily communication, information conveyed through language and voice accounts for 7% and 38% respectively, while information conveyed through facial expressions accounts for as much as 55%. Therefore, facial expression recognition has gradually become a hot topic in the field of computer vision. Accurately recognizing human expressions allows for a precise understanding of human psychology and emotions, enabling better services to be provided to people.

[0003] Research in facial expression recognition began relatively early, but early studies primarily focused on psychology and biology. With the development of modern computer vision, facial expression recognition using computers has become possible. The basic steps of facial expression recognition include image preprocessing, facial feature extraction, and expression classification. Based on the different representations of facial expression features, it is further divided into static image facial expression recognition and dynamic sequence facial expression recognition. Early facial expression recognition mainly employed traditional statistical learning methods, utilizing manual feature extraction and shallow learning to extract features; however, with the continuous development of computer hardware, deep learning methods have demonstrated their powerful capabilities in facial expression recognition.

[0004] How to better apply facial recognition technology is also an important issue. Technology cannot remain only in research; it must be put into application to provide better services to humanity. Existing common deep learning-based facial expression recognition technologies are trained on datasets, and the final result depends on the structure of the dataset and the model. Models trained on datasets collected in the laboratory have high recognition accuracy but poor generalization ability, and it is difficult to reproduce the laboratory environment in real-life applications, thus hindering their effectiveness. Models trained on datasets collected in the real world have some generalization ability, but the recognition accuracy is poor. Moreover, different camera angles from which facial expressions are captured also affect the accuracy of expression recognition, but currently, multi-view facial expression datasets are scarce and few in number, which brings great difficulties to the training of deep learning. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-view facial expression recognition method based on digital humans, enabling model training on small sample facial expression datasets and improving facial expression recognition accuracy in real-world scenarios.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A multi-view facial expression recognition method based on digital humans includes the following steps:

[0008] Obtain a dataset of real-world facial expressions;

[0009] Digital virtual faces with various expressions from multiple angles were generated using the MetaHuman tool to construct a digital human expression dataset.

[0010] An angle classification model was constructed to classify the angles of the input image, and the angle classification model was trained based on a digital human facial expression dataset;

[0011] An expression classification model is constructed to classify input images into different expressions. The network weights of the angle classification model are fixed. The expression classification model is trained based on the digital human expression dataset and the real-world expression dataset, combined with the classification results of the angle classification model.

[0012] Based on the few-shot learning method, the network parameters of the expression classification model are adjusted by using the outputs of the trained angle classification model and expression classification model to obtain the optimal expression classification model.

[0013] The system recognizes multi-view facial expressions based on angle classification and optimal expression classification models.

[0014] The generated digital virtual face includes digital virtual face data from seven angles: 0°, 15°, 30°, 45°, 60°, 75°, and 90°.

[0015] The generated digital virtual face includes seven expressions: angry, disgusted, fearful, happy, neutral, sad, and surprised.

[0016] The specific method for constructing the digital human facial expression dataset is as follows:

[0017] A dataset of facial expressions collected from the real world is acquired and input into an expression feature extraction encoder. The encoder encodes the facial feature group of the current expression based on the different movements of the facial expression units. The MetaHuman tool is used to generate digital humans with different features. Using facial expression action units, the expression action units on the digital human face are controlled to correspond one-to-one with the expression action units in the recognized real-world dataset, thus reproducing the facial expressions of the real-world human face. Based on the 3D characteristics of the digital human face, the face of the digital human is rotated to obtain expression images from different angles. The acquired images are labeled with expression and angle information to generate a digital human expression dataset.

[0018] The angle classification model consists of six convolutional layers, three pooling layers, and one fully connected layer. The input image is passed through three blocks consisting of two 3×3 convolutional layers and one fully connected layer, and then flattened to obtain an encoded feature vector of length 864. This vector is then input into the fully connected layer and activated by the softmax function to output the angle classification result.

[0019] The angle classification results include front, half-front, and side.

[0020] The expression classification model consists of five blocks. The input image first passes through a 1×1 convolutional layer, an average pooling layer, a 1×1 convolutional layer, a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and two 3×3 convolutional layers, respectively. The outputs of these four layers are then merged into a tensor of size (48, 48, 384). The input is then passed to a block containing two 3×3 convolutional layers and a global pooling layer. The output of this block is merged with the reshaped tensor of size (24, 24, 1) from the angle classification model into a tensor of size (24, 24, 129). This tensor then passes through three blocks containing three 3×3 convolutional layers and a global pooling layer. Finally, the output tensor is flattened to obtain a feature vector of length 4608. The feature vector is then input into a fully connected layer, activated by the softmax function, and the expression classification result is output.

[0021] In the few-shot learning method, the feature vectors output by the angle classification model and expression classification model of the input image are truncated and input into the cosine distance metric and the Euclidean distance metric, respectively, to calculate the distance index. The distance index is then multiplied by a scale factor and summed before being passed through a softmax layer to obtain the final output.

[0022]

[0023] Among them, cos<a,b> and euc<a,b> Let f represent the cosine distance and Euclidean distance between a and b, respectively. θ (x) represents the feature vector output by the facial expression classification model, where x is the input image and w k These are the parameters of the fully connected layer in the distance metric, where α and β are scale factors.

[0024] The distance index calculated by the cosine distance metric is:

[0025]

[0026] The distance index calculated by the Euclidean distance meter is:

[0027]

[0028] Among them, s i,j w is the distance index calculated between i and j. j f represents the parameters of the fully connected layer in the distance metric. θ (x i x is the feature vector output by the image after processing by the facial expression classification model. i The input image.

[0029] During training, both the angle classification model and the expression classification model use the cross-entropy loss function to calculate the loss between the model output and the true value, and then use the Adam optimizer to optimize the parameters in the model and update the weights until the optimal model is obtained.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) This invention uses digital human technology to generate multi-angle, multi-expression facial images for training. The digital human expression dataset can greatly improve the problems of small data volume, uneven expression distribution, and low quality of existing multi-view expression datasets.

[0032] (2) The facial expression recognition model of the present invention adopts a newly designed conditional concatenated convolutional neural network, which combines an angle classification model and an image classification network, allowing the network to learn facial expression features based on angles and improving the performance of multi-view facial expression recognition.

[0033] (3) The present invention uses a method of training by mixing digital human multi-view datasets and real-world expression datasets, which can increase the amount of information in the model and improve the accuracy of multi-view expression recognition in real-world scenarios.

[0034] (4) The present invention uses a few-shot learning method to transfer the model to a real application scenario, which can quickly deploy the model in a real scenario and quickly and accurately identify expressions from multiple perspectives in the scenario. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method of the present invention;

[0036] Figure 2 A flowchart for generating a digital human facial expression dataset;

[0037] Figure 3 This is a flowchart illustrating the process of facial expression recognition in a practical application scenario.

[0038] Figure 4 This is a network structure diagram of the Inception module in the facial expression classification module of this invention;

[0039] Figure 5This is a network structure diagram of the overall posture classification module and expression classification model in this invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0041] This embodiment provides a multi-view facial expression recognition method based on digital humans, including the following steps:

[0042] 1) Obtain real-world facial expression datasets, such as the FER2013 dataset and the RAF-DB dataset.

[0043] 2) Based on the MetaHuman tool, generate digital virtual faces with various expressions (anger, disgust, fear, happiness, neutrality, sadness, surprise) from multiple angles (0°, 15°, 30°, 45°, 60°, 75°, 90°) to construct a digital human expression dataset.

[0044] Specifically, such as Figure 2 As shown, a real facial expression dataset is input into the facial expression feature extraction encoder, which encodes the facial feature group of the current expression based on the different movements of its facial expression units. The MetaHuman tool is used to generate digital humans with different features (such as different races, genders, ages, etc.). Using the facial expression action units, the facial expression action units on the digital human face are controlled according to the facial feature group to correspond one-to-one with the facial expression action units in the recognized real-world dataset, thus reproducing the facial expressions of real-world faces on the digital human face (that is, transferring the facial feature groups of the previously extracted expressions to the digital human face). Based on the 3D characteristics of the digital human face, the face of the digital human is rotated to obtain facial expression images from different angles. The acquired images are labeled with expression and angle information to generate a digital human expression dataset.

[0045] 3) Construct an angle classification model to classify the angles of the input image, and train the angle classification model based on the digital human expression dataset.

[0046] like Figure 5 As shown, the angle classification model consists of six convolutional layers, three pooling layers, and one fully connected layer. The input image is passed through three blocks consisting of two 3×3 convolutional layers and one fully connected layer, and then flattened to obtain an encoded feature vector of length 864. This vector is then input into the fully connected layer and activated by the softmax function to output the angle classification result (front, half-front, and side).

[0047] The output of the angle classification model is represented as: A(x i)=f θ (x i ), x i The input image.

[0048] 4) Construct an expression classification model to classify input images into different expressions, and fix the network weights of the angle classification model. Based on the digital human expression dataset and the real-world expression dataset, train the expression classification model by combining the classification results of the angle classification model.

[0049] like Figure 5 As shown, the facial expression classification model consists of five blocks. See [link / reference]. Figure 4 As shown, the input image first passes through a 1×1 convolutional layer, an average pooling layer, a 1×1 convolutional layer, a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and two 3×3 convolutional layers, respectively. The outputs of these four layers are then merged into a tensor of size (48,48,384). A block containing two 3×3 convolutional layers and a global pooling layer is input, and its output is merged with the reshaped tensor of size (24,24,1) from the angle classification model to form a tensor of size (24,24,129). This tensor then passes through three blocks containing three 3×3 convolutional layers and a global pooling layer. Finally, the output tensor is flattened to obtain a feature vector of length 4608. The feature vector is then input into a fully connected layer, activated by the softmax function, and the expression classification result is output.

[0050] When training the facial expression classification model, facial expression images from the digital human dataset and the real-world dataset are mixed together and input into the multi-view angle classification module along with the facial angles output from the angle classification model. This allows the module to identify facial expression types from multiple perspectives based on the input image and its angle information. i =f θ (x i ,A(x i ))

[0051] 5) Based on the few-shot learning method, the network parameters of the expression classification model are adjusted using the outputs of the trained angle classification model and expression classification model to obtain the optimal expression classification model.

[0052] Specifically, the trained model is deployed to a real-world scenario, a small sample multi-view facial expression dataset is collected for that scenario, and the model is adjusted and trained based on the few-shot learning method to adapt the model to the facial expressions in that scenario and achieve better robustness.

[0053] In the few-shot learning method, the feature vectors output by the angle classification model and expression classification model trained on the input image are truncated and input into the cosine distance metric and the Euclidean distance metric, respectively, to calculate the distance index. The distance index is then multiplied by a scale factor, summed, and passed through a softmax layer to obtain the final output.

[0054]

[0055] Among them, cos<a,b> and euc<a,b> Let f represent the cosine distance and Euclidean distance between a and b, respectively. θ (x) represents the feature vector output by the facial expression classification model, where x is the input image and w k These are the parameters of the fully connected layer in the distance metric, where α and β are scale factors.

[0056] The distance index calculated by the cosine distance metric is:

[0057]

[0058] The distance index calculated by the Euclidean distance meter is:

[0059]

[0060] Among them, s i,j w is the distance index calculated between i and j. j f represents the parameters of the fully connected layer in the distance metric. θ (x i x is the feature vector output by the image after processing by the facial expression classification model. i The input image.

[0061] Both the angle classification model and the expression classification model use the cross-entropy loss function to calculate the loss between the model output and the true value during training. Then, the Adam optimizer is used to optimize the parameters in the model and update the weights until the optimal model is obtained.

[0062] Table 1 below illustrates the model training process of this invention: First, the pose classification module is trained; then, its network weights are fixed, and the multi-view feature extraction module continues to be trained. The image is first input into the pose classification module, and then the pose information output from this module, along with the image, is input into the multi-view feature extraction module. In this case, pose information is input after block 2 in the expression classification network. That is, after passing through blocks 1 and 2, the image is combined with the feature-extracted pose information, and then the remaining parts of the network continue to be trained.

[0063] Table 1 Training process of the model proposed in this invention

[0064]

[0065] 6) Recognize multi-view facial expressions based on angle classification model and optimal expression classification model.

[0066] Specifically, such as Figure 3 As shown, facial expression images are captured by a camera, facial detection and segmentation are performed, and the images are preprocessed into grayscale images with a pixel value of 48×48. The images are then input into the optimal recognition model obtained above for multi-view facial expression recognition to obtain the expression recognition results.

[0067] This embodiment presents an experimental study on the recognition accuracy of facial expressions in five poses using the model and training method proposed above on the KDEF dataset.

[0068] The KDEF dataset, proposed by Lundqvist et al., is a multi-view facial expression dataset containing 4900 images from 70 balanced individuals (35 men and 35 women). It includes seven basic facial expressions: anger (AN), disgust (DI), fear (FE), happiness (HA), sadness (SA), surprise (SU), and neutral (NE). Facial images are categorized into five head poses (-90°, -45°, 0°, 45°, and 90°).

[0069] This embodiment trains four models for comparison: the baseline model, the baseline model (FL) that learns few-shot learning on KDEF, the baseline model (HT) trained on a mixed dataset, and the baseline model (HTFL) that simultaneously trains on a mixed dataset and performs few-shot learning.

[0070] As shown in Table 2, the proposed method (baseline + HTFL) achieves the best results across all five poses. Adding few-shot learning to the baseline model improves accuracy across all five viewpoints. This is because few-shot learning allows the model to learn the distribution on the test set, leading to better results. Furthermore, using a hybrid training method on the baseline model also improves accuracy, particularly on the lateral (90-degree) viewpoint, where the improvement is approximately 20%, matching the accuracy achieved with few-shot learning. This demonstrates that while the hybrid training method cannot obtain the distribution on the test set like few-shot learning, introducing a new dataset enhances the model's pose robustness, bringing its performance to the level of few-shot learning. When using both few-shot learning and hybrid training, it can be seen that the recognition accuracy on frontal faces is not much different from that using only few-shot learning (78.14% and 81.39%), but there is a significant improvement on non-frontal face images, with an improvement of about 3.4% on semi-frontal (45 degrees) and about 13.8% on side (90 degrees). This also proves the high robustness of the algorithm proposed in this invention in multi-view expression recognition.

[0071] Table 2 Comparison of Model Recognition Results

[0072]

[0073] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multi-view facial expression recognition method based on digital humans, characterized in that, Includes the following steps: Obtain a dataset of real-world facial expressions; Digital virtual faces with various expressions from multiple angles are generated using the MetaHuman tool to construct a digital human expression dataset; the specific method for constructing the digital human expression dataset is as follows: Acquire a dataset of facial expressions collected from the real world, input the real facial expression dataset into the facial expression feature extraction encoder, and encode the facial feature group of the current expression based on the different actions of its facial expression units; Use the MetaHuman tool to generate digital humans with different characteristics; Using facial expression action units, the expression action units on the digital human face are controlled according to the facial feature group to correspond one-to-one with the expression action units in the recognized real-world dataset, thus reproducing the facial expressions of the real world on the digital human face. Based on the 3D characteristics of the digital human face, the digital human face is rotated to obtain expression images from different angles. The acquired images are labeled with expression and angle information to generate a digital human expression dataset. An angle classification model was constructed to classify the angles of the input image, and the angle classification model was trained based on a digital human facial expression dataset; An expression classification model is constructed to classify input images into different expressions. The network weights of the angle classification model are fixed. The expression classification model is trained based on the digital human expression dataset and the real-world expression dataset, combined with the classification results of the angle classification model. Based on the few-shot learning method, the network parameters of the expression classification model are adjusted using the outputs of the trained angle classification model and expression classification model to obtain the optimal expression classification model. In this few-shot learning method, the feature vectors output by the trained angle classification model and expression classification model of the input image are truncated and input into a cosine distance metric and a Euclidean distance metric respectively to calculate the distance index. The distance index is then multiplied by a scale factor, summed, and passed through a softmax layer to obtain the final output. Among them, cos<a,b> and euc<a,b> Let a and b represent the cosine distance and Euclidean distance, respectively. This represents the feature vector output by the facial expression classification model. x For the input image, These are the parameters of the fully connected layer in the distance metric. α , β Scale factor; The system recognizes multi-view facial expressions based on angle classification and optimal expression classification models.

2. The multi-view facial expression recognition method based on digital humans according to claim 1, characterized in that, The generated digital virtual face includes digital virtual face data from seven angles: 0˚, 15˚, 30˚, 45˚, 60˚, 75˚, and 90˚.

3. The multi-view facial expression recognition method based on digital humans according to claim 1, characterized in that, The generated digital virtual face includes seven expressions: angry, disgusted, fearful, happy, neutral, sad, and surprised.

4. The multi-view facial expression recognition method based on digital humans according to claim 1, characterized in that, The angle classification model consists of six convolutional layers, three pooling layers, and one fully connected layer. The input image is processed through three convolutional layers consisting of two convolutional layers and one fully connected layer. After a block consisting of three convolutional layers and one fully connected layer, it is flattened to obtain an encoded feature vector of length 864. This vector is then input into the fully connected layer, activated by the softmax function, and outputs the angle classification result.

5. The multi-view facial expression recognition method based on digital humans according to claim 4, characterized in that, The angle classification results include front, half-front, and side.

6. The multi-view facial expression recognition method based on digital humans according to claim 1, characterized in that, The facial expression classification model consists of five blocks. The input image first passes through a layer 1 of each block. One convolutional layer, one average pooling layer, and one 1-layer convolutional layer. 1 convolutional layer, 1 layer 1 convolutional layer and 3 layers 3 convolutional layers, one layer of 1 1 convolutional layer and two 3 layers Three convolutional layers are used, and the four outputs are then combined into a tensor of size (48, 48, 384). The input is a tensor containing two convolutional layers. The block with 3 convolutional layers and 1 global pooling layer merges its output with the reshaped tensor of size (24,24,1) of the angle classification model into a tensor of size (24,24,129). This merged output is then passed through three blocks containing three convolutional layers. The block consists of a 3-layer convolutional layer and a global pooling layer. Finally, the output tensor is flattened to obtain a feature vector of length 4608. The feature vector is then input into a fully connected layer, activated by the softmax function, and the expression classification result is output.

7. The multi-view facial expression recognition method based on digital humans according to claim 1, characterized in that, The distance index calculated by the cosine distance metric is: The distance index calculated by the Euclidean distance meter is: in, For the calculated i , j Distance indicators between These are the parameters of the fully connected layer in the distance metric. The feature vector output by the image after processing by the facial expression classification model. The input image.

8. The multi-view facial expression recognition method based on digital humans according to claim 1, characterized in that, During training, both the angle classification model and the expression classification model use the cross-entropy loss function to calculate the loss between the model output and the true value, and then use the Adam optimizer to optimize the parameters in the model and update the weights until the optimal model is obtained.

Images