An expression recognition method and system based on uncertainty learning

Abstract

The application discloses an expression recognition method and system based on uncertainty learning, comprising the following steps: S1, obtaining a face image to be recognized; S2, preprocessing the input face image to obtain a standardized face image; S3, initializing a deep learning training condition; S4, training an expression recognition model by using a plurality of modules, including a face feature extraction module, a multi-dimensional feature classification module, a latent label distribution learning module and an attention consistency module, and updating model weight parameters; and S5, obtaining an output expression category of the face image through the expression recognition model. The expression recognition method and system based on uncertainty learning solve the uncertainty problem of face expressions in a wild environment by combining a latent label distribution and an attention consistency method, improve the accuracy of expression recognition by mining the latent label distribution, and enhance the interpretability of the model by using a generated attention map method.

Description

Technical Field

[0001] This invention belongs to the field of image processing and pattern recognition, specifically relating to an expression recognition method and system based on uncertainty learning. Background Technology

[0002] Facial expressions are one of the most important ways humans express emotions, reflecting our psychological state and personality traits. In recent years, facial expression recognition has attracted widespread attention and become a research hotspot in academia. Furthermore, facial expression recognition has shown great potential in industrial fields and has been widely applied in areas such as surveillance security and healthcare. Due to the high similarity between different facial expressions, and the strong subjectivity of manually labeled expression images, the uncertainty of facial expressions has become a key challenge in recent years. We propose a facial expression recognition method based on uncertainty learning, combining latent label distribution learning and attentional consistency, to address the uncertainty problem in facial expression labeling. Summary of the Invention

[0003] This invention proposes an expression recognition method and system based on uncertainty learning to solve the problems mentioned in the background section. The technical solution steps of this invention are as follows:

[0004] Step 1: Use a camera and image acquisition software to acquire the user's face image, and preprocess the input face image using a face detector and key point detector to obtain a preprocessed standardized face image;

[0005] Step 1.1: Use OpenCV, a face detector, to detect faces in the original image to be identified. If a face region is detected, crop a rectangular face region as the face image to be identified.

[0006] Step 1.2: Use the face landmark detector Dlib to detect face landmarks in the face image to be recognized, perform face alignment based on the face landmarks, and then obtain a standardized face image through image normalization.

[0007] Step 2: Initialize deep learning training conditions, obtain and load the pre-trained weight parameters of the deep neural network model, set deep learning training parameters, and preprocess the large-scale facial expression dataset used for deep learning training.

[0008] Step 2.1: Use the MS-Celeb-1M face recognition dataset to pre-train the deep neural network model, obtain the model weight parameters of the deep neural network model, use the pre-trained weight parameters as initial values ​​to initialize the deep neural network model, and set the initial learning rate and parameter optimization algorithm for deep learning.

[0009] The deep neural network model is a ResNet18 residual neural network model; the initial learning rate is 1e-4; and the parameter optimization algorithm is the Adam algorithm.

[0010] Step 2.2: Train the deep neural network model using publicly available facial expression datasets collected in the field environment. For the face image samples in the dataset, re-divide them into training set and validation set. Standardized face expression images obtained through image preprocessing are used as face image samples to be identified.

[0011] The image preprocessing methods include image enhancement methods such as grayscale transformation, random horizontal flipping, and adding Gaussian noise;

[0012] Step 3: Train the facial expression recognition model using deep learning methods, and update the weight parameters of the facial expression recognition model through multiple modules, including a face feature extraction module, a multi-dimensional feature classification module, a latent label distribution learning module, and an attention consistency module;

[0013] Step 3.1: Face feature extraction module. The face image samples to be identified are processed by the face feature extraction network of the main branch and multiple auxiliary branches of the expression recognition model to extract face features from the face image samples.

[0014] The face feature extraction network is a ResNet18 residual neural network, and the main branch and auxiliary branch share the network weight parameters;

[0015] Step 3.2: Multidimensional feature classification module: The extracted facial features are processed through the main branch and multiple auxiliary branches of the expression recognition model to obtain the predicted probability value and attention map of each category in each branch. The predicted expression categories of the main branch and auxiliary branches are adjusted, and the weight parameters of the deep neural network model are updated.

[0016] The main branch of the facial expression recognition model includes a multi-class classifier; the multi-class classifier can obtain the probability prediction value of the face image to be recognized for each class;

[0017] The auxiliary branches of the expression recognition model include a binary classifier, and the number of auxiliary branches equals the number of expression categories. The binary classifier can obtain the predicted probability values ​​of the face image to be recognized for the target category and other categories. The target category is the expression category classified by the auxiliary branch where the binary classifier is located, and the target category of each auxiliary branch is different. The attention map is a visual heatmap generated by weighted fusion of the output probability value of each expression category of the expression recognition model with the face image to be recognized, which can reflect the region of attention of the model on the face image for any predicted expression category.

[0018] Step 3.2.1: Calculate the cross-entropy loss value of the main branch and adjust the weight parameters of the main branch of the deep neural network model:

[0019]

[0020] Where N is the number of facial expression image samples, y i and p i These represent the label value and the model prediction value of the i-th image sample, respectively;

[0021] Step 3.2.2: Calculate the cross-entropy loss values ​​of multiple auxiliary branches and adjust the weight parameters of the auxiliary branches in the deep neural network model:

[0022]

[0023] Where C is the number of facial expression image samples;

[0024] Step 3.3: The latent distribution learning module obtains the predicted probability value of each auxiliary branch for its respective target category, concatenates multiple predicted probability values, smooths the probability distribution to obtain the latent label distribution, and then uses the latent label distribution as the real label to guide the learning of the main branch and update the weight parameters of the deep neural network model.

[0025] Step 3.3.1: Concatenate the probability predictions of multiple auxiliary branches for the target category:

[0026]

[0027] in These are the probability predictions of multiple auxiliary branches for their respective target categories after softmax;

[0028] Step 3.3.2: Use the probability distribution after smoothing the auxiliary branches as the latent label distribution:

[0029]

[0030] Where T is a hyperparameter, and the probability distribution becomes smoother when T > 1;

[0031] Step 3.3.3: Calculate the JS divergence loss values ​​for the main branch prediction distribution and the latent label distribution, and adjust the weight parameters of the main branch of the deep neural network model:

[0032]

[0033] in The predicted probability distribution after Softmax for different prediction categories in the main branch;

[0034] Step 3.4: Attention Consistency Module. Obtain the attention map of each auxiliary branch for its respective target category, and the attention map of the main branch for all categories. Concatenate the attention maps of multiple auxiliary branches to maintain the attention between the main branch and the auxiliary branches. Figure 1 Consistency, update the weight parameters of the deep neural network model;

[0035] Step 3.4.1: Concatenate the attention graphs of multiple auxiliary branches for the target category:

[0036]

[0037] in An attention graph for multiple auxiliary branches for their respective target categories;

[0038] Step 3.4.2: Calculate the Euclidean loss values ​​of the attention maps for the main branch and auxiliary branches, and adjust the weight parameters of the deep neural network model:

[0039]

[0040] in Attention maps for different prediction categories in the main branch;

[0041] Step 4: Use the deep neural network model with updated weight parameters as the expression recognition model. Input the face image to be recognized into the expression recognition model, obtain the model's prediction values ​​for different expression categories, and output the expression category with the highest predicted probability as the expression recognition result.

[0042] An expression recognition system based on uncertainty learning, characterized in that it includes a detection module, a preprocessing module, an input module, a training module, and an output module;

[0043] Detection module: Detects face regions from the original image to be recognized using the OpenCV face detector, and outputs a cropped rectangular face image to the preprocessing module.

[0044] Preprocessing module: Obtains the coordinates of facial landmarks through the face landmark detector Dlib, preprocesses the face image to be recognized, and outputs the standardized face image after face alignment to the input module;

[0045] Input module: preprocesses facial expression images from the expression dataset, initializes the pre-trained expression recognition model, sets the training parameters for deep learning, and outputs the preprocessed dataset and the initialized expression recognition model to the training module;

[0046] Training module: Includes a cloud server, which trains an initial expression recognition model using facial expression image samples from a preprocessed facial expression dataset, and outputs the trained expression recognition model to the output module;

[0047] Output module: Based on the standardized face images in the preprocessing module, the trained expression recognition model, input from the training module, outputs the predicted face expression category.

[0048] The expression recognition method and system based on uncertainty learning of the present invention have the following beneficial effects:

[0049] The present invention proposes an expression recognition method and system based on uncertainty learning. By combining latent label distribution and attention consistency methods, it solves the problem of uncertainty in facial expressions in the wild environment. It improves the accuracy of expression recognition by mining latent label distribution, and enhances the interpretability of the model by using the generation of attention maps, thus providing a more reliable and efficient solution for the application of expression recognition technology. Attached Figure Description

[0050] Figure 1 This is a flowchart of the expression recognition method based on uncertainty learning according to the present invention;

[0051] Figure 2 This is a model framework diagram for the uncertainty-based facial expression recognition of the present invention;

[0052] Figure 3 This is a system flowchart for expression recognition based on uncertainty learning according to the present invention; Detailed Implementation

[0053] The present invention will be further described in detail below with reference to preferred embodiments. More details are set forth in the following description in order to provide a full understanding of the present invention. However, the present invention can obviously be implemented in many other ways different from those described herein. Those skilled in the art can make similar extensions and derivations based on actual application situations without departing from the spirit of the present invention. Therefore, the scope of protection of the present invention should not be limited by the content of this specific embodiment.

[0054] An expression recognition method based on uncertainty learning, such as Figure 1 As shown, it includes the following steps:

[0055] S1. Obtain the user's facial image using the camera and the image acquisition software built;

[0056] S2: Use a face detector and a keypoint detector to preprocess the input face image and obtain a preprocessed standardized face image;

[0057] S2.1: Use OpenCV, a face detector, to detect faces in the original image to be identified. If a face region is detected, crop a rectangular face region as the face image to be identified.

[0058] S2.2: Use the face landmark detector Dlib to detect face landmarks in the face image to be recognized, perform face alignment based on the face landmarks, and then obtain a standardized face image through image normalization.

[0059] S3: Initialize deep learning training conditions, obtain and load the pre-trained weight parameters of the deep neural network model, set deep learning training parameters, and preprocess the large-scale facial expression dataset for deep learning training.

[0060] S3.1: Use the MS-Celeb-1M face recognition dataset to pre-train the deep neural network model, obtain the model weight parameters of the deep neural network model, use the pre-trained weight parameters as initial values ​​to initialize the deep neural network model, and set the initial learning rate and parameter optimization algorithm for deep learning.

[0061] The deep neural network model is a ResNet18 residual neural network model; the initial learning rate is 1e-4; and the parameter optimization algorithm is the Adam algorithm.

[0062] S3.2: The deep neural network model is trained using publicly available facial expression datasets collected in the field environment. For the facial image samples in the dataset, they are re-divided into training set and validation set. Standardized facial expression images obtained through image preprocessing are used as facial image samples to be identified.

[0063] The image preprocessing methods include image enhancement methods such as grayscale transformation, random horizontal flipping, and adding Gaussian noise;

[0064] S4: Train the facial expression recognition model using deep learning methods, and update the weight parameters of the facial expression recognition model through multiple modules, including a face feature extraction module, a multi-dimensional feature classification module, a latent label distribution learning module, and an attention consistency module;

[0065] S4.1: Face feature extraction module. The face image samples to be identified are processed by the face feature extraction network of the main branch and multiple auxiliary branches of the expression recognition model to extract face features from the face image samples.

[0066] The face feature extraction network is a ResNet18 residual neural network, and the main branch and auxiliary branch share the network weight parameters;

[0067] S4.2: Multidimensional feature classification module. The extracted facial features are processed through the main branch and multiple auxiliary branches of the expression recognition model to obtain the predicted probability value and attention map of each category in each branch. The predicted expression categories of the main branch and auxiliary branches are adjusted, and the weight parameters of the deep neural network model are updated.

[0068] The main branch of the facial expression recognition model includes a multi-class classifier; the multi-class classifier can obtain the probability prediction value of the face image to be recognized for each class;

[0069] The auxiliary branches of the facial expression recognition model include a binary classifier, and the number of auxiliary branches equals the number of facial expression categories. The binary classifier can obtain the predicted probability values ​​of the face image to be recognized for the target category and other categories. The target category is the facial expression category classified by the auxiliary branch where the binary classifier is located, and the target category of each auxiliary branch is different. The attention map is a visual heatmap generated by weighted fusion of the output probability values ​​of the facial expression recognition model for each facial expression category and the face image to be recognized. It can reflect the region on the face image that the model focuses on for any predicted facial expression category.

[0070] S4.2.1: Calculate the cross-entropy loss value of the main branch and adjust the weight parameters of the main branch of the deep neural network model:

[0071]

[0072] Where N is the number of facial expression image samples, y i and p i These represent the label value and the model prediction value of the i-th image sample, respectively;

[0073] S4.2.2: Calculate the cross-entropy loss values ​​of multiple auxiliary branches and adjust the weight parameters of the auxiliary branches in the deep neural network model:

[0074]

[0075] Where C is the number of facial expression image samples;

[0076] S4.3: Latent distribution learning module, obtains the predicted probability value of each auxiliary branch for its respective target category, concatenates multiple predicted probability values, smooths the probability distribution and uses it as the latent label distribution, and then uses the latent label distribution as the real label to guide the learning of the main branch and update the weight parameters of the deep neural network model;

[0077] S4.3.1: Concatenate the probability predictions of multiple auxiliary branches for the target category:

[0078]

[0079] in These are the probability predictions of multiple auxiliary branches for their respective target categories after softmax;

[0080] S4.3.2: The probability distribution after smoothing the auxiliary branches and concatenating them is used as the latent label distribution:

[0081]

[0082] Where T is a hyperparameter, and the probability distribution becomes smoother when T > 1;

[0083] S4.3.3: Calculate the JS divergence loss values ​​for the main branch prediction distribution and the latent label distribution, and adjust the weight parameters of the main branch of the deep neural network model:

[0084]

[0085] in The predicted probability distribution after Softmax for different prediction categories in the main branch;

[0086] S4.4: Attention Consistency Module. This module obtains the attention map of each auxiliary branch for its respective target category, as well as the attention map of the main branch for all categories. It then concatenates the attention maps of multiple auxiliary branches to maintain the attention between the main branch and the auxiliary branches. Figure 1 Consistency, update the weight parameters of the deep neural network model;

[0087] S4.4.1: Concatenate the attention graphs of multiple auxiliary branches for the target category:

[0088]

[0089] in An attention graph for multiple auxiliary branches for their respective target categories;

[0090] S4.4.2: Calculate the Euclidean loss values ​​of the attention maps for the main branch and auxiliary branches, and adjust the weight parameters of the deep neural network model:

[0091]

[0092] in Attention maps for different prediction categories in the main branch;

[0093] S5: Use the deep neural network model with updated weight parameters as the expression recognition model. Input the face image to be recognized into the expression recognition model, obtain the model's prediction values ​​for different expression categories, and output the expression category with the highest predicted probability as the expression recognition result.

[0094] An expression recognition system based on uncertainty learning, which, as Figure 3 As shown, it includes the following steps:

[0095] E1. Obtain the user's facial image using a camera and image acquisition software.

[0096] E2. The detection module uses the OpenCV face detector to detect face regions from the original image to be identified and outputs the cropped rectangular face image to the preprocessing module.

[0097] E3. The preprocessing module obtains the coordinates of facial key points through the facial key point detector Dlib, preprocesses the face image to be recognized, and outputs the standardized face image after face alignment to the input module.

[0098] E4. The input module preprocesses the facial expression images in the expression dataset, initializes the pre-trained expression recognition model, sets the training parameters for deep learning, and outputs the preprocessed dataset and the initialized expression recognition model to the training module.

[0099] E5. The training module includes a cloud server, which trains the initial expression recognition model using facial expression image samples from the preprocessed facial expression dataset, and outputs the trained expression recognition model to the output module.

[0100] E6. The output module predicts the facial expression category based on the standardized face image in the preprocessing module and the trained expression recognition model input from the training module.

[0101] E7. Output the predicted probability value of each expression based on the expression recognition model, and output the expression category with the highest predicted probability.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to further limit the present invention. All equivalent changes made based on the description and drawings of the present invention are within the protection scope of the present invention.

Claims

1. An expression recognition method based on uncertainty learning, characterized in that, The specific steps are as follows: Step 1: Use a camera and image acquisition software to acquire the user's face image, and preprocess the input face image using a face detector and key point detector to obtain a preprocessed standardized face image; Step 2: Initialize deep learning training conditions, obtain and load the pre-trained weight parameters of the deep neural network model, set deep learning training parameters, and preprocess the large-scale facial expression dataset used for deep learning training. Step 3: Train the facial expression recognition model using deep learning methods, updating the model's weight parameters through multiple modules, specifically including a face feature extraction module, a multi-dimensional feature classification module, a latent label distribution learning module, and an attention consistency module; the sub-steps of the facial expression recognition model training process in Step 3 are as follows: Step 3.1: Face feature extraction module. The face image samples to be identified are processed by the face feature extraction network of the main branch and multiple auxiliary branches of the expression recognition model to extract face features from the face image samples. The face feature extraction network is a ResNet18 residual neural network, and the main branch and auxiliary branch share the network weight parameters; Step 3.2: Multidimensional feature classification module: The extracted facial features are processed through the main branch and multiple auxiliary branches of the expression recognition model to obtain the predicted probability value and attention map of each category in each branch. The predicted expression categories of the main branch and auxiliary branches are adjusted, and the weight parameters of the deep neural network model are updated. The main branch of the facial expression recognition model includes a multi-class classifier; the multi-class classifier can obtain the probability prediction value of the face image to be recognized for each class; The auxiliary branch of the facial expression recognition model includes a binary classifier, and the number of auxiliary branches is equal to the number of facial expression categories; the binary classifier can obtain the predicted probability values ​​of the face image to be recognized for the target category and other categories; The target category is the expression category classified by the auxiliary branch of the binary classifier, and the target category of each auxiliary branch is different. The attention map is a visual heatmap generated by weighted fusion of the output probability value of each expression category of the expression recognition model with the face image to be recognized. It can reflect the area on the face image that the model pays attention to for any predicted expression category. Step 3.3: The latent distribution learning module obtains the predicted probability value of each auxiliary branch for its respective target category, concatenates multiple predicted probability values, smooths the probability distribution to obtain the latent label distribution, and then uses the latent label distribution as the real label to guide the learning of the main branch and update the weight parameters of the deep neural network model. Step 3.4: Attention Consistency Module: Obtain the attention map of each auxiliary branch for its respective target category, and the attention map of the main branch for all categories, concatenate the attention maps of multiple auxiliary branches, maintain the consistency of the attention maps of the main branch and auxiliary branches, and update the weight parameters of the deep neural network model; Step 4: Use the deep neural network model with updated weight parameters as the expression recognition model. Input the face image to be recognized into the expression recognition model, obtain the model's prediction values ​​for different expression categories, and output the expression category with the highest predicted probability as the expression recognition result.

2. The expression recognition method based on uncertainty learning according to claim 1, characterized in that, The sub-steps for preprocessing the face image in step 1 are as follows: Step 1.1: Use OpenCV, a face detector, to detect faces in the original image to be identified. If a face region is detected, crop a rectangular face region as the face image to be identified. Step 1.2: Use the face landmark detector Dlib to detect face landmarks in the face image to be identified, perform face alignment based on the face landmarks, and then obtain a standardized face image through image normalization.

3. The expression recognition method based on uncertainty learning according to claim 1, characterized in that, The sub-step for initializing the deep learning training conditions in step 2 is as follows: Step 2.1: Use the MS-Celeb-1M face recognition dataset to pre-train the deep neural network model, obtain the model weight parameters of the deep neural network model, use the pre-trained weight parameters as initial values ​​to initialize the deep neural network model, and set the initial learning rate and parameter optimization algorithm for deep learning. The deep neural network model is a ResNet18 residual neural network model; the initial learning rate is 1e-4; and the parameter optimization algorithm is the Adam algorithm. Step 2.2: Train the deep neural network model using publicly available facial expression datasets collected in the field environment. For the face image samples in the dataset, re-divide them into training set and validation set. Standardized face expression images obtained through image preprocessing are used as face image samples to be identified. The image preprocessing methods include image enhancement techniques such as grayscale transformation, random horizontal flipping, and adding Gaussian noise.

4. The expression recognition method based on uncertainty learning according to claim 3, characterized in that, The sub-step in step 3.2 that adjusts the model parameters based on the loss value is as follows: Step 3.2.1: Calculate the cross-entropy loss value of the main branch and adjust the weight parameters of the main branch of the deep neural network model: Where N is the number of facial expression image samples, y i and p i These represent the label value and the model prediction value of the i-th image sample, respectively; Step 3.2.2: Calculate the cross-entropy loss values ​​of multiple auxiliary branches and adjust the weight parameters of the auxiliary branches in the deep neural network model: Where c is the number of facial expression image samples.

5. The expression recognition method based on uncertainty learning according to claim 4, characterized in that, The sub-step in step 3.3 that adjusts the model parameters based on the loss value is as follows: Step 3.3.1: Concatenate the probability predictions of multiple auxiliary branches for the target category: in These are the probability predictions of multiple auxiliary branches for their respective target categories after softmax; Step 3.3.2: Use the probability distribution after smoothing the auxiliary branches as the latent label distribution: Where T is a hyperparameter, and the probability distribution becomes smoother when T>1; Step 3.3.3: Calculate the JS divergence loss values ​​for the main branch prediction distribution and the latent label distribution, and adjust the weight parameters of the main branch of the deep neural network model: in The predicted probability distribution after Softmax for different prediction categories in the main branch.

6. The expression recognition method based on uncertainty learning according to claim 4, characterized in that, The sub-step in step 3.4 that adjusts the model parameters based on the loss value is as follows: Step 3.4.1: Concatenate the attention graphs of multiple auxiliary branches for the target category: in An attention graph for multiple auxiliary branches for their respective target categories; Step 3.4.2: Calculate the Euclidean loss values ​​of the attention maps for the main branch and auxiliary branches, and adjust the weight parameters of the deep neural network model: in Attention maps for different prediction categories in the main branch.

7. An expression recognition system based on uncertainty learning, implementing the expression recognition method based on uncertainty learning as described in any one of claims 1-6, characterized in that, It includes a detection module, a preprocessing module, an input module, a training module, and an output module; Detection module: Detects face regions from the original image to be identified using the OpenCV face detector, and outputs a cropped rectangular face image to the preprocessing module; Preprocessing module: Obtains the coordinates of facial landmarks through the face landmark detector Dlib, preprocesses the face image to be recognized, and outputs the standardized face image after face alignment to the input module; Input module: preprocesses facial expression images from the expression dataset, initializes the pre-trained expression recognition model, sets the training parameters for deep learning, and outputs the preprocessed dataset and the initialized expression recognition model to the training module; Training module: Includes a cloud server, which trains an initial expression recognition model using facial expression image samples from a preprocessed facial expression dataset, and outputs the trained expression recognition model to the output module; Output module: Based on the standardized face images in the preprocessing module, the trained expression recognition model, input from the training module, outputs the predicted face expression category.

Images