An expression recognition method under head pose influence based on semi-supervised learning

Abstract

The application discloses an expression recognition method under head posture influence based on semi-supervised learning, comprising the following steps: 1, re-dividing a face data set according to head posture yaw angle size; 2, performing different pretreatments on the face image data set according to front face and side face; 3, constructing a DC-SSL double-consistency-constrained semi-supervised expression recognition network, performing supervised learning on the pretreated front face image, performing unsupervised learning on the pretreated side face image, and performing constraint by using spatial consistency. The application significantly improves the test accuracy of the model in each angle interval based on only labeled front face samples and unlabeled side face samples, and effectively alleviates the problem of difficult recognition of side face expressions caused by difficult data labeling under head posture change.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and image processing, specifically a facial expression recognition method based on semi-supervised learning under the influence of head pose. Background Technology

[0002] Facial expressions are a crucial way for humans to convey emotional information. Therefore, facial expression recognition is widely used in fields such as human-computer interaction, autism detection, and safe driving. Facial expressions are generally categorized into seven basic types: neutral, happy, sad, surprised, fearful, disgusted, and angry. In practical applications, facial expression recognition in unconstrained natural scenes is easily affected by head posture. This presents a significant challenge and is currently the primary focus of facial expression recognition research.

[0003] Existing deep learning models primarily use mappings between frontal and non-frontal faces to obtain a unified description of pose robustness. However, these models struggle to adapt to non-frontal face situations, especially under extreme poses. They may lack robustness to head diversity. Additionally, coupled-scaled Gaussian process regression models using 2D geometric head pose normalization have been developed to improve expression recognition. These models emphasize the dependency between frontal and non-frontal faces. While effectively extracting head pose information, they suffer from drawbacks such as large network size, lengthy training time, and difficulty in capturing sensitivity to noise and variations. Summary of the Invention

[0004] The present invention addresses the shortcomings of the existing technology by proposing a semi-supervised learning-based facial expression recognition method under the influence of head posture. This method can fully utilize the feature information of the front and side faces, thereby solving the problem of data annotation difficulties under changes in head posture and improving the model's recognition accuracy for side face data.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0006] The present invention provides a method for facial expression recognition based on semi-supervised learning under the influence of head pose, characterized by the following steps:

[0007] Step 1: Construct a dataset based on head pose angle classification;

[0008] Step 1.1: Use the EPNP algorithm to map the coordinates of two-dimensional pixels in M ​​face images to three-dimensional space to obtain the coordinates of three-dimensional pixels in each face image. Then, based on the coordinates of three-dimensional pixels in each face image, obtain the head pose yaw angle corresponding to each face image.

[0009] Step 1.2: Set K angle intervals T = {t1, t2, ..., t...} k …t K}, where t kLet K represent the k-th angle interval, and K represent the total number of angle intervals.

[0010] Each face image is assigned to a corresponding angle interval based on its head pose yaw angle, thus obtaining a head pose face image dataset S = {S1, S2, ..., S...} within each angle interval. k …S K}, where S k Represents the k-th angle interval t k The dataset of head pose facial images within the dataset, and S k,m Represents the k-th angle interval t k The m-th head pose face image in the image, M k This represents the number of head pose face images within the k-th angle interval;

[0011] Step 1.3: Split the head pose face image dataset S into a frontal face image dataset. And K-1 profile datasets S-S1, where, Let M1 represent the m1-th face image within the first angular interval t1, where M1 represents the maximum number of face images with the largest head pose within the first angular interval; let the k′-th profile dataset be denoted as... Among them, S k′,m′ M represents the m-th face image within the k′-th angle interval. k′ This represents the number of head pose face images within the k′-th angular interval; k′∈[1,K-1];

[0012] Step 1.4: Process the frontal face dataset S1 and the k′-th profile face dataset S1 respectively. k′ Each face image undergoes random erasure to obtain a processed frontal weak augmentation dataset. and the k′-th profile weak augmentation dataset in, and These represent the m1th weakly enhanced frontal face image and the m′th weakly enhanced profile face image within the k′ angle interval, respectively.

[0013] For X respectively wa and Perform a horizontal flip to obtain the flipped frontal weakly enhanced face dataset. and the k′-th profile weakly enhanced face dataset after flipping in, and This represents the m1th flipped frontal weakly enhanced image and the m′th flipped profile weakly enhanced image within the k′ angle interval;

[0014] For Sk′ Each face image in the dataset undergoes a strong enhancement operation to obtain the k′-th enhanced profile face dataset. in, This represents the m-th enhanced image within the k′-th angle interval;

[0015] Step 2: Construct a DC-SSL double consistency constraint semi-supervised facial expression recognition network, which consists of a backbone network, a Softmax classifier, a class activation mapping attention module, and a semi-supervised module;

[0016] Step 2.1: The backbone network is a ResNet50 network, which includes, in sequence, a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, a fifth convolutional block, and a sixth fully connected block;

[0017] The first convolutional block has an m1×m2 kernel, the second convolutional block consists of one residual block, the third convolutional block consists of three residual blocks, the fourth convolutional block consists of four residual blocks, the fifth convolutional block consists of six residual blocks, and the sixth fully connected block contains global average pooling and a fully connected layer; each residual block contains two 1×1 convolutional kernels and one 3×3 convolutional kernel.

[0018] Step 2.1, The data are input into the backbone network for processing, and after passing through five convolutional blocks, the m1-th frontal weak enhancement feature map is output. Image m1 of the flipped frontal face with weak enhancement feature map The m′th profile weak enhancement feature map and the m′-th flipped profile weak enhancement feature map The m′th profile image with enhanced features

[0019] Step 2.2: The Softmax classifier respectively... Perform expression category determination to obtain Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Thus, X is obtained. wa The corresponding set of predicted probability distributions for facial expression categories X′ wa The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories

[0020]

[0021] Step 2.3, the class activation mapping attention module will... After backpropagation in the Softmax classifier, the weight of the m1th frontal face is obtained. and the weight of the m1th flipped front face and the weight of the m′th side face The m′-th flip side profile weight

[0022] The activation mapping attention module uses equation (1) to obtain the m1-th frontal face activation mapping. The m1st flipped frontal face activation map The m′-th side face activation map The m′-th flipped side profile activation map

[0023]

[0024] In equation (1), T represents transpose;

[0025] Step 2.4: The semi-supervised module uses equation (2) to generate the m'th weakly enhanced profile image within the k'th angle interval. Available pseudo tags

[0026]

[0027] In equation (2), max(·) represents taking the maximum predicted probability of the sample, and θ represents the threshold.

[0028] Step 3: Train the DC-SSL double consistency constraint semi-supervised facial expression recognition network based on the overall loss function. When the number of training iterations reaches the set number or the overall loss function L... total When convergence occurs, training stops, thus obtaining the optimal facial expression feature extraction network, which is used to identify the category of facial expression images.

[0029] The facial expression recognition method based on semi-supervised learning under the influence of head pose, as described in this invention, is characterized in that the overall loss function in step 3 is constructed according to the following process;

[0030] Step 3.1: Use equation (3) to establish the supervised classification network loss L cls :

[0031]

[0032] In equation (3), It is a probability distribution P wa The maximum probability distribution in yes The corresponding tags;

[0033] Step 3.2: Use equation (4) to establish a supervised face-flipping attention consistency loss L. cam and loss of attentional consistency when turning one's face

[0034]

[0035] In equation (4), ||·||2 represents the calculation of the second norm;

[0036] Step 3.3: Use equation (5) to establish the unsupervised semantic consistency loss.

[0037]

[0038] Step 3.4: Use equation (6) to establish the overall loss function L total :

[0039]

[0040] In equation (6), lam is the spatial consistency constraint weight parameter, and u is the semantic consistency constraint weight parameter.

[0041] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the expression recognition method, and the processor is configured to execute the program stored in the memory.

[0042] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the facial expression recognition method.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] 1. This invention reorganizes the AffectNet dataset from two dimensions: facial expression category and head yaw angle, and proposes the AffectNet-Yaw dataset, which fills the gap of lacking a dataset that can reflect the robustness of the model in the yaw angle.

[0045] 2. This invention proposes a joint learning method, which reduces the problem of low quality of unlabeled side face data caused by head posture rotation during model training, and fully expands the value of limited labeled side face data.

[0046] 3. This invention uses semantic consistency constraints on the side face dataset to correct the erroneous annotation problem caused by the annotator's perception of the same expression from different angles, thereby filtering out noise-free side face images and solving the problem of difficult annotation of side face data.

[0047] 4. This invention proposes a semi-supervised expression recognition method. It uses a flipped feature attention mechanism in the semi-supervised network to establish a class activation mapping difference loss between the same type of expression and its flipped image in the same model. By reducing the loss, the model's ability to learn and represent expression categories is improved, thereby improving the model's expression recognition accuracy under various head poses. Attached Figure Description

[0048] Figure 1 This is a framework diagram of the semi-supervised learning-based facial expression recognition method under the influence of head pose proposed in this invention.

[0049] Figure 2 This is a structural diagram illustrating the spatial and semantic consistency of the present invention. Detailed Implementation

[0050] In this embodiment, a facial expression recognition method based on semi-supervised learning under the influence of head pose is described, such as... Figure 1 As shown, the procedure is as follows:

[0051] Step 1: Construct a dataset based on head pose angle classification;

[0052] Step 1.1: Using the EPNP algorithm, the coordinates of the two-dimensional pixels in the M face images are mapped to three-dimensional space to obtain the coordinates of the three-dimensional pixels in each face image. Based on the coordinates of the three-dimensional pixels in each face image, the head pose yaw angle corresponding to each face image is obtained. In this example, all face datasets in the AffectNet dataset are selected for head pose estimation, and the data is reorganized according to the size of their yaw angles. That is, face images with yaw angles are assigned to their corresponding angle intervals. Then, the frontal face images are aligned to increase the feature difference between the frontal and side faces. Finally, all data are uniformly cropped to a size of 224×224.

[0053] Step 1.2: Set K angle intervals T = {t1, t2, ..., t...} k …t K}, where t k Let K represent the k-th angle interval, and K represent the total number of angle intervals; in the specific embodiment, K is 5.

[0054] Each face image is assigned to a corresponding angle interval based on its head pose yaw angle, thus obtaining a head pose face image dataset S = {S1, S2, ..., S...} within each angle interval. k …S K}, where S k Represents the k-th angle interval t k The dataset of head pose facial images within the dataset, and S k,m Represents the k-th angle interval t k The m-th head pose face image in the image, M k This represents the number of head pose face images within the k-th angle interval.

[0055] Step 1.3: Split the head pose face image dataset S into a frontal face image dataset. And K-1 profile datasets S-S1, where, Let M1 represent the m1-th face image within the first angular interval t1, where M1 represents the maximum number of face images with the largest head pose within the first angular interval; let the k′-th profile dataset be denoted as... Among them, S k′,m′ M represents the m-th face image within the k′-th angle interval. k′ The number of head pose face images in the k′-th angle interval is represented; k′∈[1,K-1]; S1 corresponds to the frontal face in the embodiment, and S-S1 corresponds to the side face in the embodiment.

[0056] Step 1.4: Process the frontal face dataset S1 and the k′-th profile face dataset S1 respectively. k′ Each face image undergoes random erasure to obtain a processed frontal weak augmentation dataset. and the k′-th profile weak augmentation dataset in, and Let m1 represent the frontal weakly enhanced image and m′ side weakly enhanced image in the k′ angle interval, respectively.

[0057] For X respectively wa and Perform a horizontal flip to obtain the flipped frontal weakly enhanced face dataset. and the k′-th profile weakly enhanced face dataset after flipping in, and This represents the m1th flipped frontal weakly enhanced image and the m′th flipped profile weakly enhanced image within the k′ angle interval;

[0058] For S k′Each face image in the dataset undergoes a strong enhancement operation to obtain the k′-th enhanced profile face dataset. in, This represents the m′ strongly enhanced image within the k′ angle interval; weak, strong enhancement, and flip operations correspond to the preprocessing in the embodiment. Strong enhancement is sharpening, color inversion, and random large-area cropping, which cause a lot of disturbance to the image.

[0059] Step 2: Construct a DC-SSL double consistency constraint semi-supervised facial expression recognition network, which consists of a backbone network, a Softmax classifier, a class activation mapping attention module, and a semi-supervised module;

[0060] Step 2.1: The backbone network is a ResNet50 network, which includes, in sequence, the first convolutional block, the second convolutional block, the third convolutional block, the fourth convolutional block, the fifth convolutional block, and the sixth fully connected block;

[0061] The first convolutional block has an m1×m2 kernel, the second convolutional block consists of one residual block, the third convolutional block consists of three residual blocks, the fourth convolutional block consists of four residual blocks, the fifth convolutional block consists of six residual blocks, and the sixth fully connected block contains global average pooling and a fully connected layer. Each residual block contains two 1×1 convolutional kernels and one 3×3 convolutional kernel.

[0062] Step 2.1, The data are input into the backbone network for processing, and after passing through five convolutional blocks, the m1-th weakly enhanced frontal face feature map is output. Image m1 of the flipped frontal face with weak enhancement feature map The m′th profile weak enhancement feature map and the m′-th flipped profile weak enhancement feature map The m′th profile image with enhanced features In this embodiment, the preprocessed front and side face images are processed through four convolutional layers. The feature maps obtained from each layer are downsampled through a max-pooling layer, and the final model output is a feature map of size 1×2048×7×7.

[0063] Step 2.2, the Softmax classifier is used for... Perform expression category determination to obtain Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Thus, X is obtained. waThe corresponding set of predicted probability distributions for facial expression categories X′ wa The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories In this embodiment, the model output feature map is input into the global average pooling layer AavgPool for processing, and finally the activation function softmax gives the predicted class probability value.

[0064] Step 2.3: The class activation mapping attention module will... After backpropagation in the Softmax classifier, the weights of the m1-th frontal face are obtained. and the weight of the m1th flipped front face and the weight of the m′th side face The m′-th flip side profile weight

[0065] The class activation mapping attention module uses equation (1) to obtain the m1-th frontal face activation mapping. The m1st flipped frontal face activation map The m′-th side face activation map The m′-th flipped side profile activation map In this embodiment, this is achieved by performing element-wise multiplication of the global average pooling output x with the fully connected output weights fc_weights, resulting in a reshaped fully connected layer with a shape of 1×7×2048×1×1. Finally, for each category, a summation operation on its second-dimensional channels yields a single class activation map with a shape of 1×7×7×7, serving as the model's spatial consistency output. The data represent the batch size, number of categories, map height, and width, respectively, and this output represents the model's region of interest for that category on the input image.

[0066]

[0067] In equation (1), T represents transpose.

[0068] Step 2.4: The semi-supervised module uses equation (2) to generate the m-th weakly enhanced profile image within the k′-th angle interval. Available pseudo tags In this embodiment, pseudo-labels are generated through joint learning, which reduces the risk that the classification and selection errors of side face samples during training will gradually spread as the model trains and eventually have a negative impact on the model performance, while also obtaining high-quality and effective data for semi-supervised learning.

[0069]

[0070] In equation (2), max(·) represents taking the maximum predicted probability of the sample, and θ represents the threshold.

[0071] Step 3: Construct the loss function and iteratively train the network;

[0072] Step 3.1: Use equation (3) to establish the supervised classification network loss L cls :

[0073]

[0074] In equation (3), It is a probability distribution P wa The maximum probability distribution in yes The corresponding tag.

[0075] Step 3.2: Use equation (4) to establish a supervised face-flipping attention consistency loss L. cam and loss of attentional consistency when turning one's face Spatial consistency structure such as Figure 2 As shown, images of the same category and their flipped images have theoretically consistent feature maps under the same model, and this is used to establish a loss to measure the deviation of the map in practice.

[0076]

[0077] In equation (4), ||·||2 represents the calculation of the second norm.

[0078] Step 3.3: Use equation (5) to establish the unsupervised semantic consistency loss. Semantic consistency structure such as Figure 2 As shown in the example, unlabeled samples with high confidence are selected and pseudo-labels are added to them to augment the labeled data and enhance the semi-supervised learning process.

[0079]

[0080] Step 3.4: Use equation (6) to establish the overall loss function L total :

[0081]

[0082] In equation (6), lam is the spatial consistency constraint weight parameter, and u is the semantic consistency constraint weight parameter.

[0083] Step 3.5: Based on the overall loss function L total The DC-SSL double consistency constraint semi-supervised facial expression recognition network is trained until the number of training iterations reaches a set number or the overall loss function L. total Upon convergence, training stops, resulting in the optimal facial expression feature extraction network used for classifying facial expression images. In this embodiment, the Adam optimizer is used, with the spatial hyperparameter lam set to 5, the semantic hyperparameter u set to 1, and the initial learning rate set to 1e-4. The learning rate decays to 0.9 times its original value every 30 iterations, and the training iterations are repeated 500 times to obtain the trained model.

[0084] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0085] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0086] To verify the effectiveness of the method of the present invention, two methods that have achieved state-of-the-art (SOTA) results on AffectNet were selected for comparison and verification. Table 1 shows the comparison of the expression recognition accuracy of the two methods at various angles.

[0087] Table 1 Comparison with other algorithms

[0088]

[0089] This invention primarily utilizes spatial consistency and semantic consistency constraints to jointly learn and optimize a model from labeled frontal face data and unlabeled side-face data. This method makes full use of the limited labeled side-face data and helps alleviate the problem of easily obtainable frontal face data but difficult-to-label side-face data, thereby improving the model's expression recognition accuracy under various head poses.

Claims

1. A facial expression recognition method based on semi-supervised learning under the influence of head pose, characterized in that, Includes the following steps: Step 1: Construct a dataset based on head pose angle classification; Step 1.1: Use the EPNP algorithm to map the coordinates of two-dimensional pixels in M ​​face images to three-dimensional space to obtain the coordinates of three-dimensional pixels in each face image. Then, based on the coordinates of three-dimensional pixels in each face image, obtain the head pose yaw angle corresponding to each face image. Step 1.2: Set K angle intervals T = {t1, t2, ..., t...} k ···t K }, where t k Let K represent the k-th angle interval, and K represent the total number of angle intervals. Each face image is assigned to a corresponding angle interval based on its head pose yaw angle, thus obtaining a head pose face image dataset S = {S1, S2, ..., S...} within the angle interval. k ···S K }, where S k Represents the k-th angle interval t k The dataset of head pose facial images within the dataset, and S k,m Represents the k-th angle interval t k The m-th head pose face image in the image, M k This represents the number of head pose face images within the k-th angle interval; Step 1.3: Split the head pose face image dataset S into a frontal face image dataset. And K-1 profile datasets S-S1, where, Let M1 represent the m1-th face image within the first angular interval t1, where M1 represents the maximum number of face images with the largest head pose within the first angular interval; let the k′-th profile dataset be denoted as... Among them, S k′,m′ M represents the m-th face image within the k′-th angle interval. k′ This represents the number of head pose face images within the k′-th angular interval; k′∈[1,K-1]; Step 1.4: Process the frontal face dataset S1 and the k′-th profile face dataset S1 respectively. k′ Each face image undergoes random erasure to obtain a processed frontal weak augmentation dataset. and the k′-th profile weak augmentation dataset in, and These represent the m1th weakly enhanced frontal face image and the m′th weakly enhanced profile face image within the k′ angle interval, respectively. For X respectively wa and Perform a horizontal flip to obtain the flipped frontal weakly enhanced face dataset. and the k′-th profile weakly enhanced face dataset after flipping in, and This represents the m1th flipped frontal weakly enhanced image and the m′th flipped profile weakly enhanced image within the k′ angle interval; For S k′ Each face image in the dataset undergoes a strong enhancement operation to obtain the k′-th enhanced profile face dataset. in, This represents the m-th enhanced image within the k′-th angle interval; Step 2: Construct a DC-SSL double consistency constraint semi-supervised facial expression recognition network, which consists of a backbone network, a Softmax classifier, a class activation mapping attention module, and a semi-supervised module; Step 2.1: The backbone network is a ResNet50 network, which includes, in sequence, a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, a fifth convolutional block, and a sixth fully connected block; The first convolutional block has an m1×m2 kernel, the second convolutional block consists of one residual block, the third convolutional block consists of three residual blocks, the fourth convolutional block consists of four residual blocks, the fifth convolutional block consists of six residual blocks, and the sixth fully connected block contains global average pooling and a fully connected layer; each residual block contains two 1×1 convolutional kernels and one 3×3 convolutional kernel. Step 2.1, The data are input into the backbone network for processing, and after passing through five convolutional blocks, the m1-th frontal weak enhancement feature map is output. Image m1, flipped frontal face with weak enhancement feature map The m′th profile weak enhancement feature map and the m′-th flipped profile weak enhancement feature map The m′th profile image with enhanced features Step 2.2: The Softmax classifier respectively... Perform expression category determination to obtain Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Corresponding expression category prediction probability Thus, X is obtained. wa The corresponding set of predicted probability distributions for facial expression categories X′ wa The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories The corresponding set of predicted probability distributions for facial expression categories Step 2.3, the class activation mapping attention module will... After backpropagation in the Softmax classifier, the weight of the m1th frontal face is obtained. and the weight of the m1th flipped front face and the weight of the m′th side face The m′-th flip side profile weight The activation mapping attention module uses equation (1) to obtain the m1-th frontal face activation mapping. The m1st flipped face activation map The m′-th side face activation map The m′-th flipped side profile activation map In equation (1), T represents transpose; Step 2.4: The semi-supervised module uses equation (2) to generate the m'th weakly enhanced profile image within the k'th angle interval. Available pseudo tags In equation (2), max(·) represents taking the maximum predicted probability of the sample, and θ represents the threshold. Step 3: Train the DC-SSL double consistency constraint semi-supervised facial expression recognition network based on the overall loss function. When the number of training iterations reaches the set number or the overall loss function L... total When convergence occurs, training stops, thus obtaining the optimal facial expression feature extraction network, which is used to identify the category of facial expression images.

2. The facial expression recognition method based on semi-supervised learning under the influence of head pose according to claim 1, characterized in that, The overall loss function in step 3 is constructed according to the following process; Step 3.1: Use equation (3) to establish the supervised classification network loss L cls : In equation (3), It is a probability distribution P wa The maximum probability distribution in yes The corresponding tags; Step 3.2: Use equation (4) to establish a supervised face-flipping attention consistency loss L. cam and loss of attentional consistency when turning one's face In equation (4), ||·||2 represents the calculation of the second norm; Step 3.3: Use equation (5) to establish the unsupervised semantic consistency loss. Step 3.4: Use equation (6) to establish the overall loss function L total : In equation (6), lam is the spatial consistency constraint weight parameter, and u is the semantic consistency constraint weight parameter.

3. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the facial expression recognition method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the facial expression recognition method according to claim 1 or 2.

Images