A face parsing and emotion recognition method based on multi-task collaborative network

Abstract

The present invention discloses a face parsing and emotion recognition method based on a multi-task collaborative network. The steps of the present invention are as follows: 1. Preprocessing of experimental data; 2. Constructing an MPENet network model; Step 2.1 using ResNet18 as the backbone network of the encoder to extract the semantic information of the input image; Step 2.2 constructing an edge perception branch, and adding a detail perception module DPM and a feature fusion module FFM in the edge perception branch; Step 2.3 constructing a segmentation branch for outputting the result of face parsing and supervising face parsing; Step 2.4 constructing a classification branch for face emotion recognition; 3. Using inter-task consistency learning loss function and intra-task loss function to train MPENet; 4. Using the trained MPENet network model to conduct experiments, and verifying the model effect on the CelebAMask_HQ dataset. The present invention integrates face parsing and face emotion recognition in one network, has high real-time performance and accuracy, and can be deployed on mobile devices and the like.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a face parsing and emotion recognition method based on a multi-task collaborative network. Background Art

[0002] Face parsing is a fine-grained semantic segmentation task, often used in photo editing and beautification. Facial emotion recognition is a classification task with applications in fields such as human-computer interaction and mental health assessment. This method develops a new multi-task collaborative network that can simultaneously perform face parsing and emotion recognition. Compared to other methods, this method significantly improves inference speed and accuracy, and can be deployed on mobile devices such as mobile phones. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this paper proposes a face parsing and emotion recognition method based on a multi-task collaborative network. This method achieves face parsing and emotion recognition. The paper proposes a deep learning model called MPENet, which has the following specific steps:

[0004] Step 1: Preprocessing of experimental data;

[0005] Step 2: Build the MPENet network model;

[0006] Step 3: Train the MPENet network model;

[0007] Step 4: Use the trained MPENet network model to conduct experiments on multiple face parsing datasets and evaluate the experimental results.

[0008] The step 1 specifically includes the following steps:

[0009] Step 1.1: To improve the generalization ability of the model, the image is first normalized.

[0010] Step 1.2: Crop the normalized image to a size of 512×512.

[0011] Step 1.3 performs data augmentation on the cropped image by random rotation and random scaling.

[0012] Step 1.4 divides the set into training, validation and test sets.

[0013] The step 2 comprises the following steps:

[0014] Step 2.1 uses ResNet18 as the backbone network of the encoder to extract the semantic information of the input image;

[0015] Step 2.2 builds the edge perception branch and adds the detail perception module (DPM) and feature fusion module (FFM) to the edge perception branch.

[0016] The second layer features of ResNet18 first pass through the detail perception module DPM, and its output is fused with the third layer features of ResNet18 that have been upsampled by 2 times through the feature fusion module FFM to obtain fused features I;

[0017] Furthermore, the fusion feature I is passed through the detail perception module DPM again, and its output is fused with the fourth layer feature of ResNet18 after 4 times upsampling through the feature fusion module FFM to obtain the fusion feature II; finally, the fusion feature II is passed through the detail perception module DPM and 4 times upsampling again, and then sent to two Detai Heads respectively to obtain the face binary classification boundary map. and face multi-classification boundary map The Detail Head consists of a 3×3 convolution layer, a batch norm layer, a ReLU activation function, and a 1×1 convolution.

[0018] The main structure of the detail perception module DPM is as follows:

[0019] For the input feature X, it first passes through a global maximum pooling layer and two 1×1 convolution layers to obtain spatial attention. Figure I The input feature X is processed through a global average pooling layer and two 1×1 convolution layers to obtain spatial attention. Figure II . Spatial Attention Figure I With spatial attention Figure II Add them together and then pass them through the softmax function to get the final spatial attention map. Then multiply the spatial attention map with the input feature X to get the output feature y.

[0020] Furthermore, the output feature y is used as the new input feature and is passed through a global maximum pooling layer to obtain channel attention. Figure I , the input features are passed through a global average pooling layer to obtain channel attention Figure II . Figure II With channel attention Figure II The final channel attention map is obtained by adding them together, and then passing through a 1×1 convolution layer and a softmax function. The channel attention map is then multiplied by the input features to obtain the features that finally pass through the detail perception module.

[0021] The main structure of the feature fusion module is as follows:

[0022] Input features Z1 and Z2. First, the two features are concatenated, and then passed through a global average pooling layer, a 1×1 convolution layer, and a softmax function to obtain a branch attention map. The branch attention map is expanded according to the channel dimension and multiplied with Z1 and Z2 according to the set dimension index to obtain the final fused output feature Z. For example: if the dimension of the branch attention map is 512, then the weight values ​​corresponding to channels 0-255 after expansion are multiplied with Z1, and the weight values ​​corresponding to channels 256-511 are multiplied with Z2.

[0023] Step 2.3 builds a segmentation branch to output the results of face parsing and supervise face parsing.

[0024] For the fifth layer features of the encoder ResNet18, a five-layer decoder with the same structure is designed. Each decoder consists of a 3×3 convolution layer and upsampling. After the five layers of the decoder with the same structure, the input features are restored to the original resolution, that is, the supervised face parsing result is obtained.

[0025] The fifth-layer features of the ResNet18 encoder are first upsampled 8x to obtain feature Y2. Feature Y1 from the edge-perceiving branch, which passes through the final detail-perceiving module (DPM), and the 8x upsampled feature Y2 are then fed into the dual-graph adaptive learning module (DGALM). The features from the DGALM module are then passed through a Segment Head (Seg Head) to obtain the final face parsing result. The Seg Head consists of a 3x3 convolution, a batch norm layer, a ReLU activation function, and a 1x1 convolution.

[0026] The main structure of the dual-graph adaptive learning module is as follows: first, the feature Y1 and the feature Y2 are concatenated, and the concatenated features are subjected to two 1×1 convolutions to obtain the semantic feature map Z semantic and detail feature map Z detail In the boundary perception branch, the binary face boundary is obtained. Then Scale to 1 / 4 of the original size, and then use the two-class face boundary to convert Z semantic and Z detail Distinguishing between boundary pixels and non-boundary pixels, the specific formula is as follows:

[0027]

[0028] [Z detail_edge ,Z detail_noneedge ]=Z detail ⊙[Mask,A-Mask]

[0029] [Z semantic_edge ,Z semantic_noneedge]=Z semantic ⌒[Mask,A-Mask]

[0030] Among them, ⌒ represents the matrix dot product, Z detail_noneedge is the detail feature map excluding boundary pixels, Z detail_edge It is a feature map containing boundary pixel details. semantic_noneedge is the semantic feature map excluding boundary pixels, Z semantic_edge Is a semantic feature map containing boundary pixels. A is a matrix containing only elements 1. argmax dim＝2 Indicates the index of the maximum value along the second dimension of the feature.

[0031] Furthermore, the results of face supervision parsing are obtained in the segmentation branch Then Scale it to 1 / 4 of its original size and select Topk elements as the components of the face, representing the vertices of the graph.

[0032]

[0033]

[0034] Among them, Z graph_semantic is the semantic component of the face, Z graph_detail It is the component of facial details, Z semantic_noneedge is a semantic feature map that does not contain boundaries, Z detail_edge is the detail feature map containing the boundary, and C is the number of channels of the feature.

[0035] Furthermore, after a layer of graph convolution, graph reasoning is performed, and the message passing of the graph neural network is used to establish long-range interaction between the pixels of different facial components, and the obtained and

[0036] Furthermore, the mapping matrices P1 and P2 are constructed to map the features into the original geometric space. The specific implementation is as follows:

[0037]

[0038]

[0039] Furthermore, the transpose of the mapping matrix is ​​multiplied by the features after graph reasoning, and the features are mapped back to the original geometric space. The final feature output result is X out .

[0040]

[0041]

[0042]

[0043] in, Represents the semantic feature map mapped back to the original geometric space; Represents the detail feature map mapped back to the original geometric space; Represents a feature concatenation operation.

[0044] Step 2.5 builds a classification branch for facial emotion recognition.

[0045] For the last layer (fifth layer) of the encoder ResNet18 output, the feature S = [s1, s2, ..., s C ],in Will s i It is regarded as an image block input to the transformer layer, and then sent to the transformer layer. Finally, the output features pass through a layer of MLP to obtain the result of facial emotion recognition.

[0046] Described step 3 comprises the following steps:

[0047] Step 3.1: Construct the intra-task loss function.

[0048] First, the loss function of the segmentation branch mainly includes the loss of supervised face parsing and the loss of output face parsing. The cross entropy loss function is used as follows:

[0049]

[0050] Furthermore, to construct the loss function of the boundary perception branch, we use the cross entropy loss function, as follows:

[0051]

[0052] Furthermore, to construct the loss function for facial emotion recognition, we use the cross entropy loss function, as follows:

[0053]

[0054] Furthermore, the total intra-task loss function is:

[0055]

[0056] Step 3.2: Construct an inter-task consistency loss function.

[0057] Keep Channel 0 remains unchanged, and the calculated Seg 2-joint-3Represents the binary face boundary.

[0058] Then the dice coefficient is used to calculate the task consistency loss function between the binary boundary task and the multi-classification boundary task.

[0059]

[0060] Furthermore, the consistency loss function between the binary classification boundary task, the multi-classification boundary task and the face parsing task is calculated. In the second dimension direction, we get the index of the maximum value and get Then use the boundary location algorithm, for The pixel points at the boundary are assigned a value of 1, and the other non-boundary pixel points are assigned a value of 0. and Multiplying them, we can get

[0061] Then calculate

[0062] Then, the DICE coefficient is used to calculate the task consistency loss function between the parsing task and the binary classification boundary task, and the consistency loss function between the parsing task and the multi-classification boundary task.

[0063]

[0064]

[0065] Furthermore, the overall inter-task consistency loss function is .

[0066]

[0067] The step 4 specifically includes the following steps:

[0068] Step 4.1: Introduce the F1 coefficient to evaluate the performance of face parsing and emotion recognition, which is defined as follows:

[0069]

[0070] Compared with the prior art, the beneficial results of the present invention are:

[0071] This paper achieves face parsing and facial emotion recognition by building a deep learning model based on MPENet. The addition of a boundary-aware branch further refines the face parsing results. Furthermore, a dual-image adaptive learning module is added to establish dependencies between different facial components. MPENet achieves an FPS of 92.9 on an RTX 3090 GPU, with only 11.63M model parameters, demonstrating high real-time performance and enabling deployment on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is the network structure diagram of MPENet.

[0073] Figure 2 This is an example of the comparison between MPENet and other models.

[0074] Figure 3 This is the example effect of MPENet ablation experiment. DETAILED DESCRIPTION

[0075] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0076] In order to solve the problems encountered in face parsing and facial expression recognition, the present invention designs a new multi-task collaborative learning network for face parsing and facial emotion recognition. Specifically, MPENet consists of a shared encoder and three downstream branches (classification branch, segmentation branch and edge perception branch). In the classification branch, we designed a transformer module to convert the features extracted by the shared encoder into embedding-level features for facial expression recognition. In the edge perception branch, we use multi-classification face boundaries and binary classification boundaries to extract face boundary information to help the face parsing task better locate face boundaries. In the segmentation branch, we use a dual-graph adaptive learning module to fuse the edge information and semantic information of the image to infer the relationship between different feature regions and capture more contextual relationships. At the same time, we design an additional decoder as the supervised output of face parsing, so as to obtain a more refined parsing graph. Finally, we designed a consistency learning loss function between tasks to make the tasks cooperate with each other and improve the overall accuracy of the model.

[0077] Example 1 Preprocessing of experimental data.

[0078] (1) Normalize the data.

[0079] (2) Crop the image to a size of 512×512.

[0080] (3) Perform data augmentation on the cropped image, performing random rotation and random scaling.

[0081] (4) Divide the dataset into training set, validation set and test set.

[0082] Example 2 constructs an MPENet network model.

[0083] (1) ResNet18 is used as the backbone network of the encoder to extract semantic information

[0084] (2) Constructing the boundary perception branch. The second layer features of ResNet are first processed by DPM, and then processed by FFM together with the third layer features that have been upsampled by 2 times. Furthermore, the fused features are processed by DPM again, and then processed by FFM together with the fourth layer features that have been upsampled by 4 times of ResNet18. Finally, the fused features are processed by DPM and four times upsampled again, and then sent to two detail heads respectively to obtain the face binary classification boundary map and the face multi-classification boundary map.

[0085] (3) For the fifth layer features of the encoder ResNet18, we designed a five-layer decoder structure. Each decoder consists of a 3×3 convolution layer and upsampling. After the five-layer decoder structure, the input features are restored to the original resolution, and the supervised face parsing results can be obtained.

[0086] (4) For the fifth layer features of the decoder ResNet18, we first perform 8x upsampling, and then send the feature X after the last DPM in the edge perception branch and the feature Y after 8x upsampling into DGALM. The features after the DGALM module are then passed through a Seghead to obtain the final face parsing result.

[0087] (5) The last layer of features output by the encoder ResNet18 passes through a layer of transformer and then a layer of MLP to obtain the final facial emotion classification result.

[0088] Example 3: Training the DA-Net network model.

[0089] (1) SGD optimization is used as the optimization method.

[0090] (2) The ResNet18 network weights of the MPENet encoder use the weights pre-trained on the ImageNet dataset.

[0091] Example 4 uses the trained MPENet network model to conduct experiments on the public face dataset CelebAMask_HQ and evaluates the experimental results.

[0092] (1) Table 1 below compares the performance of MPENet with current mainstream semantic segmentation frameworks on the CelebAMask_HQ dataset. Our model achieves an average F1 coefficient of 85.9%, and the Mean F1 of facial emotion segmentation reaches 80.04%. For a detailed comparison of the performance of MPENet and other methods, see Table 1.

[0093] Table 1. Comparison of results of MPENet and other models

[0094]

[0095]

[0096] (2) Table 2 below shows the ablation experiment of MPENet on the CelebAMask_HQ dataset. It can be seen that each module of MPENet can improve the model accuracy.

[0097] Table 2. Ablation experiments of MPENet

[0098]

[0099]

[0100] (3) Table 2 below shows the performance comparison of MPENet with other models. It can be seen that MPENet is at the leading level in both inference speed and parsing accuracy, with an FPS of 92.9 and a model parameter count of only 11.6.

[0101] Table 3. Performance comparison of MPENet and other models

[0102]

Claims

1. A face parsing and emotion recognition method based on a multi-task collaborative network, characterized by The steps include: Step 1: Preprocessing of experimental data; the experimental data is face image data; Step 2: Build the MPENet network model; Step 2.1 uses ResNet18 as the backbone network of the encoder to extract the semantic information of the input image; Step 2.2 builds the edge perception branch and adds the detail perception module DPM and feature fusion module FFM to the edge perception branch; Step 2.3: Build a segmentation branch to output the results of face parsing and supervise face parsing. For the fifth layer features of the encoder ResNet18, a five-layer decoder with the same structure is designed. Each decoder consists of a 3×3 convolution layer and upsampling. After the five layers of the decoder with the same structure, the input features are restored to the original resolution, that is, the supervised face parsing result is obtained. The fifth layer features of the encoder ResNet18 are first upsampled by a factor of 8 to obtain feature Y2. Then, the feature Y1 in the edge perception branch that has passed through the last detail perception module DPM and the 8-fold upsampled feature Y2 are sent together to the dual-image adaptive learning module DGALM. The features passed through DGALM are then passed through a Seg Head to obtain the final face parsing result. The structure of the dual-graph adaptive learning module is as follows: ① Concatenate feature Y1 and feature Y2, and the concatenated features undergo two 1×1 convolutions to obtain the semantic feature map Z semantic and detail feature map Z detail ; In the boundary perception branch, obtain the binary classification face boundary Then Scale to 1 / 4 of the original size, and then use the two-class face boundary to convert Z semantic and Z detail Distinguishing between boundary pixels and non-boundary pixels, the specific formula is as follows: [WITH detail_edge ,WITH detail_noneedge ]=Z detail ⊙[Mask,A-Mask] [WITH semantic_edge ,WITH semantic_noneedge ]=Z semantic ⊙[Mask,A-Mask] Among them, ⊙ represents the matrix dot product, Z detail_noneedge is the detail feature map excluding boundary pixels, Z detail_edge It is a feature map containing boundary pixel details; Z semantic_noneedge is the semantic feature map excluding boundary pixels, Z semantic_edge Is a semantic feature map containing boundary pixels; A is a matrix containing only element 1; arg max dim＝2 Indicates the index of the maximum value along the second dimension of the feature; ② Obtain the results of face supervision analysis in the segmentation branch Then Scale it to 1 / 4 of its original size and select Topk elements as the components of the face, representing the vertices of the graph; Among them, Z graph_semantic is the semantic component of the face, Z graph_detail It is the component of facial details, Z semantic_noneedge is a semantic feature map that does not contain boundaries, Z detail_edge is the detail feature map including the boundary, C is the number of channels of the feature; ③After a layer of graph convolution, we get and ④ Construct mapping matrices P1 and P2 to map the features to the original geometric space. The specific implementation is as follows: ⑤ Multiply the transposed matrix of the mapping matrix with the features after graph reasoning, and map the features back to the original geometric space. The final feature output result is X out ; in, Represents the semantic feature map mapped back to the original geometric space; Represents the detail feature map mapped back to the original geometric space; Represents feature concatenation operation; Step 2.4 builds a classification branch for facial emotion recognition; Step 3: Train the MPENet network model; Including building the task loss function, the specific implementation is as follows: 3-1-1. The loss function of the segmentation branch mainly includes the loss of supervised face parsing Seg True And the loss Seg of output face analysis Pre , using the cross entropy loss function, as follows: 3-1-2. Construct the loss function of the boundary perception branch and use the cross entropy loss function as follows: 3-1-3. Construct the loss function for facial emotion recognition and use the cross entropy loss function as follows: 3-1-4. The total intra-task loss function is: Among them, λ0, λ1 and λ2 are proportional coefficients; It also includes constructing an inter-task consistency loss function, which is implemented as follows: 3-2-1. First maintain Channel 0 remains unchanged, and the calculated Seg 2-joint-3 Represents the binary face boundary; 3-2-2. Use the dice coefficient to calculate the task consistency loss function between the binary boundary task and the multi-class boundary task; 3-2-3. Calculate the consistency loss function between the binary classification boundary task, the multi-classification boundary task, and the face parsing task; (1) First, follow In the second dimension direction, we get the index of the maximum value and get (2) Then use the boundary positioning algorithm, for The pixel points at the boundary are assigned a value of 1, and the other non-boundary pixel points are assigned a value of 0; and Multiplying them, we can get (3) Calculate (4) The dice coefficient is used to calculate the task consistency loss function between the parsing task and the binary classification boundary task, and the consistency loss function between the parsing task and the multi-classification boundary task, respectively. Specifically: (5) The overall inter-task consistency loss function is: Step 4: Use the trained MPENet network model to conduct experiments on multiple face parsing datasets and evaluate the experimental results.

2. The face parsing and emotion recognition method based on a multi-task collaborative network according to claim 1 is characterized in that Step 2.2 is implemented as follows: The second layer features of ResNet18 first pass through the detail perception module DPM, and its output is fused with the third layer features of ResNet18 that have been upsampled by 2 times through the feature fusion module FFM to obtain fused features I; The fusion feature I is passed through the detail perception module DPM again, and its output is fused with the fourth layer feature of ResNet18 after 4 times upsampling through the feature fusion module FFM to obtain the fusion feature II; finally After the fusion feature II is again passed through the detail perception module DPM and 4 times upsampling, it is sent to two detail heads respectively to obtain the face binary classification boundary map and face multi-classification boundary map 3. A face parsing and emotion recognition method based on a multi-task collaborative network according to claim 1 or 2, characterized in that The structure of the detail perception module DPM is as follows: For the input feature X, it first passes through a global maximum pooling layer and two 1×1 convolution layers to obtain the spatial attention map I; the input feature X passes through a global average pooling layer and two 1×1 convolution layers to obtain the spatial attention map II; the spatial attention map I and the spatial attention map II are added together, and then passed through the softmax function to obtain the final spatial attention map; the spatial attention map is then multiplied by the input feature X to obtain the output feature y; The output feature y is used as the new input feature, and a global maximum pooling layer is passed through it to obtain the channel attention map I. The input feature is passed through a global average pooling layer to obtain the channel attention map II. The channel attention map II is added to the channel attention map II, and then a 1×1 convolution layer and a softmax function are passed through to obtain the final channel attention map. The channel attention map is then multiplied by the input feature to obtain the final feature after the detail perception module.

4. A face parsing and emotion recognition method based on a multi-task collaborative network according to claim 1 or 2, characterized by feature fusion The module is structured as follows: For the input features Z1 and Z2, the two features are first concatenated, and then passed through a global average pooling layer, a 1×1 convolution layer and a softmax function to obtain a branch attention map; the branch attention map is expanded according to the channel dimension and multiplied with Z1 and Z2 according to the set dimension index to obtain the final fused output feature Z.

5. The face parsing and emotion recognition method based on a multi-task collaborative network according to claim 1 is characterized in that Step 2.5 is implemented as follows: For the last layer feature S=[s1,s2,...,s C ],in Will s i It is regarded as an image block input to the transformer layer, and then sent to the transformer layer. Finally, the output features pass through a layer of MLP to obtain the result of facial emotion recognition.

Images