Emotion recognition method based on facial feature and key point fusion network

Abstract

The invention discloses an emotion recognition method based on a facial feature and key point fusion network, and relates to the technical field of facial recognition. Comprising the steps of constructing an emotion recognition model based on a facial feature and key point fusion network LFFNet, obtaining a to-be-recognized face image, inputting the to-be-recognized face image into the emotion recognition model, extracting facial features through a processing layer, and extracting facial key points through a key point extraction module MobileFaceNet; fusing the facial features and each facial key point through a feature fusion module to obtain similarity mapping; splicing and integrating all similarity mappings through a multi-scale fusion module to obtain deep features; and performing full-connection processing on the deep features through a full-connection layer FC to obtain an emotion recognition result. According to the method, the obtained fusion information is aggregated again while the face key points and the face features are subjected to multi-scale fusion, and valuable information of the face key points and the face features is reserved and utilized to the maximum extent.

Description

Technical Field

[0001] The present invention relates to the field of facial recognition technology, and in particular to an emotion recognition method based on a facial feature and key point fusion network. Background Art

[0002] The application prospects of emotion recognition technology are extremely broad. First, in the field of education, it can analyze students' emotional reactions, help teachers adjust teaching strategies, and achieve personalized teaching. In the medical field, this technology can be used to assist in the diagnosis of mental illnesses such as depression and anxiety, providing doctors with important diagnostic basis. In terms of human-computer interaction, facial expression recognition can enable machines to understand human needs more intelligently and provide a more natural and smooth interactive experience. In addition, in the field of security, emotion recognition technology can be applied to identity authentication to improve the security of the system. With the continuous advancement of technology, emotion recognition will play an important role in more fields and bring more convenience to people's lives.

[0003] In the prior art, an emotion recognition method based on a convolutional neural network is proposed, which specifically includes: collecting a facial image dataset containing various emotion labels, designing a suitable convolutional neural network structure according to the data characteristics and using a training set for training; inputting the images in the facial image dataset into the trained network, and obtaining the emotion classification results through layer-by-layer calculation.

[0004] The defect of the above-mentioned prior art is that in the process of using convolutional neural networks for emotion recognition, the convolutional neural network adopts single feature information, which makes it unable to fully recognize the emotional feature information on the face image, and thus causes the emotion recognition result to be inaccurate. Summary of the invention

[0005] Based on this, it is necessary to provide an emotion recognition method based on facial features and key point fusion network to address the above technical problems.

[0006] The embodiment of the present invention provides an emotion recognition method based on a facial feature and key point fusion network, comprising:

[0007] Construct an emotion recognition model based on the facial feature and key point fusion network LFFNet; the emotion recognition model includes the backbone network Swin Transformer, the multi-scale fusion module and the fully connected layer FC; the backbone network Swin Transformer includes multiple processing layer blocks, feature fusion modules and key point extraction module MobileFaceNet;

[0008] Obtain a face image to be identified, input the face image to be identified into the emotion recognition model, extract facial features of the face image through the processing layer, and extract facial key points of the face image through the key point extraction module MobileFaceNet; fuse the facial features extracted by each processing layer and the facial key points extracted by each key point extraction module MobileFaceNet through the feature fusion module to obtain a similarity map of pixels at each corresponding position in the two facial feature maps extracted by adjacent processing layers; splice and integrate all similarity maps according to the processing order of the processing layers through the multi-scale fusion module to obtain deep features; perform full connection processing on the deep features through the fully connected layer FC to obtain the emotion recognition result.

[0009] Optionally, the backbone network Swin Transformer further includes a windowed multi-head self-attention module W_MSA and a shifted windowed multi-head self-attention module SW_MSA, wherein the windowed multi-head self-attention module W_MSA is used to calculate self-attention within each processing layer, and the shifted windowed multi-head self-attention module SW_MSA is used to make self-attention interact between different processing layers, including:

[0010]

[0011] in, is the output feature of the lth windowed multi-head self-attention module W_MSA and the shifted windowed multi-head self-attention module SW_MSA, z l is the output feature of the lth multi-layer perception module MLP, and LN is the layer normalization operation.

[0012] Optionally, the key point extraction module MobileFaceNet is a 68-point model, and the key points of the face image are extracted by the key point extraction module MobileFaceNet, including:

[0013] Detect 68 key points of facial image data including eyes, eyebrows, nose, mouth and facial contour;

[0014] A heat map is generated for each key point, and the coordinates of the key point are determined based on the peak position of each heat map.

[0015] Optionally, the facial features extracted by each processing layer and the facial key points extracted by each key point extraction module MobileFaceNet are fused through a feature fusion module, including:

[0016] The pixel attention mechanism aligns the feature map of facial feature x extracted by the backbone network Swin Transformer and the feature map of facial key point y extracted by the key point extraction module MobileFaceNet through convolution;

[0017] Perform dot multiplication on the pixels at each corresponding position of the two feature maps to obtain a similarity map that quantifies the importance of the facial key point feature at each pixel position relative to the facial feature x;

[0018] The similarity map is processed by the Sigmoid function to obtain the feature influence value between 0 and 1 for each pixel on the feature map. If the feature influence value is close to 1, the contribution of the y feature to the fusion output at the pixel point is higher; if the feature influence value is close to 0, the contribution of the x feature to the fusion output at the pixel point is higher.

[0019] Align the feature map of facial feature x with the feature map of facial key point y, and add the weighted feature influence values ​​to get the final fusion feature. The formula is:

[0020] σ=Sigmoid(f x (v x )·f y (v y ))

[0021] output=σv x +(1-σ)v y

[0022] Among them, v x is the corresponding pixel in the feature map of the facial feature, v y is the corresponding pixel in the feature map of the facial key point, f is the alignment operation, and σ is the possibility that the two pixels belong to the same object.

[0023] Optionally, the multi-scale fusion module includes a plurality of 1×1 convolutional layers and a plurality of 3×3 convolutional layers, and all similarity maps are concatenated and integrated according to the processing order of the processing layers through the multi-scale fusion module, including:

[0024] Resize different input features to the same size through 1×1 convolutional layers and upsampling operations;

[0025] The result of the first processing layer is passed to the next layer. For each subsequent processing layer, the output of the previous processing layer is added to its own input and fused through a 3×3 convolution to obtain deeper features.

[0026] The outputs of all processing layers are spliced ​​and integrated according to the processing order of the processing layers to obtain the deep features of the face image.

[0027] Optionally, the deep features are fully connected through the fully connected layer FC, including:

[0028] The deep features of the face image are flattened into a one-dimensional vector, and the one-dimensional vector is subjected to feature dimensionality reduction and feature abstraction to obtain the probability distribution of expression categories in the face image.

[0029] Compared with the prior art, the above-mentioned emotion recognition method based on facial feature and key point fusion network provided by the embodiment of the present invention has the following beneficial effects:

[0030] The present invention fuses the facial features extracted by each processing layer and the facial key points extracted by each key point extraction module MobileFaceNet through a feature fusion module, and then splices and integrates all similarity maps according to the processing order of the processing layers through a multi-scale fusion module. This can solve the problem in the prior art that convolutional neural networks cannot fully identify emotional feature information on facial images through a single feature. While multi-scale fusion of facial key points and facial features is performed, the obtained fused information is aggregated again to retain and utilize the valuable information of both to the greatest extent.

[0031] In addition, all similarity maps are spliced ​​and integrated according to the processing order of the processing layers through the multi-scale fusion module to obtain deep features and realize deep multi-scale feature fusion, which provides more powerful performance and expressiveness for facial emotion recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A model structure diagram of an emotion recognition method based on a facial feature and key point fusion network provided in one embodiment;

[0033] Figure 2 A processing flow chart of a Swin Transformer of an emotion recognition method based on a facial feature and key point fusion network provided in one embodiment;

[0034] Figure 3 A schematic diagram of a feature fusion module of an emotion recognition method based on a facial feature and key point fusion network provided in one embodiment;

[0035] Figure 4 A schematic diagram of a multi-scale fusion module of an emotion recognition method based on a facial feature and key point fusion network provided in one embodiment;

[0036] Figure 5 A recall rate result diagram of an emotion recognition method based on a facial feature and key point fusion network provided in one embodiment;

[0037] Figure 6 A confusion matrix comparison diagram of an emotion recognition method based on a facial feature and key point fusion network provided in one embodiment;

[0038] Figure 7 A comparison diagram of an emotion recognition method based on a facial feature and key point fusion network provided in an embodiment on a RAFDB data set;

[0039] Figure 8 A comparison diagram of an emotion recognition method based on a facial feature and key point fusion network provided in an embodiment on a SFEW data set;

[0040] Fig. 9 A comparison chart of recall rate results of an emotion recognition method based on a facial feature and key point fusion network provided in an embodiment; DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] In one embodiment, a method for emotion recognition based on a facial feature and key point fusion network is provided, the method comprising:

[0043] 1. Model design

[0044] 1.1 Model Structure

[0045] Construct an emotion recognition model based on facial features and key point fusion network LFFNet. The entire model framework is as follows Figure 1 As shown, it includes: backbone network Swin Transformer, multi-scale fusion module and fully connected layer FC. The backbone network Swin Transformer includes multiple processing layer blocks (block1, block2, block3 and block4), feature fusion module and key point extraction module MobileFaceNet. Four processing layers are set in MobileFaceNet, and key point extraction module MobileFaceNet is set between adjacent processing layers. The adjacent processing layers and key point extraction module MobileFaceNet are connected through feature fusion module, and the output of each processing layer is connected with multi-scale fusion module.

[0046] Obtain a face image to be identified, input the face image to be identified into the emotion recognition model, extract facial features of the face image through the processing layer, and extract facial key points of the face image through the key point extraction module MobileFaceNet; fuse the facial features extracted by each processing layer and the facial key points extracted by each key point extraction module MobileFaceNet through the feature fusion module to obtain a similarity map of pixels at each corresponding position in the two facial feature maps extracted by adjacent processing layers; splice and integrate all similarity maps according to the processing order of the processing layers through the multi-scale fusion module to obtain deep features; perform full connection processing on the deep features through the fully connected layer FC to obtain the emotion recognition result.

[0047] By sequentially transferring and concatenating the feature maps output from different levels, deeper features are obtained layer by layer, thus achieving deep multi-scale fusion. In fine-grained recognition tasks such as facial expression recognition, this hierarchical fusion structure can focus on both facial detail features and global information, enabling the model to learn more discriminative features.

[0048] 1.2 Backbone Network Swin Transformer

[0049] Facial features mainly focus on the texture and color information of the face, such as skin color, facial wrinkles, etc. These features capture the visual attributes of the face and are very useful for identifying specific facial expressions or age, gender, etc. of an individual. These features can be obtained through artificial design methods or automatically learned through deep learning models.

[0050] The Transformer network can capture the relationship between any two positions in the input sequence without being restricted by the length of the sequence, thus effectively solving the performance degradation problem encountered by traditional sequence processing models when processing long sequences. However, due to the computational complexity of its self-attention mechanism, the traditional Transformer model faces huge computational and storage overheads when processing large-size images, and due to the limitation of window size, local information may be lost, resulting in an insufficiently detailed overall understanding of the image. Therefore, the introduction of a local window mechanism allows the model to focus only on local areas in each attention operation, avoiding the complexity of global attention calculations while ensuring that the model can capture rich local detail information. Figure 2As shown in the figure, the backbone network Swin Transformer also includes a windowed multi-head self-attention module (Window-based Multi-head Self-Attention, W_MSA) and a shifted windowed multi-head self-attention module (Shifted Window Multi-head Self-Attention, SW_MSA). Among them, the self-attention is calculated inside each processing layer through the windowed multi-head self-attention module W_MSA, and the self-attention is made to interact between different processing layers through the shifted windowed multi-head self-attention module SW_MSA, which enhances the expression ability of the model. The calculation formula includes:

[0051]

[0052] in, is the output feature of the lth windowed multi-head self-attention module W_MSA and the shifted windowed multi-head self-attention module SW_MSA, z l is the output feature of the lth multi-layer perception module MLP, and LN is the layer normalization operation.

[0053] Facial expression recognition needs to accurately capture subtle changes in the face, including small movements in local areas such as the eyes, mouth corners, and eyebrows, which are key clues to judging expressions. Swin Transformer can more effectively focus on these fine-grained local features through a windowed self-attention mechanism. By using a local window mechanism, Swin Transformer allows the model to focus only on local areas in each attention operation, avoiding the complexity of global attention calculations while ensuring that the model can capture rich details. At the same time, considering the needs of facial expression recognition in practical applications, such as the need to run on resource-constrained devices such as mobile phones and embedded systems, the design of Swin Transformer significantly reduces the demand for computing resources by limiting the scope of self-attention and adopting hierarchical processing. Therefore, Swin Transformer is used as the backbone network for facial feature extraction, making full use of its advantages in capturing fine-grained features and computational efficiency to improve recognition accuracy and efficiency.

[0054] 1.3 Key Point Extraction Module MobileFaceNet

[0055] Facial landmarks refer to key points on the face, which usually correspond to the prominent features of the face, such as the contours of the eyes, nose, mouth, and the edges of the chin and cheeks. Facial landmarks provide a sparse representation of key facial areas in facial image analysis. This sparse representation is less sensitive to various intrinsic changes in the face, such as skin color, gender, age, and different background expressions, thereby effectively reducing intra-class differences.

[0056] The model used by MobileFaceNet is a 68-point model, which detects 68 key points of facial image data including eyes, eyebrows, nose, mouth and facial contours, generates a heat map for each key point, and determines the coordinates of the key point based on the peak position of each heat map.

[0057] 1.4 Feature Fusion Module

[0058] In the facial expression recognition task, facial key points and facial features each play different roles. Facial features can pay detailed attention to the key areas indicated by facial key points, while facial key points provide support for capturing the global information of facial features. The careful fusion of these two features is very important for improving the accuracy of expression recognition. To this end, a feature fusion module based on the pixel attention mechanism (PA) is used to accurately integrate effective features from different branches and fuse the valuable information in facial key points and facial features at the pixel level.

[0059] Specifically, the pixel attention mechanism first aligns the feature map of facial feature x extracted from the backbone network Swin Transformer and the feature map of facial key point y extracted from the key point extraction module MobileFaceNet through convolution, and then performs point multiplication on the pixels at each corresponding position of the two feature maps to obtain a similarity map, which quantifies the importance of feature y relative to x at each pixel position. After further mapping through the Sigmoid function, a value between 0 and 1 is obtained, where pixels close to 1 indicate that at this position, feature y contributes more critically to the fusion output; and values ​​close to 0 mean that feature x is more important at this position. Finally, the two feature maps are added with the calculated weights to obtain the final fusion feature. This mechanism not only retains the features of x, but also effectively integrates the supplementary information provided by y, thereby enhancing the expressive power of the model. The calculation method is:

[0060] σ=Sigmoid(f x (v x )·f y (v y ))

[0061] output=σv x +(1-σ)v y

[0062] Among them, v x is the corresponding pixel in the feature map of the facial feature, v y is the corresponding pixel in the feature map of the facial key point, f is the alignment operation, and σ is the possibility that the two pixels belong to the same object.

[0063] In facial expression recognition, facial key points usually focus on the key areas of expression, while facial features cover broader global information. When the similarity between facial key points and facial features at a certain position is high, it indicates that the features captured by the two at that point are highly consistent, and the features of the facial key points at that position show more significant importance. In this case, it is reasonable to rely on the information provided by the facial key points to enrich the final feature representation. On the contrary, if the similarity is low, it indicates that more facial feature information should be retained. At the same time, since facial key points are a fine-grained feature, this pixel-level fusion method can more effectively fuse facial key points and facial features than other fusion methods. Figure 3 As shown, the basic structure of the feature fusion module is demonstrated.

[0064] 1.5 Multi-scale fusion module based on residual connection

[0065] Multi-scale fusion technology can comprehensively utilize information obtained at different scales to improve recognition performance. The core idea behind this technology is that data of different scales often reveals different features of the image: small-scale data can provide rich details, while large-scale data provides more context and global vision. By effectively fusing these multi-level information, more comprehensive and accurate image analysis results can be achieved. By introducing a pooling layer that can generate fixed-size feature representations, the network can accept input images of any size. The Pyramid Pooling Module (PPM) applies pooling operations of different scales to a single feature map, and adjusts the resulting feature map to a uniform size through upsampling so that it can be fused with the original feature map.

[0066] Inspired by the above network structure, a multi-scale fusion module is proposed. The specific structure is as follows Figure 4As shown in the figure, firstly, different input features are adjusted to the same size through 1×1 convolution and upsampling operations, and then the result of the first processing layer is passed to the next layer. For each subsequent processing layer, the output of the previous processing layer is added to its own input and fused through a 3×3 convolution to obtain deeper features. Finally, the outputs of all processing layers are spliced ​​and integrated according to the processing order of the processing layers to obtain the deep features of the face image. The deep features of the face image are flattened into a one-dimensional vector, and the one-dimensional vector is subjected to feature dimensionality reduction and feature abstraction to obtain the probability distribution of expression categories in the face image.

[0067] In facial expression recognition, it is often necessary to accurately understand subtle changes in the face. The output of the low-level network focuses more on the detailed features of the face (such as edges, contours, etc.), while the output of the high-level network focuses more on global information (such as the semantics of the overall expression). By fusing these different levels of features, a more comprehensive facial expression representation can be obtained. At the same time, different expressions may have more significant differences at certain feature levels. By combining multi-level information, the model can learn more discriminative features and improve the accuracy and robustness of expression recognition.

[0068] 2. Experimental results and analysis

[0069] 2.1 Experimental environment and optimization strategy

[0070] In this experiment, we used the Ubuntu 16.04 operating system, an Intel Core i9 CPU, 32GB of RAM, and an NVIDIA RTX 3060 GPU with 12GB of video memory. The CUDA version was 10.2, and the cuDNN version was v8.4. In terms of programming languages, we used Python 3.7 and PyTorch 1.7.1.

[0071] In terms of model optimization, SAM (Sharpness-Aware Minimization) is used as the model optimization strategy. Traditional deep learning optimization methods, such as SGD or Adam, focus on minimizing the loss function on the training set. However, these methods sometimes cause the model to perform poorly on unseen data. SAM alleviates this problem by considering the sharpness of the loss, that is, the sensitivity of the loss to changes in model parameters. Specifically, SAM calculates the sharpness of the loss by adding a small perturbation to the model parameters. If a model is not very sensitive to changes in the loss in the neighborhood of its parameters, then the model is more likely to perform well on unseen data, thereby promoting the generalization ability of the model. Adam is used as the basic optimization method, the perturbation coefficient rho is set to 0.05, and the remaining hyperparameters continue to use the strategy in Chapter 3. The batchsize is set to 64, the initial learning rate is 3.5e-5, the weight decay is 1e-4, and the total number of training iterations is set to 100. The learning rate adjustment strategy uses ExponentialLR, and the gamma value is set to 0.98.

[0072] 2.2 Ablation Experiment

[0073] In order to evaluate the effectiveness of LFFNet, an ablation experiment was conducted on the RAFDB dataset. The experimental results include the selection of the backbone network, the impact of each module on the overall model, and confusion matrix analysis.

[0074] (1) Comparison of backbone network selection: The performance differences of three different backbone networks, ResNet18, VGGNet, and Swin Transformer, in facial feature extraction are compared. The experimental results are shown in Table 1.

[0075] Table 1 Feature extraction effects of different backbone networks

[0076] method Accuracy (%) Parameters(M) VGGNet 86.22 138 ResNet18 86.70 11.18 Swin Transformer 87.32 28

[0077] As can be seen from the table, Swin Transformer outperforms ResNet18 and VGGNet with an accuracy of 87.32%. This result shows that Swin Transformer can effectively capture more comprehensive facial features with its ability to capture long-distance dependencies. However, ResNet18 and VGGNet, as traditional CNN architectures, have difficulty capturing global dependencies in images due to the inherent limitations of convolution operations. In terms of the number of parameters, VGGNet is a typical relatively deep convolutional neural network with relatively more parameters, while the subsequent ResNet model has achieved better results with fewer parameters by introducing residual connections, while Swin Transformer uses a windowed attention method to achieve the best accuracy while maintaining an acceptable number of parameters.

[0078] (2) Verification of the effectiveness of each module: The proposed network model LFFNet makes full use of the complementary relationship between facial key points and facial features, and the multi-scale fusion module uses information at different scales to improve recognition performance. In order to verify the performance of each proposed module, the feature fusion module and the multi-scale fusion module are successively integrated into the Swin Transformer backbone network, and ablation experiments are performed on the RAFDB dataset. The experimental results are shown in Table 2. It can be observed that with the gradual introduction of different modules, the performance of the network model is effectively improved. Specifically, after adding the feature fusion module, the accuracy of the model increased by 1.28%. This module can achieve detailed fusion of facial key points and facial features at the pixel level, making the most of their valuable information. Furthermore, after adding the multi-scale fusion module, the accuracy of the model was further improved by 0.54%. This module uses residual connections and pyramid structures to more deeply fuse the output features of each stage, so that the model obtains a richer feature representation.

[0079] Table 2 Validity verification results of each module of the model

[0080] method Accuracy (%) Swin Transformer 87.32 Swin Transformer+Feature Fusion Module 88.60 Swin Transformer+Feature Fusion Module+Multi-Scale Fusion Module 89.14

[0081] (3) Confusion Matrix Analysis: In order to explore the details of the model performance, we further compared the performance of the proposed model LFFNet with the baseline network Swin Transformer on various facial expression categories. The comparison results are shown in Tables 3 and Figure 5 As shown in the results, we can see that LFFNet outperforms the baseline network in all expression categories. This is mainly due to the effective fusion of the model in capturing facial features and facial key points, which provides richer clues for facial expression recognition. In addition, by using a multi-scale information fusion mechanism, this model can learn more discriminative features, thereby achieving better results in facial expression recognition tasks. Figure 6 As shown, the confusion matrix of Swin Transformer and model LFFNet on the RAFDB dataset is shown, confirming the improvement of the overall performance of the model.

[0082] Table 3 Recall results of Swin Transformer and LFFNet on different expression categories

[0083]

[0084] 2.3 Comparison with existing models

[0085] In order to evaluate the effectiveness of the proposed model, the best results of the model are compared with several existing methods on the RAFDB and SFEW datasets. The experimental results are shown in Table 4 and Table 5. Figure 7 and Figure 8 As shown in the figure, it can be seen that on the RAFDB dataset, the proposed LFFNet model achieved the best accuracy of 89.14%, which exceeded other methods. This shows that the model has stronger performance in the expression recognition task. Similarly, on the SFEW dataset, LFFNet also achieved the best accuracy of 60.44%, which is better than other comparison methods.

[0086] Among them, some models use similar methods to handle expression recognition tasks. The RAN model decomposes facial images into multiple regions through facial key points and introduces regional bias loss to guide the model to focus on the most important areas. The ACNN model divides the input facial feature map into multiple sub-feature maps, assigns attention weights to each feature sub-map and the global feature map, and then simply concatenates all feature vectors into one vector. The MANet model uses a dual-branch network to extract and fuse multi-scale features and local features. The VTFF model uses RGB images and LBP images as input, selects the most important features through the attention module, and then integrates these features using the Transformer encoder.

[0087] Table 4 Comparison with other methods on the RAFDB dataset

[0088] method Accuracy (%) years DLP-CNN 84.22 2019 gACNN 85.07 2019 SCN 88.14 2020 MANet 88.36 2021 DACL 87.78 2021 VTFF 88.14 2021 RAN 86.90 2021 MixAugment 87.54 2022 LFFNet(ours) 89.14 2024

[0089] Table 5 Comparison with other methods on the SFEW dataset

[0090] method Accuracy (%) years DLP-CNN 51.05 2019 IPFR 55.10 2019 LDL-ALSG 56.50 2020 RAN 56.40 2021 MANet 59.40 2021 LFFNet(ours) 60.44 2024

[0091] Other models explore expression recognition from different perspectives. The SCN model starts from the perspective of labels, arguing that facial expressions are difficult to annotate with high quality due to the subjectivity of annotations and the uncertainty of facial images. Therefore, SCN uses an attention mechanism to weight each sample, assigns lower importance weights to uncertain facial images and modifies their labels. The LDL-ALSG model starts from the perspective of expression distribution, establishes a similarity list through facial action units and facial key points, and improves facial expression recognition performance by minimizing the probability of the center image and the nearest neighbor image while minimizing the actual value and the predicted value. From the perspective of loss function, the DACL model proposes a deep attention center loss, combining the attention mechanism and the sparse center loss to adaptively select the most important subset of feature elements. From the perspective of data enhancement, the MixAugment model generates new virtual expression images by mixing different expression images to increase the diversity of training data.

[0092] Table 6 Recall results of each model on different expression categories

[0093]

[0094] To further compare the performance differences of different models, Table 6 and Fig. 9 The recall results of each model on different expression categories are shown. As can be seen from the table, the proposed LFFNet model achieved recall rates of 88.24%, 96.20%, 90.79% and 67.57% on neutral, happy, sad and disgusted expression categories, respectively, showing a high recognition ability. It performed poorly in the classification of surprise, fear and anger expressions. But overall, although the models showed certain differences in different expression categories, the LFFNet model achieved excellent results in most cases, proving its excellent performance in expression recognition tasks. At the same time, the exploration and optimization of each model in different aspects provide rich ideas and methods for research in the field of expression recognition.

[0095] 3 Summary

[0096] An emotion recognition model based on the facial feature and key point fusion network LFFNet is proposed. It aims to fully extract and fuse facial features and facial key point information by using self-attention mechanism, feature fusion method and multi-scale technology to solve the problem that a single feature is difficult to fully capture scene details. The model uses Swin Transformer as the backbone network and uses the windowed self-attention mechanism to finely capture the rich detail information of facial images without significantly increasing the computational complexity. The feature fusion module optimizes the information fusion between facial key points and facial features at the pixel level, ensuring the maximum retention and utilization of valuable information. The multi-scale fusion module realizes deep multi-scale fusion by combining pyramid structure and residual connection, further enriching the feature representation. Through ablation experiments, the improvement effect of each module on the model performance is verified, and competitive results are achieved in comparison with the existing technology on two public facial expression recognition datasets RAFDB and SFEW.

[0097] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An emotion recognition method based on facial features and key point fusion network, characterized in that: include: Construct an emotion recognition model based on the facial feature and key point fusion network LFFNet; the emotion recognition model includes a backbone network SwinTransformer, a multi-scale fusion module and a fully connected layer FC; the backbone network SwinTransformer includes multiple processing layer blocks, a feature fusion module and a key point extraction module MobileFaceNet; Obtain a face image to be identified, input the face image to be identified into the emotion recognition model, extract facial features of the face image through the processing layer, and extract facial key points of the face image through the key point extraction module MobileFaceNet; The facial features extracted by each processing layer and the facial key points extracted by each key point extraction module MobileFaceNet are fused through the feature fusion module to obtain the similarity map of the pixel points at each corresponding position in the two facial feature maps extracted by the adjacent processing layers; all similarity maps are spliced ​​and integrated according to the processing order of the processing layers through the multi-scale fusion module to obtain the deep features; the deep features are fully connected through the fully connected layer FC to obtain the emotion recognition results.

2. The emotion recognition method based on facial feature and key point fusion network as claimed in claim 1, characterized in that: The backbone network Swin Transformer also includes a windowed multi-head self-attention module W_MSA and a shifted windowed multi-head self-attention module SW_MSA. The windowed multi-head self-attention module W_MSA is used to calculate self-attention within each processing layer. The shifted windowed multi-head self-attention module SW_MSA is used to make self-attention interact between different processing layers, including: in, is the output feature of the lth windowed multi-head self-attention module W_MSA and the shifted windowed multi-head self-attention module SW_MSA, z l is the output feature of the lth multi-layer perception module MLP, and LN is the layer normalization operation.

3. The emotion recognition method based on facial feature and key point fusion network as claimed in claim 1, characterized in that: The key point extraction module MobileFaceNet is a 68-point model, and the facial key points of the face image are extracted by the key point extraction module MobileFaceNet, including: Detect 68 key points of facial image data including eyes, eyebrows, nose, mouth and facial contour; A heat map is generated for each key point, and the coordinates of the key point are determined based on the peak position of each heat map.

4. The emotion recognition method based on facial feature and key point fusion network as claimed in claim 1, characterized in that: The facial features extracted by each processing layer and the facial key points extracted by each key point extraction module MobileFaceNet are fused through the feature fusion module, including: The pixel attention mechanism aligns the feature map of facial feature x extracted by the backbone network SwinTransformer and the feature map of facial key point y extracted by the key point extraction module MobileFaceNet through convolution; Perform dot multiplication on the pixels at each corresponding position of the two feature maps to obtain a similarity map that quantifies the importance of the facial key point feature at each pixel position relative to the facial feature x; The similarity map is processed by the Sigmoid function to obtain the feature influence value between 0 and 1 for each pixel on the feature map. If the feature influence value is close to 1, the contribution of the y feature to the fusion output at the pixel point is higher; if the feature influence value is close to 0, the contribution of the x feature to the fusion output at the pixel point is higher. Align the feature map of facial feature x with the feature map of facial key point y, and add the weighted feature influence values ​​to get the final fusion feature. The formula is: σ=Sigmoid(f x (v x )·f y (v y )) output=σv x +(1-σ)v y Among them, v x is the corresponding pixel in the feature map of the facial feature, v y is the corresponding pixel in the feature map of the facial key point, f is the alignment operation, and σ is the possibility that the two pixels belong to the same object.

5. The emotion recognition method based on facial feature and key point fusion network as claimed in claim 1, characterized in that: The multi-scale fusion module includes multiple 1×1 convolutional layers and multiple 3×3 convolutional layers. All similarity maps are spliced ​​and integrated according to the processing order of the processing layers through the multi-scale fusion module, including: Resize different input features to the same size through 1×1 convolutional layers and upsampling operations; The result of the first processing layer is passed to the next layer. For each subsequent processing layer, the output of the previous processing layer is added to its own input and fused through a 3×3 convolution to obtain deeper features. The outputs of all processing layers are spliced ​​and integrated according to the processing order of the processing layers to obtain the deep features of the face image.

6. The emotion recognition method based on facial feature and key point fusion network as claimed in claim 1, characterized in that: The fully connected processing of the deep features through the fully connected layer FC includes: The deep features of the face image are flattened into a one-dimensional vector, and the one-dimensional vector is subjected to feature dimensionality reduction and feature abstraction to obtain the probability distribution of expression categories in the face image.

Images