Facial expression recognition method and system based on feature enhancement

Abstract

The application provides a facial expression recognition method and system based on feature enhancement. The method comprises the following steps: obtaining an expression image dataset, the expression image dataset being an N-class expression image dataset after labeling and classification; performing data enhancement on each expression image in the expression image dataset; building a convolutional neural network with ResNet as a backbone network; inputting the data-enhanced expression image dataset into the trained convolutional neural network for feature extraction; outputting N expression categories through the last fully connected layer; and using a loss function obtained by jointly optimizing a softmax loss function and a feature enhancement extraction loss function as the loss function of the convolutional neural network. The facial expression recognition method provided by the application focuses on the separability and distinguishability in the feature extraction process, and fills the technical gap in the prior art that ignores the inter-class similarity and intra-class difference of the existing facial expression images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a facial expression recognition method and system based on feature enhancement. Background Technology

[0002] Facial expressions are an important component of human communication, conveying a wealth of emotional and intentional information. Facial expression recognition has wide applications in many fields, including human-computer interaction, sentiment analysis, and mental health. However, due to the complexity and diversity of facial expressions, accurately recognizing them remains a challenging problem.

[0003] With the rapid development of computer vision and artificial intelligence, facial expression recognition has made some progress. Traditional facial expression recognition methods are mainly based on hand-designed feature extraction and classifiers, but these methods often rely on the experience of domain experts and have certain limitations in dealing with the diversity and variability of facial expressions. In recent years, the rise of deep learning technology has brought new opportunities to facial expression recognition. By utilizing models such as deep convolutional neural networks (CNNs), features in facial images can be automatically learned and extracted, thereby improving the accuracy of facial expression recognition.

[0004] However, existing facial expression recognition methods still face several challenges. First, in real-world environments, collected facial image data is often affected by various factors, such as facial occlusion, changes in lighting intensity during acquisition, inconsistent facial poses, and low image resolution, all of which make expression recognition more difficult. Second, facial expressions exhibit significant inter-class similarity and intra-class differences. Inter-class similarity can be understood as the strong common features between different categories of facial expressions; for example, subtle changes in the face can represent two completely different categories when people express different expressions. Intra-class differences can be understood as the significant variations in outward expressions for the same category of expression due to the vast differences between different populations, cultures, and regions. Furthermore, changes in facial expressions are not influenced by a single location but typically involve changes in multiple facial regions and their synergistic effects.

[0005] Furthermore, most previous studies on facial expression recognition were conducted in controlled laboratory environments, which cannot reflect the complexities of real-life scenarios and thus have certain limitations. Therefore, facial expression recognition conducted in real-world scenarios can make the model more generalizable and have greater practical significance. Summary of the Invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0007] Therefore, the first objective of this invention is to propose a facial expression recognition method based on feature enhancement.

[0008] The second objective of this invention is to propose a facial expression recognition system based on feature enhancement.

[0009] A third objective of this invention is to provide a computer device.

[0010] A fourth objective of this invention is to provide a computer-readable storage medium.

[0011] To achieve the above objectives, the first aspect of the present invention provides a facial expression recognition method based on feature enhancement, comprising: acquiring an expression image dataset, wherein the expression image dataset is an N-class expression image dataset after labeling and classification; performing data enhancement on each expression image in the expression image dataset; constructing a convolutional neural network with ResNet as the backbone network; inputting the data-enhanced expression image dataset into the trained convolutional neural network for feature extraction, so as to extract morphological features, texture features, and color features from each data-enhanced expression image; and outputting N expression categories through a final fully connected layer; wherein the loss function of the convolutional neural network is a loss function jointly optimized by the softmax loss function and the feature enhancement extraction loss function;

[0012] The formula for the softmax loss function is as follows:

[0013]

[0014] Among them, y i ∈{1,...,K}, The input is in d-dimensional space; m is the total number of samples in the mini-batch. These are the weights of the last fully connected layer; It is a bias parameter;

[0015] The formula for the feature enhancement extraction loss function is as follows:

[0016]

[0017] in, The y-th depth feature i Each class center is randomly sampled from a d-dimensional Gaussian distribution; x i ∈X, representing the i-th input vector, where X is the input vector space; y i ∈Y, where Y is the label space; M is the dimension of Y; σ c The standard deviation representing the class center;

[0018] The formula for the loss function after joint optimization is as follows:

[0019]

[0020] Where λ1 is Hyperparameters.

[0021] Preferably, after the step of inputting the data-enhanced facial expression image dataset into a trained convolutional neural network for feature extraction to extract morphological, texture, and color features from each data-enhanced facial expression image, the method further includes: constructing a spatial attention module; constructing a channel attention module; building a deep network based on the fusion learning of spatial and channel attention; feeding the features extracted by the convolutional neural network into the spatial attention module of the trained deep network to extract spatial features; feeding the extracted spatial features into the channel attention module of the trained deep network to extract channel features; normalizing the extracted spatial and channel features using the log-softmax function; and before the step of outputting N facial expression categories through the last fully connected layer, the method further includes: accelerating the training and convergence speed of the deep network through a batch normalization layer; wherein, the loss function of the deep network is the total network loss function obtained by jointly optimizing the softmax loss function, the feature enhancement extraction loss function, and the fusion loss function;

[0022] The formula for the fusion loss function is as follows:

[0023]

[0024] Where N is the data size of each batch, C is the channel size of the attention map, and k is the number of attention branches. Let be the variance of the i-th sample in the j-th channel;

[0025] The formula for the total network loss function is as follows:

[0026]

[0027] Where λ2 is Hyperparameters.

[0028] Preferably, before performing data augmentation on each facial expression image in the facial expression image dataset, the method further includes: cropping each facial expression image in the facial expression image dataset to obtain a corresponding face image; scaling each face image to a length of m pixels according to the aspect ratio of the original image; if the image is not square, then after scaling, filling the short side with pixels with a grayscale value of 0 to a length of m pixels to obtain a square image with a side length of m pixels.

[0029] The second aspect of the present invention also provides a facial expression recognition system based on feature enhancement, comprising: an acquisition unit for acquiring an expression image dataset, wherein the expression image dataset is an N-class expression image dataset after labeling and classification; a data enhancement unit for performing data enhancement on each expression image in the expression image dataset; a convolutional neural network construction unit for constructing a convolutional neural network with ResNet as the backbone network; a feature extraction unit for inputting the data-enhanced expression image dataset into the trained convolutional neural network for feature extraction, thereby extracting morphological features, texture features, and color features from each data-enhanced expression image; and an expression category output unit for outputting N expression categories through the last fully connected layer; wherein the loss function of the convolutional neural network is a loss function jointly optimized by the softmax loss function and the feature enhancement extraction loss function.

[0030] The formula for the softmax loss function is as follows:

[0031]

[0032] Among them, y i ∈{1,...,K}, The input is in d-dimensional space; m is the total number of samples in the mini-batch. These are the weights of the last fully connected layer; It is a bias parameter;

[0033] The formula for the feature enhancement extraction loss function is as follows:

[0034]

[0035] in, The y-th depth feature i Each class center is randomly sampled from a d-dimensional Gaussian distribution; x i ∈X, representing the i-th input vector, where X is the input vector space; y i ∈Y, where Y is the label space; M is the dimension of Y; σ c The standard deviation representing the class center;

[0036] The formula for the loss function after joint optimization is as follows:

[0037]

[0038] Where λ1 is Hyperparameters.

[0039] Preferably, the facial expression recognition system based on feature enhancement further includes: a spatial attention module construction unit for constructing a spatial attention module; a channel attention module construction unit for constructing a channel attention module; a deep network construction unit for constructing a deep network based on the fusion learning of spatial and channel attention; a spatial feature extraction unit for feeding the features extracted by the convolutional neural network into the spatial attention module to extract spatial features; a channel feature extraction unit for feeding the extracted spatial features into the channel attention module to extract channel features; a normalization processing unit for normalizing the extracted spatial features and channel features using the log-softmax function; and a BN unit for accelerating the training and convergence speed of the deep network through BN layers; wherein the loss function of the deep network based on the fusion learning of spatial and channel attention is the total network loss function obtained by jointly optimizing the softmax loss function, the feature enhancement extraction loss function, and the fusion loss function.

[0040] The formula for the fusion loss function is as follows:

[0041]

[0042] Where N is the data size of each batch, C is the channel size of the attention map, and k is the number of attention branches. Let be the variance of the i-th sample in the j-th channel;

[0043] The formula for the total network loss function is as follows:

[0044]

[0045] Where λ2 is Hyperparameters.

[0046] Preferably, the feature-enhanced facial expression recognition system further includes: a cropping unit, used to crop each facial expression image in the facial expression image dataset to obtain the corresponding face image; a scaling unit, used to scale each face image, scaling it to the length of the longer side being m pixels according to the aspect ratio of the original image; if the image is not square, then after scaling, the shorter side is filled with pixels with a grayscale value of 0 to a length of m pixels, so as to obtain a square image with a side length of m pixels.

[0047] The third aspect of the present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the feature-enhanced facial expression recognition method as described in any of the above technical solutions.

[0048] The fourth aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the feature-enhanced facial expression recognition method as described in any of the above technical solutions.

[0049] The beneficial effects of this invention are:

[0050] (1) The facial expression recognition method and system based on feature enhancement provided by the present invention realizes the use of residual learning to help solve problems such as gradient vanishing and gradient explosion in neural networks by building a convolutional neural network with ResNet as the backbone network.

[0051] (2) The facial expression recognition method and system based on feature enhancement provided by this invention effectively enhances the recognition ability of convolutional neural networks for deep features by designing a feature enhancement extraction loss function. Specifically, the loss function of the convolutional neural network adopts a loss function that is jointly optimized by the softmax loss function and the feature enhancement extraction loss function, thereby achieving the goal of adapting to the specificity of facial expression images. It emphasizes the separability and discriminability in the feature extraction process, filling the technical gap that most previous studies only focused on the single representation of features when performing feature extraction, ignoring the inter-class similarity and intra-class differences in facial expression images.

[0052] (3) The facial expression recognition method and system based on feature enhancement provided by the present invention proposes a multi-head attention network on the basis of feature enhancement. By passing the features extracted by the convolutional neural network through attention modules of different branches, different attention regions are obtained to learn the key regions that affect the change of expression. Furthermore, the multi-branch attention network is merged, and the attention network is forced to pay attention to different regions and fuse them. Finally, the facial expression category is output, thus achieving the purpose of learning the features of multiple key regions that affect the change of expression.

[0053] Additional aspects and advantages of the invention will become apparent from the description which follows, or may be learned by practice of the invention. Attached Figure Description

[0054] Figure 1 A schematic flowchart of a feature-enhanced facial expression recognition method according to an embodiment of the present invention is shown;

[0055] Figure 2 This diagram illustrates the overall flowchart of a facial expression recognition method in the prior art.

[0056] Figure 3 A schematic diagram illustrating a feature-enhanced facial expression recognition method according to an embodiment of the present invention is shown.

[0057] Figure 4 A schematic diagram of the structure of a residual block in a ResNet backbone network according to an embodiment of the present invention is shown;

[0058] Figure 5 A schematic flowchart of a feature-enhanced facial expression recognition method according to another embodiment of the present invention is shown;

[0059] Figure 6 A schematic diagram of a network structure for fusion feature enhancement and multi-head attention according to an embodiment of the present invention is shown;

[0060] Figure 7 This invention illustrates a grayscale image of an expression in a real-world scene, representing an embodiment of the invention.

[0061] Figure 8 An example of a data-processed facial expression image according to an embodiment of the present invention is shown;

[0062] Figure 9 A schematic block diagram of a feature-enhanced facial expression recognition system according to an embodiment of the present invention is shown. Detailed Implementation

[0063] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0064] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0065] Figure 2 A schematic flowchart illustrating the overall process of facial expression recognition methods in the prior art is shown. Figure 2 As shown in the example, when using a conventional convolutional neural network for facial expression recognition, its main idea is feature learning and label prediction. First, the facial expression image is input, and the image is subjected to data augmentation and normalization. After the convolutional features are extracted to obtain deep features, the image is finally passed through a fully connected layer as a linear classifier, and the output is the facial expression category label.

[0066] Furthermore, it is crucial to construct an efficient loss function for feature identification in convolutional neural networks. This invention innovatively designs a feature enhancement extraction loss function.

[0067] Figure 1A schematic flowchart illustrating a feature-enhanced facial expression recognition method according to an embodiment of the present invention is shown. Figure 1 As shown, a facial expression recognition method based on feature enhancement includes:

[0068] Step 102: Obtain the facial expression image dataset, which is a dataset of N types of facial expression images after labeling and classification;

[0069] Step 104: Perform data augmentation on each facial expression image in the facial expression image dataset;

[0070] Step 106: Construct a convolutional neural network with ResNet as the backbone network. The loss function of the convolutional neural network is the loss function obtained by jointly optimizing the softmax loss function and the feature enhancement extraction loss function.

[0071] Step 108: Input the data-enhanced facial expression image dataset into the trained convolutional neural network for feature extraction, in order to extract the morphological features, texture features and color features from each data-enhanced facial expression image;

[0072] Step 110: Output N types of facial expressions through the last fully connected layer;

[0073] The formula for the softmax loss function is as follows:

[0074]

[0075]

[0076] Among them, y i ∈{1,...,K}, The input is in d-dimensional space; m is the total number of samples in the mini-batch. These are the weights of the last fully connected layer; It is a bias parameter;

[0077] The formula for the feature enhancement extraction loss function is as follows:

[0078]

[0079] in, The y-th depth feature i Each class center is randomly sampled from a d-dimensional Gaussian distribution; x i ∈X, representing the i-th input vector, where X is the input vector space; y i ∈Y, where Y is the label space; M is the dimension of Y; σ c The standard deviation representing the class center;

[0080] The formula for the loss function after joint optimization is as follows:

[0081]

[0082] Where λ1 is Hyperparameters.

[0083] The facial expression recognition method based on feature enhancement provided by this invention, by building a convolutional neural network with ResNet as the backbone network, realizes the use of residual learning to help solve problems such as gradient vanishing and gradient explosion in neural networks.

[0084] Furthermore, the facial expression recognition method based on feature enhancement provided by this invention effectively enhances the recognition ability of convolutional neural networks for deep features by designing a feature enhancement extraction loss function. Specifically, the loss function of the convolutional neural network adopts a loss function that is jointly optimized from the softmax loss function and the feature enhancement extraction loss function, achieving the goal of adapting to the specificity of facial expression images. It emphasizes the separability and discriminability in the feature extraction process, filling the technical gap that most previous studies only focused on the single representation of features during feature extraction, ignoring the inter-class similarity and intra-class differences existing in facial expression images.

[0085] Furthermore, after fully extracting features, label prediction is performed, and the emoji labels are output through the classifier, that is, N emoji categories are output through the last fully connected layer.

[0086] Figure 3 A schematic process diagram of a feature-enhanced facial expression recognition method according to an embodiment of the present invention is shown. Figure 3 As shown, input facial expression image data undergoes convolutional feature extraction. A loss function, jointly optimized by the softmax loss function and the feature enhancement extraction loss function, is used to achieve deep feature extraction. Further, after sufficient feature extraction, label prediction is performed, and the facial expression label is output via a classifier. Here, deep feature extraction mainly refers to the separability and discriminability during the feature extraction process.

[0087] In this embodiment, the RAF-DB public dataset can be directly obtained, which contains 15,337 facial expression images with seven expression labels: surprise, happiness, sadness, fear, anger, disgust, and neutrality.

[0088] In this embodiment, ResNet-18 is used as the backbone of the constructed convolutional neural network for feature extraction. This network structure is widely used in image recognition, and is particularly suitable for tasks such as facial expression recognition. The ResNet-18 network structure includes several key components, including an input layer, convolutional layers, residual blocks, global average pooling layers, and fully connected layers. Figure 4 As shown, a residual block consists of two convolutional layers and a cross-layer connection, where the cross-layer connection directly adds the output of neurons in the previous layer to the output of the current layer. The basic idea behind residual blocks is that each block includes not only regular convolutional layers and activation functions, but also a shortcut connection. During training, this shortcut connection allows the network to easily learn identity mappings, leading to faster convergence and better performance in deeper networks. It also ensures that information is directly passed to subsequent layers, avoiding loss during propagation. Simultaneously, the cross-layer connection guarantees the proper transmission of gradient information.

[0089] The first residual block of ResNet-18 consists of two convolutional layers. Each convolutional layer is followed by a batch normalization layer; this structure helps improve the network's stability and convergence speed. After the batch normalization layer is a ReLU activation function layer to introduce non-linear features. Finally, the result of the cross-layer connection is added to the output of the last convolutional layer, allowing information from previous layers to be directly passed to later layers, avoiding information loss and gradient vanishing problems. The remaining three residual blocks have similar structures to the first. Each residual block includes two convolutional layers and one cross-layer connection. This cross-layer connection design allows the network to be deeper while maintaining efficient gradient propagation, thus enabling better learning of features in the image.

[0090] In ResNet-18, each residual block has the same number of input and output channels to maintain feature consistency. The first residual block has 64 input channels because the first convolutional layer of ResNet-18 outputs 64 feature maps. Subsequent convolutional layers following each residual block have 64, 128, 256, and 512 channels respectively, gradually increasing the abstraction and expressive power of the features.

[0091] In ResNet-18, each residual block has a different stride. The first residual block has a stride of 1, which keeps the spatial size of the input feature map constant. Subsequent residual blocks have a stride of 2, which helps to gradually reduce the spatial size of the feature map, thereby increasing the size of the receptive field.

[0092] The final global average pooling layer transforms the output of the last residual block into a global feature vector. This is achieved by performing average pooling on each feature map, compressing the feature map size to 1x1. The global average pooling layer captures feature information from the entire image, reducing the number of parameters and computational cost while preserving important contextual information.

[0093] The final fully connected layer is used for classification. Based on the needs of facial expression recognition, ResNet-18 outputs seven different facial expression categories in the final fully connected layer. By classifying the global feature vectors, the network can accurately classify the input image into different facial expression categories.

[0094] Specifically, in the feature extraction process, the input data is converted into feature vectors that can be fed into the convolutional neural network. Data belonging to the same expression category have common feature centers. As the number of network layers increases and image features are extracted, the feature centers are constantly changing. By learning the deep feature centers of each class, during the training process, the centers are updated and the distance between the deep features and their corresponding class centers is minimized, making features belonging to the same category more compact, as shown in formula (2).

[0095]

[0096] in, The y-th depth feature i Each class center is randomly sampled from a d-dimensional Gaussian distribution; x i ∈X, representing the i-th input vector, where X is the input vector space; y i ∈Y, where Y is the label space; M is the dimension of Y; σ c The standard deviation of the class centers; as training progresses, the feature centers... Feature point x i The distance between features decreases continuously, the distance between feature centers increases continuously, and the standard deviation of class centers increases, thereby minimizing the intra-class spacing and maximizing the inter-class spacing.

[0097] Furthermore, in facial expression classification tasks, if a convolutional neural network is trained solely using feature extraction loss, the learned deep features and feature centers may suffer from degradation. On the other hand, if only softmax loss is used as a supervision signal, the resulting deep features exhibit significant intra-class variation, failing to achieve class discriminability. Therefore, a feature enhancement loss is used to train the CNN for facial expression feature learning, and the softmax function is jointly optimized with the feature enhancement loss function for facial expression classification.

[0098] Given an image classification task with m samples and k classes, y i ∈{1,...,K}, The input is in d-dimensional space. Traditional loss functions combine fully connected layers, softmax functions, and cross-entropy loss to jointly measure the model's prediction error. The fully connected layer accepts input x. i And output vector z through linear transformation i The linear transformation is shown in formula (6).

[0099] z i =W T x i +B (6)

[0100] in, These are the weights of the last fully connected layer. This is the bias parameter. Then, the probability distribution of all classes is calculated using the softmax function, as shown in formula (7).

[0101]

[0102] Finally, cross-entropy is used to calculate the difference between the predicted and actual values, and the softmax loss function can be expressed by formula (1).

[0103]

[0104] Where m is the total number of samples in the mini-batch. The softmax loss function is minimized by stochastic gradient descent to optimize network parameters and achieve better classification. Finally, the loss function of the convolutional neural network is the loss function obtained by jointly optimizing the softmax loss function and the feature enhancement extraction loss function, as shown in Equation (3).

[0105]

[0106] Here, λ1 is a hyperparameter used to balance the two loss functions. An appropriate λ1 can significantly enhance the discriminative power of deep features.

[0107] Figure 5 A schematic flowchart illustrating another embodiment of the facial expression recognition method based on feature enhancement of the present invention is shown. Figure 5 As shown, a facial expression recognition method based on feature enhancement includes:

[0108] Step 502: Obtain the facial expression image dataset, which is a dataset of N types of facial expression images after labeling and classification;

[0109] Step 504: Perform data augmentation on each facial expression image in the facial expression image dataset;

[0110] Step 506: Construct a convolutional neural network with ResNet as the backbone network;

[0111] Step 508: Input the data-enhanced facial expression image dataset into the trained convolutional neural network for feature extraction, so as to extract the morphological features, texture features and color features from each data-enhanced facial expression image;

[0112] Step 510: Construct the spatial attention module;

[0113] Step 512, construct the channel attention module;

[0114] Step 514: Build a deep network based on spatial and channel attention fusion learning. The loss function of the deep network is the total network loss function after jointly optimizing the softmax loss function, feature enhancement extraction loss function, and fusion loss function.

[0115] Step 516: The features extracted by the convolutional neural network are fed into the spatial attention module in the trained deep network to extract spatial features.

[0116] Step 518: The extracted spatial features are fed into the channel attention module in the trained deep network to extract channel features;

[0117] Step 520: Normalize the extracted spatial features and channel features using the log-softmax function;

[0118] Step 522: Accelerate the training and convergence speed of deep networks by using BN layers;

[0119] Step 524: Output N types of expressions through the last fully connected layer.

[0120] The formula for the fusion loss function is as follows:

[0121]

[0122] Where N is the data size of each batch, C is the channel size of the attention map, and k is the number of attention branches. Let be the variance of the i-th sample in the j-th channel;

[0123] The formula for the total network loss function is as follows:

[0124]

[0125] Where λ2 is Hyperparameters.

[0126] In this embodiment, features in an image are learned through a convolutional neural network, and a feature enhancement loss function is designed to better learn the inter-class similarity and intra-class differences in facial expression images, thereby achieving feature enhancement. Furthermore, since facial expression changes are not influenced by a single location but typically involve changes in multiple facial regions and their synergistic effects, this invention proposes a multi-head attention network to learn the multiple key regions influencing expression changes, building upon feature enhancement. Features extracted by the convolutional neural network are passed through attention modules of different branches to obtain different attention regions, thus learning the key regions affecting expression changes. Specifically, this invention constructs multiple parallel attention heads, consisting of a spatial attention module and a channel attention module. First, the features extracted by the convolutional neural network are fed into the spatial attention module to extract spatial features, and then the extracted features are fed into the channel attention module to extract channel features. Different weight values ​​are assigned according to the importance of each channel, boosting channels of feature maps that play a crucial role in expression recognition and suppressing others. Further, the multi-branch attention network is merged, forcing the attention network to focus on different regions and fusing them, finally outputting the facial expression category.

[0127] Figure 6 A schematic diagram of a network structure for fusion feature enhancement and multi-head attention according to an embodiment of the present invention is shown. In a specific embodiment, the spatial attention module consists of four convolutional modules and one activation function, constructing four types of convolutional kernels: 1×1, 1×3, 3×1, and 3×3, to capture local features of different key regions. Then, the ReLU activation function is used to achieve non-linearity of the network.

[0128] For the channel attention module, adaptive average pooling is first used to downsample the input, reducing the size of the feature map, thereby reducing the number of model parameters and computational cost. Compared with conventional average pooling and max pooling, adaptive average pooling calculates the pooling output by weighted averaging of the feature map at each location. Adaptive average pooling does not require specifying the pooling window size, but dynamically calculates the pooling window size by specifying the size of the output feature map, which can retain more feature information.

[0129] Then, a small autoencoder is implemented using two linear layers and an activation function to encode the channel information. The output of spatial attention can be expressed as Equation (8):

[0130] s i =x i ×H i (ω s ,x') (8)

[0131] Among them, H i ∈{H1,...,H k} represents the spatial attention heads, i∈{1,...,k}, where k is the number of heads in a multi-head attention system, and S i ∈{S1,...,S k} represents the spatial attention feature of the output, ω s It is H i The network parameters. Similarly, the final output obtained after passing through the channel attention module can be expressed as formula (9):

[0132] c i =s i ×H i '(ω c ,s i (9)

[0133] Where i∈{1,...,k}, ω c It is H i 'Network parameters'.

[0134] Feature maps output by multi-head attention networks can capture key regions from facial expressions, but they also have certain problems. As the number of attention heads increases, the multi-head structure may learn overlapping facial regions, such as multiple attention heads focusing on the same areas like the eyes or mouth, making it difficult to coordinate the learning of attention distribution areas and wasting network resources. To address these issues, this invention constructs an attention fusion network to improve the learned features.

[0135] First, considering network computation speed and gradient stability, the log-softmax function is applied to normalize the attention features. Assume V q c represents the i-th attention head i The output vector, v m c i The m-th vector, after normalization, can be expressed as formula (10):

[0136]

[0137] Then, to avoid overlap of multi-head attention, a fusion loss function is proposed to guide the network to focus attention on different key regions and merge the normalized multi-head attention.

[0138] Finally, the learned distributed feature representations are mapped to the sample label space through a fully connected layer, and a batch normalization (BN) layer is added to accelerate the training and convergence speed of the network, outputting the expression category score.

[0139] By applying the fusion loss function, the region of focus of multi-head attention is maximized, as expressed by formula (4).

[0140]

[0141] Where N is the data size of each batch, C is the channel size of the attention map, and k is the number of attention branches. Let represent the variance of the i-th sample in the j-th channel.

[0142] The formula for the total network loss function is as follows:

[0143]

[0144] Where λ1 and λ2 are respectively and The hyperparameters were all set to 1 in the experiment.

[0145] In one embodiment of the present invention, before the step of performing data augmentation on each facial expression image in the facial expression image dataset, the method further includes: cropping each facial expression image in the facial expression image dataset to obtain a corresponding face image; scaling each face image to a length of m pixels according to the aspect ratio of the original image; if the image is not square, then after scaling, filling the short side with pixels with a grayscale value of 0 to a length of m pixels to obtain a square image with a side length of m pixels.

[0146] Grayscale images of facial expressions in real-world scenarios, such as Figure 7 As shown, unprocessed facial expression images from real-world scenes cannot be directly input into convolutional neural networks.

[0147] In this embodiment, before performing data augmentation on each facial expression image in the facial expression image dataset, the facial expression images in the dataset are uniformly normalized so that the subsequent network can better learn the features in the facial expression images.

[0148] Specifically, each facial expression image in the facial expression image dataset is cropped to obtain the corresponding face image. That is, the facial expression is cropped from the image with the face as the center. The window size should be large enough to include the entire face in the cropped image, but small enough to prevent multiple faces from appearing in the same cropped image. Then, each face image is scaled to the same size. The processed facial expression images are as follows: Figure 8 As shown.

[0149] Figure 9 A schematic block diagram of a feature-enhanced facial expression recognition system according to an embodiment of the present invention is shown. Figure 9 As shown, the feature-enhanced facial expression recognition system 900 includes:

[0150] The acquisition unit 902 is used to acquire the facial expression image dataset, wherein the facial expression image dataset is a dataset of N types of facial expression images after labeling and classification;

[0151] Data augmentation unit 904 is used to perform data augmentation on each facial expression image in the facial expression image dataset;

[0152] The convolutional neural network building unit 906 is used to build a convolutional neural network with ResNet as the backbone network.

[0153] The feature extraction unit 908 is used to input the data-enhanced facial expression image dataset into the trained convolutional neural network for feature extraction, so as to extract the morphological features, texture features and color features in each data-enhanced facial expression image;

[0154] The expression category output unit 910 is used to output N expression categories through the last fully connected layer;

[0155] The loss function used in the convolutional neural network is a loss function that is jointly optimized by the softmax loss function and the feature enhancement extraction loss function.

[0156] The formula for the softmax loss function is as follows:

[0157]

[0158] Among them, y i ∈{1,...,K}, The input is in d-dimensional space; m is the total number of samples in the mini-batch. These are the weights of the last fully connected layer; It is a bias parameter;

[0159] The formula for the feature enhancement extraction loss function is as follows:

[0160]

[0161] in, The y-th depth feature i Each class center is randomly sampled from a d-dimensional Gaussian distribution; x i ∈X, representing the i-th input vector, where X is the input vector space; y i ∈Y, where Y is the label space; M is the dimension of Y; σ c The standard deviation representing the class center;

[0162] The formula for the loss function after joint optimization is as follows:

[0163]

[0164] Where λ1 is Hyperparameters.

[0165] The facial expression recognition system based on feature enhancement provided by this invention, by building a convolutional neural network with ResNet as the backbone network, realizes the use of residual learning to help solve problems such as gradient vanishing and gradient explosion in neural networks.

[0166] Furthermore, the facial expression recognition system based on feature enhancement provided by this invention effectively enhances the recognition ability of convolutional neural networks for deep features by designing a feature enhancement extraction loss function. Specifically, the loss function of the convolutional neural network adopts a loss function jointly optimized from the softmax loss function and the feature enhancement extraction loss function, achieving the goal of adapting to the specificity of facial expression images. It emphasizes the separability and discriminability in the feature extraction process, filling the technical gap that most previous studies only focused on the single representation of features during feature extraction, ignoring the inter-class similarity and intra-class differences existing in facial expression images.

[0167] Furthermore, after fully extracting features, label prediction is performed, and the emoji labels are output through the classifier, that is, N emoji categories are output through the last fully connected layer.

[0168] In one embodiment of the present invention, such as Figure 9 As shown, the facial expression recognition system 900 based on feature enhancement further includes: a spatial attention module construction unit 912 for constructing a spatial attention module; a channel attention module construction unit 914 for constructing a channel attention module; a deep network construction unit 916 for constructing a deep network based on the fusion learning of spatial and channel attention; a spatial feature extraction unit 918 for feeding the features extracted by the convolutional neural network into the spatial attention module to extract spatial features; a channel feature extraction unit 920 for feeding the extracted spatial features into the channel attention module to extract channel features; a normalization processing unit 922 for normalizing the extracted spatial features and channel features using the log-softmax function; and a BN unit 924 for accelerating the training and convergence speed of the deep network through BN layers; wherein, the loss function of the deep network based on the fusion learning of spatial and channel attention is the total network loss function obtained by jointly optimizing the softmax loss function, the feature enhancement extraction loss function, and the fusion loss function.

[0169] The formula for the fusion loss function is as follows:

[0170]

[0171] Where N is the data size of each batch, C is the channel size of the attention map, and k is the number of attention branches. Let be the variance of the i-th sample in the j-th channel;

[0172] The formula for the total network loss function is as follows:

[0173]

[0174] Where λ2 is Hyperparameters.

[0175] In this embodiment, features in an image are learned through a convolutional neural network, and a feature enhancement loss function is designed to better learn the inter-class similarity and intra-class differences in facial expression images, thereby achieving feature enhancement. Furthermore, since facial expression changes are not influenced by a single location but typically involve changes in multiple facial regions and their synergistic effects, this invention proposes a multi-head attention network to learn the multiple key regions influencing expression changes, building upon feature enhancement. Features extracted by the convolutional neural network are passed through attention modules of different branches to obtain different attention regions, thus learning the key regions affecting expression changes. Specifically, this invention constructs multiple parallel attention heads, consisting of a spatial attention module and a channel attention module. First, the features extracted by the convolutional neural network are fed into the spatial attention module to extract spatial features, and then the extracted features are fed into the channel attention module to extract channel features. Different weight values ​​are assigned according to the importance of each channel, boosting channels of feature maps that play a crucial role in expression recognition and suppressing others. Further, the multi-branch attention network is merged, forcing the attention network to focus on different regions and fusing them, finally outputting the facial expression category.

[0176] In one embodiment of the present invention, such as Figure 9 As shown, the feature-enhanced facial expression recognition system 900 further includes: a cropping unit 926, used to crop each facial expression image in the facial expression image dataset to obtain the corresponding face image; and a scaling unit 928, used to scale each face image, scaling it to a length of m pixels according to the original image's aspect ratio; if the image is not square, after scaling, the short side is filled with pixels with a grayscale value of 0 to a length of m pixels to obtain a square image with a side length of m pixels.

[0177] In this embodiment, before performing data augmentation on each facial expression image in the facial expression image dataset, the facial expression images in the dataset are uniformly normalized so that the subsequent network can better learn the features in the facial expression images.

[0178] Specifically, each facial expression image in the facial expression image dataset is cropped to obtain the corresponding face image. That is, the facial expression is cropped from the image with the face as the center. The window size should be large enough to include the entire face in the cropped image, and small enough to prevent multiple faces from appearing in the same cropped image. Then, each face image is scaled to the same size.

[0179] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the feature-enhanced facial expression recognition method as described in any of the above embodiments.

[0180] The computer device provided by this invention, when the processor executes a computer program, constructs a convolutional neural network with ResNet as the backbone network, and realizes the use of residual learning to help solve problems such as gradient vanishing and gradient explosion in neural networks.

[0181] Furthermore, the facial expression recognition method based on feature enhancement provided by this invention effectively enhances the recognition ability of convolutional neural networks for deep features by designing a feature enhancement extraction loss function. Specifically, the loss function of the convolutional neural network adopts a loss function that is jointly optimized from the softmax loss function and the feature enhancement extraction loss function, achieving the goal of adapting to the specificity of facial expression images. It emphasizes the separability and discriminability in the feature extraction process, filling the technical gap that most previous studies only focused on the single representation of features during feature extraction, ignoring the inter-class similarity and intra-class differences existing in facial expression images.

[0182] Furthermore, after fully extracting features, label prediction is performed, and the emoji labels are output through the classifier, that is, N emoji categories are output through the last fully connected layer.

[0183] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the feature-enhanced facial expression recognition method as described in any of the above embodiments.

[0184] The computer-readable storage medium provided by this invention enables a computer program to be executed by a processor. By constructing a convolutional neural network with ResNet as the backbone network, it achieves the use of residual learning to help solve problems such as gradient vanishing and gradient explosion in neural networks.

[0185] Furthermore, the facial expression recognition method based on feature enhancement provided by this invention effectively enhances the recognition ability of convolutional neural networks for deep features by designing a feature enhancement extraction loss function. Specifically, the loss function of the convolutional neural network adopts a loss function that is jointly optimized from the softmax loss function and the feature enhancement extraction loss function, achieving the goal of adapting to the specificity of facial expression images. It emphasizes the separability and discriminability in the feature extraction process, filling the technical gap that most previous studies only focused on the single representation of features during feature extraction, ignoring the inter-class similarity and intra-class differences existing in facial expression images.

[0186] Furthermore, after fully extracting features, label prediction is performed, and the emoji labels are output through the classifier, that is, N emoji categories are output through the last fully connected layer.

[0187] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on feature enhancement, characterized in that, include: Obtain an expression image dataset, wherein the expression image dataset is an N-class expression image dataset after labeling and classification; Data augmentation is performed on each facial expression image in the facial expression image dataset; Construct a convolutional neural network with ResNet as the backbone network; The augmented facial expression image dataset is input into a trained convolutional neural network for feature extraction, in order to extract the morphological features, texture features, and color features from each augmented facial expression image. The last fully connected layer outputs N types of emojis; The loss function of the convolutional neural network is a loss function that is jointly optimized by the softmax loss function and the feature enhancement extraction loss function. The formula for the softmax loss function is as follows: （1） in, , Is d Input in 3D space; m It is a portion of the total sample size; These are the weights of the last fully connected layer; It is a bias parameter; The formula for the feature enhancement extraction loss function is as follows: （2） in, , representing the first depth feature Individual category center from d Random sampling from a Gaussian distribution; , indicating the first i input vectors, X It is the input vector space; , Y It is a tag space; M yes Y The dimension; The standard deviation representing the class center; The formula for the loss function after joint optimization is as follows: （3） in, yes hyperparameters; After inputting the augmented facial expression image dataset into a trained convolutional neural network for feature extraction—specifically, extracting morphological, texture, and color features from each augmented facial expression image—the process also includes: Construct a spatial attention module; Construct a channel attention module; Construct a deep network based on spatial and channel attention fusion learning; The features extracted by the convolutional neural network are fed into the spatial attention module in the trained deep network to extract spatial features. The extracted spatial features are fed into the channel attention module in the trained deep network to extract channel features; The extracted spatial features and channel features are normalized using the log-softmax function; and Before the step of outputting N emoji categories through the last fully connected layer, the following steps are also included: The training and convergence speed of the deep network is accelerated by using Batch Normalization (BN) layers; The loss function of the deep network is the total network loss function obtained by jointly optimizing the softmax loss function, the feature enhancement extraction loss function, and the fusion loss function. The formula for the fusion loss function is as follows: （4） Where N is the data size of each batch. C For the channel size of the attention map, k For the number of attention branches, For the first i The sample at the th j Variance across channels; The formula for the total network loss function is as follows: （5） in, yes Hyperparameters.

2. The facial expression recognition method based on feature enhancement according to claim 1, characterized in that, Before performing data augmentation on each facial expression image in the facial expression image dataset, the method further includes: Each facial expression image in the facial expression image dataset is cropped to obtain the corresponding face image; Each face image is scaled up to the original aspect ratio until the longer side is n pixels long. If the image is not square, after scaling, the shorter side is filled with pixels with a grayscale value of 0 to a length of n pixels to obtain a square image with a side length of n pixels.

3. A facial expression recognition system based on feature enhancement, characterized in that, include: An acquisition unit is used to acquire an expression image dataset, wherein the expression image dataset is an N-class expression image dataset after labeling and classification; The data augmentation unit is used to perform data augmentation on each facial expression image in the facial expression image dataset; The convolutional neural network building unit is used to build a convolutional neural network with ResNet as the backbone network. The feature extraction unit is used to input the data-enhanced facial expression image dataset into the trained convolutional neural network for feature extraction, in order to extract the morphological features, texture features and color features from each data-enhanced facial expression image; The expression category output unit is used to output N expression categories through the last fully connected layer; The loss function of the convolutional neural network is a loss function that is jointly optimized by the softmax loss function and the feature enhancement extraction loss function. The formula for the softmax loss function is as follows: （1） in, , Is d Input in 3D space; m It is a portion of the total sample size; These are the weights of the last fully connected layer; It is a bias parameter; The formula for the feature enhancement extraction loss function is as follows: （2） in, , representing the first depth feature Individual category center from d Random sampling from a Gaussian distribution; , indicating the first i input vectors, X It is the input vector space; , Y It is a tag space; M yes Y The dimension; The standard deviation representing the class center; The formula for the loss function after joint optimization is as follows: （3） in, yes hyperparameters; The aforementioned feature-enhanced facial expression recognition system further includes: Spatial attention module construction unit, used to construct spatial attention modules; Channel attention module construction unit, used to construct channel attention modules; Deep network building unit, used to build deep networks based on spatial and channel attention fusion learning; A spatial feature extraction unit is used to feed the features extracted by the convolutional neural network into the spatial attention module to extract spatial features. The channel feature extraction unit is used to send the extracted spatial features into the channel attention module to extract channel features; The normalization processing unit is used to normalize the extracted spatial features and channel features using the log-softmax function; BN units are used to accelerate the training and convergence speed of the deep network through BN layers; The loss function of the deep network based on spatial and channel attention fusion learning is the total network loss function after jointly optimizing the softmax loss function, the feature enhancement extraction loss function, and the fusion loss function. The formula for the fusion loss function is as follows: （4） Where N is the data size of each batch. C For the channel size of the attention map, k For the number of attention branches, For the first i The sample at the th j Variance across channels; The formula for the total network loss function is as follows: （5） in, yes Hyperparameters.

4. The facial expression recognition system based on feature enhancement according to claim 3, characterized in that, Also includes: The cropping unit is used to crop each facial expression image in the facial expression image dataset to obtain the corresponding face image; The scaling unit is used to scale each face image, scaling it to a length of n pixels along the original aspect ratio. If the image is not square, after scaling, the short side is filled with pixels with a grayscale value of 0 to a length of n pixels to obtain a square image with a side length of n pixels.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the feature-enhanced facial expression recognition method as described in claim 1 or 2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the feature-enhanced facial expression recognition method as described in claim 1 or 2.

Images