Facial expression recognition method, system and electronic equipment

Abstract

The present invention provides a facial expression recognition method, system, and electronic device. The method comprises: preprocessing a facial sample image to extract the coordinates of a user's facial feature points; using a convolutional neural network backbone to obtain local features and global features; using a cross-attention module 1 to fuse the aggregated global features with each local feature to obtain enhanced local features; using a graph convolutional neural network to process the enhanced local features to extract structured semantic information of the user's face; using a cross-attention module 2 to fuse the structured semantics of the user's face into each global feature to obtain a global feature with enhanced structured semantics of the user's face; using a visual self-attention model to further encode the global feature with enhanced structured semantics of the user's face; and using a cross-entropy loss to supervise the entire model training. The present invention utilizes a graph convolutional neural network to enhance the feature representation capability of the visual self-attention model for solving expression recognition tasks in natural scenes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and more specifically, relates to a facial expression recognition method, system and electronic equipment. Background Art

[0002] Facial expression recognition uses computer vision and deep learning technologies to extract and classify facial expression features from images or videos. Facial expressions are one of the primary ways humans express their emotions. Through expression recognition, we can understand a person's underlying emotional state, including happiness, sadness, disgust, anger, and so on. In recent years, facial expression recognition has seen unprecedented exploration and application in multiple fields. For example, in the field of human-computer interaction, emotional interaction plays a particularly important role. Systems can identify user emotional fluctuations and make decisions about the next interaction, thereby achieving a more humane and intelligent interactive experience. Furthermore, facial expression recognition is widely used for emotion monitoring. By analyzing changes in facial expressions in real time, it is possible to identify emotional changes over a specific time period, providing important insights for measuring mental health and diagnosing physical illnesses. This has also promoted the development of automated early detection of mental illnesses and intelligent psychological counseling and intervention.

[0003] Graph convolutional neural networks are an effective approach for addressing the challenges of facial expression recognition. By treating facial landmarks as nodes and connecting them in order of facial parts, a geometric facial landmark graph can be constructed. By performing graph convolution on this geometric facial landmark graph, information from neighboring nodes can be aggregated, enabling geometric perception of each node and improving the node's semantic expressiveness. Compared to traditional convolutional neural networks, the visual self-attention model, through its self-attention mechanism, can directly aggregate information globally to capture the global information of an image. This is more accurate and efficient in establishing long-range relationships. Furthermore, in expression recognition, the multi-head mechanism within the self-attention model not only reduces time-consuming matrix operations through parallel matrix operations but also allows for feature encoding. By learning global semantic associations, the model can capture richer and more complex features.

[0004] Although visual self-attention models have made significant progress in facial expression recognition, they still face numerous challenges. First, they cannot explicitly explore the geometric structured semantics of the face and cannot extract geometric correlation representations that aid in facial expression recognition. Second, they do not explore the semantic relationships between local and global features, particularly the local and global semantic relationships associated with facial features and the density of facial expressions. Furthermore, graph convolutional neural networks, due to their artificially defined and fixed graph structures, are unable to explore high-level semantic relationships between nodes and are also unable to adaptively establish new node connections. For these reasons, existing methods still lack robust and discriminative feature extraction capabilities. Summary of the Invention

[0005] In view of the defects of the prior art, the purpose of the present invention is to provide a facial expression recognition method, system and electronic equipment, aiming to solve the problem of poor performance of existing facial expression recognition.

[0006] To achieve the above object, in a first aspect, the present invention provides a facial expression recognition method, comprising the following steps:

[0007] Obtain the user's facial image and extract the user's facial feature points;

[0008] The convolutional neural network extracts shallow and deep features of the user's facial image. The corresponding local feature blocks are cropped from the shallow features with each facial feature point as the center to obtain multiple local features of the facial image. The deep features are then segmented and projected to obtain multiple global features of the facial image.

[0009] The similarity between the global feature and each local feature is used as the corresponding attention weight, and then the multiple global features are adaptively fused into each local feature based on the corresponding attention weight to obtain multiple enhanced local features;

[0010] Based on the graph convolutional neural network combined with multiple enhanced local features, the feature point map corresponding to the facial feature points is convolved to extract multiple structured semantic features of the user's face;

[0011] The similarity between the structured semantic features and the global features is used as the corresponding attention weight. Then, multiple structured semantic features are fused into each global feature based on the corresponding attention weights. The regions with high correlation with facial expression recognition in the global features are enhanced to obtain enhanced global features.

[0012] The user's facial expression is obtained by further aggregating and enhancing the global feature recognition based on the visual self-attention model.

[0013] In a possible implementation, shallow features and deep features of the user's facial image are extracted through a ResNet-18 network or an IR-50 network.

[0014] In a possible implementation, a local sub-block of a preset size is cut out from the shallow features with each user's facial feature point as the center. Among them, N represents the number of user face feature points, C represents the number of channels, and H local and W local Represents the size of the cropped local sub-block; finally, after feature flattening and feature mapping, the final local feature is obtained d local The dimension of the feature vector representing the local features.

[0015] Assume that the deep features of the convolutional neural network output The deep features are evenly divided and then projected through the fully connected layer to obtain the global features. in H local and W local Represents the size of deep features, h and w represent the height and width of each sub-block, d global Represents the number of channels of each global feature after projection.

[0016] In a possible implementation, the enhanced local feature X′ local for:

[0017]

[0018]

[0019] Among them, d is the channel dimension of the feature vector, Softmax(·) is the activation function used to speed up training convergence, X local is a local feature, X global is the global feature, w represents the projection of the feature using the fully connected layer, Represents matrix multiplication, G2L Attention is the attention weight of global features relative to local features.

[0020] In a possible implementation, the structured semantic feature Z local Obtained through the following steps:

[0021] The geometric prior knowledge of the user's facial feature points is used to construct a graph model G = (v, e), where v represents a node and e represents an edge. Each feature point sub-block x i ∈v is a node in the graph model, and the edge a between any two nodes ij ∈e is initialized using the geometric graph of the user's facial feature points to obtain the adjacency matrix Aij ;

[0022] The enhanced local feature X′ local and the adjacency matrix A ij Input into the graph convolutional neural network, the enhanced local features are extracted to obtain the structured semantic features Z of the face local The adjacency matrix is ​​a learnable parameter of the graph convolutional neural network, so that the graph convolutional neural network can adaptively find the association between local areas of the user's face according to the target task, reduce the connection between unrelated areas, and add edge connections to local areas with semantic associations to extract corresponding structured semantic features.

[0023] In a possible implementation, the enhanced global feature X′ global for:

[0024]

[0025]

[0026] Among them, d is the channel dimension of the feature vector, Softmax(·) is the activation function used to speed up training convergence, and Z local is a structured semantic feature, X global is the global feature, w represents the projection of the feature using the fully connected layer, Represents matrix multiplication, L2G Attention is the attention weight of structured semantic features relative to global features.

[0027] In one possible implementation, the visual self-attention model consists of M encoders, each encoder including: a multi-head self-attention MSA with long-hop connections and a multi-layer perceptron MLP;

[0028] The global features are further aggregated and enhanced based on the visual self-attention model, specifically:

[0029] MAS operation part: First, the enhanced global features The linear projection is query q, keyword k and value v as follows:

[0030] [q, k, v] = X′ global [w q , w k , w v ]

[0031] in, Explore the similarity between features by projecting them into subspaces through fully connected layers; d k and d v is the number of channels after projection;

[0032] Secondly, the self-attention weight Z is calculated based on the linear projection result global , and calculate the output Z′ by weighted summation of all values global :

[0033]

[0034] Z′ global =Z global v

[0035] MLP operation part: Z′ global Input into the multi-layer perceptron, which consists of a two-layer prior neural network and a ReLU activation function. Through linear mapping and nonlinear activation, the feature Z′ is further improved. global semantics.

[0036] In a possible implementation, the user's facial expression is recognized, specifically:

[0037] The features output by the visual self-attention model are input into the fully connected layer for mapping, and mapped to the output head of the corresponding expression category. Finally, after normalization by the softmax function, the probability distribution of each facial expression category is obtained.

[0038] In a second aspect, the present invention provides a facial expression recognition system, characterized in that it includes:

[0039] A facial feature point extraction unit, used to obtain a user's facial image and extract the user's facial feature points;

[0040] A facial feature determination unit is used to extract shallow and deep features of the user's facial image based on a convolutional neural network, crop corresponding local feature blocks from the shallow features with each facial feature point as the center to obtain multiple local features of the facial image, and divide and project the deep features to obtain multiple global features of the facial image;

[0041] A facial feature enhancement unit is configured to use the similarity between the global feature and each local feature as the corresponding attention weight, and then adaptively fuse multiple global features into each local feature in combination with the corresponding attention weight to obtain multiple enhanced local features; perform graph convolution on the feature point graph corresponding to the facial feature points based on the graph convolutional neural network in combination with the multiple enhanced local features to extract multiple structured semantic features of the user's face; and use the similarity between the structured semantic feature and the global feature as the corresponding attention weight, and then fuse the multiple structured semantic features into each global feature in combination with the corresponding attention weight, thereby enhancing the areas with a high correlation with facial expression recognition in the global features to obtain enhanced global features;

[0042] The facial expression recognition unit is used to obtain the user's facial expression by further aggregating and enhancing the global feature recognition based on the visual self-attention model.

[0043] In a third aspect, the present invention provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0045] In a fifth aspect, the present invention provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0046] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0047] The present invention provides a facial expression recognition method, system and electronic device, which fuses the facial structured semantic information extracted from a graph convolutional neural network into the embedding of a visual self-attention model, thereby improving the representation ability of the visual self-attention model. The present invention uses the prior information of facial feature points to construct a topological map, and extracts the facial structured semantic features by aggregating local areas rich in facial expression information using a graph convolutional neural network; it makes up for the lack of inductive bias of the visual self-attention model for local areas of the face, and improves the representation ability of global features. The present invention proposes a bidirectional coupling structure of a graph convolutional neural network and a visual self-attention model, in which global features improve the robustness of local features and promote the exploration of correlations between local areas; local features supplement the identifiable information lacking in global features, improve the representation ability, and improve the recognition accuracy of facial expressions.

[0048] The present invention provides a facial expression recognition method, system, and electronic device. Local features guided by facial feature points are combined with global features provided by a visual self-attention model encoder as input to a graph convolutional neural network (GCNN). This approach aims to mitigate noise interference caused by local features, thereby improving the robustness of facial structural semantics. The present invention dynamically updates the GCNN's adjacency matrix through pre-training initialization and regularization loss during training. While maintaining the inherent structured relationships between facial parts, the weights in the adjacency matrix are dynamically updated using a gradient descent algorithm, thereby improving the GCNN's adaptability to each encoder block and enhancing the accuracy of facial expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a facial expression recognition method provided by an embodiment of the present invention;

[0050] Figure 2 This is a flow chart of a facial expression recognition method using an enhanced visual self-attention model provided by an embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of a graph model provided by an example of the present invention;

[0052] Figure 4 Schematic diagram of the facial expression recognition model provided by an embodiment of the present invention;

[0053] Figure 5 This is a diagram of the facial expression recognition system architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] The present invention provides a facial expression recognition method, system, and electronic device, belonging to the field of computer vision technology. The method comprises: inputting a pre-recognized facial image into a visual self-attention model enhanced by a graph convolutional neural network for facial expression recognition; a training method comprising: preprocessing a facial sample image to extract the coordinates of facial feature points; employing a convolutional neural network backbone to obtain local and global features; employing a cross-attention module 1 to fuse the aggregated global features with each local feature to obtain enhanced local features; employing a graph convolutional neural network to process the enhanced local features to extract structural semantic information of the face; employing a cross-attention module 2 to fuse the structural semantics of the face into each global feature to obtain a global feature enhanced with structural semantics of the face; employing a visual self-attention model to further encode the global feature enhanced with structural semantics of the face; and employing a cross-entropy loss to supervise the entire model training. The present invention utilizes a graph convolutional neural network to enhance the feature representation capability of the visual self-attention model, and is used to solve the task of facial expression recognition in natural scenes.

[0056] Figure 1 : is a flow chart of a facial expression recognition method provided by an embodiment of the present invention; Figure 1 As shown, the following steps are included:

[0057] S11, obtaining a user's facial image and extracting the user's facial feature points;

[0058] S12, extracting shallow features and deep features of the user's facial image based on a convolutional neural network, cropping corresponding local feature blocks from the shallow features with each facial feature point as the center to obtain multiple local features of the facial image, and then segmenting and projecting the deep features to obtain multiple global features of the facial image;

[0059] S13, taking the similarity between the global feature and each local feature as the corresponding attention weight, and then adaptively fusing multiple global features into each local feature based on the corresponding attention weight to obtain multiple enhanced local features;

[0060] S14, based on the graph convolutional neural network combined with multiple enhanced local features, the feature point map corresponding to the facial feature points is convolved to extract multiple structured semantic features of the user's face;

[0061] S15, using the similarity between the structured semantic feature and the global feature as the corresponding attention weight, and then fusing the multiple structured semantic features into each global feature based on the corresponding attention weight, enhancing the area with a high correlation with facial expression recognition in the global feature, and obtaining an enhanced global feature;

[0062] S16, based on the visual self-attention model, the global feature recognition is further aggregated and enhanced to obtain the user's facial expression.

[0063] Specifically, the facial sample image is preprocessed to extract the user's facial feature points; the convolutional neural network extracts local features and global features; the global features are fused into the local features through the cross-attention module 1 to obtain enhanced local features; a graph convolutional neural network is constructed to extract the user's facial structured semantics; the local features are fused into the global features through the cross-attention module 2 to obtain global features with enhanced structured semantics of the user's face; a visual self-attention model is constructed to encode the global features, and after the features are input into the fully connected layer for classification, the category probability output of each sample is obtained to complete the user's facial expression recognition task.

[0064] Furthermore, the user facial feature points refer to a key point feature set containing key point position information, which is obtained by passing the preprocessed user facial image through a pre-trained user facial feature detector and performing normalization and size enlargement.

[0065] like Figure 2 As shown, an embodiment of the present invention provides a facial expression recognition method based on an enhanced visual self-attention model, comprising the steps of:

[0066] S101: Preprocess the dataset samples and extract the user's facial feature points.

[0067] With the acquisition of natural scene datasets, facial expression recognition faces two significant challenges. First, intra-class variability. Even if two facial images are labeled with the same expression category, it can be difficult for the human eye to accurately distinguish them. Image blur, lighting variations, head pose changes, and partial occlusions can cause blurred or incomplete facial information. Furthermore, due to individual differences in race, age, gender, and facial features, even the same expression can be expressed differently. Second, inter-class similarity. Facial images of users with different expression categories may exhibit only subtle differences, making accurate distinction difficult. For example, the expressions of anger and surprise can exhibit similar facial features, resulting in similar facial features such as wide-open eyes or an open mouth. To address these two issues, it's necessary to consider more than just the global appearance of the entire facial image. It's also necessary to focus on fine-grained local regions. For example, nearby facial regions often change appearance with changing expressions. Taking these regions into account often facilitates expression recognition.

[0068] Furthermore, the dataset samples were scaled to 112x112 pixels and then fed into a pre-trained facial landmark detector, MobileFaceNet. This scaled sample was fed into the MobileFaceNet detector to obtain the normalized locations of 51 facial landmarks. This landmark location information was then scaled up to the sample size to determine the location of each facial landmark in the sample image.

[0069] S102: Extract shallow features and deep features of each sample in the data sample set, and further process them to obtain corresponding local features and global features.

[0070] Furthermore, the feature extraction of the sample preferably adopts one of the following two methods;

[0071] a. Feature extraction method using ResNet-18 network

[0072] ResNet-18 is a convolutional neural network architecture that addresses the vanishing gradient problem in deep neural networks, allowing deeper networks to be trained without performance degradation. The ResNet-18 architecture consists of 18 layers, including 17 convolutional layers and one fully connected layer. It follows a building block structure that incorporates residual connections to enable information to flow from one layer to another, skipping some layers. This helps address the vanishing gradient problem and enables the network to learn more effective representations.

[0073] The ResNet-18 network structure is divided into the following parts: First, the input layer receives the pixel values ​​of the input image. This layer then passes through a convolutional layer with a kernel size of 7x7 and a stride of 2 to reduce the size of the input image. This layer has 64 filters and performs batch normalization and ReLU activation. This layer is followed by a 3x3 max pooling layer with a stride of 2 to further reduce the size of the feature map. The core of ResNet-18 lies in the design of the residual block. Each residual block consists of two convolutional layers with 3x3 filters, batch normalization, and ReLU activation. The outputs of these two convolutional layers are added to the original input of the residual block to create a shortcut connection. This addition operation is called a "skip connection" or "identity shortcut." The purpose of the skip connection is to ensure that information from earlier layers is preserved and can flow directly to later layers, making it easier for the network to learn the residual mapping.

[0074] The formula is:

[0075]

[0076] F(x)=H(x)+x

[0077] Among them, conv1(x) and conv2(x) both represent convolution operations; BN(x) represents batch normalization, which is used to normalize the mean and variance of each feature map; Relu(x) represents the rectified linear unit activation function, which sets negative values ​​to 0 and keeps positive values ​​unchanged; H(x) represents the residual map, and F(x) represents the entire operation process of the residual block.

[0078] Furthermore, a ResNet-18 network, pre-trained on the public user face dataset Ms-Celeb-1M, can be used to transfer the model to the user facial expression recognition task. Since the model parameters are already well-prepared, the training process only requires fine-tuning to extract better features. When the sample image is input into the ResNet-18 network, shallow features of size 64*32*32 and deep features of size 512*4*4 are extracted from the first and last residual blocks of the network, respectively.

[0079] b. Feature extraction method using IR-50 network

[0080] In this embodiment's feature extraction method, the network used was still pre-trained on the Ms-Celeb-1M dataset. The IR-50 network structure is similar to that of ResNet-50, with some modifications before the final fully connected layer. The first layer accepts the user's facial image as input; the next layer is the convolutional layer, typically consisting of 64 3x3 filters with a stride of 1, which extract low-level features from the image. IR-50 also features residual blocks, similar in structure to ResNet-50, to address the vanishing gradient problem and help the network learn better feature representations. However, the IR-50 residual block consists of three convolutional layers. The first convolutional layer uses a 1x1 convolution kernel to reduce the number of channels, typically to reduce computational overhead. The second convolutional layer then uses a 3x3 convolution kernel for feature extraction. Finally, the third convolutional layer uses a 1x1 convolution kernel to restore the number of channels. A long-skip connection is added between the second and third convolutional layers to enable direct information transfer regardless of the increased number of layers. Shallow features and deep features are still extracted from the first residual block and the last residual block of the network.

[0081] The features obtained by the above two feature extraction methods can both be used as the feature vectors of the samples in this embodiment.

[0082] After extracting shallow and deep features, they need to be further processed to become acceptable inputs for graph convolutional neural networks and visual self-attention models. First, using the user's facial feature points extracted by S1O1, a 5*5 size sub-block is cropped on the shallow features with 51 user facial feature points as the center. At this time, a local sub-block is obtained. Where N represents the number of user facial feature points, C represents the number of channels, and H local and W local Represents the size of the cropped local sub-block. Finally, after feature flattening and feature mapping, the final local feature is obtained where d local The dimension of the feature vector representing the local features.

[0083] Specifically, the above feature flattening and feature mapping process should be C×H local ×W local First flatten it into one dimension, N×(C×H local ×W local ), and then through a fully connected layer mapping, C×H local ×W local Mapped to d local .

[0084] Further extract the output of the last layer of the convolutional neural network And evenly divide it, and then project it through the fully connected layer to obtain the global feature in h and w represent the height and width of each sub-block, d global Represents the number of channels of each global feature after projection.

[0085] S103: Global features are fused with local features through cross attention to enhance local features.

[0086] Because local features are located and cropped based on the user's facial landmarks, facial landmark detection in natural scenes is often inaccurate, and the local features extracted in the previous step may contain noise. To improve the robustness of local features, global features are fused with local features through a cross-attention mechanism to enhance the representational power of local features.

[0087] At the same time, each graph convolutional neural network is embedded in front of each encoder in the visual self-attention model, and each graph convolutional neural network should be adapted to the corresponding encoder. In order to enhance the semantics of local features aggregated by each graph convolutional neural network and mine potential facial associations, the attention mechanism G2L is proposed. Attention , the global feature X global Fusion into local features X local Since each local feature has unequal similarity with all global features, the attention mechanism is used to explore the high semantic similarity between the global features and each local feature to obtain the enhanced local feature X′ local , the formula is as follows:

[0088]

[0089]

[0090] Among them, d is the channel dimension of the feature vector, the channel dimension of the global feature and the local feature are the same, and the scaling factor In order to normalize the attention, w means that the feature is projected using the fully connected layer. Represents matrix multiplication.

[0091] Specifically, there are multiple global features and local features. For example, if there are A and B, the attention is calculated by weighting the attention of multiple A's with each B, and then each B is fused with the multiple A's through the attention weight (mechanism). Therefore, each global feature will be calculated with the attention of multiple local features, and each global feature can pay different attention to local features in different regions.

[0092] Through the cross-attention mechanism, each local feature searches for the area in the global face that is most relevant to itself for fusion, thereby enhancing the semantic representation of local features and reducing the impact of noise.

[0093] S104: Graph convolutional neural network extracts structured semantic features of the user's face.

[0094] Graph convolutional neural networks (GCNNs) are an effective approach for addressing the problem of facial expression recognition. By treating facial landmarks as nodes and connecting them in order of facial parts, a geometric facial landmark graph can be constructed. By performing graph convolution on this geometric facial landmark graph, information from neighboring nodes is aggregated, enabling geometric perception of each node and improving the node's semantic representation. Furthermore, because each node's neighbors all belong to the same facial part, GCNNs can suppress noise introduced by inaccurate landmark location, improving the node's ability to represent facial part features. In a user's facial expression, multiple facial parts often change simultaneously. To better explore the potential relationships between facial parts, the graph can be defined as learnable.

[0095] In this way, the graph convolutional neural network can adaptively explore the relationships between different facial parts based on input node features to obtain better geometrically perceived facial features. This graph convolutional neural network-based approach effectively extracts association information between facial parts and is robust to inaccurate feature point positioning. It can better capture subtle changes and local features in a user's facial expression, thereby improving the performance of facial expression recognition.

[0096] like Figure 3 As shown, the geometric prior knowledge of the user's facial feature points is used to construct the graph model G = (v, e), and each feature point sub-block x i∈v is a node in the graph, and the edge a between any two nodes ij ∈e is initialized using the geometric user facial feature point map, such as Figure 3 As shown, s(·) is the graph initialization function, and its formula is expressed as:

[0097]

[0098] The enhanced local features are aggregated using a geometric user facial feature map to obtain structured semantic features of the user's face, while also improving the robustness and semantic representation of each enhanced local feature. Furthermore, the entire adjacency matrix is ​​set as a learnable parameter and updated using the same gradient descent method as the network parameters. This allows the graphical model to adaptively identify associations between local regions of the user's face based on the target task, reducing connections between unrelated regions and adding edges to semantically related local regions.

[0099] The graph convolutional neural network is set to two layers, one of which is the enhanced local feature X′. local , the other input is the adjacency matrix A ij , through the graph model aggregation enhanced local features to extract the user's facial structured semantics Z local At the same time, inputting the global features extracted by the visual self-attention model into the graph convolutional neural network can help the network explore the connections between different facial regions while using global features as an intermediate point to strengthen the connections between different local regions. Aggregating enhanced local features through semantic relationships makes the structured semantic features of the user's face more abstract, further enhancing the semantics of the structured semantic features of the user's face.

[0100] S105: The structured semantic features of the user's face are fused with the global features through cross attention to obtain enhanced global features.

[0101] The present invention utilizes the attention mechanism to fuse the structured semantic features of the user's face into the global features, providing context for the global features. The structured semantic features of the user's face complement each other, and the fused features have stronger representation capabilities and improved the generalization ability of the model.

[0102] Specifically, the attention mechanism L2G Attention A method similar to the aforementioned attention mechanism is used to calculate the similarity between each global feature and the structured semantic features of all users' faces. Since an expression is often closely related to a certain part of the face, the attention mechanism is used to fully explore and enhance the local areas that are discriminative for expression recognition. The specific formula is as follows:

[0103]

[0104]

[0105] The Softmax(·) function is used to normalize the attention and speed up the convergence of training.

[0106] Although the self-attention model has a good ability to capture long-range dependencies, due to the lack of inductive bias in the adaptive calculation of the self-attention mechanism and the absence of position information, its ability to extract the structured semantic features of the user's face is relatively weak. This paper uses a graph convolutional neural network to fuse the structured semantic features of the user's face with global features. Taking into account the connection between the structured semantic features of the user's face and the global features, it further enhances the semantics of the global features and helps the visual self-attention model incorporate the connection between the structured semantic features of the user's face when calculating self-attention.

[0107] S106: Visual self-attention model extracts features with excellent representation capabilities.

[0108] For specific feature fusion enhancement models, see Figure 4 Compared to traditional convolutional neural networks, the visual self-attention model, through its self-attention mechanism, can directly aggregate information globally to capture the global information of an image. This makes it more accurate and efficient in establishing long-range relationships. Secondly, the multi-head mechanism in the self-attention model not only reduces the time required for parallel matrix operations but also allows for feature representation. These self-attention modules can learn global and local relationships in an image, enabling the model to capture richer and more complex features.

[0109] Because the spatial information between each sub-block is lost when slicing, a spatial encoding is initialized and directly added to the global features to introduce spatial information. Furthermore, to facilitate feature classification, a classification vector is introduced and concatenated with the global features to learn expression features from other global feature vectors.

[0110] Furthermore, the attention mechanism mentioned above is used to integrate the user's facial structured features into the global features. This enhanced global feature is then input into an encoder of the visual self-attention model. The visual self-attention model consists of M encoders. Each encoder is composed of a multi-head self-attention (MSA) with long-hop connections and a multi-layer perceptron (MLP).

[0111] First, enter The linear projection is query q, keyword k and value v as follows:

[0112] [q, k, v] = X′ global [w q , w k , w v ]

[0113] in, The similarity between features is explored by projecting into the subspace through the fully connected layer.

[0114] Second, the self-attention weights are calculated and the output is calculated by weighted summation of all values:

[0115]

[0116] Z′ global =Z global v

[0117] Multi-head self-attention involves performing the above self-attention mechanism k times in parallel, embedding the concatenated features through a fully connected layer to form the final output. A multi-layer perceptron consists of two fully connected layers for feature projection and a Reluctant Unit (RELU) activation function for nonlinearity. The multi-head self-attention mechanism is designed to embed projections into their own subspaces, adaptively calculating and aggregating feature similarities within each subspace to increase feature diversity. It also further focuses on inter-feature correlations from multiple perspectives, enhancing model robustness.

[0118] The total loss of the entire model Specifically, it can be expressed as:

[0119]

[0120] in, It is a cross entropy loss commonly used in classification tasks. It is used to measure the difference between the model's prediction results and the true labels in classification tasks. The specific formula is as follows:

[0121]

[0122] in, represents the sample label distribution predicted by the fully connected layer after the last encoder output of the visual self-attention model; i represents the true distribution of the sample.

[0123] Furthermore, the present invention designs a graph loss function It is used to ensure that the dynamically updated graph can still maintain the original geometric user facial feature point connection relationship to a certain extent, ensuring the effectiveness of the user facial structured features extracted by the graph convolutional neural network. Its loss function is expressed as follows:

[0124]

[0125] Among them, A initial represents the adjacency matrix initialized according to the geometric user facial feature point graph, A represents the dynamically updated adjacency matrix, and * represents the Hadamard product.

[0126] In one embodiment, the RAF-DB (Real-world Affective Faces Database) expression database was used. The database contains 29,672 facial images collected from the Internet, and the expressions were annotated by 315 staff members (university students and faculty members). It contains a total of 7 expressions: anger, disgust, fear, happiness, sadness, surprise, and neutral.

[0127] The present invention selected all facial expression images and conducted experiments based on the training and test sets divided into the expression library. When the ResNet-18 network was used as the convolutional neural network backbone, the expression recognition accuracy obtained was 89.93%; when the IR-50 network was used as the convolutional neural network backbone, the expression recognition accuracy obtained was 92.24%.

[0128] In another embodiment, the FER+ (Hard-Label) expression library is used; this expression library is an extension of the original FER dataset, in which facial expression images are re-labeled into one of eight emotion types: neutral, happy, surprised, sad, angry, disgusted, fearful, and contemptuous.

[0129] The present invention selected all facial expression images in the data set for training; when the ResNet-18 network was used as the convolutional neural network backbone, the expression recognition accuracy obtained was 90.25%; when the IR-50 network was used as the convolutional neural network backbone, the expression recognition accuracy obtained was 91.92%.

[0130] Through research and comparison, the TransFER method achieved an accuracy of 90.91% on the RAF-DB dataset and 90.83% on the FER+ dataset. Meanwhile, the POSTER method achieved an accuracy of 92.05% on the RAF-DB dataset and 91.62% on the FER+ dataset. Compared to these state-of-the-art methods, the proposed method achieves superior performance, achieving the highest accuracy results to date.

[0131] The facial expression recognition results of this invention on two datasets achieve higher performance than recent facial expression recognition methods based on visual self-attention models. Even when compared with corresponding methods using different convolutional neural network backbones, the recognition accuracy remains the highest to date. In summary, the method proposed by this invention is currently the state-of-the-art method for facial expression recognition, surpassing a range of cutting-edge methods such as TransFER and POSTER.

[0132] Figure 5 : is a diagram of the facial expression recognition system architecture provided by an embodiment of the present invention; Figure 5 Shown, including:

[0133] The facial feature point extraction unit 510 is used to obtain a user's facial image and extract the user's facial feature points;

[0134] Facial feature determination unit 520 is used to extract shallow features and deep features of the user's facial image based on a convolutional neural network, crop corresponding local feature blocks from the shallow features with each facial feature point as the center to obtain multiple local features of the facial image, and segment and project the deep features to obtain multiple global features of the facial image;

[0135] Facial feature enhancement unit 530 is used to use the similarity between the global feature and each local feature as the corresponding attention weight, and then adaptively fuse multiple global features into each local feature in combination with the corresponding attention weight to obtain multiple enhanced local features; based on the graph convolutional neural network, perform graph convolution on the feature point graph corresponding to the facial feature points in combination with the multiple enhanced local features to extract multiple structured semantic features of the user's face; and use the similarity between the structured semantic feature and the global feature as the corresponding attention weight, and then fuse the multiple structured semantic features into each global feature in combination with the corresponding attention weight, thereby enhancing the areas with high correlation with facial expression recognition in the global features to obtain enhanced global features;

[0136] The facial expression recognition unit 540 is used to obtain the user's facial expression by further aggregating and enhancing the global feature recognition based on the visual self-attention model.

[0137] It should be noted that the detailed functional implementation of each of the above units can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0138] It should be understood that the above-mentioned system is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program unit in the system are similar to those described in the above-mentioned method. The working process of the system can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0139] Based on the methods described in the above embodiments, embodiments of the present invention provide an electronic device. The device may include at least one memory for storing programs and at least one processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is configured to execute the methods described in the above embodiments.

[0140] Based on the method in the above embodiment, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0141] Based on the method in the above embodiment, an embodiment of the present invention provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0142] It is understood that the processor in the embodiments of the present invention may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0143] The method steps in the embodiments of the present invention can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.

[0144] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0145] It should be understood that the various numerical numbers involved in the embodiments of the present invention are only used for the convenience of description and are not intended to limit the scope of the embodiments of the present invention.

[0146] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A facial expression recognition method, characterized in that: The following steps are involved: Obtain the user's facial image and extract the user's facial feature points; The convolutional neural network extracts shallow and deep features of the user's facial image. The corresponding local feature blocks are cropped from the shallow features with each facial feature point as the center to obtain multiple local features of the facial image. The deep features are then segmented and projected to obtain multiple global features of the facial image. The similarity between the global feature and each local feature is used as the corresponding attention weight, and then the multiple global features are adaptively fused into each local feature based on the corresponding attention weight to obtain multiple enhanced local features; Based on the graph convolutional neural network combined with multiple enhanced local features, the feature point map corresponding to the facial feature points is convolved to extract multiple structured semantic features of the user's face; The similarity between the structured semantic features and the global features is used as the corresponding attention weight. Then, multiple structured semantic features are fused into each global feature based on the corresponding attention weights. The regions with high correlation with facial expression recognition in the global features are enhanced to obtain enhanced global features. The user's facial expression is obtained by further aggregating and enhancing the global feature recognition based on the visual self-attention model.

2. The method according to claim 1, characterized in that A local sub-block of a preset size is cropped from the shallow features with each user's facial feature point as the center Among them, N represents the number of user face feature points, C represents the number of channels, and H local and W local Represents the size of the cropped local sub-block; finally, after feature flattening and feature mapping, the final local feature is obtained d local The dimension of the feature vector representing the local features; Assume that the deep features of the convolutional neural network output The deep features are evenly divided and then projected through the fully connected layer to obtain the global features. in H local and W local Represents the size of deep features, h and w represent the height and width of each sub-block, d global Represents the number of channels of each global feature after projection.

3. The method according to claim 1, characterized in that The enhanced local feature X l ′ ocal for: Among them, d is the channel dimension of the feature vector, Softmax(·) is the activation function used to speed up training convergence, X local is a local feature, X global is the global feature, w represents the projection of the feature using the fully connected layer, Represents matrix multiplication, G2L Attention is the attention weight of global features relative to local features.

4. The method according to claim 1 or 3, characterized in that The structured semantic feature Z local Obtained through the following steps: The geometric prior knowledge of the user's facial feature points is used to construct a graph model G = (v, e), where v represents a node and e represents an edge. Each feature point sub-block x i ∈v is a node in the graph model, and the edge a between any two nodes ij ∈e is initialized using the geometric graph of the user's facial feature points to obtain the adjacency matrix A ij ; The enhanced local feature X′ local and the adjacency matrix A ij Input into the graph convolutional neural network, the enhanced local features are extracted to obtain the structured semantic features Z of the face local The adjacency matrix is ​​a learnable parameter of the graph convolutional neural network, so that the graph convolutional neural network can adaptively find the association between local areas of the user's face according to the target task, reduce the connection between unrelated areas, and add edge connections to local areas with semantic associations to extract corresponding structured semantic features.

5. The method according to claim 1, characterized in that The enhanced global feature X′ global for: Among them, d is the channel dimension of the feature vector, Softmax(·) is the activation function used to speed up training convergence, and Z local is a structured semantic feature, X global is the global feature, w represents the projection of the feature using the fully connected layer, Represents matrix multiplication, L2G Attention is the attention weight of structured semantic features relative to global features.

6. The method according to claim 1, characterized in that The visual self-attention model consists of M encoders, each of which includes: a multi-head self-attention MSA with long-skip connections and a multi-layer perceptron MLP; The global features are further aggregated and enhanced based on the visual self-attention model, specifically: MAS operation part: First, the enhanced global features The linear projection is query q, keyword k and value v as follows: [q,k,v]=X′ global [w q ,w k ,w v ] in, Explore the similarity between features by projecting them into subspaces through fully connected layers; d k and d v is the number of channels after projection; Secondly, the self-attention weight Z is calculated based on the linear projection result global , and calculate the output Z′ by weighted summation of all values global : Z′ global =Z global in MLP operation part: Z′ global Input into the multi-layer perceptron, which consists of a two-layer prior neural network and a ReLU activation function. Through linear mapping and nonlinear activation, the feature Z′ is further improved. global semantics.

7. The method according to claim 1 or 6, characterized in that Recognize the user's facial expressions, specifically: The features output by the visual self-attention model are input into the fully connected layer for mapping, and mapped to the output head of the corresponding expression category. Finally, after normalization by the softmax function, the probability distribution of each facial expression category is obtained.

8. A facial expression recognition system, characterized in that: include: A facial feature point extraction unit, used to obtain a user's facial image and extract the user's facial feature points; A facial feature determination unit is used to extract shallow and deep features of the user's facial image based on a convolutional neural network, crop corresponding local feature blocks from the shallow features with each facial feature point as the center to obtain multiple local features of the facial image, and divide and project the deep features to obtain multiple global features of the facial image; A facial feature enhancement unit is configured to use the similarity between the global feature and each local feature as the corresponding attention weight, and then adaptively fuse multiple global features into each local feature in combination with the corresponding attention weight to obtain multiple enhanced local features; perform graph convolution on the feature point graph corresponding to the facial feature points based on the graph convolutional neural network in combination with the multiple enhanced local features to extract multiple structured semantic features of the user's face; and use the similarity between the structured semantic feature and the global feature as the corresponding attention weight, and then fuse the multiple structured semantic features into each global feature in combination with the corresponding attention weight, thereby enhancing the areas with a high correlation with facial expression recognition in the global features to obtain enhanced global features; The facial expression recognition unit is used to obtain the user's facial expression by further aggregating and enhancing the global feature recognition based on the visual self-attention model.

9. An electronic device, characterized in that: include: at least one memory for storing a program; At least one processor is configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is run on a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images