Facial emotion recognition method and device

Abstract

The application discloses a face emotion recognition method and device, and relates to the technical field of emotion recognition. The method comprises the following steps: acquiring human face image data; constructing a face emotion recognition model; extracting a deep feature map of the human face image data; extracting human face global semantic features from the deep feature map by using a CNN branch; reconstructing the deep feature map into a human face graph structure by using a graph construction layer of an improved GCN branch; performing twice neighborhood information aggregation on the human face graph structure by using a first graph convolution layer and a second graph convolution layer, so as to obtain face local topological features; after dimension alignment of the human face graph structure and the face local topological features, adding the human face graph structure and the face local topological features, so as to obtain face topological global features; after splicing the human face global semantic features and the face topological global features, performing multi-branch feature interaction, so as to obtain human face fusion features; and based on the human face fusion features, using a classifier to classify human face emotions.

Description

Technical Field

[0001] This application relates to the field of emotion recognition technology, and in particular to a facial emotion recognition method and apparatus. Background Technology

[0002] Facial expressions are an indispensable part of daily human communication and social interaction. We can convey rich emotional information through facial expressions, including happiness, anger, sadness, surprise, and fear. Research by psychologist Mehrabia indicates that human facial expressions carry over 55% of emotional information transmission, making accurate expression recognition crucial for understanding another person's psychological state, intentions, and needs. In recent years, with breakthroughs in artificial intelligence technology, Facial Expression Recognition (FER) has shown broad application prospects in fields such as healthcare (e.g., emotion monitoring of autistic patients), intelligent security (high-risk personnel early warning at airports), and virtual reality (immersive interactive experiences).

[0003] In existing technologies, commonly used convolutional neural network models in the field of facial expression recognition include VGG and ResNet. Both extract features from edge texture to high-level semantics step by step through multi-layer convolution stacking (VGG's consecutive 3×3 convolutions and ResNet's residual blocks), demonstrating powerful hierarchical feature extraction capabilities.

[0004] However, while existing convolutional neural networks can effectively extract local features, their local receptive field mechanism makes it difficult to model the spatial relationships between different regions of the face. Furthermore, the pooling operation weakens spatial location information during dimensionality reduction, causing subtle differences in facial expressions to be smoothed out or lost. As a result, it is difficult to accurately capture the spatial relationships of facial muscle movements, leading to inaccurate facial emotion recognition results. Summary of the Invention

[0005] Therefore, it is necessary to provide a facial emotion recognition method and device to address the aforementioned technical problems.

[0006] The present invention adopts the following technical solution: This invention provides a facial emotion recognition method, comprising: Acquire human facial image data; A facial emotion recognition model based on CNN and improved GCN is constructed. The facial emotion recognition model includes: a cascaded feature extraction layer, a feature processing layer, a feature fusion layer, and a classifier. The feature processing layer includes a parallel CNN branch and an improved GCN branch. The improved GCN branch includes: a cascaded graph construction layer, a first graph convolutional layer, a second graph convolutional layer, an activation layer, and a one-dimensional convolutional layer cascaded with the graph construction layer. The graph construction layer is dimensionally transformed by the one-dimensional convolutional layer, added to the output of the second graph convolutional layer, and then activated by the activation layer to form a residual connection spanning the first and second graph convolutional layers. In the feature extraction layer, depth feature maps of human facial image data are extracted; In the feature processing layer, a CNN branch is used to extract global semantic features of the human face from the deep feature map; a graph construction layer with an improved GCN branch is used to reconstruct the human face map structure from the deep feature map using an adjacency matrix constructed with 4 neighborhoods; the human face map structure is then aggregated twice using a first graph convolutional layer and a second graph convolutional layer to obtain local facial topological features; the human face map structure is then dimensionally aligned with the local facial topological features using a one-dimensional convolutional layer; the dimensionally aligned human face map structure and the local facial topological features are then added element-wise and activated using an activation layer to obtain global facial topological features. In the feature fusion layer, the global semantic features of the human face are concatenated with the global topological features of the face and then multi-branch feature interaction is performed through an MLP model to obtain the fused features of the human face. The human facial fusion features are used to perform emotion recognition, and the human facial emotion recognition result is obtained.

[0007] Preferably, the feature extraction layer is an EfficientNet-B0 network structure with the MBConv module as the main structure; the MBConv module includes cascaded 1×1 point convolutions, 3×3 depthwise separable convolutions, SE attention mechanism, and 1×1 point convolutions; The extraction of the depth feature map from the human facial image data specifically includes: The human facial image data is input into the EfficientNet-B0 network model; The EfficientNet-B0 network uses multiple stacked MBConv modules to perform stepwise feature extraction on human facial image data. The processing of each MBConv module includes: Channel expansion is performed sequentially using 1×1 point convolution, spatial feature extraction is performed using 3×3 depth convolution, channel feature recalibration is performed using SE attention mechanism, and then the number of channels is compressed to the number of input channels of MBConv module using 1×1 point convolution to obtain the output features of MBConv module; The input feature map of the MBConv module is added to the output feature map through residual connections; after passing through all MBConv modules, a depth feature map is output.

[0008] Preferably, the graph construction layer of the improved GCN branch is used to reconstruct the deep feature map into a human face map structure using an adjacency matrix constructed from 4 neighborhoods, specifically including: The depth feature map is denoted as Where B is the batch size and C is the number of channels. H and W The feature map is defined by its height and width; a 1×1 convolutional layer is used to perform channel projection on the depth feature map F, compressing the number of channels to [value missing]. The projected feature map is obtained. and will Perform adaptive average pooling to a fixed space size ;in, Indicates by A projection module consisting of convolution, batch normalization, and GELU activation; The feature map Flatten the facial nodes into N nodes based on their spatial location and construct a facial node feature matrix. Each node corresponds to a spatial location on the feature map; where, N = H × W Based on the spatial location of the feature map, an adjacency matrix A is constructed using a 4-neighborhood connection method, and a self-connection is added to each node to obtain an adjacency matrix à = A + I, where I is the identity matrix; The adjacency matrix is ​​symmetrically normalized to obtain the normalized adjacency matrix. Based on the normalized adjacency matrix, a human face graph structure is constructed.

[0009] Preferably, the processing steps for the first and second image convolutional layers specifically include: The first convolutional layer uses the facial node feature matrix X. (0) Using the normalized adjacency matrix as input, neighborhood information is aggregated to extract node features from... Mapping to higher dimensions The local topological features of the first layer are obtained. The second convolutional layer takes the local topological features of the first layer as input and uses the normalized adjacency matrix to aggregate neighborhood information again, thereby integrating node features from... Dimension mapping to preset output dimensions This yields the second layer of local topological features, namely, the local topological features of the face. In this configuration, each of the first and second graph convolutional layers is followed by a LayerNorm layer and a GELU activation function layer.

[0010] Preferably, the CNN branch is further cascaded with a multi-head attention module with 8 attention heads; the improved GCN branch is further cascaded with a multi-head attention module with 4 attention heads. The multi-head attention module with 8 attention heads is used to weight the global semantic features of the human face in terms of importance. The multi-head attention with 4 attention heads is used to refine the global features of the facial topology.

[0011] Preferably, fusing the global semantic features of the human face with the global topological features of the face specifically includes: The global semantic features of the human face are concatenated with the global topological features of the face to obtain the facial concatenation features; The spliced ​​features are input into a two-layer multilayer perceptron model for multi-branch feature interaction to obtain the human face fusion features. The hidden layers of the multilayer perceptron model use the GELU activation function, and a LayerNorm layer and a Dropout layer are applied after each layer.

[0012] This invention provides a facial emotion recognition device, comprising: The data acquisition module is used to acquire human facial image data; A model building module is used to build a facial emotion recognition model based on CNN and improved GCN. The facial emotion recognition model includes: a cascaded feature extraction layer, a feature processing layer, a feature fusion layer, and a classifier. The feature processing layer includes a parallel CNN branch and an improved GCN branch. The improved GCN branch includes: a cascaded graph construction layer, a first graph convolutional layer, a second graph convolutional layer, an activation layer, and a one-dimensional convolutional layer cascaded with the graph construction layer. The graph construction layer undergoes dimensionality transformation by the one-dimensional convolutional layer, is added to the output of the second graph convolutional layer, and then activated by the activation layer to form a residual connection spanning the first and second graph convolutional layers. The feature extraction module is used to extract the depth feature map of the human facial image data in the feature extraction layer; The feature processing module is used to extract global semantic features of the human face from the deep feature map using a CNN branch in the feature processing layer; to reconstruct the human face map structure using an adjacency matrix constructed with 4 neighborhoods using a graph construction layer with an improved GCN branch; to perform two neighborhood information aggregations on the human face map structure using a first graph convolutional layer and a second graph convolutional layer to obtain local facial topological features; to dimensionally align the human face map structure with the local facial topological features using a one-dimensional convolutional layer; and to add the dimensionally aligned human face map structure with the local facial topological features element-wise and activate it using an activation layer to obtain global facial topological features. The feature fusion module is used to concatenate the global semantic features of the human face with the global topological features of the face, and then perform multi-branch feature interaction through an MLP model to obtain the fused features of the human face. The emotion classification module is used to perform emotion recognition on the fused features of the human face using a classifier to obtain the human face emotion recognition result.

[0013] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described facial emotion recognition method.

[0014] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-mentioned facial emotion recognition method.

[0015] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects: In the facial emotion recognition method provided by this invention, a parallel dual-branch hybrid neural network architecture is constructed. While retaining the powerful local texture feature extraction capability of the CNN branch, an improved graph convolutional network (GCN) branch is introduced. This involves reconstructing the facial graph structure by building an adjacency matrix from the deep feature map based on 4 neighborhoods, and using cascaded first and second graph convolutional layers to aggregate neighborhood information twice. This explicitly models the spatial topological dependencies between different key facial regions such as the eyes and mouth, enabling the model to accurately capture structural changes caused by muscle movement. Simultaneously, this addresses the challenges in deep graph networks... To address the issues of information loss and oversmoothing, this invention designs a cross-layer connection structure within the GCN branch. After dimensional alignment of the original node features of the graph construction layer through one-dimensional convolution, it is added element-wise to the local facial topological features extracted through two layers of graph convolution. This results in the final global facial topological features containing both the high-order spatial relationships mined through depth propagation and the original semantic baseline information of the underlying layers. This greatly enhances the discriminative power and robustness of the features, effectively solving the problem of loss of subtle differences in facial expressions caused by missing context and weakened spatial location information in existing technologies. It also significantly improves the model's comprehensive understanding of facial emotions. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a facial emotion recognition method provided by the present invention. Figure 2 A CNN-GCN network architecture diagram for a facial emotion recognition method provided by the present invention; Figure 3 This invention provides a structural diagram of the MBConv module for a facial emotion recognition method. Figure 4 A schematic diagram of the graph construction process for a facial emotion recognition method provided by the present invention; Figure 5 This invention provides a two-layer graph convolutional structure diagram for a facial emotion recognition method. Figure 6 This is a schematic diagram of the multi-head attention mechanism structure of a facial emotion recognition method provided by the present invention; Figure 7 A schematic diagram illustrating the types and quantities of data in the dataset for a facial emotion recognition method provided by the present invention; Figure 8 A confusion matrix analysis diagram of a facial emotion recognition method provided by the present invention; Figure 9An evaluation result diagram based on F1 score for a facial emotion recognition method provided by the present invention; Figure 10 A precision-recall comparison chart of a facial emotion recognition method provided by the present invention; Figure 11 ROC-AUC score distribution diagram of a facial emotion recognition method provided by the present invention; Figure 12 This is a schematic diagram of a facial emotion recognition device provided by the present invention; Figure 13 A diagram of a computer device for implementing a facial emotion recognition method provided by the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the specification without creative effort are within the scope of protection of this application.

[0018] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0019] Figure 1 This is a schematic diagram of a facial emotion recognition method according to the present invention, which specifically includes the following steps: S101: Acquire human facial image data.

[0020] S102: Construct a facial emotion recognition model based on CNN and improved GCN; the facial emotion recognition model includes: a cascaded feature extraction layer, a feature processing layer, a feature fusion layer, and a classifier; the feature processing layer includes a parallel CNN branch and an improved GCN branch; wherein, the improved GCN branch includes: a cascaded graph construction layer, a first graph convolutional layer, a second graph convolutional layer, an activation layer, and a one-dimensional convolutional layer cascaded with the graph construction layer, and the graph construction layer is dimensionally transformed by the one-dimensional convolutional layer, added to the output of the second graph convolutional layer, and then activated by the activation layer to form a residual connection spanning the first and second graph convolutional layers.

[0021] Specifically, the model framework is a CNN-GCN hybrid network architecture, as follows: Figure 2As shown, it mainly consists of four modules: a CNN feature extraction branch, a GCN spatial modeling branch, a multi-head attention mechanism, and a feature fusion module. The specific steps are as follows: The CNN-GCN network preprocesses the input image data into a 224x224 grayscale image and extracts a 1280x7x7 deep feature map using EfficientNet-B0. The extracted deep feature maps are then input into the CNN and GCN branches for processing. The CNN branch obtains weighted global features through global average pooling and an attention mechanism. The GCN branch reconstructs the feature map into a graph structure and extracts spatial relationship features through two layers of graph convolution. The global semantic features output by the CNN branch and the facial topological global features output by the GCN branch are enhanced using a multi-head attention mechanism, then integrated through a fusion network, and finally classified by a classifier to output the final sentiment classification result.

[0022] In addition, the CNN branch is cascaded with a multi-head attention module with 8 attention heads; the improved GCN branch is cascaded with a multi-head attention module with 4 attention heads; the multi-head attention module with 8 attention heads is used to weight the global semantic features of the human face in terms of importance; the multi-head attention module with 4 attention heads is used to refine the global topological features of the face.

[0023] Specifically, to further improve the quality of feature representation, this paper introduces a multi-head self-attention mechanism in both branches. The attention module of the CNN branch uses eight attention heads to weight the importance of the globally pooled feature vectors, enabling the model to adaptively focus on the feature dimensions most critical to emotion recognition. The GCN branch uses four attention heads to refine the output of the graph convolution. The multi-head attention mechanism maps the input query (Q), key (K), and value matrix (V) to different low-dimensional subspaces through a linear transformation, with each subspace corresponding to one attention head. Each attention head independently calculates the scaled dot product attention, as shown in the formula: (1) Where dk is the dimension of the key vector. The multi-head mechanism allows the model to capture various types of dependencies in parallel across different representation subspaces, resulting in richer and more diverse feature representations. The principle diagram is shown below. Figure 5 The Scaled Dot-Product Attention mechanism, for a given query, identifies the most relevant keys from a set of keys and then weights and sums the corresponding values ​​based on their relevance to generate a focused output. Experiments show that the introduction of this attention mechanism significantly improves the discriminative power of the two branch features.

[0024] S103: In the feature extraction layer, extract the depth feature map of the human face image data.

[0025] Optionally, the feature extraction layer is structured as an EfficientNet-B0 network with MBConv modules as the main structure; the MBConv modules include cascaded 1×1 point convolutions, 3×3 depthwise separable convolutions, SE attention mechanism, and 1×1 point convolutions; The extraction of depth feature maps from human facial image data specifically includes: inputting the human facial image data into the EfficientNet-B0 network model; performing step-by-step feature extraction on the human facial image data through multiple MBConv modules stacked in the EfficientNet-B0 network, wherein the processing of each MBConv module includes: sequentially performing channel expansion through 1×1 point convolution, spatial feature extraction through 3×3 depth convolution, channel feature recalibration through SE attention mechanism, and then compressing the number of channels to the input number of the MBConv module through 1×1 point convolution to obtain the output features of the MBConv module; adding the output features of the MBConv module to the human facial image data through residual connection, and outputting the final depth feature map after passing through all MBConv modules.

[0026] Specifically, in convolutional neural network architectures, classic structures include AlexNet, ResNet, and EfficientNet. Throughout the evolution of these classic structures, innovations have been continuously made in network depth, efficiency, and architecture. Compared to other CNN architectures, EfficientNet achieves maximum performance improvement with limited computational resources by simultaneously optimizing network depth, width, and resolution, making it highly suitable for expression recognition tasks requiring high accuracy and computational efficiency. Furthermore, the MBConv module used in EfficientNet-B0 incorporates techniques such as depthwise separable convolution and squeeze-and-excitation (SE), enabling EfficientNet-B0 to maintain feature extraction capabilities while reducing computational complexity and enhancing key facial region features. Therefore, this paper selects EfficientNet-B0 as the CNN backbone network to implement deep feature extraction tasks.

[0027] The core structure of the EfficientNet-B0 network model is the MBConv module, such as Figure 3As shown, the input feature map (H×W×C dimension) is first expanded by a 1×1 point convolution, increasing the number of channels by several times (usually 4-6 times). This step provides sufficient expressive power for subsequent spatial feature extraction in the high-dimensional feature space. Then, the expanded feature map undergoes spatial feature extraction via a 3×3 depthwise separable convolution. This operation significantly reduces computational complexity and the number of parameters while maintaining effective spatial information capture. After spatial feature extraction, MBConv introduces a Squeeze-and-Excitation (SE) attention mechanism module. This module learns the interdependencies between channels through global average pooling and two fully connected layers, adaptively recalibrating the channel feature weights. This channel attention mechanism enhances the expression of important feature channels, suppresses redundant information, and further improves the network's representational ability. Finally, another 1×1 convolution compresses the number of feature channels back to the original dimension, completing the design of the inverse residual structure. The entire module directly adds the input features to the output features through a residual connection (Skip Connection). This design not only helps to alleviate the gradient vanishing problem in deep networks, but also promotes feature reuse and information flow.

[0028] The EfficientNet-B0 network structure is shown in Table 1. In the initial stage, the network uses a standard 3×3 convolutional layer for feature extraction, outputting a feature map with 32 channels and maintaining a resolution of 224×224. Subsequently, the main body of the network uses MBConv modules as the core building blocks for deep feature extraction, finally outputting a 7x7 resolution feature map with 1280 channels.

[0029] Table 1 EfficientNet-B0 Network Structure

[0030] S104: In the feature processing layer, global semantic features of the human face are extracted from the deep feature map using a CNN branch; the graph construction layer of the improved GCN branch is used to reconstruct the human face map structure by constructing an adjacency matrix of 4 neighborhoods from the deep feature map; the human face map structure is aggregated twice using a first graph convolutional layer and a second graph convolutional layer to obtain local facial topological features; the human face map structure is dimensionally aligned with the local facial topological features using a one-dimensional convolutional layer; the dimensionally aligned human face map structure and the local facial topological features are added element-wise and activated using an activation layer to obtain global facial topological features.

[0031] Optionally, by utilizing the graph construction layer of the improved GCN branch, the deep feature map is reconstructed into a human face map structure using an adjacency matrix constructed from 4 neighborhoods. Specifically, this includes: denoting the deep feature map as... Where B is the batch size and C is the number of channels. H and W The feature map is defined by its height and width; a 1×1 convolutional layer is used to perform channel projection on the depth feature map F, compressing the number of channels to a minimum. The projected feature map is obtained. and will Perform adaptive average pooling to a fixed space size ;in, Indicates by A projection module consisting of convolution, batch normalization, and GELU activation is used to project the feature map. Flatten the facial nodes into N nodes based on their spatial location and construct a facial node feature matrix. Each node corresponds to a spatial location on the feature map; where, N = H × W Based on the spatial location of the feature map, an adjacency matrix A is constructed using a 4-neighborhood connection method, and a self-connection is added to each node to obtain an adjacency matrix à = A + I, where I is the identity matrix; the adjacency matrix à is symmetrically normalized to obtain a normalized adjacency matrix; based on the normalized adjacency matrix, a human face map structure is constructed.

[0032] Optionally, the CNN branch is further cascaded with a multi-head attention module with 8 attention heads; the improved GCN branch is further cascaded with a multi-head attention module with 4 attention heads; the multi-head attention module with 8 attention heads is used to perform importance weighting on the global semantic features of the human face; the multi-head attention module with 4 attention heads is used to refine the global topological features of the face.

[0033] Optionally, the processing steps of the first and second image convolutional layers specifically include: the first image convolutional layer takes the facial node feature matrix as input and uses the normalized adjacency matrix to aggregate neighborhood information, so as to extract node features from... Mapping to higher dimensions The first layer of local topological features is obtained; the second convolutional layer takes the first layer of local topological features as input and uses the normalized adjacency matrix to aggregate neighborhood information again, so as to integrate node features from... Dimension mapping to preset output dimensions The second layer of local topological features, namely facial local topological features, is obtained; wherein, after the first and second convolutional layers, a LayerNorm layer and a GELU activation function layer are connected in sequence to normalize and nonlinearly transform the features after neighborhood aggregation.

[0034] Specifically, in graph convolutional network structures, it is necessary to convert the feature maps obtained by the convolutional neural network into graph representations to explicitly model the spatial topological relationships between local facial regions. The core idea of ​​our method is to map each location of the 7×7 spatial feature map extracted by the CNN to a node in the graph, thus establishing an effective bridge from two-dimensional spatial structure to graph topology. During graph construction, each spatial location in the 7×7 feature map is defined as an independent node in the graph, resulting in a graph structure of 49 nodes. Each node carries a feature vector with a dimension of 1280. These high-dimensional feature vectors contain rich semantic information about the corresponding spatial location, providing a sufficient representational foundation for subsequent graph convolutional learning. This node definition strategy ensures the complete preservation of the original spatial feature information during the graph structure transformation process. The choice of 4-neighborhoods instead of fully connected or 8-neighborhoods is to maintain local spatial continuity and suppress the propagation of long-distance noise, allowing information to diffuse layer by layer in a controllable manner to form a progressive receptive field, which is more suitable for modeling local facial deformations. The specific process is as follows: Figure 4 As shown.

[0035] Let the deep features output by EfficientNet be: in For batch size, This refers to the number of channels (EfficientNet-B0 output channels are typically 1280). To reduce the computational cost of graph propagation and filter out high-dimensional redundant information, this paper first uses convolution for channel projection, compressing the channel dimension to... As shown in formula (2): (2) in Indicates by The projection module consists of convolution, batch normalization, and GELU activation. Then... Perform adaptive average pooling to a fixed space size .

[0036] Will Flattened into N=49 nodes according to spatial location, the node feature matrix is ​​obtained as shown in formula (3): (3) Each node corresponds to a spatial location on the feature map, carrying a 64-dimensional feature vector. This definition method can fully preserve the spatial locality of the original feature map and provide structured input for subsequent graph convolution learning.

[0037] On a 7×7 regular grid, an adjacency matrix is ​​constructed using 4-neighbor connections. Each node establishes connections only with its direct neighbors in the four directions: up, down, left, and right. This sparse connection strategy can limit the range of information propagation, suppress the propagation of long-distance noise on the graph, and maintain spatial continuity and local correlation. To preserve the node's own information, self-connections are added, as shown in formula (4): (4) in It is an identity matrix.

[0038] To address the information aggregation bias caused by uneven node degrees, the following measures are taken: Symmetric normalization is performed, as shown in formula (5). Let the degree matrix be: (5) The normalized adjacency matrix is ​​then given by formula (6): (6) This normalization strategy can adaptively balance the contribution weights of different nodes, especially boundary nodes and internal nodes, in the aggregation process, improve the numerical stability of graph propagation, and provide a good mathematical foundation for subsequent graph convolution learning.

[0039] This graph construction method based on fixed grid topology has several advantages: First, the regular spatial structure maintains the spatial locality of the original feature map, enabling graph convolution operations to effectively capture the correlation between spatially adjacent regions; second, the fixed graph structure avoids the computational overhead of dynamic graph construction, improving the inference efficiency of the model; and finally, symmetric normalization ensures the numerical stability of information propagation, providing a good mathematical foundation for subsequent graph convolution learning.

[0040] The core of GCN is to update node features through neighborhood aggregation, which essentially allows each node to extract effective information from its neighbors. The basic aggregation formula is (7): (7) Among them: To obtain the normalized adjacency matrix, For learnable weight matrix, For bias, For non-linear activation functions, this paper uses GELU.

[0041] To address the subtle nature of facial expressions and their reliance on local region coordination, this solution designs a "progressive two-layer residual graph convolution module (wide in, narrow out)." Its core principle is to first perform graph propagation on a compact channel, then expand the expressive power and compress it into a compact graph vector. Simultaneously, cross-layer residuals are introduced to mitigate information loss and over-smoothing. Its specific structure and processing details are as follows: Figure 4 As shown.

[0042] The first layer of graph convolution is... As input, node features are mapped to a higher-dimensional intermediate space to enhance the ability to model local relationships. Specifically, this is shown in formula (8): (8) To improve generalization ability and suppress overfitting, this paper adds a lightweight Dropout to the graph convolution input, followed by LayerNorm and GELU activation for stabilization and nonlinear enhancement, as shown in Equation (9): (9) Since facial emotion recognition relies on subtle differences in local areas (such as wrinkles around the eyes and the curvature of the mouth), the first layer aggregates 4-neighborhood information to propagate and align change cues within a local area; increasing the dimensions from 64 to 256 enhances the expressive capacity and makes the modeling of local topological relationships more comprehensive.

[0043] The second layer of graph convolution further aggregates neighborhood information to obtain two-hop spatial dependencies, which enhances cross-regional collaboration while compressing semantics into a compact representation, as shown in Equation (10): (10) Similarly, LayerNorm and GELU are used for normalization and activation, as shown in formula (11): (11) On a 7×7 grid, two layers of propagation correspond to two-hop neighborhood fusion, which is sufficient to cover the collaborative regions inside local organs such as the eyes or mouth, but will not prematurely over-mix different organ regions, thereby effectively reducing the risk of over-smoothing and maintaining the semantic clarity of different facial regions as independent emotional expression units.

[0044] Cross-layer residual connections and dimensionality alignment: To alleviate information loss caused by multiple propagation and dimensionality reduction and improve training stability, this paper introduces cross-layer residual direct connections: input node features... By linear projection (equivalent to performing a linear projection on the node channels) The mapping is aligned to the output dimension 128, as shown in formula (12): (12) The final output node features are: (13) This residual path allows the original semantic baseline information to pass directly to the output, which strengthens the spatial topological relationship while avoiding excessive smoothing or compression of key discriminative information, thus improving convergence stability and generalization ability.

[0045] This improved graph convolutional layer design offers several significant advantages: in terms of feature representation, it effectively models spatial structure information, achieving a unified fusion of local details and global semantics; in terms of computational efficiency, it offers better control over computational complexity compared to complex attention mechanisms; in terms of training stability, it avoids common problems of deep networks through reasonable network depth design and regularization strategies; and in terms of generalization ability, the structured feature learning approach helps improve the model's adaptability and robustness across different visual tasks. These advantages make this method an efficient, stable, and highly expressive solution for visual feature processing.

[0046] S105: In the feature fusion layer, the global semantic features of the human face are concatenated with the global topological features of the face, and then multi-branch feature interaction is performed through the MLP model to obtain the fused features of the human face.

[0047] Optionally, the global semantic features of the human face are fused with the global topological features of the face, specifically including: concatenating the global semantic features of the human face with the global topological features of the face to obtain concatenated facial features; inputting the concatenated features into a two-layer multilayer perceptron model for multi-branch feature interaction to obtain the fused human face features; wherein the hidden layers of the multilayer perceptron model use the GELU activation function, and a LayerNorm layer and a Dropout layer are applied after each layer.

[0048] Specifically, feature fusion is the key step connecting the two branches. This paper adopts a hierarchical fusion strategy. First, the attention-enhanced CNN features and GCN features are simply concatenated. Then, a two-layer fusion network learns the interactions between multi-branch features. This paper uses the GELU activation function instead of the traditional ReLU because GELU's smoothness helps with gradient flow and training stability. LayerNorm is chosen for normalization, as it is more suitable for handling features with variable sequence lengths than BatchNorm. To prevent overfitting, a decreasing dropout strategy is used, gradually reducing the frequency from 0.2 in the first layer to 0.06 in the last layer. This ensures regularization while avoiding excessive information loss. The entire fusion process maintains the uniqueness of each branch's features while learning their collaborative relationships.

[0049] This paper presents a hybrid feature learning architecture that effectively integrates local texture and global spatial structure information in facial expression images through a CNN-GCN dual-branch design, attention enhancement mechanism, and hierarchical feature fusion. This architecture offers several significant advantages: First, the dual-branch design fully leverages the complementary strengths of CNNs in local feature extraction and GCNs in structured modeling. Second, the multi-head attention mechanism enables the model to adaptively focus on key feature dimensions, enhancing the discriminative power of feature representations. Third, the hierarchical fusion strategy learns multi-branch feature interactions through nonlinear transformations, achieving effective feature integration while maintaining computational efficiency. Finally, the composite loss function specifically addresses the data imbalance problem, improving the model's training stability and generalization performance. Overall, this architecture provides an efficient and effective feature learning paradigm for facial expression recognition tasks, laying a solid foundation for subsequent classification decisions.

[0050] S106: Use a classifier to perform emotion recognition on the fused features of the human face to obtain the human face emotion recognition result.

[0051] To further verify the effectiveness of the method of the present invention, this embodiment also provides an experimental design based on the method of the present invention, including: The effectiveness of the model was validated using the fer2013 dataset, a publicly available website from Kaggle. fer2013 contains 35,887 grayscale face images, all with a resolution of 48x48 pixels. The fer2013 dataset is typically provided as a single CSV file, containing three columns, each row representing information for one face image: `emotion`: an integer value (0 to 6) representing the emotion label for that face image; `pixels`: a comma-separated string containing 48x48=2304 integer values. These values ​​are arranged in row-major order, representing the grayscale value of each pixel in the image (typically ranging from 0 to 255); `Usage`: a string label indicating which predefined data set the image belongs to: `Training` (training set); `PublicTest` (public test set); `PrivateTest` (private test set). The training set contains 28,709 images, the public test set contains 3,589 images, and the private test set contains 3,589 images. The dataset's expression types and image counts are as follows: Figure 7 As shown.

[0052] First, the dataset is split into (Training / PublicTest / PrivateTest) based on the usage parameter in the CSV file. Then, the pixel strings in the file are converted into 48×48 grayscale images, and a mapping from numeric labels to sentiment names is provided. Next, relevant data augmentation strategies are applied, enlarging the original 48×48 images to 224×224 to fit the EfficientNet input. Random flipping and rotation are then used to increase data diversity, improve generalization ability, and simulate head tilt in real-world scenarios. Finally, normalization is performed, normalizing pixel values ​​to the [0,1] range.

[0053] This experiment was run on the Kaggle platform using the platform's default environment.

[0054] The hardware environment is as follows: CPU is Intel(R) Xeon(R) CPU @ 2.00GHz, GPU is Tesla P100-PCIE-16GB.

[0055] The software environment is as follows: Python version 3.11.13, PyTorch version 2.6.0, and CUAD version 12.5.

[0056] The training parameters were set as follows: batch size of 32, initial learning rate of 0.001, weight decay of 0.01, maximum number of training epochs of 60, and early stopping patience of 20.

[0057] The evaluation indicators are as follows: Accuracy: Used to evaluate the overall classification accuracy, calculated using formula (14): (14) Precision: Measures the proportion of samples that the model predicts as belonging to a certain class, and is calculated using formula (15): (15) Recall: measures the proportion of samples of a certain category that are correctly identified. The formula is (16): (16) F1 score: The harmonic mean of precision and recall, used to comprehensively evaluate model performance, is calculated using formula (17). (17) Where TP (True Positives) is the number of true positive instances, i.e., the number of instances correctly predicted as positive by the model. TN (True Negatives) is the number of true negative instances, i.e., the number of instances correctly predicted as negative by the model. FP (False Positives) is the number of false positive instances, i.e., the number of negative instances incorrectly predicted as positive by the model. FN (False Negatives) is the number of false negative instances, i.e., the number of positive instances incorrectly predicted as negative by the model.

[0058] This study... Figure 8 The normalized confusion matrix was used to comprehensively evaluate the performance of the proposed seven-category emotion classification model, revealing its detailed performance patterns in different emotion recognition tasks. From the category-specific accuracy analysis, the model demonstrates excellent recognition ability in categories with strong emotional polarity: Happy emotion achieves a classification accuracy of 88.6%, Surprise emotion achieves 85.3%, and Neutral emotion also achieves a good performance of 75.9%. This high accuracy pattern indicates that the model successfully captures the core feature representations of these emotion categories, validating the effectiveness of the adopted feature extraction architecture. However, the model exhibits a systematic confusion pattern when dealing with semantically similar negative emotions, a phenomenon consistent with the theoretical expectations of affective psychology.

[0059] Category specificity assessment results based on F1 score ( Figure 9 This study reveals the balanced performance distribution of the model across seven emotion recognition tasks. The "Happy" emotion category exhibits the best F1 score (0.895), indicating a near-perfect balance between precision and recall, which is highly correlated with the saliency of happy emotions in facial expressions and semantic features. "Surprise" emotion follows closely (F1=0.853), validating the discriminative advantage of positive emotion categories in the feature space. Notably, while "Disgust" emotion shows relatively good classification accuracy in the confusion matrix, its F1 score of 0.784 suggests a potential imbalance between precision and recall. The most significant performance differences are observed in "Fear" (F1=0.587) and "Sad" (F1=0.598) emotions. The low F1 scores of these two categories directly reflect the systematic confusion among negative emotions observed in the confusion matrix analysis above, further confirming the significant overlap in the feature representations of low-valence emotion categories in the emotion dimension space, thus affecting the model's discriminative ability.

[0060] Precision-Recall Comparison Analysis Figure 10This provides important insights into the model's predictive bias across different emotion categories. The "Happy" emotion exhibits extremely high precision (0.905) and recall (0.886), indicating that the model can accurately identify genuine happiness while effectively avoiding misclassifying other emotions as happy, demonstrating the sufficiency of feature learning for this category. The "Surprise" emotion shows perfect agreement between precision and recall (both 0.853), suggesting that this category achieved an ideal balance during training. However, the "Disgust" emotion exhibits significant asymmetry: a high precision of 0.851 but a recall of only 0.727. This pattern suggests that the model is relatively conservative in predicting disgust, possibly influenced by the scarcity of training samples for this category, leading the model to tend to make high-confidence predictions to maintain precision.

[0061] ROC-AUC score distribution Figure 11 The model's ability to distinguish between different emotion categories was quantified statistically. All categories achieved an AUC value above 0.8, indicating good overall discriminative performance. The AUC value for "Happy" reached 0.977, close to the theoretical upper limit of a perfect classifier, confirming the uniqueness and distinguishability of this category in the feature space. "Surprise" (AUC=0.936) and "Neutral" (AUC=0.906) emotions also demonstrated excellent discriminative ability, consistent with their superior performance in the F1 score evaluation. Further analysis is warranted. Although the F1 score for "Fear" was relatively low (0.587), its AUC value still reached 0.803. This difference suggests that while the model can distinguish fear from other categories probabilistically, there is room for optimization in setting the specific classification decision threshold.

[0062] Table 2 shows the overall performance of the proposed method on multi-class classification tasks. Through comprehensive evaluation on the test set, this paper employs three evaluation strategies: macro averaging, weighted averaging, and micro averaging to comprehensively characterize the model performance. Macro averaging results show that the model achieves a precision of 0.7347, a recall of 0.7169, and an F1 score of 0.7243 across all classes, reflecting the model's robustness in handling class imbalance. Weighted averaging considers the differences in sample distribution across classes, with precision, recall, and F1 score of 0.7286, 0.7275, and 0.7270, respectively, demonstrating relatively balanced performance. Notably, micro averaging maintains consistency across all evaluation dimensions (precision, recall, and F1 score are all 0.7275). This characteristic stems from the inherent properties of micro averaging calculations and also verifies the consistency of the evaluation results. Overall, the combined accuracy of approximately 72.7% indicates that the model has achieved an acceptable level of classification performance, laying a solid foundation for subsequent practical applications. These results confirm the effectiveness and practicality of the proposed architecture in complex multi-class classification scenarios.

[0063] Table 2 Evaluation Indicators under Different Category Averaging Methods

[0064] Table 3 presents a detailed performance breakdown of the proposed model on seven emotion classification tasks, revealing significant performance differences among different emotion categories. In terms of classification performance, the "Happy" category performs best, with precision, recall, and F1 score of 0.9048, 0.8862, and 0.8954, respectively, and a high ROC-AUC of 0.9773, indicating that positive emotions have more significant and easily identifiable feature patterns. The "Surprise" category also performs well, with all metrics remaining balanced and stable at 0.8534 and an ROC-AUC of 0.9364. Notably, the "Disgust" category, despite having the smallest sample size (only 55 samples), still achieves a high precision of 0.8511 and an F1 score of 0.7843, demonstrating the model's effectiveness in handling rare categories. In contrast, identifying negative emotions presents a greater challenge: the F1 scores for the "Fear," "Sad," and "Angry" categories are 0.5872, 0.5983, and 0.6482, respectively, significantly lower than those for positive emotions. In particular, the recall rate for the "Fear" category is only 0.5549, indicating a high false negative rate. Nevertheless, the ROC-AUC for all categories exceeds 0.8, demonstrating the model's good class discrimination ability. This performance difference reflects the inherent complexity of different emotional expressions in the feature space, providing important insights for further optimizing model architecture and data augmentation strategies.

[0065] Table 3 Performance analysis of the model under various facial expressions

[0066] Table 4 presents the performance comparison results of different models on the FER2013 dataset, validating the effectiveness of the proposed method. Experimental results show that the proposed method achieves an accuracy of 72.75% on this benchmark dataset, which is the best among all compared methods. Specifically, the traditional Attentional CNN model achieves an accuracy of 70.02%, the CNN+improvedSoftmax method reaches 70.91%, while the DAF-CNN model, which incorporates a deep attention mechanism, achieves an accuracy of 72.39%. The method using a VGG16 backbone network combined with Focal Loss achieves an accuracy of 72.49%, which is similar to the performance of the proposed method. In contrast, the improved MobileNetV2 model compromises on accuracy due to its lightweight nature, reaching only 68.96%. The proposed method improves upon the suboptimal VGG16+Focal Loss method by 0.26 percentage points and the baseline Attentional CNN model by 2.73 percentage points. This significant improvement validates the rationality and effectiveness of the proposed architecture design. These comparative results fully demonstrate the innovative contributions of this paper in feature extraction, attention mechanism design, and loss function optimization, providing valuable technical solutions for the further development of facial expression recognition. It is worth emphasizing that the performance improvement achieved on the challenging field facial expression dataset FER2013 further confirms the practical value and generalization ability of the proposed method in real-world applications.

[0067] Table 4. Experimental results of different models on the fer2013 dataset.

[0068] To address the limitation of CNNs in modeling spatial relationships between different facial regions, this paper proposes a network architecture combining CNNs and Global Networking Components (GCNs). This architecture integrates the local feature extraction capabilities of traditional CNNs with the global relationship modeling capabilities of GCNs. The CNN branches capture low-level visual features such as texture and edges, while the GCN branches focus on modeling the spatial dependencies between different facial regions. This complementarity enables the model to understand facial expressions from multiple perspectives, thereby improving the facial expression recognition rate.

[0069] Unlike traditional CNN methods that simply treat facial expression recognition as an image classification problem, our proposed method explicitly models the anatomical relationships between facial regions using a graph structure. This modeling approach aligns better with the human mechanism for recognizing expressions, which involves observing the coordinated movements of different facial muscle groups to determine emotional states. The incorporation of graph convolution principles, while maintaining the core CNN backbone, better models the spatial relationships between facial features compared to pure convolution methods. Furthermore, our network architecture employs a dual attention mechanism designed for both CNN and GCN features, adaptively adjusting the importance weights of different features. Experimental results demonstrate that this design allows the model to dynamically focus on the most relevant features when processing different types of expressions, improving accuracy by 2.73% compared to an attention-based CNN architecture.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

[0071] The above describes a facial emotion recognition method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding facial emotion recognition device, such as... Figure 11 As shown.

[0072] Figure 12 This is a schematic diagram of a facial emotion recognition device provided by the present invention. The device includes: Data acquisition module 1201 is used to acquire human facial image data; The model building module 1202 is used to build a facial emotion recognition model based on CNN and improved GCN. The facial emotion recognition model includes: a cascaded feature extraction layer, a feature processing layer, a feature fusion layer, and a classifier. The feature processing layer includes a parallel CNN branch and an improved GCN branch. The improved GCN branch includes: a cascaded graph construction layer, a first graph convolutional layer, a second graph convolutional layer, an activation layer, and a one-dimensional convolutional layer cascaded with the graph construction layer. The graph construction layer is dimensionally transformed by the one-dimensional convolutional layer, added to the output of the second graph convolutional layer, and then activated by the activation layer to form a residual connection spanning the first and second graph convolutional layers. Feature extraction module 1203 is used to extract the depth feature map of the human face image data in the feature extraction layer; The feature processing module 1204 is used to extract global semantic features of the human face from the deep feature map using a CNN branch in the feature processing layer; reconstruct the human face map structure using a graph construction layer with an improved GCN branch, constructing an adjacency matrix of 4 neighborhoods from the deep feature map; perform neighborhood information aggregation twice on the human face map structure using a first graph convolutional layer and a second graph convolutional layer to obtain local facial topological features; align the human face map structure with the local facial topological features using a one-dimensional convolutional layer; and add the dimension-aligned human face map structure with the local facial topological features element by element and activate them using an activation layer to obtain global facial topological features. The feature fusion module 1205 is used to concatenate the global semantic features of the human face with the global topological features of the face in the feature fusion layer and then perform multi-branch feature interaction through an MLP model. The emotion classification module 1206 is used to perform emotion recognition on the fused features of the human face using a classifier to obtain the human face emotion recognition result.

[0073] For specific limitations regarding a facial emotion recognition device, please refer to the limitations of a facial emotion recognition method described above, which will not be repeated here. Each module in the aforementioned facial emotion recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0074] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 A facial emotion recognition method is provided.

[0075] The present invention also provides Figure 13 The schematic diagram of the computer device shown is as follows: Figure 13 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 A facial emotion recognition method is provided.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

Claims

1. A facial emotion recognition method, characterized in that, include: Acquire human facial image data; Construct a facial emotion recognition model based on CNN and an improved GCN; The facial emotion recognition model includes: a cascaded feature extraction layer, a feature processing layer, a feature fusion layer, and a classifier; the feature processing layer includes a parallel CNN branch and an improved GCN branch; wherein, the improved GCN branch includes: a cascaded graph construction layer, a first graph convolutional layer, a second graph convolutional layer, an activation layer, and a one-dimensional convolutional layer cascaded with the graph construction layer, and the graph construction layer is dimensionally transformed by the one-dimensional convolutional layer, added to the output of the second graph convolutional layer, and then activated by the activation layer to form a residual connection spanning the first and second graph convolutional layers; In the feature extraction layer, depth feature maps of human facial image data are extracted; In the feature processing layer, a CNN branch is used to extract global semantic features of the human face from the deep feature map; a graph construction layer with an improved GCN branch is used to reconstruct the human face map structure from the deep feature map using an adjacency matrix constructed with 4 neighborhoods; the human face map structure is then aggregated twice using a first graph convolutional layer and a second graph convolutional layer to obtain local facial topological features; the human face map structure is then dimensionally aligned with the local facial topological features using a one-dimensional convolutional layer; the dimensionally aligned human face map structure and the local facial topological features are then added element-wise and activated using an activation layer to obtain global facial topological features. In the feature fusion layer, the global semantic features of the human face are concatenated with the global topological features of the face and then multi-branch feature interaction is performed through an MLP model to obtain the fused features of the human face. The human facial fusion features are used to perform emotion recognition, and the human facial emotion recognition result is obtained.

2. The facial emotion recognition method as described in claim 1, characterized in that, The feature extraction layer is structured as an EfficientNet-B0 network with the MBConv module as the main structure; the MBConv module includes cascaded 1×1 point convolutions, 3×3 depthwise separable convolutions, SE attention mechanism, and 1×1 point convolutions; The extraction of the depth feature map from the human facial image data specifically includes: The human facial image data is input into the EfficientNet-B0 network model; The EfficientNet-B0 network uses multiple stacked MBConv modules to perform stepwise feature extraction on human facial image data. The processing of each MBConv module includes: Channel expansion is performed sequentially using 1×1 point convolution, spatial feature extraction is performed using 3×3 depth convolution, channel feature recalibration is performed using SE attention mechanism, and then the number of channels is compressed to the number of input channels of MBConv module using 1×1 point convolution to obtain the output features of MBConv module; The input feature map and output feature map of the MBConv module are added together by residual connection. After passing through all MBConv modules, the depth feature map is output.

3. The facial emotion recognition method as described in claim 1, characterized in that, The graph construction layer utilizing the improved GCN branch reconstructs the deep feature map into a human face map structure using a 4-neighbor adjacency matrix, specifically including: The depth feature map is denoted as Where B is the batch size and C is the number of channels. H and W The feature map is defined by its height and width; a 1×1 convolutional layer is used to perform channel projection on the depth feature map F, compressing the number of channels to a minimum. The projected feature map is obtained. and will Perform adaptive average pooling to a fixed space size ;in, Indicates by A projection module consisting of convolution, batch normalization, and GELU activation; The feature map Flatten the facial nodes into N nodes based on their spatial location and construct a facial node feature matrix. Each node corresponds to a spatial location on the feature map; where, N = H × W Based on the spatial location of the feature map, an adjacency matrix A is constructed using a 4-neighborhood connection method, and a self-connection is added to each node to obtain an adjacency matrix à = A + I, where I is the identity matrix. The adjacency matrix is ​​symmetrically normalized to obtain the normalized adjacency matrix. Based on the normalized adjacency matrix, a human face graph structure is constructed.

4. A facial emotion recognition method as described in claim 1 or 3, characterized in that, The processing steps of the first and second graph convolutional layers specifically include: The first convolutional layer uses the facial node feature matrix X. (0) Using the normalized adjacency matrix as input, neighborhood information is aggregated to extract node features from... Mapping to higher dimensions The local topological features of the first layer are obtained; The second convolutional layer takes the local topological features of the first layer as input and uses the normalized adjacency matrix to aggregate neighborhood information again, thereby integrating node features from... Dimension mapping to preset output dimensions This yields the second layer of local topological features, namely, the local topological features of the face. In this configuration, each of the first and second graph convolutional layers is followed by a LayerNorm layer and a GELU activation function layer.

5. The facial emotion recognition method as described in claim 1, characterized in that, The CNN branch is also cascaded with a multi-head attention module with 8 attention heads; the improved GCN branch is also cascaded with a multi-head attention module with 4 attention heads. The multi-head attention module with 8 attention heads is used to weight the global semantic features of the human face in terms of importance. The multi-head attention with 4 attention heads is used to refine the global features of the facial topology.

6. The facial emotion recognition method as described in claim 1, characterized in that, The fusion of the global semantic features of the human face with the global topological features specifically includes: The global semantic features of the human face are concatenated with the global topological features of the face to obtain the facial concatenation features; The spliced ​​features are input into a two-layer multilayer perceptron model for multi-branch feature interaction to obtain the human face fusion features. The hidden layers of the multilayer perceptron model use the GELU activation function, and a LayerNorm layer and a Dropout layer are applied after each layer.

7. A facial emotion recognition device, characterized in that, include: The data acquisition module is used to acquire human facial image data; The model building module is used to build facial emotion recognition models based on CNN and improved GCN; The facial emotion recognition model includes: a cascaded feature extraction layer, a feature processing layer, a feature fusion layer, and a classifier; the feature processing layer includes a parallel CNN branch and an improved GCN branch; wherein, the improved GCN branch includes: a cascaded graph construction layer, a first graph convolutional layer, a second graph convolutional layer, an activation layer, and a one-dimensional convolutional layer cascaded with the graph construction layer, and the graph construction layer is dimensionally transformed by the one-dimensional convolutional layer, added to the output of the second graph convolutional layer, and then activated by the activation layer to form a residual connection spanning the first and second graph convolutional layers; The feature extraction module is used to extract the depth feature map of the human facial image data in the feature extraction layer; The feature processing module is used to extract global semantic features of the human face from the deep feature map using a CNN branch in the feature processing layer; to reconstruct the human face map structure using an adjacency matrix constructed with 4 neighborhoods using a graph construction layer with an improved GCN branch; to perform two neighborhood information aggregations on the human face map structure using a first graph convolutional layer and a second graph convolutional layer to obtain local facial topological features; to dimensionally align the human face map structure with the local facial topological features using a one-dimensional convolutional layer; and to add the dimension-aligned human face map structure and the local facial topological features element-wise and activate them using an activation layer to obtain global facial topological features. The feature fusion module is used to fuse the global semantic features of the human face with the global topological features of the face in the feature fusion layer to obtain the fused features of the human face. The emotion classification module is used to perform emotion recognition on the fused features of the human face using a classifier to obtain the human face emotion recognition result.

Images