An emotion recognition method based on a fusion network of global and local features

Abstract

This invention discloses an emotion recognition method based on a global and local feature fusion network, relating to the field of face recognition technology. The method includes: constructing an emotion recognition model based on a global and local feature fusion network (GLFNet); inputting a face image to be recognized into the emotion recognition model, extracting intermediate feature maps from the face image through a ResNet-18 backbone network, and feeding these intermediate feature maps into the global and local modules respectively to obtain global and local feature maps; fusing the global and local feature maps using a feature fusion module to obtain fused features; establishing residual connections between the intermediate features and the fused features to obtain the final fused features; and processing the final fused features through a fully connected (FC) layer to obtain the emotion recognition result. This invention can improve the accuracy of facial image emotion recognition results, obtain richer fused feature information, and provide more powerful performance for facial expression recognition tasks.

Description

Technical Field

[0001] This invention relates to the field of facial recognition technology, and in particular to an emotion recognition method based on a network that fuses global and local features. Background Technology

[0002] Emotion recognition technology can accurately capture users' emotions in social media, enabling personalized interactions; in human-computer interaction, it allows machines to understand human emotions more naturally, improving user experience; and in mental health assessments, it helps to identify emotional problems in a timely manner and provide effective support. With its unique advantages, emotion recognition technology shows broad application prospects in multiple fields, bringing convenience and change to people's lives and work.

[0003] In existing technologies, the focus is on capturing global or local features of the face through parallel convolutional neural networks. The specific process includes: designing a parallel convolutional neural network that includes a global network and a local network; the global network is used to capture the global expression features of the entire face, while the local network is used to capture the local expression features of the face; the face image is input into the parallel convolutional neural network to obtain global features such as the shape, texture, and overall expression changes of the entire face, as well as local features of specific regions (such as eyes and mouth).

[0004] The drawback of the aforementioned existing technology is that it ignores the synergistic effect between global and local facial features when capturing them through parallel convolutional neural networks, which reduces the overall performance of the neural network in emotion recognition tasks and leads to inaccurate emotion recognition results. Summary of the Invention

[0005] Therefore, it is necessary to provide an emotion recognition method based on a network that fuses global and local features to address the aforementioned technical problems.

[0006] This invention provides an emotion recognition method based on a global and local feature fusion network, comprising:

[0007] An emotion recognition model based on the global and local feature fusion network GLFNet is constructed; the emotion recognition model includes a backbone network ResNet-18, a global module, a local module, a feature fusion module, and a fully connected layer (FC).

[0008] The process involves acquiring a face image to be identified, inputting the face image into the emotion recognition model, extracting intermediate feature maps from the face image through the backbone network ResNet-18, performing global feature extraction on the intermediate feature maps through the global module to obtain global feature maps, and performing local feature extraction on the intermediate feature maps through the local module to obtain local feature maps.

[0009] The feature fusion module fuses the global feature map and the local feature map to obtain the fused feature; and establishes a residual connection between the intermediate feature and the fused feature to obtain the final fused feature. The final fused feature is then processed by a fully connected layer (FC) to obtain the emotion recognition result.

[0010] Optionally, the global module includes multiple branches consisting of 3x3 convolutional layers and frequency channel attention modules (FCA), with the output of the previous branch connected to the input of the next branch.

[0011] Optionally, global feature extraction is performed on the intermediate feature map using a global module, including:

[0012] The intermediate feature map of the face image is divided into multiple initial feature subsets along the channel dimension, and the initial feature subsets are input into the corresponding branches respectively;

[0013] For the first initial feature subset in the sequence, the input is processed by a branch consisting of a 3x3 convolutional layer and an attention module and then passed to the next branch.

[0014] For each subsequent branch, the input includes the processed feature subset of the previous branch's output and the corresponding initial feature subset;

[0015] The outputs of all branches are concatenated according to the order of branch processing to obtain the global feature map of the face image.

[0016] Optionally, the local module includes multiple branches consisting of 3x3 convolutional layers and frequency channel attention modules (FCAs).

[0017] Optionally, local feature extraction is performed on the intermediate feature map through a local module, which specifically includes:

[0018] The intermediate feature map of a face image is segmented into four non-overlapping subsets along the channel dimension;

[0019] Each region subset is processed by a branch consisting of a 3x3 convolutional layer and a frequency channel attention module (FCA) to obtain its own initial local feature map.

[0020] The four initial local feature maps are overlapped according to the branch processing order to obtain the local feature maps of the face image.

[0021] Optionally, the processing procedure of the frequency channel attention module (FCA) includes:

[0022] The input feature X after processing by the 3x3 convolutional layer is divided into multiple subsets along the channel dimension, denoted as [X0, X1, ..., X...]. n-1 ];

[0023] Each subset is assigned a corresponding two-dimensional DCT frequency component, and the result of the two-dimensional DCT frequency component operation is used as the compression result of the subset, as shown in the formula:

[0024]

[0025] Freq=cat([Freq0, Freq1,…,Freq n-1 ])

[0026] FCA_Attn = Sigmoid(FC(Freq))

[0027]

[0028] Among them, X i For input features, [u i v i ] is X i The corresponding frequency component two-dimensional exponent, Freq i The vector is the compressed result. FC stands for fully connected layer, cat represents the connection operation, FCA_Attn is the channel attention function, and Sigmoid represents the activation function. These are new input features;

[0029] The compressed result is processed by a fully connected layer to obtain an attention weight map containing the attention weights corresponding to the channels. Each channel of the input feature is multiplied by the attention weight at the corresponding position to obtain a local feature map.

[0030] Optionally, the global feature map and the local feature map are fused through the feature fusion module, including:

[0031] The extracted global and local features are combined, and the corresponding weights of the global and local features are calculated through a content-guided attention mechanism.

[0032] The weights of the global features and the weights of the local features are processed to obtain the fused features;

[0033] The fused features are integrated using a 1×1 convolutional layer to obtain the final fused features of the face image.

[0034] Optionally, the weights of global and local features are calculated through a content-guided attention mechanism, specifically including:

[0035] The spatial attention features W of the input features are calculated using both spatial attention and channel attention mechanisms. s Attention features W in the channel dimension c The formula is:

[0036]

[0037] in, Features are processed by global average pooling in the spatial dimension. Features are processed by global average pooling at the channel dimension. For features processed by global max pooling along the channel dimension, cat represents the join operation, ReLU is the activation function, and C... 1×1 Represents a 1×1 convolution, C 7×7 Represents a 7×7 convolution;

[0038] Attention features of spatial dimension W s Attention features W in the channel dimension c The broadcasts are summed to obtain preliminary spatial information modules (SIMs);

[0039] The input features and SIMs are concatenated and rearranged alternately using a channel shuffle operation. Finally, a 7×7 group convolutional layer is used to generate SIMs for each channel, as shown in the formula:

[0040] W x =cat([W c +W s ,X])

[0041] W = Sigmoid(GC) 7×7 (CS(W x )))

[0042] Where X represents the input feature, cat represents the connection operation, CS represents the channel shuffling operation, and GC represents the GC operation. k×k This represents a group of convolutional layers with a kernel size of k×k, and Sigmoid represents the activation function.

[0043] Set the number of groups to the number of channels C to obtain the corresponding weights of global and local features.

[0044] The emotion recognition method based on a fusion network of global and local features provided in this invention has the following advantages compared with the prior art:

[0045] This invention extracts global features from intermediate feature maps using a global module to obtain a global feature map; extracts local features from intermediate feature maps using a local module to obtain a local feature map; fuses the global and local feature maps using a feature fusion module; and establishes a residual connection between the intermediate features and the fused features to obtain the final fused features. The final fused features are then processed through a fully connected (FC) layer to obtain the emotion recognition result. This invention addresses the problem in existing technologies where parallel convolutional neural networks neglect the synergistic effect between global and local facial features during capture, thereby improving the accuracy of facial image emotion recognition results, obtaining richer fused feature information, and providing more powerful performance for facial expression recognition tasks.

[0046] Specifically, residual connections and frequency channel attention mechanisms are introduced into the global module. The frequency channel attention mechanism can dynamically adjust the weights of the feature maps, enabling the model to pay more attention to key features in facial images. By combining these two, while maintaining the simplicity of the network structure, feature information at different scales can be effectively captured, thereby improving the model's expressive power and recognition performance.

[0047] A frequency channel attention mechanism is introduced in the local module. This mechanism replaces the traditional global average pooling operation with discrete cosine transform, thereby focusing on important information from the perspective of frequency. Compared with other attention methods that consider spatial or channel dimensions, this mechanism can focus on multiple frequency information, including high-frequency information, which often contains key texture and detail information for facial expression recognition.

[0048] A content-guided attention mechanism was introduced in the feature fusion module. This mechanism can generate a weight for each channel by combining the attention mechanism and the channel shuffling operation. Using this feature, global features and local features are finely fused at the channel level. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0050] Figure 2 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0051] Figure 3 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0052] Figure 4 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0053] Figure 5 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0054] Figure 6 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0055] Figure 7 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0056] Figure 8 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0057] Figure 9 This is a schematic diagram of the structure of an emotion recognition method based on a global and local feature fusion network provided in one embodiment;

[0058] Figure 10 This is a schematic diagram of the structure of an emotion recognition method based on a network that fuses global and local features, as provided in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0060] In one embodiment, an emotion recognition method based on a global and local feature fusion network is provided, the method comprising:

[0061] 1. Model Design

[0062] 1.1 Model Structure

[0063] A sentiment recognition model based on the global and local feature fusion network GLFNet is constructed. The overall model block diagram is as follows: Figure 1 As shown, it includes: a backbone network ResNet-18, a global module, local modules, a feature fusion module, and fully connected (FC) layers. The global module has multiple parallel branches consisting of 3x3 convolutional layers and frequency channel attention (FCA) modules, with the output of one branch connected to the input of the next. Similarly, the local modules also have multiple parallel branches consisting of 3x3 convolutional layers and FCA modules.

[0064] The process involves acquiring a face image to be identified and inputting it into the emotion recognition model. The model extracts intermediate feature maps from the face image using a ResNet-18 backbone network. Then, a global module extracts global features from the intermediate feature maps, resulting in a global feature map. A local module extracts local features from the intermediate feature maps, resulting in local feature maps. Finally, a feature fusion module fuses the global and local feature maps to obtain fused features. A residual connection is established between the intermediate and fused features to obtain the final fused features. These final fused features are then processed through a fully connected (FC) layer to obtain the emotion recognition result.

[0065] The global module introduces residual connections and frequency channel attention mechanisms. The frequency channel attention mechanism can dynamically adjust the weights of feature maps, enabling the model to pay more attention to key features in facial images. By combining these two, the model can effectively capture feature information at different scales while maintaining a simple network structure, thereby improving its expressive power and recognition performance.

[0066] The local module introduces a frequency channel attention mechanism, which replaces the traditional global average pooling operation with discrete cosine transform, thereby focusing on important information from a frequency perspective. Compared with other attention methods that consider spatial or channel dimensions, this mechanism can focus on multiple frequency information, including high-frequency information, which often contains key texture and detail information for facial expression recognition.

[0067] The feature fusion module introduces a content-guided attention mechanism, which generates a weight for each channel by combining the attention mechanism and the channel shuffling operation. This feature allows for the fine fusion of global and local features at the channel level.

[0068] 1.2 Global Module

[0069] In deep learning tasks, multi-scale features enable networks to simultaneously capture detailed and contextual information from images, thereby enhancing feature representation capabilities. Furthermore, multi-scale processing can improve model robustness, thus improving generalization performance. Feature Pyramid Networks (FPNs), by establishing a top-down network structure, achieve efficient transfer of high-level information to low-level information, promoting the fusion of features at different scales. Dilated Spatial Pyramid Pooling (ASPP) utilizes dilated convolutions with different sampling rates to capture multi-scale information, effectively expanding the receptive field and capturing more comprehensive multi-scale features. Pyramid Attention Networks (PANs) enhance multi-scale feature representation by combining attention mechanisms with a pyramid structure.

[0070] However, these multi-scale fusion methods typically introduce complex structures, increasing computational costs. Therefore, by employing residual connections, multi-scale features are captured at a finer-grained level. Compared to previous methods, this approach achieves more powerful multi-scale feature fusion by improving residual connections and hierarchical feature extraction, while avoiding overly complex structures and improving training and inference efficiency.

[0071] Inspired by the aforementioned networks, improvements to residual connections and hierarchical feature extraction effectively capture feature information at different scales. The attention mechanism dynamically adjusts the weights of the feature maps, enabling the model to focus more on key features in facial images. By combining residual connections and frequency channel attention, the model's expressive power and recognition performance can be effectively improved while maintaining a simple network structure.

[0072] The global module's processing flowchart is as follows: Figure 2 As shown, the intermediate feature map of the face image is segmented into multiple initial feature subsets along the channel dimension, and each initial feature subset is input into its corresponding branch. For the first initial feature subset in the sequence, the input is processed by a branch consisting of a 3x3 convolutional layer and an attention module before being passed to the next branch; for each subsequent branch, the input includes the processed feature subset output from the previous branch and the corresponding initial feature subset; the outputs of all branches are concatenated according to the branch processing order to obtain the global feature map of the face image. Increasing the number of branches allows the output features to contain more scale information, but conversely, it increases the complexity and computational cost of the model. To balance accuracy and efficiency, the number of branches is set to 4 in this module. This method, through multi-scale feature fusion technology and attention mechanism, enables the model to extract richer global information.

[0073] 1.3 Local Modules

[0074] By using attention mechanisms, models can focus more on key features in the input data, thereby improving their expressive power. Early research mainly focused on finding dependencies in the channel or spatial dimensions to highlight key information. For example, SENet (Squeeze and Excitation Network) dynamically adjusts the weights of each channel in the feature map by introducing a channel attention mechanism. SGE (Spatial Group-wise Enhance) groups feature maps in the spatial dimension and learns the spatial attention distribution within each group. CBAM (Convolutional Block Attention Module) considers attention not only in the channel dimension but also in the spatial dimension, making the attention mechanism more comprehensive. GAM (Global Attention Mechanism) is a global attention mechanism that crosses spatial and channel dimensions, amplifying the interaction between them. SimAM (Simple, Parameter-Free Attention Module) improves model performance by introducing an effective attention mechanism without increasing network complexity.

[0075] The attention mechanisms described above typically employ a compression method to shrink features into a scalar representing the entire channel or spatial information. Global average pooling, global max pooling, and global standard deviation pooling are commonly used compression methods. However, by performing frequency domain feature decomposition on global average pooling, it is demonstrated that global average pooling is actually a special case of frequency domain analysis. Using global average pooling means retaining only the lowest frequency information in the image, while important information from other frequencies is discarded. This discarded information also retains key clues in the image and should not be ignored. Therefore, a novel attention mechanism—Frequency Channel Attention (FCA)—is proposed to re-examine the channel representation problem from the perspective of frequency analysis. It utilizes two-dimensional discrete cosine transform (DCT) to replace the traditional global average pooling operation, extending global average pooling to more frequencies and better compressing information.

[0076] Frequency channel attention mechanism, such as Figure 3 As shown, the input feature X after processing by the 3x3 convolutional layer is divided into multiple subsets along the channel dimension, denoted as [X0, X1, ..., X...]. n-1Then, a corresponding two-dimensional DCT frequency component is assigned to each subset, and the result of the two-dimensional DCT frequency component operation is used as the compression result of the subset. The compression result is processed through a fully connected layer to obtain an attention weight map containing the attention weights corresponding to the channels. Each channel of the input feature is multiplied by the attention weight at the corresponding position to obtain a local feature map.

[0077] The processing procedure of the frequency channel attention module (FCA) is shown in formulas (1)-(4), where X i For input features, [u i ,v i ] is X i The corresponding frequency component two-dimensional exponent, Freq i The vector is the compressed result. FC stands for fully connected layer, cat represents the connection operation, FCA_Attn is the channel attention function, and Sigmoid represents the activation function. These are new input features.

[0078]

[0079] Freq=cat([Freq0, Freq1,…,Freq n-1 (2)

[0080] FCA_Attn=Sigmoid(FC(Freq)) (3)

[0081]

[0082] Inspired by the above work, the FCA mechanism is introduced as the attention network of the local module. FCA converts image feature information into frequency information through DCT transformation, thereby focusing on important information from the perspective of frequency. Compared with other attention methods that consider spatial or channel dimensions, FCA can pay more attention to the details of facial expressions, because the former only focuses on low-frequency information, while the latter can focus on multiple frequency information, including high-frequency information. High-frequency information often contains key texture and detail information for facial expression recognition.

[0083] The processing flow of a local module is as follows: Figure 4As shown, firstly, the intermediate feature map of the face image is segmented into four non-overlapping region subsets along the channel dimension. Each region subset is processed by a branch consisting of a 3x3 convolutional layer and a frequency channel attention module (FCA) to obtain its own initial local feature map. Finally, the four initial local feature maps are overlapped according to the branch processing order to obtain the local feature map of the face image. In complex scenes, facial images are often affected by occlusion and lighting, which leads to instability of global features. However, expression recognition methods based on local features can process each local region independently, thereby reducing dependence on global features and improving the model's robustness to environmental changes.

[0084] 1.4 Feature Fusion Module

[0085] While global features can capture the overall contextual information of facial expressions, they may not be sensitive enough to subtle differences in expression. Local features, on the other hand, can reveal detailed changes in facial expressions, but often ignore the global background, thus limiting the overall performance of expression recognition systems. By fusing these two types of features, the model can fully utilize both overall and detailed information in the image, thereby improving its ability to interpret image content.

[0086] Early research employed fusion methods such as element-wise addition, concatenation, or hybridization, which often failed to fully leverage the potential synergies between features. Therefore, an adaptive fusion strategy, CGAFusion, utilizes Content-Guided Attention (CGA) to precisely calculate the weights of different channels, achieving more efficient feature fusion.

[0087] Content-driven attention mechanisms can be represented as follows: Figure 5 As shown, the attention features W in the spatial and channel dimensions are first calculated using spatial and channel attention mechanisms. s and W c The calculation method is shown in formulas (5) and (6), where, and Let represent the features processed by global average pooling in the spatial dimension, global average pooling in the channel dimension, and global max pooling in the channel dimension, respectively. `cat` represents the connection operation, `ReLU` is the activation function, and `C` represents the feature processed by global average pooling in the channel dimension. 1×1 Represents a 1×1 convolution, C 7×7 This represents a 7×7 convolution.

[0088]

[0089] Next, W s and W cA simple broadcast summation is performed to generate a preliminary spatial information module (SIM). To further refine these SIMs, the input features and SIMs are concatenated and rearranged alternately using a channel shuffle operation. Finally, a 7×7 group convolutional layer is used to generate SIMs for each channel. The calculation method is shown in Equations (7) and (8), where X represents the input features, cat represents the concatenation operation, CS represents the channel shuffle operation, and GC represents the channel shuffle operation. k×k This represents a group convolutional layer with a kernel size of k×k, and Sigmoid represents the activation function. Setting the number of groups to the number of channels C aims to generate a weight λ for each channel, thereby finely fusing features at the channel level.

[0090] W x =cat([W c +W s ,X]) (7)

[0091] W = Sigmoid(GC) 7×7 (CS(W x ))) (8)

[0092] Inspired by the above work, this paper proposes a content-guided attention mechanism that integrates global and local features, aiming to more finely fuse the feature information of both at the channel level.

[0093] The structure of the feature fusion module is as follows: Figure 6 As shown, the extracted global and local features are first summed, and the corresponding weights of the global and local features are calculated using a content-guided attention mechanism. Then, a weighted summation is performed on the weights of the global and local features to obtain the fused features. Furthermore, to mitigate the vanishing gradient problem, residual connections are introduced in the module to preserve information from the input features. Finally, a 1×1 convolutional layer is used to integrate the fused features, resulting in the final fused features of the face image.

[0094] 2. Experimental Results and Analysis

[0095] 2.1 Experimental Environment and Pretreatment Methods

[0096] This experiment used Ubuntu 16.04 operating system, an Intel Core i9 CPU, 32GB of RAM, and an NVIDIA RTX 3060 graphics processor (12GB of VRAM). CUDA version 10.2 and cuDNN version 8.4 were used. Python 3.7 was chosen as the programming language, and the model was implemented and trained using the PyTorch 1.7.1 deep learning framework. Specific experimental configurations are shown in Table 1.

[0097] For model optimization, Adam was used as the optimization strategy, with a batch size of 64. The initial learning rate was 3.5e-4, the weight decay was 1e-4, and the total number of iterations was set to 100. For the learning rate adjustment strategy, ExponentialLR was chosen, which is an exponentially decreasing learning rate regulator. In each training round, the learning rate was adjusted according to the decay factor gamma. In this experiment, gamma was set to 0.98.

[0098] The following preprocessing methods were adopted: (1) Random cropping and random scaling ensured that the cropped image maintained a uniform size of 224x224 pixels. (2) Random horizontal flipping generated new training samples by horizontally flipping the image, which helped the model learn to ignore the horizontal direction of the image, thus making it more robust to changes in the horizontal layout of the image. (3) Random erasure, by randomly selecting a region in the image and replacing its pixel values ​​with random values ​​to simulate the occlusion situation that may be encountered in real application scenarios, the model's adaptability to complex environments was further enhanced. The area of ​​the erasure region was set to (0.02, 0.1), which is between 2% and 10% of the original image area.

[0099] Table 1 Experimental Configuration

[0100] hardware Configuration software Version processor Intel Core i9 CUDA 10.2 Memory 32GB cuDNN 8.4 GPU NVIDIA RTX 3060 Python 3.7 Video memory 12GB Pytorch 1.7.1

[0101] 2.2 Ablation Experiment

[0102] To evaluate the effectiveness of GLFNet, ablation experiments were conducted on the RAFDB dataset. Experimental results include the selection of feature fusion strategies, the impact of attention mechanisms on feature extraction, the influence of global and local modules on the overall model, CAM visualization analysis, and confusion matrix analysis.

[0103] (1) Selection of fusion strategy: The impact of different fusion strategies on the model on the RAFDB dataset was compared. Table 2 shows the comparison results of three feature fusion strategies: concatenation fusion, additive fusion, and adaptive fusion. Concatenation fusion fuses features in the channel dimension, additive fusion directly adds two feature maps, and adaptive fusion uses the CGA mechanism to fuse global and local features.

[0104] Table 2 Experimental results of different fusion strategies

[0105]

[0106]

[0107] As shown in the table, the adaptive fusion strategy exhibits the best performance, achieving an accuracy of 88.94%, higher than the other two fusion methods. Concatenation fusion, by simply stitching feature maps together along the channel dimension, retains original feature information but also significantly increases the feature dimensionality. This may dilute the influence of some key features, thus reducing the model's recognition ability. Additive fusion, by directly adding two feature maps, fuses information from different features to some extent, but this method ignores the complex interactions between features. This simple linear superposition may not fully explore and utilize the complementarity and dependence between features, thus limiting the model's ability to learn and represent the inherent structure of the data. In contrast, adaptive fusion adjusts the weights of the two features in the final fusion result based on their actual contributions, better capturing the inherent relationships between complex data.

[0108] (2) Impact of attention mechanisms on feature extraction: The impact of different attention mechanisms on feature extraction performance was compared on the RAFDB dataset. By evaluating various attention mechanisms, including SE, CBAM, GAM, SimAM, and FCA, the aim was to explore the impact of various mechanisms on feature extraction performance. The experimental results are shown in Table 3.

[0109] As shown in the table, the feature extraction module integrating the FCA mechanism achieves the highest accuracy, reaching 88.46%, while its parameter count and computational complexity are not significantly increased compared to other methods. This result demonstrates that the FCA mechanism, by effectively mining and utilizing frequency domain information, can enhance model performance without adding extra computational burden. In contrast, other attention mechanisms mainly extract features from spatial or channel dimensions, failing to fully consider the importance of high-frequency information, which may lead to shortcomings in capturing key texture and detail information of facial expressions.

[0110] Table 3 Feature extraction results using different attention mechanisms

[0111] method Accuracy (%) Parameter (M) SE 87.10 42.98 CBAM 87.94 42.99 GAM 86.86 108.22 SimAM 87.23 42.66 FCA 88.46 42.98

[0112] (3) Validation of global and local modules: On the RAFDB dataset, the performance of global and local modules relative to the baseline model (ResNet18) was compared. The experimental results are shown in Table 4.

[0113] Table 4. Validation results of the fusion model

[0114] global module Local module Accuracy (%) - - 87.70 √ - 88.26 - √ 88.46 √ √ 88.94

[0115] As shown in the table, using either the global module or the local module alone improves the model's accuracy compared to the baseline model. The best performance was achieved when both the global and local modules were used simultaneously, confirming their complementary nature in enhancing the overall model performance. By fully utilizing global information and local details, the model can capture more comprehensive and accurate information from images, thereby improving the accuracy of facial expression recognition.

[0116] (4) Confusion Matrix Analysis: To gain a deeper understanding of the specific performance of the models, the recognition performance of GLFNet and the benchmark ResNet-18 on different facial expression categories was further compared. Table 5 and Figure 7 Showing the recall rates for both. Figure 8 The confusion matrix of the two is shown.

[0117] As shown in the table, GLFNet outperforms ResNet-18 in most expression categories, especially in neutral, surprised, disgusted, happy, and sad expressions. However, GLFNet performs poorly in expressions of fear and anger. This may be because the RAFDB dataset has relatively few samples of fear and anger, only 283 and 705 images respectively, accounting for only 2.3% and 5.7% of the total dataset. This may have resulted in the model not fully learning the features of these two expressions during training, thus affecting recognition performance.

[0118] Table 5. Recall rates of ResNet-18 and GLFNet across different facial expression categories.

[0119]

[0120] (5) CAM Visualization Analysis: To more clearly demonstrate the working mechanism of GLFNet in facial expression recognition, Gradient Weighted Class Activation Mapping (Grad-CAM) technology was used for visualization analysis. Grad-CAM is an advanced visualization method that generates heatmaps by analyzing gradient information in the model. These heatmaps can intuitively show the image regions that the model focuses on when making predictions. Compared to the baseline model, GLFNet pays more attention to key parts of the expression, such as the eyes and corners of the mouth, when making predictions. This shows that through adaptive fusion of global and local information, GLFNet can capture richer expression cues, thereby improving the accuracy of facial expression recognition.

[0121] 2.3 Comparison with existing models

[0122] Table 6 shows the comparison with other methods on the RAFDB dataset.

[0123] method Accuracy (%) years DLP-CNN 84.22 2019 gACNN 85.07 2019 SCN 88.14 2020 RAN 86.90 2021 DACL 87.78 2021 MANet 88.36 2021 MixAugment 87.54 2022 GLFNet(ours) 88.94 2024

[0124] Table 7 shows the comparison with other methods on the SFEW dataset.

[0125] method Accuracy (%) years DLP-CNN 51.05 2019 IPFR 55.10 2019 LDL-ALSG 56.50 2020 RAN 56.40 2021 MANet 59.40 2021 GLFNet(ours) 59.53 2024

[0126] To evaluate the effectiveness of the proposed model GLFNet, its best results are compared with several existing methods on the RAFDB and SFEW datasets. Experimental results are shown in Tables 6 and 7. Figure 9 and Figure 10 As shown, the proposed model achieves the best accuracy of 88.94% on the RAFDB dataset, surpassing other methods. This demonstrates the model's superior performance in emotion recognition tasks. Similarly, on the SFEW dataset, the proposed model also achieves the best accuracy of 59.53%, outperforming other comparative methods. This result further proves the generalization ability of the proposed model on different datasets.

[0127] 3. Summary

[0128] Based on multiple attention mechanisms, a network model GLFNet that fuses global and local features is proposed. This model aims to fully utilize the feature capture capabilities of attention mechanisms to address the shortcomings of previous methods that neglected the synergistic effect between global and local features, resulting in insufficient feature extraction and fusion. Residual connections and frequency channel attention mechanisms are introduced into the global module. The frequency channel attention mechanism dynamically adjusts the weights of feature maps, enabling the model to focus more on key features in facial images. This combination effectively captures feature information at different scales while maintaining a simple network structure, improving the model's expressive power and recognition performance. Simultaneously, a frequency channel attention mechanism is introduced into the local module. This mechanism replaces the traditional global average pooling operation with discrete cosine transform, focusing on important information from a frequency perspective. Compared to other attention methods that consider spatial or channel dimensions, this mechanism can focus on multiple frequency information, including high-frequency information, which often contains key texture and detail information for facial expression recognition. A content-guided attention mechanism is introduced into the feature fusion module. This mechanism, by combining attention mechanisms and channel shuffling operations, generates a weight for each channel individually, using this feature to finely fuse global and local features at the channel level.

[0129] To verify the effectiveness of the proposed method, a series of ablation experiments were conducted to explore the role of each component, and the performance advantages of GLFNet were visually demonstrated using visualization techniques. On the two publicly available facial expression recognition datasets, RAFDB and SFEW, GLFNet achieved higher accuracies, reaching 88.94% and 59.53% respectively, both outperforming other comparative models.

[0130] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. An emotion recognition method based on a network fusing global and local features, characterized in that, include: An emotion recognition model based on the global and local feature fusion network GLFNet is constructed; the emotion recognition model includes a backbone network ResNet-18, a global module, a local module, a feature fusion module, and a fully connected layer (FC). The global module includes multiple branches consisting of 3x3 convolutional layers and frequency channel attention modules (FCA), and the output of the previous branch is connected to the input of the next branch. The process involves acquiring a face image to be identified, inputting the face image into the emotion recognition model, extracting intermediate feature maps from the face image through the backbone network ResNet-18, performing global feature extraction on the intermediate feature maps through the global module to obtain global feature maps, and performing local feature extraction on the intermediate feature maps through the local module to obtain local feature maps. The step of performing global feature extraction on the intermediate feature map through a global module includes: The intermediate feature map of the face image is divided into multiple initial feature subsets along the channel dimension, and the initial feature subsets are input into the corresponding branches respectively; For the first initial feature subset in the sequence, the input is processed by a branch consisting of a 3x3 convolutional layer and an attention module and then passed to the next branch. For each subsequent branch, the input includes the processed feature subset of the previous branch's output and the corresponding initial feature subset; The outputs of all branches are concatenated according to the order of branch processing to obtain the global feature map of the face image; The feature fusion module fuses the global feature map and the local feature map to obtain the fused feature; and establishes a residual connection between the intermediate feature and the fused feature to obtain the final fused feature. The final fused feature is then processed by a fully connected layer (FC) to obtain the emotion recognition result.

2. The emotion recognition method based on a global and local feature fusion network as described in claim 1, characterized in that, The local module includes multiple branches consisting of 3x3 convolutional layers and frequency channel attention modules (FCA).

3. The emotion recognition method based on a global and local feature fusion network as described in claim 2, characterized in that, The step of extracting local features from the intermediate feature map using local modules includes: The intermediate feature map of a face image is segmented into four non-overlapping subsets along the channel dimension; Each region subset is processed by a branch consisting of a 3x3 convolutional layer and a frequency channel attention module (FCA) to obtain its own initial local feature map. The four initial local feature maps are overlapped according to the branch processing order to obtain the local feature maps of the face image.

4. The emotion recognition method based on a global and local feature fusion network as described in claim 3, characterized in that, The processing procedure of the frequency channel attention module (FCA) includes: The input feature X after processing by the 3x3 convolutional layer is divided into multiple subsets along the channel dimension, denoted as follows: ; Each subset is assigned a corresponding two-dimensional DCT frequency component, and the result of the two-dimensional DCT frequency component operation is used as the compression result of the subset, as shown in the formula: ； ； ； ； in, As input features, for The corresponding frequency component two-dimensional exponent, This is the compressed vector. It is a fully connected layer. For connection operations, It's about channel attention. This represents the activation function. These are new input features; The compressed result is processed by a fully connected layer to obtain an attention weight map containing the attention weights corresponding to the channels. Each channel of the input feature is multiplied by the attention weight at the corresponding position to obtain a local feature map.

5. The emotion recognition method based on a global and local feature fusion network as described in claim 1, characterized in that, The process of fusing global and local feature maps through a feature fusion module includes: The extracted global and local features are combined, and the corresponding weights of the global and local features are calculated through a content-guided attention mechanism. The weights of the global features and the weights of the local features are processed to obtain the fused features; The fused features are integrated using a 1×1 convolutional layer to obtain the final fused features of the face image.

6. The emotion recognition method based on a global and local feature fusion network as described in claim 5, characterized in that, The calculation of the corresponding weights of global and local features through the content-guided attention mechanism includes: Attention features of the input features in the spatial dimension are calculated using spatial attention and channel attention mechanisms, respectively. Attention features in the channel dimension The formula is: ； ； in, Features are processed by global average pooling in the spatial dimension. Features are processed using global average pooling at the channel dimension. Features are processed using global max pooling at the channel dimension. Indicates a connection operation. For activation function, Represents a 1×1 convolution. Represents a 7×7 convolution; Attention features of spatial dimension Attention features in the channel dimension The broadcasts are summed to obtain preliminary spatial information modules (SIMs); The input features and SIMs are concatenated and rearranged alternately using a channel shuffle operation. Finally, a 7×7 group convolutional layer is used to generate SIMs for each channel, as shown in the formula: ； ； in, Indicates input features, Indicates a connection operation. This indicates a channel shuffling operation. This represents a group convolutional layer with a kernel size of 7×7. Indicates the activation function; Set the number of groups to the number of channels. This yields the weights of the global and local features.

Images