Emotion recognition method based on global feature and local feature fusion network

Abstract

The invention discloses an emotion recognition method based on a global feature and local feature fusion network, and relates to the technical field of face recognition. Comprising the following steps: constructing an emotion recognition model based on a global feature and local feature fusion network GLFNet; inputting a face image to be recognized into the emotion recognition model, extracting an intermediate feature map of the face image through the backbone network ResNet-18, and respectively sending the intermediate feature map into the global module and the local module to obtain a global feature map and a local feature map; the feature fusion module fuses the global feature map and the local feature map to obtain fused features; residual connection is established between the intermediate feature and the fusion feature to obtain a final fusion feature; and performing full connection layer FC processing on the final fusion feature to obtain an emotion recognition result. According to the method, the accuracy of a facial image emotion recognition result can be improved, richer fusion feature information is obtained, and higher performance is provided for a facial expression recognition task.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition, and in particular to an emotion recognition method based on a global feature and local feature fusion network. Background Art

[0002] Emotion recognition technology can accurately capture user emotions in social media and achieve personalized interactions; in human-computer interaction, it enables machines to understand human emotions more naturally and improve user experience; in mental health assessment, it helps to timely identify emotional problems and provide effective support. With its unique advantages, emotion recognition technology has shown broad application prospects in many fields, bringing convenience and changes to people's lives and work.

[0003] In the prior art, the focus is on capturing the global features or local features of the face through parallel convolutional neural networks. The specific process includes: designing a parallel convolutional neural network including a global network and a local network; the global network is used to capture the global expression features of the entire face, and the local network is used to capture the local expression features of the face; the face image is input into the parallel convolutional neural network to obtain global features such as the shape, texture and overall expression changes of the entire face and local features of specific areas (such as eyes and mouth).

[0004] The defect of the above-mentioned prior art is that in the process of capturing the global features or local features of the face through parallel convolutional neural networks, the synergistic effect between the two is ignored, which reduces the overall performance of the neural network in the emotion recognition task, and thus leads to inaccurate emotion recognition results. Summary of the invention

[0005] Based on this, it is necessary to provide an emotion recognition method based on a global feature and local feature fusion network to address the above technical problems.

[0006] The embodiment of the present invention provides an emotion recognition method based on a global feature and a local feature fusion network, comprising:

[0007] Construct an emotion recognition model based on the global feature and local feature fusion network GLFNet; the emotion recognition model includes a backbone network ResNet-18, a global module, a local module, a feature fusion module and a fully connected layer FC;

[0008] Obtain a face image to be identified, input the face image to be identified into the emotion recognition model, extract the intermediate feature map of the face image through the backbone network ResNet-18; perform global feature extraction on the intermediate feature map through the global module to obtain a global feature map; perform local feature extraction on the intermediate feature map through the local module to obtain a local feature map;

[0009] The global feature map and the local feature map are fused through the feature fusion module to obtain the fused feature; and a residual connection is established between the intermediate feature and the fused feature to obtain the final fused feature, and the final fused feature is processed by the fully connected layer FC to obtain the emotion recognition result.

[0010] Optionally, the global module includes multiple groups of branches consisting of 3x3 convolutional layers and frequency channel attention modules FCA, and the outputs of the previous group of branches are connected to the inputs of the latter group of branches.

[0011] Optionally, a global feature extraction is performed on the intermediate feature map through a global module, including:

[0012] The intermediate feature map of the face image is divided into multiple initial feature subsets along the channel dimension, and the initial feature subsets are input into the corresponding branches respectively;

[0013] For the first initial feature subset in the sequence, the input is processed by the branch consisting of a 3x3 convolutional layer and an attention module and then passed to the next branch;

[0014] For each subsequent branch, the input includes the processed feature subset output by the previous branch and the corresponding initial feature subset;

[0015] The outputs of all branches are concatenated in the order of branch processing to obtain the global feature map of the face image.

[0016] Optionally, the local module includes multiple branches consisting of 3x3 convolutional layers and frequency channel attention modules FCA.

[0017] Optionally, local features are extracted from the intermediate feature map through a local module, which specifically includes:

[0018] The intermediate feature map of the face image is split into four non-overlapping region subsets along the channel dimension;

[0019] Each region subset is processed by a branch consisting of a 3x3 convolutional layer and a frequency channel attention module FCA to obtain its own initial local feature map;

[0020] The four initial local feature maps are overlapped in the branch processing order to obtain the local feature map of the face image.

[0021] Optionally, the processing process of the frequency channel attention module FCA includes:

[0022] The input feature X after 3x3 convolutional layer processing is divided into multiple partial subsets along the channel dimension, represented as [X0, X1…, X n-1 ];

[0023] A corresponding two-dimensional DCT frequency component is allocated to each partial subset, and the operation result of the two-dimensional DCT frequency component is used as the compression result of the partial subset. The formula is:

[0024]

[0025] Freq=cat([Freq0, Freq1,…,Freq n-1 ])

[0026] FCA_Attn = Sigmoid(FC(Freq))

[0027]

[0028] Among them, X i is the input feature, [u i , v i ] is X i The corresponding two-dimensional index of the frequency component, Freq i is the compressed vector, FC is the fully connected layer, cat is the connection operation, FCA_Attn is the channel attention, Sigmoid represents the activation function, is the new input feature;

[0029] The compression result is processed through a fully connected layer to obtain an attention weight map containing the attention weights corresponding to the channels, and each channel of the input feature is multiplied by the attention weight of the corresponding position to obtain a local feature map.

[0030] Optionally, the global feature map and the local feature map are fused through a feature fusion module, including:

[0031] The extracted global features and local features are summed up, and the corresponding weights of the global features and local features are calculated through the content-guided attention mechanism;

[0032] The weights of global features and local features are processed to obtain fused features;

[0033] The fused features are integrated through a 1×1 convolutional layer to obtain the final fused features of the face image.

[0034] Optionally, the corresponding weights of global features and local features are calculated through a content-guided attention mechanism, specifically including:

[0035] The attention features W of the input features in the spatial dimension are calculated respectively through the spatial attention mechanism and the channel attention mechanism. s and the channel dimension attention feature W c , the formula is:

[0036]

[0037] in, is the feature processed by global average pooling in the spatial dimension, is the feature of global average pooling processing in the channel dimension, is the feature of the global maximum pooling process in the channel dimension, cat represents the connection operation, ReLU is the activation function, and C 1×1 represents 1×1 convolution, C 7×7 Represents 7×7 convolution;

[0038] The attention feature W in the spatial dimension s and the channel dimension attention feature W c Perform broadcast addition to obtain preliminary spatial information modules SIMs;

[0039] The input features and SIMs are connected and rearranged in an alternating manner through the channel shuffle operation Channel Shuffle, and finally a 7×7 group convolution layer is used to generate SIMs for each channel. The formula is:

[0040] W x =cat([W c +W s ,X])

[0041] W=Sigmoid(GC 7×7 (CS(W x )))

[0042] Among them, X represents the input feature, cat represents the concatenation operation, CS represents the channel shuffle operation, GC k×k represents a group convolutional layer with a kernel size of k×k, and Sigmoid represents the activation function;

[0043] The number of groups is set to the number of channels C, and the corresponding weights of global features and local features are obtained.

[0044] Compared with the prior art, the above-mentioned emotion recognition method based on the global feature and local feature fusion network provided by the embodiment of the present invention has the following beneficial effects:

[0045] The present invention performs global feature extraction on an intermediate feature map through a global module to obtain a global feature map; performs local feature extraction on the intermediate feature map through a local module to obtain a local feature map; fuses the global feature map and the local feature map through a feature fusion module; and establishes a residual connection between the intermediate feature and the fused feature to obtain a final fused feature, and processes the final fused feature through a fully connected layer FC to obtain an emotion recognition result; the present invention can solve the problem in the prior art that the synergy between the global and local features of the face is ignored in the process of capturing the global and local features of the face by a parallel convolutional neural network, can improve the accuracy of the emotion recognition result of the facial image, obtain richer fused feature information, and provide more powerful performance for the facial expression recognition task.

[0046] Specifically, residual connections and frequency channel attention mechanisms are introduced in the global module. The frequency channel attention mechanism can dynamically adjust the weights of feature maps so that the model can pay more attention to the key features in facial images. The combination of the two can effectively capture feature information of different scales while keeping the network structure simple, thereby improving the model's expression ability and recognition performance.

[0047] A frequency channel attention mechanism is introduced in the local module. This mechanism replaces the traditional global average pooling operation with discrete cosine transform, thereby focusing on important information from the frequency perspective. Compared with other attention methods that consider the spatial dimension or channel dimension, this mechanism can focus on multiple frequency information including high-frequency information, and high-frequency information often contains key texture and detail information for facial expression recognition.

[0048] A content-guided attention mechanism is introduced in the feature fusion module. This mechanism can generate a separate weight for each channel by combining the attention mechanism and the channel shuffle operation. This feature is used to finely fuse global features with local features at the channel level. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0050] Figure 2 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0051] Figure 3 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0052] Figure 4 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0053] Figure 5 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0054] Figure 6 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0055] Figure 7 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0056] Figure 8 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0057] Fig. 9 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in one embodiment;

[0058] Fig.10 A schematic diagram of the structure of an emotion recognition method based on a global feature and local feature fusion network provided in an embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] In one embodiment, a method for emotion recognition based on a global feature and local feature fusion network is provided, the method comprising:

[0061] 1. Model design

[0062] 1.1 Model Structure

[0063] Construct an emotion recognition model based on the global feature and local feature fusion network GLFNet. The entire model block diagram is as follows: Figure 1 As shown in the figure, it includes: backbone network ResNet-18, global module, local module, feature fusion module and fully connected layer FC. In the global module, multiple groups of branches consisting of 3x3 convolutional layers and frequency channel attention modules FCA are arranged in parallel, and the output of the previous group of branches is connected to the input of the next group of branches; in the local module, multiple groups of branches consisting of 3x3 convolutional layers and frequency channel attention modules FCA are arranged in parallel.

[0064] Obtain the face image to be recognized, input the face image to be recognized into the emotion recognition model, extract the intermediate feature map of the face image through the backbone network ResNet-18, then extract the global feature of the intermediate feature map through the global module to obtain the global feature map; extract the local feature of the intermediate feature map through the local module to obtain the local feature map. Finally, the global feature map and the local feature map are fused through the feature fusion module to obtain the fused feature; and a residual connection is established between the intermediate feature and the fused feature to obtain the final fused feature, and the final fused feature is processed through the fully connected layer FC to obtain the emotion recognition result.

[0065] Among them, the global module introduces residual connection and frequency channel attention mechanism. The frequency channel attention mechanism can dynamically adjust the weight of the feature map, so that the model can pay more attention to the key features in the facial image. Through the combination of these two, while keeping the network structure simple, it can effectively capture feature information of different scales and improve the expression ability and recognition performance of the model.

[0066] The local module introduces a frequency channel attention mechanism, which replaces the traditional global average pooling operation with discrete cosine transform, thereby focusing on important information from the frequency perspective. Compared with other attention methods that consider spatial dimensions or channel dimensions, this mechanism can focus on multiple frequency information including high-frequency information, and high-frequency information often contains key texture and detail information for facial expression recognition.

[0067] The feature fusion module introduces a content-guided attention mechanism, which generates a weight for each channel by combining the attention mechanism and the channel shuffle operation. This feature is used to finely fuse global features with local features at the channel level.

[0068] 1.2 Global Modules

[0069] In deep learning tasks, multi-scale features enable the network to capture both detailed information and contextual information of the image, thereby enhancing the representation ability of the features. In addition, multi-scale processing can also enhance the robustness of the model, thereby improving the generalization performance of the model. The Feature Pyramid Network (FPN) achieves effective transmission of high-level information to low-level information by establishing a top-down network structure, and promotes the fusion of features of different scales. Atrous Spatial Pyramid Pooling (ASPP) uses atrous convolutions with different sampling rates to capture multi-scale information, thereby effectively expanding the receptive field and capturing more comprehensive multi-scale features. The Pyramid Attention Network (PAN) enhances the multi-scale representation of features by combining the attention mechanism with the pyramid structure.

[0070] However, these multi-scale fusion methods usually introduce more complex structures, which increases the computational cost. Therefore, by adopting the residual connection method, multi-scale features are captured at a finer level. Compared with previous methods, it achieves more powerful multi-scale feature fusion by improving residual connections and hierarchical feature extraction, while avoiding overly complex structures and improving training and reasoning efficiency.

[0071] Inspired by the above network, by improving residual connections and hierarchical feature extraction, feature information of different scales can be effectively captured, while the attention mechanism dynamically adjusts the weights of feature maps, allowing the model to pay more attention to key features in facial images. By combining residual connections and frequency channel attention mechanisms, the model's expression ability and recognition performance can be effectively improved while keeping the network structure simple.

[0072] The processing flow chart of the global module is as follows Figure 2 As shown in the figure, the intermediate feature map of the face image is divided into multiple initial feature subsets along the channel dimension, and the initial feature subsets are input into the corresponding branches respectively. For the first initial feature subset in the sequence, the input is processed by the branch consisting of a 3x3 convolution layer and an attention module and then passed to the next branch; for each subsequent branch, the input includes the processed feature subset output by the previous branch and the corresponding initial feature subset; the outputs of all branches are spliced ​​in the order of branch processing to obtain the global feature map of the face image. Increasing the number of branches can make the output features contain more scale information, but in turn increases the complexity and computational overhead of the model. In order to balance accuracy and efficiency, the number of branches is set to 4 in this module. This method allows the model to extract richer global information through multi-scale feature fusion technology and attention mechanism.

[0073] 1.3 Local Modules

[0074] By using the attention mechanism, the model can pay more attention to the key features in the input data, thereby improving the model's expressiveness. Early research mainly focused on finding dependencies in the channel dimension or spatial dimension to highlight key information. For example, SENet (Squeeze and Excitation Network) dynamically adjusts the weights of each channel of the feature map by introducing a channel attention mechanism. SGE (Spatial Group-wise Enhance) groups feature maps in the spatial dimension and learns the spatial attention distribution within each group. CBAM (Convolutional Block Attention Module) not only considers attention in the channel dimension, but also introduces attention in the spatial dimension, making the attention mechanism more comprehensive. GAM (Global Attention Mechanism) is a global attention mechanism across spatial-channel dimension information, which can amplify the interaction between spatial-channel dimensions. SimAM (Simple, Parameter-Free Attention Module) improves the performance of the model by introducing an effective attention mechanism without increasing the complexity of the network.

[0075] The above-mentioned attention mechanism usually uses a compression method to compress the features into a scalar to represent the entire channel or spatial information. Global average pooling, global maximum pooling and global standard deviation pooling are relatively common compression methods. However, by performing frequency domain feature decomposition on global average pooling, it is proved that global average pooling is actually a special case processing method in frequency domain analysis. Using global average pooling means that only the lowest frequency information in the image is retained, while important information from other frequencies is discarded. This discarded information also retains the key clues of the image and should not be ignored. Therefore, a new attention mechanism, Frequency Channel Attention (FCA), is proposed to re-examine the channel representation problem from the perspective of frequency analysis. The two-dimensional discrete cosine transform (DCT) is used to replace the traditional global average pooling operation, and the global average pooling is extended to more frequencies, which better compresses the information.

[0076] Frequency channel attention mechanism Figure 3 As shown in the figure, the input feature X after the 3x3 convolution layer is divided into multiple partial subsets along the channel dimension, represented as [X0, X1…, X n-1], and then assign the corresponding two-dimensional DCT frequency component to each partial subset, and the operation result of the two-dimensional DCT frequency component is used as the compression result of the partial subset. The compression result is processed by the fully connected layer to obtain an attention weight map containing the attention weights corresponding to the channels. Each channel of the input feature is multiplied by the attention weight of the corresponding position to obtain a local feature map.

[0077] The processing process of the frequency channel attention module FCA is shown in formulas (1)-(4), where X i is the input feature, [u i ,v i ] is X i The corresponding two-dimensional index of the frequency component, Freq i is the compressed vector, FC is the fully connected layer, cat is the connection operation, FCA_Attn is the channel attention, Sigmoid represents the activation function, is the new input feature.

[0078]

[0079] Freq=cat([Freq0, Freq1,…,Freq n-1 ]) (2)

[0080] FCA_Attn=Sigmoid(FC(Freq)) (3)

[0081]

[0082] Inspired by the above work, the FCA mechanism is introduced as the attention network of the local module. FCA converts image feature information into frequency information through DCT transformation, thereby focusing on important information from the frequency perspective. Compared with other attention methods that consider the spatial dimension or channel dimension, FCA can pay more attention to the details of facial expressions, because the former only focuses on low-frequency information, while the latter can focus on multiple frequency information including high-frequency information, and high-frequency information often contains key texture and detail information for facial expression recognition.

[0083] The processing flow of the local module is as follows Figure 4As shown in the figure, first, the intermediate feature map of the face image is divided into four non-overlapping region subsets along the channel dimension. Each region subset is processed by a branch consisting of a 3x3 convolutional layer and a frequency channel attention module FCA to obtain its own initial local feature map. Finally, the four initial local feature maps are overlapped in the order of branch processing to obtain the local feature map of the face image. In complex scenes, facial images are often affected by occlusion and illumination, which will lead to the instability of global features. The expression recognition method based on local features can process each local area independently, thereby reducing the dependence on global features and improving the robustness of the model to environmental changes.

[0084] 1.4 Feature Fusion Module

[0085] Although global features can capture the contextual information of the overall expression, they may not be sensitive enough when dealing with subtle differences in expression, while local features can reveal detailed changes in facial expressions, but they often ignore the global context, thus limiting the overall performance of the expression recognition system. By fusing these two types of features, the model can fully utilize the overall information and detailed information in the image, thereby improving the ability to parse the image content.

[0086] In early studies, the fusion methods used, such as element-wise addition, concatenation or mixing, often fail to fully exploit the potential synergy between features. Therefore, an adaptive fusion strategy, CGAFusion, uses the content-guided attention (CGA) mechanism to accurately calculate the weights of different channels and achieve more efficient feature fusion.

[0087] The content-guided attention mechanism can be expressed as Figure 5 As shown, firstly, the attention features W in the spatial dimension and channel dimension are calculated through the spatial attention mechanism and the channel attention mechanism. s and W c , the calculation method is shown in formula (5) and formula (6), where, and They represent the features processed by the global average pooling of the spatial dimension, the global average pooling of the channel dimension, and the global maximum pooling of the channel dimension, respectively. cat represents the connection operation, ReLU is the activation function, and C 1×1 represents 1×1 convolution, C 7×7 Represents a 7×7 convolution.

[0088]

[0089] Next, W s and W cA simple broadcast addition is performed to generate a preliminary spatial information module (SIMs). To further refine these SIMs, the input features and SIMs are concatenated and rearranged in an alternating manner through a channel shuffle operation. Finally, a 7×7 group convolution layer is used to generate SIMs for each channel. The calculation method is shown in formula (7) and formula (8), where X represents the input feature, cat represents the concatenation operation, CS represents the channel shuffle operation, and GC represents the channel shuffle operation. k×k represents a group convolutional layer with a kernel size of k×k, and Sigmoid represents the activation function. The number of groups is set to the number of channels C, in order to generate a weight λ for each channel separately, so as to finely fuse features at the channel level.

[0090] W x =cat([W c +W s ,X]) (7)

[0091] W=Sigmoid(GC 7×7 (CS(W x ))) (8)

[0092] Inspired by the above work, the global features and local features are fused based on the content-guided attention mechanism, aiming to more finely integrate the feature information of the two at the channel level.

[0093] The structure of the feature fusion module is as follows: Figure 6 As shown in the figure, the extracted global features and local features are first summed up, and the corresponding weights of the global features and local features are calculated through the content-guided attention mechanism. Then, the weights of the global features and the weights of the local features are processed by weighted summation to obtain the fused features. In addition, in order to alleviate the gradient vanishing problem, the residual connection is introduced in the module to retain the information of the input features. Finally, the fused features are integrated through a 1×1 convolution layer to obtain the final fused features of the face image.

[0094] 2. Experimental results and analysis

[0095] 2.1 Experimental environment and preprocessing methods

[0096] In this experiment, the Ubuntu 16.04 operating system, Intel Core i9 CPU, 32GB memory, and NVIDIA RTX 3060 GPU (12GB video memory) were used. CUDA version 10.2, cuDNN version 8.4. Python 3.7 was selected as the programming language, and the model was implemented and trained based on the Pytorch 1.7.1 deep learning framework. The specific experimental configuration is shown in Table 1.

[0097] Adam is used as the optimization strategy for model optimization, and the batch size is set to 64. The initial value of the learning rate is 3.5e-4, the weight decay is 1e-4, and the total number of iterations is set to 100. ExponentialLR is used as the learning rate adjustment strategy, which is an exponentially decreasing learning rate regulator. In each round of training, the learning rate will be adjusted accordingly according to the decay factor gamma. In this experiment, gamma is set to 0.98.

[0098] The following preprocessing methods were used. (1) Random cropping and random scaling ensure that the cropped image maintains a uniform size of 224x224 pixels. (2) Random horizontal flipping: Generate new training samples by horizontally flipping the image to help the model learn to ignore the horizontal direction of the image, thereby having better robustness to horizontal layout changes of the image. (3) Random erasing: Randomly select an area in the image and replace its pixel values ​​with random values ​​to simulate the occlusion that may be encountered in actual application scenarios, further enhancing the model's adaptability to complex environments. The area of ​​the erased area is set to (0.02, 0.1), which varies between 2% and 10% of the original image area.

[0099] Table 1 Experimental configuration

[0100] hardware Configuration software Version processor Intel Core i9 CUDA 10.2 Memory 32GB cuDNN 8.4 GPU NVIDIA RTX3060 Python 3.7 Video Memory 12GB Pytorch 1.7.1

[0101] 2.2 Ablation Experiment

[0102] In order to evaluate the effectiveness of GLFNet, an ablation experiment was conducted on the RAFDB dataset. The experimental results include the selection of feature fusion strategy, the impact of attention mechanism on feature extraction, the impact of global and local modules on the overall model, CAM visualization analysis, and confusion matrix analysis.

[0103] (1) Selection of fusion strategy: The effects of different fusion strategies on the model on the RAFDB dataset are compared. Table 2 shows the comparison results of three feature fusion strategies: splicing fusion, additive fusion, and adaptive fusion. Splicing fusion fuses features in the channel dimension, additive fusion directly adds two feature maps, and adaptive fusion uses the CGA mechanism to fuse global features and local features.

[0104] Table 2 Experimental results of different fusion strategies

[0105]

[0106]

[0107] As can be seen from the table, the adaptive fusion strategy shows the best performance, with an accuracy of 88.94%, which is higher than the other two fusion methods. Splicing fusion can retain the original feature information by simply splicing the feature maps together in the channel dimension, but it also leads to a significant increase in the feature dimension, which may dilute the influence of certain key features and thus reduce the recognition ability of the model. Additive fusion directly adds two feature maps, which fuses the information of different features to a certain extent, but this method ignores the complex interactions between features. This simple linear superposition may not be able to fully explore and utilize the complementarity and dependency between features, thereby limiting the model's ability to learn and represent the intrinsic structure of the data. In contrast, adaptive fusion can adjust the weights of the two in the final fusion result according to the actual contribution of the features, and better capture the intrinsic connection between complex data.

[0108] (2) Impact of attention mechanism on feature extraction: The effects of different attention mechanisms on feature extraction performance are compared on the RAFDB dataset. By evaluating various attention mechanisms including SE, CBAM, GAM, SimAM, and FCA, we aim to explore the effects of various mechanisms on feature extraction performance. The experimental results are shown in Table 3.

[0109] As can be seen from the table, the feature extraction module integrated with the FCA mechanism can achieve the highest accuracy, reaching 88.46%, while the number of parameters and computational complexity are not significantly increased compared with other methods. This result shows that the FCA mechanism can enhance the performance of the model without adding additional computational burden by effectively mining and utilizing frequency domain information. In contrast, other attention mechanisms mainly extract features from the spatial or channel dimensions, and fail to fully consider the importance of high-frequency information, which may lead to deficiencies in capturing key texture and detail information of facial expressions.

[0110] Table 3. Feature extraction results using different attention mechanisms

[0111] method Accuracy (%) Parameters(M) SE 87.10 42.98 CBAM 87.94 42.99 GAM 86.86 108.22 SimAM 87.23 42.66 FCA 88.46 42.98

[0112] (3) Verification of the effectiveness of global modules and local modules: On the RAFDB dataset, a comparison experiment was conducted on the effects of the global module and the local module relative to the baseline model (ResNet18). The experimental results are shown in Table 4.

[0113] Table 4 Validity verification results of the fusion model

[0114] Global Modules Local modules Accuracy (%) - - 87.70 √ - 88.26 - √ 88.46 √ √ 88.94

[0115] As can be seen from the table, compared with the baseline model, using either the global module or the local module alone can improve the accuracy of the model. The best performance was achieved when using both the global module and the local module, which confirms that the two modules are complementary in improving the overall performance of the model. By making full use of global information and local details, the model can capture important information in the image more comprehensively and accurately, thereby improving the accuracy of facial expression recognition.

[0116] (4) Confusion Matrix Analysis: In order to gain a deeper understanding of the model performance, we further compared the recognition effects of GLFNet and the benchmark ResNet-18 on different facial expression categories. Figure 7 Show the recall results of the two. Figure 8 The confusion matrix for both is shown.

[0117] As can be seen from the table, GLFNet outperforms ResNet-18 in most expression categories, especially in neutral, surprised, disgusted, happy, and sad expressions. GLFNet performs poorly in fear and anger. This may be because in the RAFDB dataset, the number of fear and anger samples is relatively small, only 283 and 705 respectively, accounting for only 2.3% and 5.7% of the total dataset. This may have led to the model failing to fully learn the characteristics of these two types of expressions during training, thus affecting the recognition performance.

[0118] Table 5 Recall results of ResNet-18 and GLFNet on different expression categories

[0119]

[0120] (5) CAM visualization analysis: In order to more clearly demonstrate the working mechanism of GLFNet in the facial expression recognition task, the gradient weighted class activation mapping (Grad-CAM) technology is used for visualization analysis. Grad-CAM is an advanced visualization method that generates heat maps by analyzing the gradient information in the model. These heat maps can intuitively show the image areas that the model focuses on when making predictions. Compared with the baseline model, GLFNet can pay more attention to the key parts of the expression, such as the eyes and the corners of the mouth, when making predictions. This shows that through the adaptive fusion of global and local information, the GLFNet model can capture richer expression clues, thereby improving the accuracy of facial expression recognition.

[0121] 2.3 Comparison with existing models

[0122] Table 6 Comparison with other methods on the RAFDB dataset

[0123] method Accuracy (%) years DLP-CNN 84.22 2019 gACNN 85.07 2019 SCN 88.14 2020 RAN 86.90 2021 DACL 87.78 2021 MANet 88.36 2021 MixAugment 87.54 2022 GLFNet(ours) 88.94 2024

[0124] Table 7 Comparison with other methods on the SFEW dataset

[0125] method Accuracy (%) years DLP-CNN 51.05 2019 IPFR 55.10 2019 LDL-ALSG 56.50 2020 RAN 56.40 2021 MANet 59.40 2021 GLFNet(ours) 59.53 2024

[0126] In order to evaluate the effectiveness of the proposed model GLFNet, the best results of the model are compared with several existing methods on the RAFDB and SFEW datasets. The experimental results are shown in Tables 6 and 7. Fig. 9 and Fig.10 As shown in the figure. It can be seen that on the RAFDB dataset, the proposed model achieved the best accuracy of 88.94%, which exceeded other methods. This shows that the model has a stronger performance in the emotion recognition task. Similarly, on the SFEW dataset, the proposed model also achieved the best accuracy of 59.53%, which is better than other comparison methods. This result once again proves the generalization ability of the proposed model on different datasets.

[0127] 3. Summary

[0128] Based on multiple attention mechanisms, a network model GLFNet that fuses global features and local features is proposed. It aims to make full use of the feature capture ability of the attention mechanism to solve the problem that previous methods ignore the synergy between global features and local features and are insufficient in feature extraction and feature fusion. Residual connection and frequency channel attention mechanism are introduced in the global module. The frequency channel attention mechanism can dynamically adjust the weight of the feature map, so that the model can pay more attention to the key features in the facial image. Through the combination of the two, while keeping the network structure simple, it can effectively capture feature information of different scales and improve the expression ability and recognition performance of the model. At the same time, the frequency channel attention mechanism is introduced in the local module. This mechanism replaces the traditional global average pooling operation with discrete cosine transform, so as to focus on important information from the perspective of frequency. Compared with other attention methods that consider the spatial dimension or channel dimension, this mechanism can focus on multiple frequency information including high-frequency information, and high-frequency information often contains key texture and detail information for facial expression recognition. The content-guided attention mechanism is introduced in the feature fusion module. This mechanism can generate a weight for each channel by combining the attention mechanism and the channel shuffle operation. This feature is used to finely fuse the global features with the local features at the channel level.

[0129] In order to verify the effectiveness of the proposed method, a series of ablation experiments were conducted to explore the role of each component, and the performance advantages of GLFNet were intuitively demonstrated through visualization technology. On two public facial expression recognition datasets, RAFDB and SFEW, GLFNet was able to achieve higher accuracy rates, reaching 88.94% and 59.53% respectively, both better than other comparison models.

[0130] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An emotion recognition method based on a global feature and local feature fusion network, characterized in that: include: Construct an emotion recognition model based on the global feature and local feature fusion network GLFNet; the emotion recognition model includes a backbone network ResNet-18, a global module, a local module, a feature fusion module and a fully connected layer FC; Obtain a face image to be identified, input the face image to be identified into the emotion recognition model, extract the intermediate feature map of the face image through the backbone network ResNet-18; perform global feature extraction on the intermediate feature map through the global module to obtain a global feature map; perform local feature extraction on the intermediate feature map through the local module to obtain a local feature map; The global feature map and the local feature map are fused through the feature fusion module to obtain the fused feature; and a residual connection is established between the intermediate feature and the fused feature to obtain the final fused feature, and the final fused feature is processed by the fully connected layer FC to obtain the emotion recognition result.

2. The emotion recognition method based on the global feature and local feature fusion network as claimed in claim 1, characterized in that: The global module includes multiple groups of branches consisting of 3x3 convolutional layers and frequency channel attention modules FCA, and the outputs of the previous group of branches are connected to the inputs of the latter group of branches.

3. The emotion recognition method based on the global feature and local feature fusion network as claimed in claim 2, characterized in that: The extracting global features from the intermediate feature map by the global module includes: The intermediate feature map of the face image is divided into multiple initial feature subsets along the channel dimension, and the initial feature subsets are input into the corresponding branches respectively; For the first initial feature subset in the sequence, the input is processed by the branch consisting of a 3x3 convolutional layer and an attention module and then passed to the next branch; For each subsequent branch, the input includes the processed feature subset output by the previous branch and the corresponding initial feature subset; The outputs of all branches are concatenated in the order of branch processing to obtain the global feature map of the face image.

4. The emotion recognition method based on the global feature and local feature fusion network as claimed in claim 1, characterized in that: The local module includes multiple groups of branches consisting of 3x3 convolutional layers and frequency channel attention modules FCA.

5. The emotion recognition method based on the global feature and local feature fusion network as claimed in claim 4, characterized in that: The extracting local features from the intermediate feature map by using the local module includes: The intermediate feature map of the face image is split into four non-overlapping region subsets along the channel dimension; Each region subset is processed by a branch consisting of a 3x3 convolutional layer and a frequency channel attention module FCA to obtain its own initial local feature map; The four initial local feature maps are overlapped in the order of branch processing to obtain the local feature map of the face image.

6. The emotion recognition method based on the global feature and local feature fusion network as claimed in claim 5, characterized in that: The processing process of the frequency channel attention module FCA includes: The input feature X after 3x3 convolutional layer processing is divided into multiple partial subsets along the channel dimension, represented as [X0, X1…, X n-1 ]; A corresponding two-dimensional DCT frequency component is allocated to each partial subset, and the operation result of the two-dimensional DCT frequency component is used as the compression result of the partial subset. The formula is: Freq=cat([Freq0,Freq1,…,Freq n-1 ]) FCA_Attn = Sigmoid(FC(Freq)) Among them, X i is the input feature, [u i ,v i ] is X i The corresponding two-dimensional index of the frequency component, Freq i is the compressed vector, FC is the fully connected layer, cat is the connection operation, FCA_Attn is the channel attention, Sigmoid represents the activation function, is the new input feature; The compression result is processed through a fully connected layer to obtain an attention weight map containing the attention weights corresponding to the channels, and each channel of the input feature is multiplied by the attention weight of the corresponding position to obtain a local feature map.

7. The emotion recognition method based on the global feature and local feature fusion network as claimed in claim 1, characterized in that: The fusing of the global feature map and the local feature map by the feature fusion module includes: The extracted global features and local features are summed up, and the corresponding weights of the global features and local features are calculated through the content-guided attention mechanism; The weights of global features and local features are processed to obtain fused features; The fused features are integrated through a 1×1 convolutional layer to obtain the final fused features of the face image.

8. The emotion recognition method based on the global feature and local feature fusion network as claimed in claim 7, characterized in that: The calculation of the corresponding weights of global features and local features through the content-guided attention mechanism includes: The attention features W of the input features in the spatial dimension are calculated respectively through the spatial attention mechanism and the channel attention mechanism. S and the channel dimension attention feature W c , the formula is: in, is the feature processed by global average pooling in the spatial dimension, is the feature of global average pooling processing in the channel dimension, is the feature of the global maximum pooling process in the channel dimension, cat represents the connection operation, ReLU is the activation function, and C 1×1 represents 1×1 convolution, C 7×7 Represents 7×7 convolution; The attention feature W in the spatial dimension S and the channel dimension attention feature W c Perform broadcast addition to obtain preliminary spatial information modules SIMs; The input features and SIMs are connected and rearranged in an alternating manner through the channel shuffle operation Channel Shuffle, and finally a 7×7 group convolution layer is used to generate SIMs for each channel. The formula is: W x =cat([W c +W s ,X]) W=Sigmoid(GC 7×7 (CS(W x ))) Among them, X represents the input feature, cat represents the concatenation operation, CS represents the channel shuffle operation, GC k×k represents a group convolutional layer with a kernel size of k×k, and Sigmoid represents the activation function; The number of groups is set to the number of channels C, and the corresponding weights of global features and local features are obtained.

Images