Facial expression recognition method based on grid attention and pyramid segmentation attention

Abstract

A facial expression recognition method based on grid attention and pyramid segmentation attention belongs to the technical field of deep learning image processing and emotion calculation, and comprises the following steps: introducing a grid attention mechanism and a pyramid segmentation attention mechanism on a ResNet101 large model, capturing local expression detail features through a grid attention module, establishing multi-scale global feature association by using a pyramid segmentation attention mechanism; hierarchical segmentation is carried out on the backbone network, and information from multiple hierarchies is effectively fused; and dynamically balancing the contribution degree of each level of features through learnable parameters, and integrating the extracted features to judge the expression category. Experiments show that the model achieves the recognition accuracy superior to that of a traditional model on a public data set in natural scenes such as complex illumination and posture change. According to the method, a solution with high robustness is provided for facial expression recognition in a complex environment, and the method has important application value in the fields of intelligent human-computer interaction, mental health assessment and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning image processing and affective computing, and particularly relates to a facial expression recognition method based on grid attention and pyramid split attention. BACKGROUND

[0002] With the continuous development of technology and the continuous improvement of automation level, people's demand for human-computer interaction is also becoming more and more intense. Facial expression recognition (FER) helps machines to understand human emotions from facial expressions. This makes it a core task of human-computer interaction. In addition, FER also has strong expression understanding ability and has wide application prospects in the fields of psychology, intelligent monitoring, virtual reality, synthetic animation, etc.

[0003] In recent years, FER has developed rapidly due to its increasing attention. Early FER work uses manual features to analyze facial expressions. However, FER algorithms based on artificial features are usually only applicable to specific FER tasks, and have defects such as weak feature generalization ability, poor environmental adaptability, etc., and the recognition accuracy significantly decreases in real scenes. With the development of deep learning technology, in order to improve the robustness of the network, convolutional neural networks (CNNs) are introduced into FER. Although it improves the feature expression ability, its local convolution characteristics lead to insufficient global context modeling ability, making it difficult to capture cross-regional correlation features of facial muscle movements. In recent years, although visual Transformer (ViT) has performed well in image classification tasks, its architecture completely dependent on self-attention needs a 300 million-level dataset to be trained to obtain spatial sensitivity bias. However, for FER tasks, such a huge dataset is usually not available. Most existing FER methods use a fixed hierarchical feature fusion strategy, which cannot coordinate the complementary relationship between low-order features and high-order semantic information, and the entire model only uses one same attention mechanism, which is difficult to balance the multi-scale characteristics of expression recognition, and cannot achieve the best performance of the FER model.

[0004] Applying artificial intelligence to the facial expression recognition task not only helps intelligent devices to analyze facial expression features in real time, improves emotional interaction efficiency, and enables the system to complete emotional state analysis within milliseconds. It also realizes cross-platform emotional state monitoring and feedback, and provides accurate emotional perception support for digital interaction that lacks face-to-face communication. SUMMARY

[0005] In order to make up for the above shortcomings, the present application provides a face expression recognition method based on grid attention and pyramid split attention. By improving the ResNet101 backbone network, the grid attention and pyramid split attention mechanism are innovatively integrated, the former accurately captures local expression features such as inter-brow wrinkles and mouth corner curvature through fine-grained feature gridding processing, and the latter realizes global semantic association across levels through multi-scale feature pyramid construction. The network hierarchical architecture is adopted, the low-order features of facial muscle movement are extracted in the shallow network, the high-order representation of emotional semantics is modeled in the deep network, and the contribution degree of different levels of features is adaptively weighted through learnable parameters. The recognition accuracy of the method on the public datasets such as RAF-DB and AffectNet is better than that of the traditional model, and a reliable expression recognition solution is provided for intelligent interaction system.

[0006] The face expression recognition method based on grid attention and pyramid split attention of the present application comprises the following steps:

[0007] 1) Obtain a face expression database, and pre-process the face expression images: detect, align and crop the face expression images using the MTCNN method to remove redundant background information; divide the training set and the test set;

[0008] 2) Data augmentation is performed on the training set in step 1), and the data augmentation includes: scaling the image size, randomly flipping the image, randomly cropping the image, image data normalization, Mixup, Cutout and CutMix;

[0009] 3) Construct a face expression recognition system based on grid attention and pyramid split attention, which specifically comprises the following contents:

[0010] 3.1 Construct a grid attention module, comprising the following steps:

[0011] 3.1.1 The grid attention considers the features from different face regions, and this block uses a matrix point generator operator to calculate the similarity between different grids:

[0012]

[0013] Each channel is squeezed into a scalar after the attention mechanism using adaptive pooling, and the channel is expanded back to the original shape, and the mathematical expression is:

[0014]

[0015] Wherein: AdaptiveAvgPool2d represents an adaptive pooling technique for converting the operand matrix into a scalar; Ones is used to generate a matrix with all elements equal to 1 in shape;

[0016] ​3.1.2 Using the attention weights obtained for each grid pattern, a global attention matrix is formed by reconnecting these weights to the shape of the original matrix, expressed as:

[0017] Out = Cat([Y0, Y1,..., Y S-1 ]);

[0018] Where: Ungrid is the inverse operation of grid, which puts these grid attentions back to the shape of the original face image I, and applies the residual network technique to fuse the features between the original image and the weighted feature map of the backbone network, expressed as:

[0019]

[0020] Where: FT1 represents the feature transformation network of the original face image; FT2 represents the feature transformation network of the weighted feature map ~ I 0; RFN represents the residual feature fusion network; the obtained feature map is forwarded to the candidate backbone network;

[0021] 3.2 Building a basic ResNet101 backbone network model, including the following steps:

[0022] ResNet101 is a deep residual convolutional neural network architecture, whose core feature is to build a deep network with a multi-level residual structure. By introducing a residual module with a skip connection, the gradient decay and explosion problems that easily occur in the training process of a deep network are effectively alleviated, so that the network depth can reach 101 layers while still achieving stable training and efficient feature learning.

[0023] 3.2.1 The basic building block of the network is an improved residual block, which contains a main path and a residual path for parallel processing; the main path adopts a bottleneck structure, the first layer of 1x1 convolution kernel performs channel reduction, followed by a batch normalization layer and a ReLU activation function; the middle 3x3 convolution layer performs spatial feature extraction; the last layer of 1x1 convolution kernel realizes channel dimensionality increase, and batch normalization and ReLU activation are applied again; the residual path adds the input features and the output of the main path element by element, enhancing the nonlinear fusion ability;

[0024] 3.2.2 The network architecture is divided into five feature processing stages according to the spatial resolution: stage1 to stage5. After the input RGB image is processed by the 7x7 convolution kernel of stage1 and the 3x3 max pooling completes the preliminary down-sampling, stage2 to stage5 respectively stack 3, 4, 23, 3 residual blocks, and the first residual block of each stage realizes the spatial resolution reduction of the feature map by 1x1 convolution, with the down-sampling rate being 4, 8, 16, 32 in turn; through the progressive feature abstraction mechanism, each stage realizes multi-scale feature expression from edge texture to high-level semantics;

[0025] 3.2.3 Each two layers in the five layers of ResNet101 model are used as the input of local CNN, which extracts hierarchical multi-scale and multi-variety local features; the global CNN extracts the overall facial expression through global average pooling and fully connected layer; the combination of learned global and local information further improves the accuracy;

[0026] 3.3 Constructing the pyramid split attention module, including the following steps:

[0027] 3.3.1 The input feature map X is divided into S parts, for each split part, there are common channels, through the above split mode, the feature maps extracted by the five feature processing stages from stage1 to stage5 can be processed in parallel at multiple scales, for each split part, the multi-scale spatial information is learned independently, and the cross-channel interaction is established in a local manner, the relationship between the multi-scale kernel size and the group size is:

[0028]

[0029] Where: K is the kernel size; G is the group size;

[0030] The multi-scale feature map generation function is as follows:

[0031] F i =Conv(k i ×k i ,G i )(X i )i=0,1,2…S-1,F=Cat([F0,F1,…,F S-1 ]);

[0032] The entire multi-scale preprocessed feature map can be obtained by multi-scale feature map concatenation;

[0033] 3.3.2 By extracting channel attention weight information from the multi-scale preprocessed feature map, different scale attention weight vectors are obtained; by SEWeight, attention weights are obtained from input feature maps with different scales, and different scale context information is fused; the mathematical expression of the attention weight vector Zi is:

[0034] Z i =SEWeight(F i ),i=0,1,2…S-1,

[0035] Where: Z i is the attention value from F i ;

[0036] To realize the interaction of attention information, without destroying the original channel attention vector, cross-dimension fusion is used to obtain the whole multi-scale attention weight vector Z in a cascading manner, and the mathematical expression is:

[0037]

[0038] Wherein: is a concat operator, which uses soft attention across channels to adaptively select different spatial scales, recalibrates the channel attention of features, and fuses and splices in a cascading manner to obtain the whole channel attention vector:

[0039]

[0040] Wherein: att represents the multi-scale channel weight after attention interaction, and the recalibrated weight of multi-scale channel attention atti is multiplied by the feature map of the corresponding scale Fi:

[0041] Y i =F i ⊙att i ,i=1,2,3,…S-1;

[0042] Wherein: ⊙ represents channel-wise multiplication; Y i is a feature map with obtained multi-scale channel-wise attention weight, and the obtained refined output is written as:

[0043] Out=Cat([Y0,Y1,…,Y S-1 ]);

[0044] 4) Use the data augmented training set data in step 2) to train the model in step 3), and then input the test set image of step 1) into the trained model to verify the recognition effect.

[0045] The aligned facial expression image in step 1) is represented as:

[0046] I C×H×W ,

[0047] Wherein: C is the channel of the image, and the RGB mode is 3; H is the height; W is the width; The image will be cropped into h×w grids and then forwarded to the feature extraction network, and the set of these grids is represented as:

[0048]

[0049] Wherein: h and w represent that the input image I is divided into h×w grids; represents the image grid; In order to learn a low feature from different facial regions for attention calculation, the low feature extraction process is represented as:

[0050]

[0051] wherein: LFN is a low-level feature extraction network; is a feature map obtained by using the low feature extraction network from I i,j .

[0052] The last layer of the ResNet101 backbone network model described in step 3.2 is a full connection layer, and the size of the output is a C-dimensional vector, wherein C is the number of categories, the expression is classified, and the categories are divided into: neutral, happy, angry, sad, fear, surprise and disgust, so C = 7, and the data in the vector is represented by a random variable z i , and when multi-classification is performed, a distribution of probabilities is expected to be output, so normalization processing is performed by using Softmax, wherein the output probability q i is obtained from z i by using a SoftMax function:

[0053]

[0054] For small sample data, a CrossEntropyLoss loss function is used in the output layer for classification, and the CrossEntropyLoss loss function is improved from a cross-entropy function, and the expression is:

[0055]

[0056] wherein: x represents the output of the model; y represents the true label; C represents the number of categories; and n represents the number of samples.

[0057] The present application provides a solution with strong robustness for face expression recognition in a complex environment, and has important application value in the fields of intelligent human-computer interaction and psychological health assessment. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 It is a network structure of a face expression recognition system based on grid attention and pyramid split attention;

[0059] Figure 2 It is a model block diagram of the grid attention module;

[0060] Figure 3 It is a low-level feature extraction network block diagram;

[0061] Figure 4 It is a residual feature fusion network block diagram;

[0062] Figure 5 It is a model block diagram of the pyramid split attention module. DETAILED DESCRIPTION

[0063] The application will be described below with reference to the accompanying drawings.

[0064] As Figure 1 shown, the application is a face expression recognition method based on grid attention and pyramid split attention, which includes the following steps:

[0065] 1) Obtain a face expression database, and pre-process the face expression images: use the MTCNN method to detect, align and crop the face expression images, and remove redundant background information; divide the training set and the test set;

[0066] 2) Data augmentation is performed on the training set in step 1), and the data augmentation includes: scaling the image size, randomly flipping the image, randomly cropping the image, image data normalization, Mixup, Cutout and CutMix;

[0067] 3) Construct a face expression recognition system based on grid attention and pyramid split attention, which specifically includes the following contents:

[0068] 3.1 Construct a grid attention module (as Figure 2 shown), which includes the following steps:

[0069] The grid attention module is used to alleviate the weakness of the convolution filter;

[0070] 3.1.1 The grid attention considers the features from different face regions, and this block uses a matrix point generation operator to calculate the similarity between different grids:

[0071]

[0072] Each channel is squeezed into a scalar after the attention mechanism using adaptive pooling, and the channel is expanded back to the original shape, and the mathematical expression is:

[0073]

[0074] Where: AdaptiveAvgPool2d represents an adaptive pooling technique that converts the operand matrix to a scalar; Ones is used to generate a matrix with all elements equal to 1 in shape;

[0075] 3.1.2 Use the attention weights of each grid pattern obtained to form a global attention matrix by reconnecting these weights to the shape of the original matrix, and the expression is:

[0076] Out = Cat ([Y0, Y1, …, Y S-1 ]);

[0077] ​Where: Ungrid is the inverse operation of grid, focusing attention back on the shape of the original face image I, and applying residual network technology to fuse features between the weighted feature maps of the original image and the backbone network. Its mathematical expression is:

[0078]

[0079] like Figure 4 As shown, RFN represents the residual feature fusion network, and the feature maps obtained are... Forwarded to the backbone network; where: FT1 represents the feature transformation network of the original face image; FT2 represents the feature transformation network of the weighted feature mapping; RFN represents the residual feature fusion network; the obtained feature map It was forwarded to the candidate backbone network.

[0080] 3.2 Constructing the basic ResNet101 backbone network model includes the following steps:

[0081] ResNet101 is a deep residual convolutional neural network architecture whose core feature is the use of multi-layer residual structures to build deep networks. By introducing residual modules with skip connections, it effectively alleviates the gradient decay and explosion problems that easily occur during the training of deep networks, enabling stable training and efficient feature learning even when the network depth reaches 101 layers.

[0082] 3.2.1 The basic building block of the network is an improved residual block, which includes a main path and a residual path for parallel processing. The main path adopts a bottleneck structure, with a 1×1 convolutional kernel in the first layer performing channel reduction, followed by batch normalization and ReLU activation. The middle 3×3 convolutional layers perform spatial feature extraction. The last 1×1 convolutional kernel achieves channel dimensionality enhancement, and batch normalization and ReLU activation are applied again. The residual path adds the input features to the output of the main path element-wise to enhance the nonlinear fusion capability.

[0083] 3.2.2 The network architecture is divided into five feature processing stages, from stage 1 to stage 5, based on spatial resolution. After the input RGB image is initially downsampled by a 7×7 convolution kernel in stage 1 and a 3×3 max pooling, stages 2 to 5 stack 3, 4, 23, and 3 residual blocks, respectively. The first residual block in each stage is subjected to a 1×1 convolution to halve the spatial resolution of the feature map—the downsampling rates are 4, 8, 16, and 32, respectively. Each stage uses a progressive feature abstraction mechanism to achieve multi-scale feature representation from edge texture to high-level semantics.

[0084] 3.2.3 In the ResNet101 model, every two layers are used as input to a local CNN, which extracts hierarchical, multi-scale, and diverse local features; the global CNN extracts the overall facial expression through global average pooling and fully connected layers; the learned global and local information are combined to further improve accuracy.

[0085] 3.3 Construct a pyramid-shaped attention module, such as Figure 5 As shown, it includes the following steps:

[0086] 3.3.1 The input feature map X is divided into S parts. For each split part, it has... Each public channel, through this splitting method, can process feature maps extracted from the five feature processing stages (stage 1 to stage 5) in parallel at multiple scales. For each split, multi-scale spatial information is learned independently, and cross-channel interaction is established locally. The relationship between the multi-scale kernel size and the group size is as follows:

[0087]

[0088] Where: K is the kernel size; G is the group size;

[0089] The multi-scale feature map generation function is as follows:

[0090] F i =Conv(k i ×k i G i (X) i )i=0,1,2…S-1,F=Cat([F0,F1,…,F S-1 ]);

[0091] The entire multi-scale preprocessed feature map can be obtained by cascading multi-scale feature maps;

[0092] 3.3.2 By extracting channel attention weight information from the multi-scale preprocessed feature maps, attention weight vectors at different scales are obtained; attention weights are obtained from input feature maps at different scales using SEWeight, and context information at different scales is fused; the mathematical expression for the attention weight vector Zi is:

[0093] Z i =SEWeight(F i ), i = 0, 1, 2…S-1,

[0094] Among them: Z i It is from F i Attention value;

[0095] To achieve interaction of attention information, cross-dimensional directions are fused without destroying the original channel attention vectors, and the entire multi-scale attention weight vector Z is obtained in a cascade manner. Its mathematical expression is:

[0096]

[0097] in: It is the concat operator, which uses soft attention across channels to adaptively select different spatial scales, fuses and concatenates the channel attentions recalibrated by features in a concatenated manner, and thus obtains the entire channel attention vector:

[0098]

[0099] Where: att represents the multi-scale channel weights after attention interaction, which are the recalibrated weights of the multi-scale channel attention atti multiplied by the feature map of the corresponding scale Fi:

[0100] Y i =F i ⊙att i i = 1, 2, 3, ..., S-1;

[0101] Where: ⊙ represents channel-by-channel multiplication; Y i This refers to the feature map with the obtained multi-scale channel-wise attention weights, and the refined output is written as:

[0102] Out = Cat([Y0,Y1,…,Y) S-1 ]);

[0103] 4) Use the training set data expanded in step 2) to train the model in step 3), and then input the test set images from step 1) into the trained model to verify the recognition effect.

[0104] The mesh-based attention module described in step 3.1 is used to mitigate the weaknesses of the convolutional filter.

[0105] The aligned facial expression image mentioned in step 1) is represented as follows:

[0106] I C×H×W ,

[0107] Where: C is the image's pass-through RGB mode of 3; H is the height; W is the width; the image will be cropped into an h×w grid and then forwarded to the feature extraction network. The set of these grids is represented as:

[0108]

[0109] Where: h and w represent that the input image I is divided into an h×w grid; This represents the image grid; to learn a low-level feature from different face regions for attention calculation, the low-level feature extraction process is represented as:

[0110]

[0111] Among them: such as Figure 3 As shown, LFN is a low-level feature extraction network; It uses a low feature extraction network from I i,j The obtained feature maps are forwarded to mesh-based attention computation to weight their importance to the FER task.

[0112] Step 3.2 describes the last layer of the ResNet101 backbone network model as a fully connected layer, whose output is a C-dimensional vector, where C is the number of categories. For classifying facial expressions, the categories are: neutral, happy, angry, sad, fearful, surprised, and disgusted, so C = 7. The data in the vector is represented by the random variable z. i This indicates that in multi-class classification, we want the output to conform to a probability distribution, so we use Softmax for normalization, where the output probability q is... i By z i Calculated using the SoftMax function:

[0113]

[0114] For small sample data, the CrossEntropyLoss loss function is used for classification in the output layer. The CrossEntropyLoss loss function is an improvement on the cross-entropy function, and its expression is:

[0115]

[0116] Where: x represents the model output; y represents the true label; C represents the number of categories; and n represents the number of samples.

Claims

1. A facial expression recognition method based on grid attention and pyramid segmentation attention, characterized in that, Includes the following steps: 1) Obtain a facial expression database and preprocess the facial expression images: Use the MTCNN method to detect, align, and crop the facial expression images to remove unnecessary background information; Divide the dataset into training and testing sets; 2) Perform data augmentation on the training set in step 1). Data augmentation includes: scaling the image size, randomly flipping the image, randomly cropping the image, image data normalization, Mixup, Cutout, and CutMix. 3) Construct a facial expression recognition system based on grid attention and pyramid segmentation attention, specifically including the following: 3.1 Construct a grid-based attention module, including the following steps: 3.1.1 Mesh-based attention considers features from different face regions. This block uses a matrix point generation operator to calculate the similarity between different meshes: Adaptive pooling is used to squeeze each channel into a scalar after the attention mechanism, and then expands the channels back to their original shape. The mathematical expression for this is: in: AdaptiveAvgPool2d represents an adaptive pooling technique that converts the operand matrix into a scalar; Ones is used to generate... A matrix whose shape has all elements equal to 1; 3.1.2 Using the attention weights of each obtained grid pattern, a global attention matrix is ​​formed by reconnecting these weights to the shape of the original matrix. Its expression is: Out=Cat([Y0,Y1,…,Y S-1 ]); in: Ungrid is the inverse operation of grid, focusing attention back on the shape of the original face image I. It applies residual network techniques to fuse features between the weighted feature maps of the original image and the backbone network. Its mathematical expression is: Where: FT1 represents the feature transformation network of the original face image; FT2 represents the feature transformation network of the weighted feature map ~ I0; RFN represents the residual feature fusion network; and the obtained feature map It was forwarded to the candidate backbone network; 3.2 Constructing the basic ResNet101 backbone network model includes the following steps: 3.2.1 The basic building block of the network is an improved residual block, which includes a main path and a residual path for parallel processing. The main path adopts a bottleneck structure, with a 1×1 convolutional kernel in the first layer performing channel reduction, followed by batch normalization and ReLU activation. The middle 3×3 convolutional layers perform spatial feature extraction. The last 1×1 convolutional kernel achieves channel dimensionality enhancement, and batch normalization and ReLU activation are applied again. The residual path adds the input features to the output of the main path element-wise to enhance the nonlinear fusion capability. 3.2.2 The network architecture is divided into five feature processing stages, from stage 1 to stage 5, based on spatial resolution. After the input RGB image is initially downsampled by a 7×7 convolution kernel in stage 1 and a 3×3 max pooling, stages 2 to 5 stack 3, 4, 23, and 3 residual blocks, respectively. The first residual block in each stage is subjected to a 1×1 convolution to halve the spatial resolution of the feature map—the downsampling rates are 4, 8, 16, and 32, respectively. Each stage uses a progressive feature abstraction mechanism to achieve multi-scale feature representation from edge texture to high-level semantics. 3.2.3 In the ResNet101 model, every two layers are used as input to a local CNN, which extracts hierarchical, multi-scale, and diverse local features; the global CNN extracts the overall facial expression through global average pooling and fully connected layers; the learned global and local information are combined to further improve accuracy. 3.3 Constructing the pyramid-shaped attention module includes the following steps: 3.3.1 The input feature map X is divided into S parts. For each split part, it has... Each public channel, through this splitting method, can process feature maps extracted from the five feature processing stages (stage 1 to stage 5) in parallel at multiple scales. For each split, multi-scale spatial information is learned independently, and cross-channel interaction is established locally. The relationship between the multi-scale kernel size and the group size is as follows: Where: K is the kernel size; G is the group size; The multi-scale feature map generation function is as follows: F i =Conv(k i ×k i ,G i )(X i )i=0,1,2…S-1,F=Cat([F0,F1,…,F S-1 ]); The entire multi-scale preprocessed feature map can be obtained by cascading multi-scale feature maps; 3.3.2 By extracting channel attention weight information from the multi-scale preprocessed feature maps, attention weight vectors at different scales are obtained; attention weights are obtained from input feature maps at different scales using SEWeight, and context information at different scales is fused; the mathematical expression for the attention weight vector Zi is: Z i =SEWeight(F i ),i=0,1,2…S-1, Among them: Z i It is from F i Attention value; To achieve interaction of attention information, cross-dimensional directions are fused without destroying the original channel attention vectors, and the entire multi-scale attention weight vector Z is obtained in a cascade manner. Its mathematical expression is: in: It is the concat operator, which uses soft attention across channels to adaptively select different spatial scales, fuses and concatenates the channel attentions recalibrated by features in a concatenated manner, and thus obtains the entire channel attention vector: Where: att represents the multi-scale channel weights after attention interaction, which are the recalibrated weights of the multi-scale channel attention atti multiplied by the feature map of the corresponding scale Fi: Y i =F i ⊙att i ,i=1,2,3,…S-1; Where: ⊙ represents channel-by-channel multiplication; Y i This refers to the feature map with the obtained multi-scale channel-wise attention weights, and the refined output is written as: Out=Cat([Y0,Y1,…,Y S-1 ]); 4) Use the training set data expanded in step 2) to train the model in step 3), and then input the test set images from step 1) into the trained model to verify the recognition effect.

2. The facial expression recognition method based on grid attention and pyramid segmentation attention as described in claim 1, characterized in that: The aligned facial expression image mentioned in step 1) is represented as follows: I C×H×W ， Where: C is the image's pass-through RGB mode of 3; H is the height; W is the width; the image will be cropped into an h×w grid and then forwarded to the feature extraction network. The set of these grids is represented as: Where: h and w represent that the input image I is divided into an h×w grid; This represents the image grid; to learn a low-level feature from different face regions for attention calculation, the low-level feature extraction process is represented as: Wherein: LFN is a low-level feature extraction network; It uses a low feature extraction network from I i,j The obtained feature map.

3. The facial expression recognition method based on grid attention and pyramid segmentation attention as described in claim 1, characterized in that, Step 3.2 describes the last layer of the ResNet101 backbone network model as a fully connected layer, whose output is a C-dimensional vector, where C is the number of categories. The expressions are classified into: neutral, happy, angry, sad, fearful, surprised, and disgusted, so C = 7. The data in the vector is represented by the random variable z. i This indicates that in multi-class classification, we want the output to conform to a probability distribution, so we use Softmax for normalization, where the output probability q is... i By z i Calculated using the SoftMax function: For small sample data, the CrossEntropyLoss loss function is used for classification in the output layer. The CrossEntropyLoss loss function is an improvement on the cross-entropy function, and its expression is: Where: x represents the model output; y represents the true label; C represents the number of categories; and n represents the number of samples.

Images