Facial Expression Recognition Method Based on Attention-Modulated Contextual Spatial Information

Abstract

The present invention discloses a facial expression recognition method based on attention-modulated context space information. The specific steps are as follows: S1: Obtain a public dataset of natural scene facial expression images to be trained, and preprocess the facial expression images; S2: Construct an attention-modulated context space information network model for natural scene facial expression recognition; S3: Use the preprocessed facial expression images to train the context space information (ACSI) network model; S4: Repeat the model training in step S3 until the set number of training times is reached to obtain a trained deep residual network model, and use the trained deep residual network model for facial expression recognition. The combination of context convolution and coordinated attention can significantly improve the performance of facial expression recognition. On the publicly available facial expression dataset, ACSI has higher recognition performance compared with similar algorithms.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic facial expression recognition, and relates to a facial expression recognition algorithm, in particular to a facial expression recognition method based on attention modulation of context space information. Background Art

[0002] Facial expressions have rich feature information, and human face expression recognition has been widely applied in aspects such as human-computer interaction and mental health assessment. Traditional expression recognition methods can be divided into two categories. One is the expression recognition method based on Facial Action Units (AU). Usually, the Facial Expression Recognition (FER) task is transformed into an AU detection task. AU is a small but distinguishable muscle movement related to expressions. However, it is difficult to detect local changes in the human face using this method, and factors such as illumination or pose changes will also reduce the performance of AU detection. The other is to design features manually to represent the human face image and train an expression classifier to achieve expression recognition. However, in natural scenes, affected by uncontrollable factors, the performance of the human face expression recognition method based on manually designed features is limited. In recent years, deep learning-based human face expression recognition has become a research hotspot, and related work has shifted from controlled laboratory scenes to natural scenes and made certain progress. Convolutional Neural Network (CNN) is the mainstream model for expression recognition, and CNN has strong generalization in the expression recognition task. Subsequently, various improvement methods have emerged. Among these methods, on the one hand, to solve the problem of incomplete expression features, Zhao Z et al. designed a symmetric structure to learn multi-scale features in the residual block, retaining human face expression information at the granularity level; Li Y et al. proposed Slide-Patch (SP), sliding a window on each feature map to extract the global features of facial expressions; Fan X et al. proposed a Hierarchical Scale Net (HSNet) for human face expression recognition, in which a dilated inception block was added to enhance the extraction of kernel scale information. Liang X et al. used a dual-branch network for expression recognition, where one branch uses CNN to capture local edge information, and the other branch applies Vision Transformer to obtain a better global representation; Mao Junyu et al. proposed using pyramid convolution units composed of convolutional kernels of different sizes to extract expression features, improving the non-linear expression ability of the model.However, these methods improve the completeness of extracted facial features by adding auxiliary network layers or adopting branching structures. Furthermore, to address the issue of blurred classification boundaries between expression classes, Xie S et al. proposed a module called the Salient Expressional Region Descriptor (SERD) to highlight salient features related to expression and improve feature representation. Gera D et al. proposed a novel Spatio-Channel Attention Net (SCAN) that obtains local and global attention for each channel and spatial location, processing facial features in the spatial and channel dimensions rather than directly performing feature dimensionality reduction. Wang Z et al. designed an attention branch using a U-Net-like architecture to highlight subtle local expression information. Song Yuqin et al. used the CBAM attention mechanism after extracting multi-scale features to filter facial features and enhance the expression of effective features. These methods improve model performance by adding auxiliary network layers or using branching structures to extract more subtle deep facial expression features. However, these methods ignore the potential contextual relationships between local facial regions, and the complex network structure is not conducive to model lightweighting.

[0003] A Chinese patent document (application number: 202010537198.2) discloses a facial expression recognition method based on a deep residual network. First, the deep residual network model is used to extract multi-scale features from the magnified facial expression image, and then the extracted features are subjected to dimensionality reduction and compression processing, and the processed features are used for expression classification. This method has three defects: ① The standard convolution kernel with a fixed receptive field is used in the residual network, which cannot obtain a wide range of facial expression information; ② While removing redundant information through the dimensionality reduction and compression feature scheme, some important information related to the expression is lost; ③ It performs well on a controlled laboratory dataset, but the recognition performance on an uncontrolled dataset needs to be verified. The above points limit the completeness of the expression features extracted by this method, and the feature representation ability needs to be improved.

[0004] A Chinese patent document (application number: 202110133950.1) discloses a dynamic expression recognition method and system based on a representation stream embedding network. This method embeds a differentiable representation stream layer in a convolutional neural network to extract dynamic expression features from video sequences and weights the output features using spatial attention weights. This method has two drawbacks: 1) it only uses spatial attention and does not perform feature optimization from a channel dimension; 2) it involves the acquisition and processing of video data, which is complex and leads to high operating costs.

[0005] The existing methods have the following deficiencies: 1) In the feature extraction stage, only global or local features of facial expressions are considered, resulting in limited feature completeness; 2) In the feature processing stage, the features are dimensionally reduced and compressed, leading to blurred inter-class classification boundaries. Summary of the Invention

[0006] The present invention provides an expression recognition method based on attention-modulated contextual spatial information, and proposes a new natural scene facial expression recognition model, called the Attention-modulated Contextual Spatial Information (ACSI) model. Contextual convolution is used to replace the standard convolution in the residual network to construct the Contextual Convolution Residual Networks CoResNet18 and CoResNet50 for extracting multi-scale features, and more subtle expression information can be obtained without increasing the network complexity; Coordinate attention is embedded in each residual block in CoResNet to focus on significant features, enhance the useful information related to expressions and suppress redundant information in the input feature map, and effectively reduce the sensitivity of deep convolution to face occlusion and pose changes.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is: The expression recognition method based on attention-modulated contextual spatial information specifically includes the following steps:

[0008] S1: Obtain a public dataset of natural scene facial expression images to be trained, and preprocess the facial expression images.

[0009] S2: Construct an attention-modulated contextual spatial information network model for natural scene facial expression recognition.

[0010] S3: Use the preprocessed facial expression images to train the contextual spatial information (ACSI) network model.

[0011] S4: Repeat the model training in step S3 until the set number of training times is reached to obtain a trained deep residual network model, and use the trained deep residual network model for facial expression recognition.

[0012] With the above technical solution, an expression recognition model based on attention-modulated context space information is constructed. First, convolutional kernels with a lower dilation rate are used to capture local context information. Second, convolutional kernels with a higher dilation rate are used to merge global context information, extracting discriminative local features and relevant global features of the face, ensuring the complementarity of expression feature information. Finally, a coordinated attention mechanism is used to assign attention weights to the extracted features, increasing the feature differences between expression classes and strengthening the feature representation ability. Experiments are carried out on the AffectNet-7 and RAF_DB datasets to verify the effectiveness of the ACSI model, and compared with similar models, the proposed model has better recognition performance.

[0013] As a preferred technical solution of the present invention, the step S2 specifically includes the following steps:

[0014] S21: Replace the middle convolutional layer of the residual block with a context convolutional block to form a context convolutional residual module and construct a context convolutional residual network;

[0015] S22: Use Coordinate Attention (CA) to construct a coordinated attention module to assign attention weights to the multi-scale features extracted by CoResNet to strengthen the feature representation ability.

[0016] With the above technical solution, first, use context convolution to replace the standard convolution in the convolutional residual block, and construct a context convolutional residual network (CoResNet) as the feature extraction part, using convolutional kernels with different dilation rates to capture local and merge global context information; second, embed the coordinated attention module into CoResNet as the feature processing part, assign attention weights to the extracted features, highlight significant features, increase the feature differences between expression classes, and finally form the ACSI model for facial expression recognition.

[0017] As a preferred technical solution of the present invention, the step S21 is specifically:

[0018] S211: The context convolutional block receives the input feature map M in , which applies convolutional kernels with different dilation rates D = {d1, d2, d3,..., d n} at different levels L = {1, 2, 3,..., n}, that is, level i (level i represents the context convolution of the i-th level in the context convolutional block) has a dilation rate

[0019] S212: At different levels level of the context convolution, the context convolution outputs multiple feature maps For all i ∈ L, each graph has a width W out and a height H out ;

[0020] S213: Retaining the residual structure combines the correlations between layers to obtain a context convolutional residual module;

[0021] S214: Adjust the level of the context convolutional block in each layer according to the size of the feature map, thereby constructing a context convolutional residual network. The context convolutional residual network (CoResNet) constructed in step S2 includes CoResNet18 and CoResNet50; in CoResNet18, each context residual module consists of a context convolutional residual module and a 1×1 standard convolutional layer; in CoResNet50, each context residual module consists of a context convolutional residual module and two 1×1 standard convolutional layers; the context convolutional residual module is used for multi-scale feature extraction, and the 1×1 standard convolutional layer is used for channel transformation; according to the size of the input feature map, different levels of context convolutional blocks are used in each context residual module. A context convolutional block with level 4 (level = 4) is used in the first context convolutional residual module, level = 3 in the first context convolutional residual module, level = 2 in the third context convolutional residual module, and level = 1 in the last context convolutional residual module. Here, when level = n, the convolutional kernel in the context convolutional block has a dilation rate size di = i, i = 1, …, n - 1, n.

[0022] As a preferred technical solution of the present invention, the learnable parameters and the number of floating-point operations of the context convolution in step S21 are calculated using formulas (1) and (2);

[0023] params = M in ·K w ·K h ·M out (1);

[0024] FLOPs = M in ·K h ·K w ·M out ·W out ·H out (2);

[0025] where, M in and M out represent the number of input and output feature maps, K w and K h represent the width and height of the convolutional kernel, and finally, W out and H outIndicates the width and height of the output feature map.

[0026] As a preferred technical solution of the present invention, the step S22 is specifically as follows:

[0027] S221: Denote the features extracted by CoResNet as X. First, use average pooling kernels of size (H, 1) and (1, W) to encode each channel along the horizontal coordinate direction and the vertical coordinate direction. The encoded output of the c-th channel with height h is calculated by formula (3):

[0028]

[0029] where (H, 1) and (1, W) respectively represent the sizes of the pooling kernels used along the horizontal coordinate direction and the vertical coordinate direction; H represents the height of the average pooling kernel used along the horizontal coordinate direction; W represents the width of the average pooling kernel used along the vertical coordinate direction; numerically, H and W are also equal to the height and width of the input feature map; this formula (3) calculates the encoded output of the c-th channel at height h in the horizontal coordinate direction, and sums the input features along width i;

[0030] The encoded output of the c-th channel with width w is calculated by formula (4):

[0031]

[0032] This formula (4) calculates the encoded output of the c-th channel at width w in the vertical coordinate direction, and sums the input features along height j, 0 ≤ j ≤ H;

[0033] S222: Aggregate the features of the two transformations in step S221 along the two spatial directions, and return a pair of direction-aware attention maps;

[0034] S223: Concatenate the pair of direction-aware attention maps generated in step S222, and send them into a 1×1 convolutional transformation function F;

[0035] f = δ(F([y h ,y w ))(5);

[0036] where [, ] represents the concatenation operation along the spatial dimension, δ is the non-linear sigmoid activation function, and f ∈ R C / r×(H+W) is the intermediate feature map encoding the spatial information in the horizontal and vertical directions; in order to reduce the complexity of the model, an appropriate reduction rate r is used to reduce the number of channels of f;

[0037] S224: Continue to decompose f into two separate tensors f h ∈R C / r×H and f w ∈R C / r×W along the spatial dimension, and use two 1×1 convolutional transforms F h and F w to transform f h and f w into tensors with the same number of channels respectively, using formulas (6) and (7);

[0038] m h = δ(F h (f h ))(6);

[0039] m w = δ(F w (f w ))(7);

[0040] where δ is the sigmoid function, and the output m h and m w are used as attention weights. Finally, the output Z of the coordinated attention module is shown by formula (8):

[0041]

[0042] where z c (i,j) is the output, x c (i,j) is the input, and are attention weights.

[0043] The above technical solution is adopted to focus on significant features and enhance the feature differences between expression classes. Therefore, the coordinated attention mechanism is used to embed the coordinated attention module (CA) in the context convolutional residual network for feature processing, so as to enhance the expression-related information in the input feature map and suppress redundant information. Embedding coordinated attention in the network can capture the long-range dependencies between input features along one spatial direction and retain the position information of the expression-related face regions in the other spatial direction. Then, the obtained feature map is encoded into a pair of direction-aware and position-sensitive attention maps, which are applied to the input feature map to enhance subtle expression information; adding the CA module after each context convolutional block and CoResNet to filter key scale features and emphasize significant face regions to enhance the feature representation ability, thereby improving the recognition performance.

[0044] As a preferred technical solution of the present invention, the specific steps of step S1 are as follows: First, adjust the size of the input image to 256×256, then crop it from top, bottom, left, right and the center to obtain five face images of 224×224, which have the same expression labels, and then randomly flip it horizontally with a probability of 0.5.

[0045] As a preferred technical solution of the present invention, the specific steps of step S3 are as follows:

[0046] S31: Perform multi-scale feature extraction and context space information integration on the input face expression image through a context convolutional residual network (CoResNet);

[0047] S32: Embed an attention module in each context convolutional residual module to focus on significant scale features, use coordinated attention on the output features of CoResNet to perform attention weighting on the extracted features, and capture the correlation of expression information and retain accurate face key region information in two spatial directions respectively;

[0048] S33: Perform downsampling operation on the attention-weighted features, and perform classification processing on the downsampled features.

[0049] As a preferred technical solution of the present invention, the context space information network model (ACSI) in step S3 includes a convolutional layer, a bn layer, a relu layer, a Maxpool layer, four context residual modules, a coordinated attention (CA) module, a global average pooling layer, an fc layer and a Softmax classification layer connected in sequence; the convolutional layer extracts features from the input face expression image through a 3*3 standard convolution operation; the bn layer performs batch normalization on the extracted features to prevent gradient disappearance or explosion; then the relu layer performs nonlinear activation on it; the max pooling layer is used for feature dimensionality reduction; the four context convolutional modules are used to extract multi-scale face features from the dimensionality-reduced features; the coordinated attention (CA) module embedded in the context convolutional module is used to focus on features of different scales; the CA module after the output feature layer of CoResNet performs attention weighting on the output features; the global average pooling layer and the fc layer perform downsampling operations, and the downsampled face expression features are then classified by a Softmax classifier.

[0050] As a preferred technical solution of the present invention, the input of the Softmax classifier is a vector of arbitrary real numbers, and the output is a vector, where the value of each element is between (0, 1) and their sum is 1. There is an array, and the calculation formula of its Softmax is formula (9):

[0051]

[0052] Where, xi denotes the i-th element, and softmax(x i ) represents the output value of the i-th element in the values taken by softmax. I is the number of elements, that is, the number of classification categories. Through the Softmax function, the output values of multi-classification can be converted into a probability distribution ranging from [0, 1] and whose sum is 1.

[0053] As a preferred technical solution of the present invention, before training the attention modulation context space information network model (ACSI) using the facial expression dataset in step S3, a large facial dataset MS-CELEB-1M with more than 10 million data (including 10 million facial images of nearly 100,000 subjects) is used as the training set to pre-train ACSI. Then, the facial expression datasets AffectNet-7 and RAF_DB are respectively input into the pre-trained ACSI model. After forward propagation, the output values (predicted class probabilities) are obtained. According to the output values, the cross-entropy loss function is used to calculate the loss value of the ACSI model. The calculation formula of the cross-entropy loss function is shown in formula (10):

[0054]

[0055] Among them, p(x) refers to the true class probability, and q(x) is the predicted class probability of the model;

[0056] In step S4, the network weights are updated by backpropagation according to the loss value of the ACSI model calculated according to formula (10), and the training is repeated until the set number of training times is reached to end the training, and the trained attention modulation context space information network model ACSI model is obtained.

[0057] Compared with the prior art, the beneficial effects of the facial expression recognition method based on attention modulation context space information of the present invention are as follows:

[0058] (1) Replace some convolutional layers in the residual network with context convolutional blocks composed of convolutional kernels with different dilation rates, access the context space information of the facial image on multiple network layers, extract more robust multi-scale expression features, and at the same time maintain the same number of parameters and computational costs as the standard convolutional layer of the same size;

[0059] (2) Use a new attention mechanism, that is, coordinated attention, which can capture the dependencies between discriminative local features along one spatial direction, and at the same time retain the precise position information of the key facial regions along the other spatial direction, so as to reduce the sensitivity of the deep network to occlusion and pose changes and strengthen the feature representation ability;

[0060] (3) The effectiveness and reliability of the constructed model for facial expression recognition in an uncontrolled environment were verified on two large natural environment facial expression image datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flowchart of the facial expression recognition method based on attention-modulated context space information of the present invention;

[0062] Figure 2 is a block diagram of the attention-coordinated context space information network (ACSI) model in the facial expression recognition method based on attention-modulated context space information of the present invention;

[0063] Figure 3 is a schematic diagram of the context convolution block in the facial expression recognition method based on attention-modulated context space information of the present invention;

[0064] Figure 4 is a schematic diagram of the structure of the coordinated attention module in the facial expression recognition method based on attention-modulated context space information of the present invention;

[0065] Figure 5 is the t-SNE visualization result of the features extracted by the baseline method and the ACSI50 model on the AffectNet-7 dataset; where (a) is the t-SNE visualization schematic diagram of the features extracted by the baseline method on the AffectNet-7 dataset; (b) is the t-SNE visualization schematic diagram of the features extracted by the ACSI50 model on the AffectNet-7 dataset;

[0066] Figure 6 is the t-SNE visualization result of the features extracted by the baseline method and the ACSI50 model in the facial expression recognition method based on attention-modulated context space information of the present invention on the RAF-DB; where (a) is the t-SNE visualization schematic diagram of the features extracted by the baseline method on the RAF-DB; (b) is the t-SNE visualization result schematic diagram of the features extracted by the ACSI50 model on the RAF-DB;

[0067] Figure 7 is a schematic diagram of the attention visualization result on the example facial expression images in the RAF_DB dataset in the facial expression recognition method based on attention-modulated context space information of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0069] Embodiment: As Figure 1 shown, the facial expression recognition method based on attention-modulated context space information specifically includes the following steps:

[0070] S1: Obtain the public dataset of natural scene face expression images to be trained, and preprocess the face expression images;

[0071] The specific steps of step S1 are as follows: First, adjust the input image size to 256×256, then perform cropping on its top, bottom, left, right, and center to obtain five 224×224 face images with the same expression label, and then randomly horizontally flip them with a probability of 0.5;

[0072] S2: Construct an attention modulation context space information network model for natural scene face expression recognition; First, use context convolution to replace the standard convolution in the convolution residual block to construct a context convolution residual network (CoResNet) as the feature extraction part, and capture local and merge global context information with convolution kernels of different dilation rates; Second, embed the coordinated attention module into CoResNet as the feature processing part, assign attention weights to the extracted features, highlight significant features, and increase the feature differences between expression classes, and finally form the ACSI model for face expression recognition;

[0073] The specific steps of step S2 include the following steps:

[0074] S21: Use the context convolution block to replace the middle convolution layer of the residual block to form a context convolution residual module and construct a context convolution residual network; In the deep face expression recognition task, multi-scale features are very important. While describing global semantic information, they can capture richer local detail information. The context convolution block contains convolution kernels with different dilation rates and can extract multi-scale features through receptive fields of different sizes; In CNN, the standard convolution only uses convolution kernels with a fixed receptive field, and its convolution kernel size is usually, because increasing the convolution kernel size will bring an increase in the number of parameters and calculation time; The number of learnable parameters (weights) and floating-point operations of the standard convolution can be calculated using formulas (1) and (2); Like the standard convolution layer, all convolution kernels in the context convolution block are independent and allow parallel execution. Different from the standard convolution layer, the same-sized context convolution has the ability to integrate context information while maintaining a similar number of parameters and calculation costs; Therefore, the context convolution block can be used as a direct replacement for the standard convolution layer to better complete feature extraction;

[0075] As Figure 2 shown, the specific steps of step S21 are as follows:

[0076] S211: The context convolution block receives the input feature map M in , which applies convolution kernels with different dilation rates D = {d1, d2, d3,..., d at different levels L = {1, 2, 3,..., n}n}, that is, level i (level i indicates that the convolutional kernel on the i-th level of the context convolution block has a dilation rate from level 1 to level n the dilation rate increases successively. Such an operation can extract increasingly extensive context information. Among them, the convolutional kernel with a lower dilation rate is responsible for capturing information about the local details of the face from the input feature map, while the convolutional kernel with a higher dilation rate is responsible for merging global context information, thereby helping the entire facial expression recognition process;

[0077] S212: At different levels level of the context convolution, the context convolution outputs multiple feature maps M outi , for all i ∈ L, each map has a width W out and a height H out ;

[0078] S213: Maintain the residual structure to combine the inter-layer correlations and obtain the context convolution residual module;

[0079] S214: Adjust the level (level) of the context convolution block in each layer according to the size of the feature map, thereby constructing the context convolution residual network; the context convolution residual network (CoResNet) constructed in step S2 includes CoResNet18 and CoResNet50; in CoResNet18, each context residual module is composed of a context convolution residual module and a 1*1 standard convolutional layer; in CoResNet50, each context residual module is composed of a context convolution residual module and two 1*1 standard convolutional layers; the context convolution residual module is used for multi-scale feature extraction, and the 1*1 standard convolutional layer is used for channel transformation; according to the size of the input feature map, different levels of context convolution blocks are used in each context residual module; the schematic diagram of the context convolution block is as Figure 3As shown, in the first context convolution residual module, a context convolution block at level 4 (level = 4) is used. In the first context convolution residual module, level = 3; in the third context convolution residual module, level = 2; and in the last context convolution residual module, level = 1. Here, when level = n, the convolution kernels in the context convolution block have dilation rates di = i, where i = 1, …, n - 1, n. Different from the previous work of network cascading, this technical solution directly integrates context convolution into the widely used residual network, improves the residual blocks in ResNet18 and ResNet50 respectively, and obtains the corresponding CoResNet18 and CoResNet50. CoResNet mainly consists of 4 network layers, and each layer has context convolution residual blocks at different levels. Since the size of the feature map decreases as the network layer is farther from the input, in this paper, the level of the context convolution block in each layer is adjusted according to the size of the feature map. CoConv4, that is, the context convolution block at level 4, is used in the first layer, CoConv3 in the second layer, and CoConv2 in the third layer. Since the resolution of the feature map input to the last layer has been reduced to 7×7, it is no longer reasonable to use context convolution at this time, so only a standard convolution is used, also denoted as CoConv1. The convolution parameter situations of CoConv at different levels are shown in Table 1.

[0080] Table 1 Convolution Parameter Situations of Context Convolution Residual Blocks

[0081]

[0082] In step S21, the learnable parameters and the number of floating-point operations of the context convolution are calculated using formulas (1) and (2).

[0083] params = M in ·K w ·K h ·M out (1);

[0084] FLOPs = M in ·K h ·K w ·M out ·W out ·H out (2);

[0085] Among them, M in and M out represent the number of input and output feature maps, K w and K h represent the width and height of the convolution kernel. Finally, W out and H outIndicates the width and height of the output feature map;

[0086] S22: Use Coordinate Attention (CA) to construct a Coordinate Attention Module (whose structure is as Figure 4 shown) to assign attention weights to the multi-scale features extracted by CoResNet to enhance the feature representation ability;

[0087] The specific steps of step S22 are as follows:

[0088] S221: Denote the features extracted by CoResNet as X. First, use average pooling kernels of size (H,1) and (1,W) to encode each channel along the horizontal coordinate direction and the vertical coordinate direction. The encoded output of the c-th channel with height h is calculated by formula (3):

[0089]

[0090] where (H,1) and (1,W) represent the sizes of the pooling kernels along the horizontal coordinate direction and the vertical coordinate direction respectively; H represents the height of the average pooling kernel along the horizontal coordinate direction; W represents the width of the average pooling kernel along the vertical coordinate direction; numerically, H and W are also equal to the height and width of the input feature map; this formula (3) calculates the encoded output of the c-th channel at height h in the horizontal coordinate direction, and performs a summation operation on the input features along width i;

[0091] The encoded output of the c-th channel with width w is calculated by formula (4):

[0092]

[0093] This formula (4) calculates the encoded output of the c-th channel at width w in the vertical coordinate direction, and performs a summation operation on the input features along height j, 0 ≤ j ≤ H;

[0094] S222: Aggregate the features of the two transformations in step S221 along the two spatial directions, and return a pair of direction-aware attention maps;

[0095] S223: Connect the pair of direction-aware attention maps generated in step S222, and send them into a 1×1 convolutional transformation function F;

[0096] f = δ(F([y h ,y w ))(5);

[0097] Among them, [,] represents the concatenation operation along the spatial dimension, δ is the non-linear sigmoid activation function, and f ∈ R C / r×(H+W) is the intermediate feature map for encoding spatial information in the horizontal and vertical directions; to reduce the complexity of the model, an appropriate reduction rate r is used to reduce the number of channels of f;

[0098] S224: Continue to decompose f into two separate tensors f h ∈ R C / r×H and f w ∈ R C / r×W , and use two 1×1 convolutional transforms F h and F w to transform f h and f w into tensors with the same number of channels respectively, using formulas (6) and (7);

[0099] m h = δ(F h (f h )) (6);

[0100] m w = δ(F w (f w )) (7);

[0101] Among them, δ is the sigmoid function, and the output m h and m w serve as attention weights. Finally, the output Z of the coordinated attention module is shown by formula (8):

[0102]

[0103] where z c (i, j) is the output, x c (i, j) is the input, and are the attention weights;

[0104] To focus on significant features and enhance the feature differences between expression classes, a coordinated attention mechanism is adopted, and a coordinated attention module (CA) is embedded in the context convolutional residual network for feature processing to enhance the expression-related information in the input feature map and suppress redundant information. As shown in the figure, embedding coordinated attention in the network can capture the long-range dependencies between input features along one spatial direction and retain the position information of the expression-related face regions in the other spatial direction. Then, the obtained feature map is encoded into a pair of direction-aware and position-sensitive attention maps, which are applied to the input feature map to enhance subtle expression information; a CA module is added after each context convolutional block and CoResNet to filter key-scale features and emphasize significant face regions to enhance the feature representation ability, thereby improving the recognition performance.

[0105] S3: Train the context spatial information (ACSI) network model using the preprocessed facial expression images; the context spatial information network model (ACSI) in step S3 includes a convolutional layer, a bn layer, a relu layer, a Maxpool layer, four context residual modules, a coordinated attention (CA) module, a global average pooling layer, an fc layer, and a Softmax classification layer connected in sequence; the convolutional layer extracts features from the input facial expression images through a 3*3 standard convolution operation; the bn layer performs batch normalization on the extracted features to prevent gradient disappearance or explosion; then the relu layer performs non-linear activation on it; the max pooling layer is used for feature dimensionality reduction; the four context convolutional modules are used to extract multi-scale facial features from the dimensionality-reduced features; the coordinated attention (CA) module embedded in the context convolutional module is used to focus on features at different scales; the CA module after the CoResNet outputs features performs attention weighting on the output features; the global average pooling layer and the fc layer perform downsampling operations, and the downsampled facial expression features are then classified by a Softmax classifier;

[0106] The specific steps of step S3 are as follows:

[0107] S31: Perform multi-scale feature extraction and context spatial information integration on the input facial expression images through a context convolutional residual network (CoResNet);

[0108] S32: Embed an attention module in each context convolutional residual module to focus on significant-scale features, and use coordinated attention to perform attention weighting on the features extracted by the CoResNet output features, capturing the correlation of expression information and retaining the precise face key region information in two spatial directions respectively;

[0109] S33: Perform a downsampling operation on the attention-weighted features, and use a Softmax classifier to perform classification processing on the downsampled features;

[0110] The input of the Softmax classifier is a vector of arbitrary real numbers, and the output is a vector, where the value of each element is between (0, 1) and their sum is 1. There is an array, and the calculation formula of Softmax is formula (9):

[0111]

[0112] where, x i represents the i-th element, and softmax(x i ) represents the output value of the i-th element in the softmax value. I is the number of elements, that is, the number of classification categories; through the Softmax function, the output values of multi-classification can be converted into a probability distribution with a range of [0, 1] and their sum is 1;

[0113] Before training the attention modulation context space information network model (ACSI) using the facial expression dataset in step S3, a large facial dataset MS-CELEB-1M with more than 10 million data (including 10 million facial images of nearly 100,000 subjects) was used as the training set to pre-train ACSI. Then, the facial expression datasets AffectNet-7 and RAF_DB were respectively input into the pre-trained ACSI model, and the output value (predicted class probability) was obtained through forward propagation. According to the output value, the loss value of the ASCP model was calculated using the cross-entropy loss function; the calculation formula of the cross-entropy loss function is shown in formula (10):

[0114]

[0115] where, p(x) refers to the true class probability, and q(x) is the predicted class probability of the model;

[0116] S4: Repeat the model training in step S3 until the set number of training times is reached, and the trained deep residual network model is obtained. Use the trained deep residual network model for facial expression recognition; step S4 updates the network weights by backpropagation according to the loss value of the ACSI model calculated by formula (10), and repeats the training until the training ends when the set number of training times is reached, and the trained attention modulation context space information network model ACSI model is obtained.

[0117] Specific application examples: Adopting the above technical solutions, in order to verify the effectiveness of the ASCP model proposed in this paper, experiments were conducted on two public facial expression databases, AffectNet and RAF-DB. Both of these databases provide face images in natural scenes. Among them, the AffectNet database is one of the largest databases in the field of facial emotion computing research, with approximately 440,000 face images, including AffectNet-7 and AffectNet-8 (with the addition of the "contempt" category); the RAF-DB database includes 7 basic facial expressions and 12 compound facial expressions, with a total of approximately 30,000 face images. As shown in Table 2, the face images of 7 basic facial expressions (happy, surprised, sad, angry, disgusted, fearful, and neutral) in the AffectNet-7 and RAF-DB databases were used as the training set in this experiment. Since the test set was not available, the test was conducted on the corresponding validation set to evaluate the performance of the proposed model.

[0118] In step S1, during the image preprocessing stage, first, the input image size was adjusted to 256×256, and then it was cropped top, bottom, left, right, and in the center to obtain five face images of 224×224, which had the same expression label. Then, it was randomly horizontally flipped with a probability of 0.5; the model was implemented using pytorch, and the model training was carried out on an NVIDIA GeForce GTX1650 GPU. During the training process, the SGD algorithm was used for optimization, with the momentum set to 0.9, the initial learning rate to 0.01, the learning rate decreased to 0.1 of the original every 20 iterations, the total number of iterations to 60, and the batch size to 16.

[0119] Table 2 Details of the experimental datasets, including the expression categories, the numbers of the training set and the test set

[0120]

[0121] The experimental results of the proposed facial expression recognition method based on attention-modulated context spatial information on the AffectNet-7 and RAF-DB validation sets are shown in Table 3. Among them, CoResNet18 and CoResNet50 (the baseline models of this paper) are context convolutional residual networks, while CoResNet18_CA_a and CoResNet50_CA_a respectively embed the coordinated attention module after the feature output layer of CoResNet18 and CoResNet50; CoResNet18_CA_b and CoResNet50_CA_b embed the coordinated attention module in each context convolutional residual block of the corresponding CoResNet.

[0122] Table 3 Recognition accuracy of the ACSI model on the AffectNet-7 and RAF-DB validation sets

[0123]

[0124]

[0125] As can be seen from Table 3, on the AffectNet-7 validation set, the facial expression recognition accuracy of ACSI18 increased by 1.70% compared to CoResNet18, and increased by 1.36% and 1.30% compared to CoResNet18_CA_a and CoResNet18_CA_b respectively. The facial expression recognition accuracy of ACSI50 increased by 2.03% compared to CoResNet50, and increased by 0.80% and 0.25% compared to CoResNet50_CA_a and CoResNet50_CA_b respectively. On the RAF_DB validation set, the facial expression recognition accuracy of ACSI18 increased by 1.89% compared to CoResNet18, and increased by 1.23% and 1.14% compared to CoResNet18_CA_a and CoResNet18_CA_b respectively; the facial expression recognition accuracy of ACSI50 increased by 1.79% compared to CoResNet50, and increased by 0.35% and 0.06% compared to CoResNet50_CA_a and CoResNet50_CA_b respectively; the above experimental results demonstrate the effectiveness and generalization of the algorithm in this paper.

[0126] To further illustrate the effectiveness of the context space information (ACSI) network model constructed in the proposed expression recognition method based on attention modulation of context space information, the constructed context space information (ACSI) network model was compared with other similar models in recent years on the datasets AffectNet-7 and RAF-DB, as shown in Tables 4 and 5. As can be seen from Table 4, the ACSI50 proposed in this paper improved by 1.61% compared with FMPN, 0.97% compared with OADN, 0.75% compared with Ensemble CNN, and 0.52% compared with the DDA-Loss method on AffectNet-7. As can be seen from Table 5, the ACSI50 proposed in this paper improved by 2.5% compared with FSN, 0.91% compared with CNN, 0.76% compared with DLP-CNN, and 0.33% compared with pACNN on RAF_DB. The results show that the recognition accuracy of the proposed model has been improved to a certain extent on AffectNet-7 and RAF-DB, and it has certain competitiveness compared with similar models. Since these models fail to solve the problems of limited feature completeness or fuzzy inter-class classification boundaries well, the recognition performance is low. The proposed model in this paper can extract multi-scale facial expression features using context convolution, embed a coordinated attention module in the network to enable the network to focus on more discriminative expression features, and combine the inter-layer correlations better through the residual structure, ultimately improving the recognition performance.

[0127] Table 4 Performance comparison of models on AffectNet-7

[0128]

[0129]

[0130] Table 5 Performance comparison of models on RAF-DB

[0131] Method Acc.% FSN 81.10 CNN 82.69 DLP-CNN 82.84 pACNN 83.27 ACSI18 83.47 ACSI50 83.60

[0132] To prove the inter-class difference of the expression features extracted by the ACSI model, in this section, the features extracted by the ACSI50 model on the AffectNet-7 and RAF-DB validation sets were visualized by t-SNE, and the results are as Figure 5 and Figure 6 shown. Figure 2-6Seven basic facial expression classifications are shown, including Anger, Digust, Fear, Happiness, Sadness, Surprise, and Neutral. As can be seen from the figure, compared with the baseline model, the features extracted by the ACSI50 model have the distribution characteristics of relatively dispersed between classes and relatively concentrated within classes.

[0133] To further study the role of the attention module in the model, the Class Activation Map (CAM) method is used to visualize the attention map generated by the attention in this paper. The CAM method is used to visualize the activated parts of different expressions, mapping the weights of the output layer to the convolutional feature map to identify the importance of different regions of the face image. Specifically, the CAM is used to visualize the facial activation regions for the proposed network ACSI to obtain the attention map. To display the attention region on the original image, generally, the attention map is adjusted to the same size as the input image, and the attention map is visualized on the original image through the COLORMAP_JET color mapping. The specific steps of this technical solution when used are as follows: First, the visualization attention map is adjusted to the same size as the input image, and the attention map is visualized on the original image through color mapping; Figure 7 Shows the attention maps of different expression images in RAF_DB. There are 7 columns in this figure, each column showing one of the seven expressions. From left to right, they are Anger, Digust, Fear, Happiness, Sadness, Surprise, and Neutral. The first row shows the original aligned facial images, and the second row shows the attention visualization of the corresponding expression images by the ACSI50 model. From Figure 7 it can be clearly seen that in the presence of occlusion and pose changes, the attention module used in this paper enables the network to focus on more discriminative face regions; the results show that the combination of context convolution and coordinated attention can significantly improve the performance of expression recognition. On the publicly available expression dataset, compared with similar algorithms, ACSI has higher recognition performance.

[0134] The above is only a preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An expression recognition method based on attention modulation of context space information, characterized in that, Specifically, it includes the following steps: S1: Obtain a public dataset of natural scene facial expression images to be trained, and preprocess the facial expression images; S2: Construct an attention modulation context space information network model ACSI for natural scene facial expression recognition; S3: Use the preprocessed facial expression images to train the context space information network model ACSI; S4: Repeat the model training in step S3 until the set number of training times is reached to obtain a trained deep residual network model, and use the trained deep residual network model for facial expression recognition; The specific steps of step S2 are as follows: S21: Replace the middle convolutional layer of the residual block with a context convolutional block to form a context convolutional residual module and construct a context convolutional residual network; S22: Use coordinated attention to construct a coordinated attention CA module to assign attention weights to the multi-scale features extracted by the context convolutional residual network CoResNet constructed in step S21 to enhance the feature representation ability; The specific steps of step S21 are as follows: S211: The context convolution block receives the input feature map M in , which applies convolution kernels D = {d1, d2, d3,..., d n} with different dilation rates at different levels L = {1, 2, 3,..., n}, that is, the convolution kernel at level i has a dilation rate of S212: At the level of context convolution, context convolution outputs multiple feature maps for different levels of convolution For all i ∈ L, each map has a width W out and a height H out ; S213: Keep the residual structure to combine the inter-layer correlations to obtain a context convolutional residual module; S214: Adjust the level of the context convolutional block in each layer according to the size of the feature map, so as to construct a context convolutional residual network; The learnable parameters and the number of floating-point operations of the context convolution in step S21 are calculated using formulas (1) and (2); params = M in ·K w ·K h ·M out (1); FLOPs = M in ·K h ·K w ·M out ·W out ·H out (2); Among them, M in and M out represent the number of input and output feature maps, K w and K h represent the width and height of the convolutional kernel. Finally, W out and H out represent the width and height of the output feature map; The specific steps of step S22 are as follows: S221: Denote the features extracted by CoResNet as X. First, use average pooling kernels of size (H, 1) and (1, W) to encode each channel along the horizontal and vertical coordinate directions. The encoded output of the c-th channel with height h is calculated by Equation (3): The formula (3) calculates the encoded output of the c-th channel at a height of h in the horizontal coordinate direction, summing the input features along the width i; the encoded output of the c-th channel with a width of w is calculated by formula (4): This formula (4) calculates the encoded output of the c-th channel at height w in the vertical coordinate direction, and performs a summation operation on the input features along height j, 0 ≤ j ≤ H; S222: Aggregate the features of the two transformations in step S221 along the two spatial directions and return a pair of direction-aware attention maps; S223: Connect the pair of direction-aware attention maps generated in step S222 and send them into a 1×1 convolutional transformation function F; f = δ(F([y h , y w )) (5); where [,] represents the concatenation operation along the spatial dimension, δ is the non-linear sigmoid activation function, and f ∈ R C / r×(H+W) is the intermediate feature map for encoding spatial information in the horizontal and vertical directions; S224: Continue to decompose f into two separate tensors f h ∈R C / r×H and f w ∈R C / r×W , and use two 1×1 convolutional transforms F h and F w to transform f h and f w into tensors with the same number of channels respectively, using formulas (6) and (7); m h = δ(F h (f h )) (6); m w = δ(F w (f w )) (7); where δ is the sigmoid function, and the output m h and m w are used as the attention weights. Finally, the output Z of the coordinated attention module is shown in Equation (8): where z c (i, j) is the output, x c (i, j) is the input, and are the attention weights; The context space information network model ACSI in step S3 includes a convolutional layer, a bn layer, a relu layer, a Maxpool layer, four context residual modules, a coordinated attention CA module, a global average pooling layer, an fc layer, and a Softmax classification layer connected in sequence; the convolutional layer extracts features from the input facial expression images through a 3*3 standard convolution operation; the bn layer performs batch normalization on the extracted features to prevent gradient disappearance or explosion; then the relu layer performs non-linear activation on it; the max pooling layer is used for feature dimensionality reduction; the four context convolutional modules are used to extract multi-scale facial features from the dimensionality-reduced features; the coordinated attention CA module embedded in the context convolutional module is used to focus on features of different scales; the CA module after the CoResNet outputs features performs attention weighting on the output features; the global average pooling layer and the fc layer perform downsampling operations, and the downsampled facial expression features are then classified by a Softmax classifier.

2. The method for facial expression recognition based on attention modulation of context space information according to claim 1, characterized in that The specific steps of step S1 are as follows: First, adjust the size of the input image to 256×256, then perform cropping on its top, bottom, left, right, and center to obtain five face images of 224×224, which have the same expression labels, and then horizontally flip them with a probability of 0.

5.

3. The facial expression recognition method based on attention modulation of context space information according to claim 1, wherein The specific steps of step S3 are as follows: S31: Perform multi-scale feature extraction and context space information integration on the input face expression image through the context convolutional residual network CoResNet; S32: Embed an attention module in each context convolutional residual module to focus on significant scale features, use coordinated attention on the output features of CoResNet to perform attention weighting on the extracted features, and capture the correlation of expression information and retain the information of key face regions in two spatial directions respectively; S33: Perform downsampling operations on the attention-weighted features and classify the downsampled features.

4. The method for facial expression recognition based on attention modulation of context space information according to claim 3, wherein The input of the Softmax classifier is a vector of arbitrary real numbers, and the output is a vector, where the value of each element is between (0, 1) and their sum is 1. There is an array, and the calculation formula of its Softmax is formula (9): Among them, x i represents the i-th element, and softmax(x i ) represents the output value of the i-th element among the values taken by softmax. I is the number of elements, that is, the number of classification categories. By using the Softmax function, the output values of multi-classification can be converted into a probability distribution with a range of [0, 1] and the sum of 1.

5. The method for facial expression recognition based on attention modulation of context space information according to claim 3, wherein Before training the attention modulation context space information network model ACSI using the face expression dataset in step S3, a large face dataset MS-CELEB-1M with more than 10 million data is used as the training set to pre-train ACSI. Then, the face expression datasets AffectNet-7 and RAF_DB are respectively input into the pre-trained ACSI model, and the output values are obtained through forward propagation. According to the output values, the cross-entropy loss function is used to calculate the loss value of the ACSI model; the calculation formula of the cross-entropy loss function is shown in formula (10): Among them, p(x) refers to the true class probability, and q(x) is the predicted class probability of the model; In step S4, the network weights are updated by backpropagation according to the loss value of the ACSI model calculated by formula (10), and the training is repeated until the set number of training times is reached, and then the training ends to obtain the trained attention modulation context space information network model ACSI model.

Images