Lightweight facial expression recognition method and system based on attention mechanism

Abstract

The present application relates to a kind of lightweight facial expression recognition method and system based on attention mechanism, comprising the following steps: establishing convolution model, the picture of data set is cropped, and the picture is preprocessed, and the picture after pre-processing is input into convolution model;Picture is extracted in convolution model, attention mechanism is recalibrated and down-sampling operation is carried out, and the feature map of final output is obtained;Vector in feature map is classified into expression, and recognition result is obtained;Establish loss function model, use recognition result to train model parameter and test, complete facial expression recognition.Ghost convolution in GhostNet network is introduced to reduce the parameter amount of pointwise convolution, in order to eliminate the interference of expression irrelevant factors, the improved coordinate attention mechanism can simultaneously focus on the position information and context information of expression image, increase the weight of key area of facial expression image, improve the performance of identification, compared with conventional expression recognition method, model parameter amount is less, and recognition accuracy is higher.

Description

Technical Field

[0001] This invention relates to the field of computer vision facial image processing, and more specifically, to a lightweight facial expression recognition method and system based on an attention mechanism. Background Technology

[0002] The primary way machines discern human emotions is by detecting global facial information and then analyzing key information related to facial expressions. Therefore, how to quickly and accurately extract key information from facial expressions is a pressing issue that needs to be addressed in the field of facial expression recognition.

[0003] Early facial expression recognition methods largely relied on traditional image processing and machine learning. However, these methods suffer from the inability to extract deep-level features from facial expression images, especially when the images contain complex background information. In such cases, the accuracy drops significantly, failing to meet the demands of current facial expression recognition tasks. In recent years, with the rapid development of deep learning in computer vision, numerous excellent convolutional neural network (CNN) models have been applied to facial expression recognition. By stacking convolutional layers, adding attention mechanisms, and improving loss functions, considerable recognition results have been achieved. However, these CNN methods typically have a large number of parameters and complex model structures. For real-time tasks like facial expression recognition, existing methods are not a suitable choice.

[0004] In existing technologies, Chinese invention patents provide a method and apparatus for obtaining an expression recognition model, an expression recognition method and apparatus, a storage medium, and an electronic device. The method for obtaining the expression recognition model includes: acquiring multiple sets of first training data, wherein each set of first training data includes: an image, a face corresponding to the image, and an expression corresponding to the face; constructing an initial expression recognition model based on a face recognition model; and training the initial expression recognition model using the multiple sets of first training data through deep learning to obtain the expression recognition model. This invention solves the problems of low accuracy, poor generalization, and poor stability in facial expression recognition due to the expression recognition model in related technologies. However, it cannot eliminate interference from irrelevant factors, and the model has a large number of parameters, making modeling and computation cumbersome. Summary of the Invention

[0005] To address the shortcomings of existing facial expression recognition methods, such as significant interference from irrelevant factors and a large number of model parameters, this invention provides a lightweight facial expression recognition method and system based on an attention mechanism.

[0006] To achieve the above-mentioned objectives, the technical solution adopted is as follows:

[0007] A lightweight facial expression recognition method based on an attention mechanism includes the following steps:

[0008] S1: Build a convolutional model, crop the images in the dataset, preprocess the images, and input the preprocessed images into the convolutional model;

[0009] S2: The image undergoes lightweight Ghost convolution feature extraction, attention mechanism recalibration, and downsampling operations in the convolution model to obtain the final output feature map;

[0010] S3: Perform facial expression classification on the vectors in the feature map to obtain the recognition result;

[0011] S4: Establish a loss function model, use the recognition results to train the model parameters and test them to complete facial expression recognition.

[0012] In the above scheme, Ghost convolution is introduced to reduce the number of parameters in pointwise convolution. In order to eliminate the interference of factors irrelevant to the expression, an attention mechanism is designed to simultaneously pay attention to the positional information and contextual information of the expression image, thereby increasing the weight of key regions of the facial expression image and improving the recognition performance. Compared with conventional expression recognition methods, the model has fewer parameters and higher recognition accuracy.

[0013] Preferably, in step S1, the input image is randomly cropped to 48×48 and then rotated and flipped horizontally at a small angle.

[0014] In the above scheme, the images in the dataset are first adjusted by cropping them to 48×48 pixels and then preprocessed, i.e., a series of data augmentation operations are used to increase the training data. Specifically, the image cropping involves randomly cropping the input images, rotating them at a small angle, and flipping them horizontally to prevent overfitting due to insufficient training data.

[0015] Preferably, in step S2, lightweight feature extraction includes the following steps: extracting features using 1×3 and 3×1 convolutions, and then extracting features using Ghost convolutions; repeating the above operations, and then concatenating the features with the input image to obtain a feature map; then performing a shuffling operation on the channels to obtain the final output feature map.

[0016] Preferably, in step S2, the downsampling operation includes the following steps: using a 3×3 depthwise convolution with a stride of 2 to perform a downsampling operation, the size of the output feature map is halved; using a global average pooling layer and a 1×1 convolution to replace the final downsampling operation, resulting in an output feature map of 1024×1×1.

[0017] In the above scheme, a 3×3 convolution is first used to extract shallow features, with 3 channels as input and 32 channels as output. The model then enters the Lightweight Feature Extraction (LFE) module, which uses 1×3 and 3×1 convolutions to extract features, and then replaces pointwise convolution with a Ghost convolution module. This process is repeated, and the features are then concatenated with the input feature map to obtain a new feature map. Channel shuffling is then performed to enhance information exchange, resulting in the final output feature map. The improved Attention Module (LCA) then recalibrates the input features. Next, a 3×3 depthwise convolution with a stride of 2 is used for downsampling, halving the size of the final output feature map. Finally, a global average pooling layer and a 1×1 convolution replace the final downsampling module at the end of the model, resulting in a final output feature map of 1024×1×1.

[0018] Preferably, in step S3, the features obtained by the global average pooling layer are fed into the fully connected layer. Specifically, 1024 vectors are mapped into 7 vectors and fed into the classifier for expression classification to obtain the final recognition result.

[0019] Preferably, in step S4, the loss function model is built using the PyTorch framework, validated using the Fer2013 dataset, and supervised using the loss function, specifically as follows:

[0020]

[0021] Where e represents the number of expression categories, and N is the total number of samples. W These are weight parameters. y For tags, x For input data.

[0022] A lightweight facial expression recognition system based on an attention mechanism utilizes step S2 of the aforementioned lightweight facial expression recognition method based on an attention mechanism. The convolutional model in step S2 includes a feature extraction module, an attention module, and a downsampling module. The output of the feature extraction module is electrically connected to the input of the attention module, and the output of the attention module is electrically connected to the input of the downsampling module.

[0023] Preferably, in the feature extraction module, the 3×3 ordinary convolution is first decomposed and replaced by two decomposed convolutions, 1×3 and 3×1, concatenated. The 1×1 pointwise convolution after the depthwise convolution is also replaced. The Ghost convolution is used to significantly reduce the parameters. Finally, the two are concatenated.

[0024] In the above scheme, the ordinary 3×3 convolution is first decomposed and replaced by two concatenated decomposed convolutions, 1×3 and 3×1. To further reduce the number of network parameters, the 1×1 pointwise convolution after the depthwise convolution is also replaced by a novel Ghost convolution that significantly reduces the number of parameters. The working principle of this convolution is as follows: First, a 1×1 convolution is performed on the input feature map, where the number of convolution kernels is half of the input channels, generating a feature map x1∈R. C / 2×H×W Then, a 3×3 depthwise convolution is used for feature extraction to obtain the feature map x2∈R. C / 2×H×W Finally, the two are concatenated, as described by the formula: F = concat(g(x1), g(x2)), F ∈ R C×H×W Using this novel Ghost convolution instead of 1×1 convolution reduces the number of network parameters while enriching the receptive field of the network, making it easier for the network to extract facial detail information.

[0025] To extract depth features, this module repeats the above convolution process twice, and the final generated feature map is Fo∈R. C×H×W To better integrate shallow features, the input features Fi∈R of this module are... C×H×W With output features Fo∈R C ×H×W The output feature Mo∈R is obtained by concatenating and fusing the features. 2C×H×W The purpose of using concatenation is twofold: first, to increase the dimensionality of channels; and second, to reduce pixel-by-pixel addition operations, thereby accelerating the model's recognition speed. Finally, channel shuffling is used on feature Mo to enhance the exchange of information between features and improve feature expressive power.

[0026] The formula for calculating the number of parameters in a typical convolution is: K × K × Ci × Co. The convolution calculation formula for this module is: 1 × K × Ci + K × 1 × Ci + Ci × Ci / 2 + K × K × Ci / 2. K is the kernel size, Ci is the number of input channels, and Co is the number of output channels. As can be seen from the above formulas, the method proposed in this invention significantly reduces the number of parameters.

[0027] Preferably, the attention module includes a context feature extraction branch and a coordinate attention branch;

[0028] For the coordinate attention branch, the feature information is first obtained through adaptive global max pooling and average pooling. Then, the features of the two are concatenated and a 1×1 convolution is used to generate a feature map. The feature map is then split into features in two directions and the information is re-encoded using a 1×1 convolution. Finally, the attention map is generated using the Sigmoid function.

[0029] For the context feature extraction branch, dilated convolution is first used to extract context features, and then 1×1 convolution is used to aggregate relevant information of the channels to generate a feature map. Finally, the generated attention feature map is fused with the coordinate attention branch feature map by multiplying the two feature maps element by element.

[0030] The above scheme primarily utilizes two parallel branches for attention: a contextual feature extraction branch and a coordinate attention branch. For the coordinate attention branch, adaptive global max pooling and average pooling are first applied to obtain feature information Za and Zm. Then, these two features are concatenated, and a 1×1 convolution is used to generate a feature map Mc=F. 1×1 (concat(Za,Zm)), then split the feature map into features in two directions, re-encode the information using a 1×1 convolution, and finally use the Sigmoid function to generate an attention map: Ch=F 1×1 (fh), Cw=F 1×1 (fw). Where Ch∈R C×H×1 ,Cw∈R C×1×W .

[0031] For the context feature extraction branch, dilated convolution is first used to extract context features. Dilated convolution does not increase the number of network parameters, but it can increase the receptive field. Then, 1×1 convolution is used to aggregate the relevant information of each channel to generate a feature map Mg=F. 1×1 (F) dilated (x)), Mg ∈R 1×H×W Finally, the generated attention feature map Mg is fused with the coordinate attention branch feature map. Specifically, the two feature maps are multiplied element-wise: Ao = Ch * Cw * Mg, Ao ∈ R. C×H×W .

[0032] The improved coordinate attention of this invention can not only capture channel correlation and the location information of encoded features, but also use dilated convolution to extract contextual features, which is beneficial to improving the overall performance of the model.

[0033] Preferably, in the downsampling module, a 3×3 depthwise convolution is used instead of a pooling layer, and then the convolution outputs a feature map.

[0034] In the above scheme, the model proposed in this invention uses a 3×3 depthwise convolution instead of a pooling layer, solving the problem of key information loss caused by using pooling layers with a small computational cost. The feature map output by this convolution is: D=F 3×3 (Ao), D∈R C×H / 2×W / 2 .

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] This invention provides a lightweight facial expression recognition method and system based on an attention mechanism. It introduces Ghost convolution to reduce the number of parameters in pointwise convolution. In order to eliminate the interference of expression-irrelevant factors, the coordinate attention mechanism is improved so that it can simultaneously pay attention to the positional information and contextual information of the expression image, thereby increasing the weight of key regions of the facial expression image and improving the recognition performance. Compared with conventional expression recognition methods, the model has fewer parameters and higher recognition accuracy. Attached Figure Description

[0037] Figure 1 This is a flowchart of the method of the present invention;

[0038] Figure 2 This is a flowchart illustrating the overall operation of the present invention;

[0039] Figure 3 This is a diagram of the overall model structure of the present invention;

[0040] Figure 4 This is a diagram of the improved attention module of the present invention;

[0041] Figure 5 This is a diagram of the feature extraction module of the present invention;

[0042] Figure 6 This is a schematic diagram of the Ghost convolution introduced in this invention. Detailed Implementation

[0043] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] Example 1

[0046] like Figure 1 and Figure 2 As shown, a lightweight facial expression recognition method based on an attention mechanism includes the following steps:

[0047] S1: Build a convolutional model, crop the images in the dataset, preprocess the images, and input the preprocessed images into the convolutional model;

[0048] S2: The image undergoes feature extraction, attention mechanism recalibration, and downsampling operations in the convolutional model to obtain the final output feature map;

[0049] S3: Perform facial expression classification on the vectors in the feature map to obtain the recognition result;

[0050] S4: Establish a loss function model, use the recognition results to train the model parameters and test them to complete facial expression recognition.

[0051] In the above scheme, Ghost convolution is introduced to reduce the number of parameters in pointwise convolution. In order to eliminate the interference of expression-irrelevant factors, the coordinate attention mechanism is improved so that it can simultaneously pay attention to the positional information and contextual information of the expression image, thereby increasing the weight of key regions of the facial expression image and improving the recognition performance. Compared with conventional expression recognition methods, the model has fewer parameters and higher recognition accuracy.

[0052] Preferably, in step S1, the input image is randomly cropped to 48×48 and then rotated and flipped horizontally at a small angle.

[0053] In the above approach, the images in the dataset are first adjusted by cropping them to 48×48 pixels and then preprocessed, i.e., a series of data augmentation operations are used to increase the training data. Specifically, the image cropping involves randomly cropping the input images, rotating them at a small angle, and flipping them horizontally to prevent overfitting due to insufficient training data.

[0054] Preferably, in step S2, lightweight feature extraction includes the following steps: extracting features using 1×3 and 3×1 convolutions, and then extracting features using Ghost convolutions; repeating the above operations, and then concatenating the features with the input image to obtain a feature map; then performing a shuffling operation on the channels to obtain the final output feature map.

[0055] Preferably, in step S2, the downsampling operation includes the following steps: using a 3×3 depthwise convolution with a stride of 2 to perform a downsampling operation, the size of the output feature map is halved; using a global average pooling layer and a 1×1 convolution to replace the final downsampling operation, resulting in an output feature map of 1024×1×1.

[0056] In the above scheme, a 3×3 convolution is first used to extract shallow features, with 3 channels as input and 32 channels as output. The model then enters the Lightweight Feature Extraction (LFE) module, which uses 1×3 and 3×1 convolutions to extract features, and then replaces pointwise convolution with a Ghost convolution module. This process is repeated, and the features are then concatenated with the input feature map to obtain a new feature map. Channel shuffling is then performed to enhance information exchange, resulting in the final output feature map. The improved Attention Module (LCA) then recalibrates the input features. Next, a 3×3 depthwise convolution with a stride of 2 is used for downsampling, halving the size of the final output feature map. Finally, a global average pooling layer and a 1×1 convolution replace the final downsampling module at the end of the model, resulting in a final output feature map of 1024×1×1.

[0057] Preferably, in step S3, the features obtained by the global average pooling layer are fed into the fully connected layer. Specifically, 1024 vectors are mapped into 7 vectors and fed into the classifier for expression classification to obtain the final recognition result.

[0058] Preferably, in step S4, the loss function model is built using the PyTorch framework, validated using the Fer2013 dataset, and supervised using the loss function, specifically as follows:

[0059]

[0060] Where e represents the number of expression categories, and N is the total number of samples. W These are weight parameters. y For tags, x For input data.

[0061] Example 2

[0062] like Figures 3-6 As shown, a lightweight facial expression recognition system based on an attention mechanism utilizes step S2 of the aforementioned lightweight facial expression recognition method based on an attention mechanism. The convolutional model in step S2 includes a feature extraction module, an attention module, and a downsampling module. The output of the feature extraction module is electrically connected to the input of the attention module, and the output of the attention module is electrically connected to the input of the downsampling module.

[0063] Preferably, in the feature extraction module, the 3×3 ordinary convolution is first decomposed and replaced by two decomposed convolutions, 1×3 and 3×1, concatenated. The 1×1 pointwise convolution after the depthwise convolution is also replaced. The Ghost convolution is used to significantly reduce the parameters. Finally, the two are concatenated.

[0064] In the above scheme, the ordinary 3×3 convolution is first decomposed and replaced by two concatenated decomposed convolutions, 1×3 and 3×1. To further reduce the number of network parameters, the 1×1 pointwise convolution after the depthwise convolution is also replaced by a novel Ghost convolution that significantly reduces the number of parameters. The working principle of this convolution is as follows: First, a 1×1 convolution is performed on the input feature map, where the number of convolution kernels is half of the input channels, generating a feature map x1∈R. C / 2×H×W Then, a 3×3 depthwise convolution is used for feature extraction to obtain the feature map x2∈R. C / 2×H×W Finally, the two are concatenated, as described by the formula: F = concat(g(x1), g(x2)), F ∈ R C×H×W Using this novel Ghost convolution instead of 1×1 convolution reduces the number of network parameters while enriching the receptive field of the network, making it easier for the network to extract facial detail information.

[0065] To extract depth features, this module repeats the above convolution process twice, and the final generated feature map is Fo∈R. C×H×WTo better integrate shallow features, the input features Fi∈R of this module are... C×H×W With output features Fo∈R C ×H×W The output feature Mo∈R is obtained by concatenating and fusing the features. 2C×H×W The purpose of using concatenation is twofold: first, to increase the dimensionality of channels; and second, to reduce pixel-by-pixel addition operations, thereby accelerating the model's recognition speed. Finally, channel shuffling is used on feature Mo to enhance the exchange of information between features and improve feature expressive power.

[0066] The formula for calculating the number of parameters in a typical convolution is: K × K × Ci × Co. The convolution calculation formula for this module is: 1 × K × Ci + K × 1 × Ci + Ci × Ci / 2 + K × K × Ci / 2. K is the kernel size, Ci is the number of input channels, and Co is the number of output channels. As can be seen from the above formulas, the method proposed in this invention significantly reduces the number of parameters.

[0067] Preferably, the attention module includes a context feature extraction branch and a coordinate attention branch;

[0068] For the coordinate attention branch, the feature information is first obtained through adaptive global max pooling and average pooling. Then, the features of the two are concatenated and a 1×1 convolution is used to generate a feature map. The feature map is then split into features in two directions and the information is re-encoded using a 1×1 convolution. Finally, the attention map is generated using the Sigmoid function.

[0069] For the context feature extraction branch, dilated convolution is first used to extract context features, and then 1×1 convolution is used to aggregate relevant information of the channels to generate a feature map. Finally, the generated attention feature map is fused with the coordinate attention branch feature map by multiplying the two feature maps element by element.

[0070] The above scheme primarily utilizes two parallel branches for attention: a contextual feature extraction branch and a coordinate attention branch. For the coordinate attention branch, adaptive global max pooling and average pooling are first applied to obtain feature information Za and Zm. Then, these two features are concatenated, and a 1×1 convolution is used to generate a feature map Mc=F. 1×1 (concat(Za,Zm)), then split the feature map into features in two directions, re-encode the information using a 1×1 convolution, and finally use the Sigmoid function to generate an attention map: Ch=F 1×1 (fh), Cw=F 1×1 (fw). Where Ch∈R C×H×1 ,Cw∈R C×1×W .

[0071] For the context feature extraction branch, dilated convolution is first used to extract context features. Dilated convolution does not increase the number of network parameters, but it can increase the receptive field. Then, 1×1 convolution is used to aggregate the relevant information of each channel to generate a feature map Mg=F. 1×1 (F) dilated (x)), Mg ∈R 1×H×W Finally, the generated attention feature map Mg is fused with the coordinate attention branch feature map. Specifically, the two feature maps are multiplied element-wise: Ao = Ch * Cw * Mg, Ao ∈ R. C×H×W .

[0072] The improved attention mechanism of this invention can not only capture channel correlation and the location information of encoded features, but also use dilated convolution to extract contextual features, which helps to improve the overall performance of the model.

[0073] Preferably, in the downsampling module, a 3×3 depthwise convolution is used instead of a pooling layer, and then the convolution outputs a feature map.

[0074] In the above scheme, the model proposed in this invention uses a 3×3 depthwise convolution instead of a pooling layer, solving the problem of key information loss caused by using pooling layers with a small computational cost. The feature map output by this convolution is: D=F 3×3 (Ao), D∈R C×H / 2×W / 2 .

[0075] Example 3

[0076] The data and parameters of the loss function model are as follows:

[0077] The model of this invention was built using the PyTorch framework, and the effectiveness of the invention was verified using the Fer2013 dataset. This dataset contains a total of 35,887 facial expression images, of which 28,709 are in the training set and 3,589 are in the test set. It contains a total of 7 types of facial expressions: happy, angry, disgusted, fearful, sad, neutral, and surprised.

[0078] During the model training phase, the network's hyperparameters are adjusted by observing the loss on the validation set, and SGD with momentum is used as the optimizer with a momentum of 0.9 and a weight decay coefficient of 5e-4.

[0079] During the training phase, the Dropout function is used to prevent overfitting of the network and improve the generalization performance of the model.

[0080] Supervised training is performed using a loss function, and the trained model is then saved and validated on a test set.

[0081] The formula for calculating the loss function is:

[0082]

[0083] Where e represents the number of expression categories, N is the total number of samples, W is the weight parameter, y is the label, and x is the input data.

[0084] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A lightweight facial expression recognition method based on an attention mechanism, characterized in that, Includes the following steps: S1: Build a convolutional model, crop the images in the dataset, preprocess the images, and input the preprocessed images into the convolutional model; S2: The image undergoes lightweight Ghost convolution feature extraction, attention mechanism recalibration, and downsampling operations in the convolution model to obtain the final output feature map; In step S2, lightweight feature extraction includes the following steps: extracting features using 1×3 and 3×1 convolutions, and then using Ghost convolutions instead of pointwise convolutions to extract features; repeating the above operations, and then concatenating the feature map with the input image; then performing a shuffling operation on the channels to obtain the final output feature map; In step S2, the downsampling operation includes the following steps: using a 3×3 depthwise convolution with a stride of 2 to perform the downsampling operation, the size of the output feature map is halved; using a global average pooling layer and a 1×1 convolution to replace the last downsampling operation, resulting in an output feature map of 1024×1×1; The attention mechanism includes a context feature extraction branch and a coordinate attention branch; For the coordinate attention branch, the feature information is first obtained through adaptive global max pooling and average pooling. Then, the features of the two are concatenated and a 1×1 convolution is used to generate a feature map. The feature map is then split into features in two directions and the information is re-encoded using a 1×1 convolution. Finally, the attention map is generated using the Sigmoid function. For the context feature extraction branch, dilated convolution is first used to extract context features, and then 1×1 convolution is used to aggregate relevant information of the channels to generate a feature map. Finally, the generated attention feature map is fused with the coordinate attention branch feature map by multiplying the two feature maps element by element. S3: Perform facial expression classification on the vectors in the feature map to obtain the recognition result; S4: Establish a loss function model, use the recognition results to train the model parameters and test them to complete facial expression recognition.

2. The lightweight facial expression recognition method based on an attention mechanism according to claim 1, characterized in that, In step S1, the input image is randomly cropped to 48×48, and then rotated and flipped horizontally at a small angle.

3. The lightweight facial expression recognition method based on an attention mechanism according to claim 2, characterized in that, In step S3, the features obtained from the global average pooling layer are fed into the fully connected layer. Specifically, 1024 vectors are mapped into 7 vectors and fed into the classifier for expression classification to obtain the final recognition result.

4. A lightweight facial expression recognition method based on an attention mechanism according to claim 2, characterized in that, In step S4, the loss function model is built using the PyTorch framework, validated using the Fer2013 dataset, and supervised using the loss function, specifically as follows: Where e represents the number of expression categories, and N is the total number of samples. W These are weight parameters. y For tags, x For input data.

5. A lightweight facial expression recognition system based on an attention mechanism, employing the lightweight facial expression recognition method based on an attention mechanism as described in claim 1, characterized in that, The convolutional model in step S2 of the method includes a feature extraction module, an attention module, and a downsampling module; the output of the feature extraction module is electrically connected to the input of the attention module; the output of the attention module is electrically connected to the input of the downsampling module. The attention module includes a context feature extraction branch and a coordinate attention branch; For the coordinate attention branch, the feature information is first obtained through adaptive global max pooling and average pooling. Then, the features of the two are concatenated and a 1×1 convolution is used to generate a feature map. The feature map is then split into features in two directions and the information is re-encoded using a 1×1 convolution. Finally, the attention map is generated using the Sigmoid function. For the context feature extraction branch, dilated convolution is first used to extract context features, and then 1×1 convolution is used to aggregate relevant information of the channels to generate a feature map. Finally, the generated attention feature map is fused with the coordinate attention branch feature map by multiplying the two feature maps element by element.

6. A lightweight facial expression recognition system based on an attention mechanism according to claim 5, characterized in that, In the feature extraction module, the 3×3 ordinary convolution is first decomposed and replaced by two decomposed convolutions, 1×3 and 3×1, concatenated. The 1×1 pointwise convolution after the depthwise convolution is also replaced. The Ghost convolution is used to significantly reduce the parameters. Finally, the two are concatenated.

7. A lightweight facial expression recognition system based on an attention mechanism according to claim 5, characterized in that, In the downsampling module, a 3×3 depthwise convolution is used instead of a pooling layer, and then the convolution outputs a feature map.

Images