A student expression recognition method based on Gabor convolution and Transformer
By combining Gabor convolution and Transformer to design student expression recognition models, the accuracy of expression recognition in complex environments is solved, and efficient expression recognition effect is achieved.
Patent Information
- Application Number
- CN202311214324.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-20
- Publication Date
- 2025-08-19
- Estimated Expiration
- 2043-09-20
AI Technical Summary
Existing expression recognition methods are difficult to accurately identify student expressions in complex environments, especially the traditional method has limited recognition performance, while deep learning methods ignore the relationship between local features and global features.
The student expression recognition model based on Gabor convolution and Transformer is adopted, and local features are extracted through the GC-Block module, and global dependencies are learned in combination with the Transformer module to build GVT-Block for deeper feature extraction and classification.
In complex environments, high expression recognition accuracy is achieved, which improves the generalization ability and robustness of the model and significantly improves the accuracy of students' expression recognition.
Smart Images
Figure CN117275065B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision, and in particular relates to a design of an expression recognition model method, specifically to a student expression recognition model based on Gabor convolution and Transformer. Background Art
[0002] With the rapid development of computer vision technology and hardware, researchers have begun studying facial expression recognition (FER). This technology analyzes the information contained in facial expressions to better understand human behavior and provide targeted responses. Numerous methods have been proposed to address this problem. Generally speaking, current mainstream expression recognition methods can be roughly divided into two categories: traditional methods and deep learning-based methods.
[0003] Traditional FER tasks rely on local feature extraction based on geometry and appearance. Facial feature extraction algorithms extract local, global, or mixed features as input to a classifier to help it recognize facial expressions. Geometry-based feature extraction methods primarily focus on intrinsic facial features, such as the eyes, eyebrows, forehead, nose, and mouth. Predefined geometric landmarks are used to describe the shape and position of facial components, which are then extracted to form feature vectors representing facial geometry. Because facial expressions affect the relative shape and position of various facial features, facial expressions can be recognized by measuring the displacement of important facial components.
[0004] While traditional methods can effectively extract static facial features, their performance is very limited in real-time recognition environments. In recent years, the development of deep learning algorithms has significantly improved the accuracy and robustness of facial expression recognition. Compared to traditional methods, deep learning algorithms can automatically learn more abstract and high-level feature representations, better capturing the nuances of facial expressions. Furthermore, deep learning algorithms are capable of processing large amounts of data, making them more suitable for real-world facial expression recognition challenges.
[0005] Currently, many deep learning-based expression recognition methods have been proposed and widely used. Some researchers have proposed expression recognition methods based on convolutional neural networks (CNNs). These methods use operations such as convolution to extract local features from images, thereby enabling expression recognition. Mainstream deep learning methods focus on directly extracting deep features using CNNs, but they often overlook the relationship between local and global features.
[0006] This paper applies deep learning technology to facial expression recognition and designs an expression recognition model based on Gabor convolution and Transformer. The facial image is captured by a camera and the expression recognition model is used to identify facial expressions. Summary of the Invention
[0007] This paper studies the design and implementation of a student learning status feedback system in an online education environment. To address the problem of inability to accurately identify student expressions in complex environmental changes, the present invention provides a student expression recognition model based on Gabor convolution and Transformer. This model combines the ideas of Gabor convolutional networks and Transformer, and can still ensure a high accuracy rate of expression recognition in complex environments.
[0008] The present invention is achieved through the following technical solutions:
[0009] A student expression recognition model based on Gabor convolution and Transformer
[0010] Step 1: The GC-Block module processes the input image
[0011] The GC-Block module consists of two 5x5 Gabor convolutional layers. Each Gabor convolutional layer is followed by a batch normalization layer to prevent overfitting, improve training speed, and improve generalization. A GELU activation function is added after the batch normalization layer to perform a nonlinear transformation on the normalized feature map. This combination of convolutional layers, normalization, and activation functions extracts local features from the input data and maps them into higher-dimensional, more abstract feature representations.
[0012] The main purpose of the GC-Block module is to generate enhanced feature maps through Gabor convolution, which can better capture the texture information and edge features of facial expression images, and extract feature maps with sufficient prior knowledge as input for subsequent backbones. When the first Gabor convolution is used to extract features, setting the expand parameter to true expands the input tensor into a larger tensor for convolution, ensuring that the Gabor filter can convolve the entire input. The Nscale parameter specifies the number of Gabor filter directions at each scale. In Gabor filters, different scales and directions result in different filter responses, so using multiple filters at each scale can better capture image features in different directions. The convolution process is shown in Equation 1.
[0013]
[0014] Among them, F is the input feature map, C iis the i-th GoF, and the specific calculation process is shown in Formula 2.
[0015]
[0016] Where (n) represents F and C i,u The nth channel of , k represents different directions, and F i,k It's F i The k-th direction response, Represents the convolution process. The directional information in the feature is significantly enhanced through convolution.
[0017] A batch normalization layer is added after the Gabor convolution layer to prevent overfitting, improve training speed, and enhance generalization. The batch normalization layer also adds the GELU activation function to perform nonlinear transformations on the normalized feature maps. The calculation process is shown in Equations 3 and 4.
[0018]
[0019] f=GeLu(f′) (Formula 4)
[0020] Among them, BN is the normalization layer, and GeLu() is the activation function GeLu.
[0021] GeLu is an efficient, smooth activation function with good approximation capabilities. GeLU's negative non-zero gradient can retain more information, making it more suitable for the dot product calculation of the input vector in the multi-head self-attention mechanism in the Transformer, and eliminating the problem of neuron death. The smoothness of the feature gives it more nonlinear features and stronger representation capabilities, making it better suited to large-scale processing tasks in the Transformer, as these tasks require processing large amounts of nonlinear and high-dimensional image data. The continuity of the function also allows it to calculate derivatives in analytical form without the need for numerical approximation methods, thereby calculating gradients faster and accelerating model training.
[0022] The calculation process of the GeLu activation function can be described as: for each input x, it is multiplied by a binomial distribution φ(x), that is, GeLu(x) = x·φ(x). Because φ(x) cannot be calculated directly, the specific formula for the GeLu activation function approximation for the standard normal distribution is shown in Formula 5.
[0023]
[0024] From the formula, we can see that the GeLu activation function is a combination of the hyperbolic tangent function tanh and an approximate value.
[0025] Step 2 uses the feature extraction block GVT-Block, which includes the Transformer module and the GC-Basic-Block module, and introduces residual connections to perform finer-grained feature extraction and dimension upgrading on the feature map through the backbone.
[0026] The backbone uses 3*3 Gabor convolution to perform fine-grained feature extraction and dimension upgrading on the feature map. The GVT-Block module consists of the Transformer module and the GC-Basic-Block module.
[0027] The purpose of the Transformer module is to learn local and global information using input tensors with fewer parameters. It is desirable to model long-range non-local dependencies while having an effective H*W receptive field. A more widely used method for modeling long-range dependencies is dilated convolution. However, this method requires careful selection of the dilation rate; otherwise, weights are applied to zero-filled areas instead of valid spatial regions. Another promising solution is the self-attention mechanism. Compared with dilated convolution, it can learn longer-range feature dependencies in image sequences and can process longer image sequence data. Parallel computing also improves computational efficiency. Among the self-attention mechanisms, the visual Transformer with multi-head self-attention has been shown to be very effective in visual recognition tasks.
[0028] Before entering the Transformer module, the input feature map is first dimensionally increased. A 3x3 Gabor convolution is used to further extract local features of the image and map these features to a higher dimension, enabling the network to better learn the data's characteristics. Following this dimension increase, max pooling is performed to reduce the size of the feature map. Max pooling reduces the spatial dimensions and number of feature maps by selecting the maximum value, thereby reducing the model's computational complexity and parameter count. It also enhances significant features in the input feature map, improving the model's performance. The specific process is shown in Equation 6.
[0029] X = maxpool(BN(GC conv 3*3 (f))) (Formula 6)
[0030] Among them, maxpool is the maximum pooling operation. In order to enable the Transformer module to learn a global representation with spatial inductive bias, the input feature map is first unfolded. Unfolding is to change the size of the input feature map from [B, H, W, d] to [B, P, N, d], that is, to divide the feature map into small blocks (patches), and expand each slice into a column vector and then combine them for input into the Transformer. If B = C = 1, and the width and height of the patch W = H = 3, then you can imagine splitting the image into 9 columns, and the pixels in each column are extracted from the same position in each patch, so there are num_patches pixels in each column. We send each column into the Transformer, and use the self-attention mechanism to calculate the attention of each pixel between columns, so that the receptive field can be expanded to H*W. The specific operation of unfolding is shown in Formula 7:
[0031] X′=unfold(X)=R H*W*d →R N*P*d (Formula 7)
[0032] Among them, p = H*W, N = HW / p is the number of patches, h≤N and w≤N are the height and width of the patch respectively. For each P∈{1,…,p}, Transformer is applied to process the relationship between patches to obtain X G ∈R P×N×d The formula is shown in Formula 8:
[0033] X G (p) = Transformer(X U (p)),1≤p≤P (Formula 8)
[0034] Then, a fold operation is performed to convert [B, P, N, d] back to [B, H, W, d]. This process is used to obtain the attention-weighted representation between image vectors and obtain the global dependency relationship of the feature map. The expression is shown in Formula 9:
[0035] X = fold(X′) = R N*P*d →R H*W*d (Formula 9)
[0036] After obtaining the feature map that extracts the image's attention-weighted representation, it is input into the GC-Basic-Block module. The GC-Basic-Block module consists of a 5*5 Gabor convolution and a 3*3 Gabor convolution.
[0037] These two convolutional layers can learn different levels of feature representation: the first convolutional layer extracts features, and the second integrates them, resulting in a richer representation of features. Furthermore, residual connections transfer input features directly to the output, preventing information loss. This optimizes the gradient flow within the network, reduces vanishing and exploding gradients, and enables deeper model training while improving the model's expressiveness.
[0038] Step 3: Backbone is repeated multiple times to build a deeper network
[0039] The dimension-raising module and the GC-Block module form the backbone and are repeated multiple times to build a deeper network.
[0040] Step 4: Connect global average pooling and fully connected layers to achieve image classification
[0041] Global average pooling and fully connected layers are used to implement image classification. Global average pooling can reduce the number of model parameters and prevent overfitting, while fully connected layers can better perform transfer learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 Diagram of the structure of the expression recognition model;
[0043] Figure 2 Schematic diagram of the GC-Block module structure;
[0044] Figure 3 Schematic diagram of the Transformer module structure;
[0045] Figure 4 GC-Basic-Block module structure diagram;
[0046] Figure 5 RAF-DB data distribution map;
[0047] Figure 6 Fer2013Plus training set data and distribution diagram;
[0048] Figure 7 Comparison of experimental parameters, computational complexity, model size, and accuracy of various models on the RAF-DB and Fer2013Plus datasets;
[0049] Figure 8 Model training accuracy chart under RAF-DB dataset;
[0050] Figure 9 Model training loss value graph under RAF-DB dataset;
[0051] Figure 10 Model training accuracy chart under Fer2013Plus dataset;
[0052] Figure 11 Model training loss value graph under Fer2013Plus dataset;
[0053] Figure 12 Confusion matrix diagram of each model in the RAF-DB dataset;
[0054] Figure 13 Confusion matrix diagram of each model in the Fer2013Plus dataset. DETAILED DESCRIPTION
[0055] Step 1: The GC-Block module processes the input image
[0056] The overall model structure of the student expression recognition model based on Gabor convolution and Transformer is as follows Figure 1 As shown in the figure. The GC-Block module consists of two 5*5 Gabor convolutional layers. The specific structure is as follows Figure 2 As shown in the figure, a batch normalization layer is added after each Gabor convolution layer to prevent overfitting, improve training speed and generalization ability. The GELU activation function is added after the batch normalization layer to perform nonlinear transformations on the normalized feature maps.
[0057] The primary purpose of the GC-Block module is to generate enhanced feature maps through Gabor convolution, which better captures texture information and edge features in facial expression images. This allows for the extraction of feature maps with sufficient prior knowledge as input for subsequent backbones. When the first Gabor convolution is used for feature extraction, setting the expand parameter to true expands the input tensor into a larger tensor for convolution, ensuring that the Gabor filter can convolve the entire input. The nscale parameter specifies the number of Gabor filter directions at each scale. The convolution process is shown in Equation 1.
[0058]
[0059] Among them, F is the input feature map, C i is the i-th GoF, and the specific calculation process is shown in Formula 2.
[0060]
[0061] Where (n) represents F and C i,u The nth channel of , k represents different directions, and F i,k It's F i The k-th direction response, Represents the convolution process. The directional information in the feature is significantly enhanced through convolution.
[0062] A batch normalization layer is added after the Gabor convolution layer to prevent overfitting, improve training speed, and enhance generalization. The batch normalization layer also adds the GELU activation function to perform nonlinear transformations on the normalized feature maps. The calculation process is shown in Equations 3 and 4.
[0063]
[0064] f=GeLu(f′) (Formula 4)
[0065] Among them, BN is the normalization layer, and GeLu() is the activation function GeLu.
[0066] GeLu is an efficient, smooth activation function with good approximation capabilities. Its negative non-zero gradient retains more information, making it more suitable for computing the dot product of input vectors in the Transformer's multi-head self-attention mechanism and eliminating the neuron death problem.
[0067] The calculation process of the GeLu activation function can be described as: for each input x, it is multiplied by a binomial distribution φ(x), that is, GeLu(x) = x·φ(x). Because φ(x) cannot be calculated directly, the specific formula for the GeLu activation function approximation for the standard normal distribution is shown in Formula 5.
[0068]
[0069] From the formula, we can see that the GeLu activation function is a combination of the hyperbolic tangent function tanh and an approximate value.
[0070] Step 2 uses the feature extraction block GVT-Block, which includes the Transformer module and the GC-Basic-Block module, and introduces residual connections to perform finer-grained feature extraction and dimension upgrading on the feature map through the backbone.
[0071] The backbone uses 3*3 Gabor convolution to perform fine-grained feature extraction and dimension upgrading on the feature map. The GVT-Block module consists of the Transformer module and the GC-Basic-Block module.
[0072] The purpose of the Transformer module is to use the input tensor with fewer parameters to learn local and global information. The Transformer module structure is as follows Figure 3 As shown in Figure 2. We want to model long-range non-local dependencies while having an effective H*W receptive field.
[0073] Before entering the Transformer module, the input feature map is first dimensionally increased. A 3x3 Gabor convolution is used to further extract local features of the image and map these features to a higher dimension, enabling the network to better learn the data's characteristics. Following this dimension increase, max pooling is performed to reduce the size of the feature map. Max pooling reduces the spatial dimensions and number of feature maps by selecting the maximum value, thereby reducing the model's computational complexity and parameter count. It also enhances significant features in the input feature map, improving the model's performance. The specific process is shown in Equation 6.
[0074] X = maxpool(BN(GC conv 3*3 (f))) (Formula 6)
[0075] Among them, maxpool is the maximum pooling operation. In order to enable the Transformer module to learn a global representation with spatial inductive bias, the input feature map is first unfolded, such as Figure 3 As shown. Unfolding changes the input feature map size from [B, H, W, d] to [B, P, N, d], that is, dividing the feature map into small blocks (patches), and expanding each slice into a column vector and then combining them for easy input into the Transformer. If B = C = 1, and the width and height of the patch W = H = 3, then you can imagine splitting the image into 9 columns, and the pixels in each column are extracted from the same position in each patch, so there are num_patches pixels in each column. We send each column into the Transformer, and use the self-attention mechanism to calculate the attention of each pixel between columns, so that the receptive field can be expanded to H*W. The specific operation of unfolding is shown in Formula 7:
[0076] X′=unfold(X)=R H*W*d →R N*P*d (Formula 7)
[0077] Among them, p = H*W, N = HW / p is the number of patches, h≤N and w≤N are the height and width of the patch respectively. For each P∈{1,…,p}, Transformer is applied to process the relationship between patches to obtain X G ∈R P×N×d The formula is shown in Formula 8:
[0078] X G (p) = Transformer(X U (p)),1≤p≤P (Formula 8)
[0079] Then, a fold operation is performed to convert [B, P, N, d] back to [B, H, W, d]. This process is used to obtain the attention-weighted representation between image vectors and obtain the global dependency relationship of the feature map. The expression is shown in Formula 9:
[0080] X = fold(X′) = R N*P*d →R H*W*d (Formula 9)
[0081] After obtaining the feature map that extracts the weighted representation of the image attention, the feature map is input into the GC-Basic-Block module. The GC-Basic-Block module consists of a 5*5 Gabor convolution and a 3*3 Gabor convolution. The GC-Basic-Block module structure is as follows Figure 4 shown.
[0082] Step 3: Backbone is repeated multiple times to build a deeper network
[0083] The dimension-raising module and the GC-Block module form the backbone and are repeated multiple times to build a deeper network.
[0084] Step 4: Connect global average pooling and fully connected layers to achieve image classification
[0085] Global average pooling and fully connected layers are used to implement image classification. Global average pooling can reduce the number of model parameters and prevent overfitting, while fully connected layers can better perform transfer learning.
[0086] Example 1:
[0087] In order to test the performance of the method of the present invention, we will use two relatively representative data sets, RAF-DB and Fer2013Plus, to verify the recognition effect of the present invention. The RAF-DB data distribution is as follows: Figure 5 As shown, the Fer2013Plus training set data and distribution are as follows Figure 6As shown. The machine used an RTX A5000 (24GB) 1170MHz GPU with CUDA 11.3. The deep learning framework used was PyTorch 1.10.0, and the programming language was Python 3.7. The experimental parameters were as follows: cross-entropy loss was used as the loss function, SGD was selected as the optimizer, momentum was 0.9, weight_decay was 1e-4, label_smooth was 0.01, and mixup_alpha was 1.0. The learning rate was initialized to 0.05 at the beginning of training, and a cosine annealing learning rate decay strategy was used. The number of training iterations for the Fer2013Plus dataset and the RAF-DB dataset was set to 500, and the number of training iterations for the RAF-DB dataset was set to 300. The batch size for each epoch was 64.
[0088] To better test the performance of our method, we conducted comparative experiments with GCN, ViT-SE, and Mobile-ViT-xxs on the RAF-DB and Fer2013Plus datasets. The main focus was on the number of parameters, computational complexity, model size, changes in validation set accuracy and loss under different datasets, and the ability to classify facial expressions. These are explained below:
[0089] Parameter quantity, computational complexity, and model size:
[0090] Depend on Figure 7It can be seen that the present invention is not only much smaller than the GCN and ViT-SE models in terms of parameter quantity, computational complexity, and model size, but also has higher accuracy than these two models on both datasets. Compared with the baseline model GCN, the present invention has an accuracy that is 1.96% higher on the RAF-DB dataset and 3.03% higher than the ViT-SE model. On the Fer2013 dataset, the accuracy of the present invention is 2.69% higher than the baseline model GCN and 2.26% higher than the ViT-SE model. Compared with Mobile-ViT-xxs, the present invention has little difference in parameter quantity, computational complexity, and model size, and the accuracy of the two models on the RAF-DB dataset is very similar. However, on the Fer2013Plus dataset, the accuracy of the present invention is 87.38%, which is 0.73% higher than the Mobile-ViT-xxs model. The reason is that the image resolution in the Fer2013Plus dataset is 48*48. Resizing the images to 128*128 and inputting them into Mobile-ViT-xxs results in a loss of image detail and quality, severely impacting feature extraction using traditional convolution kernels. However, the Gabor convolution of our proposed method prioritizes the representation of image texture and edge information, effectively learning key features even with partial image distortion. Comparative experimental results demonstrate that our model achieves significant results.
[0091] Changes in validation set accuracy and loss values
[0092] Figure 8 The figure shows the change in accuracy of each model after 300 rounds of training on the RAF-DB validation set. The horizontal axis is the model training round (Epoch) and the vertical axis is the recognition accuracy (Accuracy). Figure 9 Is the loss value change, the vertical axis is the loss value (Loss). Figure 8 It can be seen that the accuracy curve of the GCN model has the slowest convergence speed and the curve oscillation amplitude is large, the effect is not good enough and the model is not stable enough. Although the curve change of the ViT-SE model is very stable and the convergence speed is fast, there is a large gap in accuracy compared with the present invention. Although the Mobile-ViT-xxs model has a very good effect at the end of the training round, the curve change fluctuates greatly, the convergence speed is slow, the stability is poor and there is a problem of overfitting. The training curve of the present invention is smoother and more stable than other models, and the convergence speed is faster. It has basically converged in the training round of 300 Epochs, and the accuracy is more significant than other models.
[0093] exist Figure 9As can be seen in the figure, the loss curves of the Mobile-ViT-xxs and GCN models fluctuate greatly, indicating unstable and slow convergence. Compared with other models, the loss of the proposed method decreases faster and more steadily. By the end of the training round, the proposed method has also basically converged, with a smaller loss value.
[0094] Figure 10 It is the change in accuracy of each model on the Fer2013Plus validation set after 500 rounds of training, where the horizontal axis is the number of model training rounds and the vertical axis is the recognition accuracy. Figure 11 Is the loss value change, the vertical axis is the loss value. Figure 10 and Figure 11 It can be seen that although GCN and ViT-SE have a faster convergence speed, their accuracy is not high and the model performance is limited. The curves of Mobile-ViT-xxs and GCN models fluctuate greatly, the model training stability is not high and the effect is not ideal. Overall, the curve of the present invention changes smoothly, the convergence speed is faster and the accuracy is the highest, which shows that the model has a strong generalization ability, can be well extended to new data sets and adapt to different data distributions, while also avoiding the problem of overfitting, and has the best effect compared with other models. Compared with the RAF-DB data set, Fer2013Plus has many facial occlusions and angle-shifted expression images, so it can verify the robustness of a model in complex environments. The present invention has the highest accuracy compared with other models on this data set, which proves that the present invention is more robust. In terms of the change of Loss value, the decline rate of the present invention is faster and more stable, the convergence speed is faster, and the effect is significant compared with other models.
[0095] The ability to classify facial expressions Figure 12 As can be seen, GCN and ViT-SE have poor classification results on category 5, with neither exceeding 80%. GCN, ViT-SE, and Mobile-ViT-xxs all have poor classification results on category 1, with each failing to exceed 60%. The present invention performs best, achieving an accuracy of 71%, and also achieves a maximum accuracy of 97% on category 3. Although Mobile-ViT-xxs outperforms the present invention on categories 4 and 5, overall, the present invention achieves more balanced classification results across all categories and achieves higher classification accuracy.
[0096] The Fer2013Plus dataset contains many facial images with occlusion and angle offset. Although some images belong to different categories, the expression changes in the images are very similar, which makes it more likely for the model to make misjudgments and makes it easier to reflect the accuracy and robustness of the model. Figure 13It can be seen that GCN and ViT-SE have very poor classification effects on category 9, and are both incorrectly identified as category 1. Except for the present invention, the classification effects of other models on categories 6, 7, and 8 are also poor, with accuracy rates of less than 80%, and some images are incorrectly identified as category 1. In contrast, the present invention has better classification effects on all categories, with recognition accuracy rates exceeding 80%, and the recognition effects of each category are very balanced, proving that the present invention has better effects.
[0097] The present invention has a more significant effect than other models, thereby verifying the superior effect of the present invention.
Claims
1. A student expression recognition method based on Gabor convolution and Transformer, characterized in that: Here are the steps: 1) The GC-Block module processes the input image; 2) Using the feature extraction block GVT-Block, which includes the Transformer module and the GC-Basic-Block module, and introducing residual connections, it uses the backbone to perform finer-grained feature extraction and dimension upgrading on the feature map; The backbone uses 3*3 Gabor convolution to perform finer-grained feature extraction and dimension upgrading on the feature map. The GVT-Block module consists of the Transformer module and the GC-Basic-Block module. The Transformer module uses input tensors with fewer parameters to learn local and global information, build a model of long-range non-local dependencies, and have an effective H*W receptive field; Before entering the Transformer module, the input feature map is first upgraded in dimension. A 3*3 Gabor convolution is used to further extract local features of the image and map the features to a higher dimension. After the dimensionality upgrade operation, a maximum pooling operation is performed to reduce the size of the feature map. The maximum pooling operation reduces the spatial dimension and number of feature maps by selecting the maximum value. The specific process is shown in Formula 6: X = maxpool(BN(GC conv3*3 (f))) (Formula 6) Among them, maxpool is the maximum pooling operation. In order to enable the Transformer module to learn a global representation with spatially induced bias, the input feature map is first unfolded. Unfolding is to change the input feature map size from [B, H, W, d] to [B, P, N, d], split the feature map into small patches, and expand each slice into a column vector and then combine them for input into the Transformer. Each column is sent to the Transformer, and the self-attention mechanism is used to calculate the attention of each pixel between columns to expand the receptive field. The specific operation of unfolding is shown in Formula 7: X′=unfold(X)=R H*W*d →R N*P*d (Formula 7) Among them, p = H*W, N = HW / p is the number of patches, h≤N and w≤N are the height and width of the patch respectively. For each P∈{1,…,p}, Transformer is applied to process the relationship between patches to obtain X G ∈R P×N×d ; The formula is shown in Formula 8: X G (p) = Transformer(X U (p)),1≤p≤P (Formula 8) Then, a fold operation is performed to restore [B, P, N, d] to [B, h, W, d]. Through this process, the attention weighted representation between the image vectors is obtained, and the global dependency relationship of the feature map is obtained. The expression is shown in Formula 9: X = fold(X′) = R N*P*d →R H*W*d (Formula 9) After obtaining the feature map that extracts the weighted representation of the image’s attention, the feature map is input into the GC-Basic-Block module; the GC-Basic-Block module consists of a 5*5 Gabor convolution and a 3*3 Gabor convolution; 3) The backbone is repeated 4 times to build a deeper network; 4) Access global average pooling and fully connected layers to achieve image classification.
2. A student expression recognition method based on Gabor convolution and Transformer according to claim 1, characterized in that, In the above 1), the specific method is: The GC-Block module consists of two 5*5 Gabor convolutional layers. A batch normalization layer is added after each Gabor convolutional layer to prevent overfitting. A GELU activation function is added after the BN layer to perform nonlinear transformation on the normalized feature map. The GC-Block module generates enhanced feature maps through Gabor convolution, captures the texture information and edge features of facial expression images, and extracts feature maps with prior knowledge as input for subsequent backbones. When the first Gabor convolution is used to extract features, the expand parameter is set to true to expand the input tensor into a larger tensor for convolution, ensuring that the Gabor filter can convolve the entire input. The Nscale parameter specifies the number of directions of the Gabor filter at each scale. The convolution process is shown in Formula 1: Among them, F is the input feature map, C i is the i-th GoF, and the specific calculation process is shown in Formula 2: Where (n) represents F and C i,u The nth channel of , k represents different directions, and F i,k It's F i The k-th direction response, Represents the convolution process, which significantly enhances the directional information in the feature through convolution; A BN layer is added after the Gabor convolution layer for normalization. The BN layer adds a GELU activation function to perform nonlinear transformation on the normalized feature map. The calculation process is shown in Formula 3 and Formula 4. f=GeLu(f′) (Formula 4) Among them, BN is the normalization layer, GeLu() is the activation function GeLu; The calculation process of the GeLu activation function is described as follows: for each input x, it is multiplied by a binomial distribution φ(x), that is, GeLu(x) = x φ(x). Because φ(x) cannot be calculated directly, the specific formula for the approximate calculation of the GeLu activation function for the standard normal distribution is shown in Formula 5: From the formula, we can see that the GeLu activation function is a combination of the hyperbolic tangent function tanh and an approximate value.