Lightweight facial expression recognition method and system based on global-local cross attention

Abstract

The present invention discloses a lightweight facial expression recognition method and system based on global-local cross-attention, which relates to artificial intelligence and proposes this solution to the technical problems in the prior art. When processing facial expression recognition, the following steps are performed: S1. Image preprocessing; S2. Feature pre-extraction; S3. Pruning processing based on cross-attention. The lightweight facial expression recognition method based on global-local cross-attention described in the present invention utilizes the system to perform facial expression recognition. Its advantages are: by combining Landmark detection with the ViT model, the performance of facial expression recognition is significantly enhanced through the cross-attention mechanism of global and local characteristics. It can focus on the information of local key areas while capturing global facial features, thereby improving the accuracy and robustness of recognition. By using the local characteristics of the landmark to guide the model to prune tokens, it can effectively discard unimportant or interfering areas, enhance the performance of FER, and reduce computational overhead.

Description

Technical Field

[0001] The present invention relates to artificial intelligence, and in particular to a lightweight facial expression recognition method and system based on global-local cross attention. Background Art

[0002] Facial expressions are one of the most powerful, natural, and universal signals for expressing human emotional state and intention. Understanding human emotional state is a fundamental prerequisite for many computer vision tasks. Due to the importance of automatic facial expression analysis in social robotics, healthcare, driver fatigue monitoring, and many other human-computer interaction systems, facial expression recognition (FER) has been extensively studied. Traditional FER methods primarily rely on handcrafted feature extraction (e.g., local binary patterns (LBP), histogram of oriented gradients (HOG), scale-invariant feature transform (SIFT), and non-negative matrix factorization (NMF)). However, these methods lack generalization in the presence of strong environmental interference.

[0003] In recent years, the performance of facial expression recognition has been significantly improved with the development of deep learning and visual transformer (ViT) technology. ViT can capture global features in facial expressions by dividing images into a series of tokens and processing these tokens. However, due to the lack of the inherent inductive bias of convolutional neural networks, they are relatively weak in capturing local details and processing local features. In addition, the ViT model has a high computational overhead when processing high-resolution images, especially in environments with limited computing resources. This poses a challenge to the application of real-time expression recognition.

[0004] Landmark detection, as a key feature modeling method in facial expression recognition, is widely used in the field of expression recognition. Landmark detection locates key points on the face (including eyes, nose, mouth, etc.) to enhance the model's attention to local areas, thereby improving sensitivity and robustness to changes in facial details. This technology has been widely studied and has made significant progress in enhancing the performance of facial expression recognition models. Thanks to the excellent local feature capture capability of Landmark detection, the application of Landmark detection in outdoor environments can more accurately extract key features of facial areas and significantly improve the accuracy and stability of expression recognition. Especially in tasks that combine global features and local detail information, Landmark detection plays a vital role in improving the robustness of the model and handling complex expression changes.

[0005] Defects and shortcomings of existing technology:

[0006] 1. The network architecture based on ViT, due to the lack of the inherent inductive bias of convolutional neural networks, is relatively weak in capturing local details and processing local features.

[0007] 2. The computational complexity of ViT is related to the square of the number of tokens. The computational overhead is high when processing high-resolution images, especially in an environment with limited computing resources. This also poses a challenge to the application of real-time expression recognition. Summary of the Invention

[0008] The purpose of the present invention is to provide a lightweight facial expression recognition method and system based on global-local cross attention to solve the problems existing in the above-mentioned prior art.

[0009] The lightweight facial expression recognition system based on global-local cross attention described in the present invention performs the following steps when processing facial expression recognition:

[0010] S1. Image preprocessing;

[0011] S2. Feature pre-extraction: The first three stages of IR-50 pre-trained on Ms-Celeb-1M are used as feature pre-extractors to obtain global features. The pre-trained MobileFaceNet is used as a landmark detector to obtain local features from the original image, and the weights are frozen during the training phase. Define the input image X∈R H×W×C , where H and W are the height and width of the image, respectively, and C is the number of channels; the image feature X is obtained in the feature pre-extractor using IR-50 pre-trained on Ms-Celeb-1M. img ∈R N×D ; Obtain landmark feature X through the trained face landmark detector MobileFaceNet lmk ∈R N×D , where N is the number of tokens and D is the feature dimension; during training, IR-50 is fine-tuned while keeping the weights of MobileFaceNet frozen to ensure correct landmark output;

[0012] S3. Pruning based on cross attention: For the global branch, the pruning module projects the image features into two feature vectors Q′ img ∈R N×D ,K′ img ∈R N×D For the local branch, the landmark feature is projected into 3 feature vectors Q lmk ∈R N ×D ,K lmk ∈R N×D , K lmk ∈R N×D :

[0013] Q′ img =Ximg W Q1 ,K′ img =X img W K1

[0014] Q lmk =X img W Q2 , K lmk =X img W K2 , V lmk =X lmk W V2

[0015] Q lmk ∈R N×D Taking the global average

[0016]

[0017] Avgpooling(·) represents global averaging along the spatial dimension;

[0018] Exchange the Q vectors of the two branches, fuse local and global features across branches, and for the global branch, use For the characteristics of strong perception of local important areas, learn the importance of local areas:

[0019]

[0020] where Attn∈R 1×N It is the local perception attention, which is used as the importance measure of each token.

[0021] The lightweight facial expression recognition method based on global-local cross attention described in the present invention utilizes the system to perform facial expression recognition.

[0022] The advantages of the lightweight facial expression recognition method and system based on global-local cross attention described in the present invention are:

[0023] (1) By combining landmark detection with the ViT model, the performance of facial expression recognition (FER) is significantly enhanced through a cross-attention mechanism of global and local features. It can capture global facial features while focusing on the information of local key areas, thereby improving the accuracy and robustness of recognition.

[0024] (2) By utilizing the local characteristics of landmarks to guide the model to prune tokens, it can effectively discard unimportant or interfering areas, enhance the performance of FER, and reduce computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the network architecture of the lightweight facial expression recognition system described in the present invention.

[0026] Figure 2 Schematic diagram of the network architecture of the pruning module in the present invention. DETAILED DESCRIPTION

[0027] like Figures 1 to 2 As shown, the lightweight facial expression recognition system based on global-local cross attention described in the present invention performs the following steps when processing facial expression recognition:

[0028] S1. Image Preprocessing: During data preprocessing, face detection and keypoint alignment are performed on the facial expression images in the dataset to ensure that the model focuses on features in the facial region. To enhance the diversity and robustness of the training data, a series of data augmentation strategies are employed during training, including random horizontal flipping, random cropping, and random erasing, thereby improving the model's generalization ability in practical applications.

[0029] S2. Feature pre-extraction: The first three stages of IR-50 pre-trained on Ms-Celeb-1M are used as feature pre-extractors to obtain global features, and the pre-trained MobileFaceNet is used as a landmark detector to obtain local features from the original image, and the weights are frozen during the training phase. Define the input image X∈R H×W×C , where H and W are the height and width of the image respectively, and C is the number of channels. In the feature pre-extractor, the image feature X is obtained by using IR-50 pre-trained on Ms-Celeb-1M. img ∈R N×D Obtain the landmark feature X through the trained face landmark detector MobileFaceNet lmk ∈R N×D , where N is the number of tokens and D is the feature dimension. During training, IR-50 is fine-tuned while keeping the weights of MobileFaceNet frozen to ensure correct landmark output.

[0030] At each stage of the global branch, global features are extracted from the original Transformer encoder and the input features are projected through three linear layers:

[0031] Q=X img W Q , K=X img W K , V=X img W V ;

[0032] Where W Q , W K , W V ∈R D×D .

[0033] The multi-head attention can then be calculated using the following formula:

[0034]

[0035] MHSA(X)=[head1;...;head h ]W O ;

[0036] in is the normalized scaling factor, head i is the self-attention output of each head, W O ∈R D×D is the output weight matrix, SA(·) is the calculation structure of self-attention, and MHSA(·) is the calculation structure of multi-head self-attention.

[0037] The final output is obtained through the multi-layer perceptron MLP and layer normalization LayerNorm, and the probability distribution is calculated through Cls Token:

[0038] Z′=LayerNorm(X+MHSA(X));

[0039] Z=MLP(Z′)=GELU(Z′W1)W2;

[0040]

[0041] Where W head represents the weight of the classification head, ClsToken is the output of the last layer Transformer of the global branch. It is the Cls Token output by the last layer Transformer of the local branch, and the output is the final classification probability.

[0042] S3. Pruning based on cross attention: For the global branch, the pruning module projects the image features into two feature vectors Q′ img ∈R N×D ,K′ img ∈R N×D For the local branch, the landmark feature is projected into 3 feature vectors Q lmk ∈R N ×D ,K lmk ∈R N×D , K lmk ∈R N×D:

[0043] Q′ img =X img W Q1 ,K′ img =X img W K1

[0044] Q lmk =X img W Q2 ,K lmk =X img W K2 ,V lmk =X lmk W V2

[0045] Q lmk ∈R N×D Taking the global average

[0046]

[0047] Avgpooling(·) represents global averaging along the spatial dimension.

[0048] Exchange the Q vectors of the two branches, fuse local and global features across branches, and for the global branch, use For the characteristics of strong perception of local important areas, learn the importance of local areas:

[0049]

[0050] where Attn∈R 1×N It is the local perception attention, which is used as the importance measure of each token.

[0051] For local branches, landmarks primarily focus on features in local areas such as the eyes, nose, and mouth, while ignoring the global nature of facial expressions. When significant posture changes, lighting, or occlusion occur, landmark detection may be inaccurate, so it is necessary to introduce global features into the local branches to compensate for this information. Furthermore, integrating global features can guide the local branches to incorporate long-range dependencies, guiding the model to focus on the correlations between local regions. The calculation process is as follows:

[0052]

[0053] Z lmk =MLP(LayerNorm(Z lmk +MHSA(Q′ img , K lmk , V lmk)));

[0054] where Z lmk is the output of the cross-fusion Transformer.

[0055] In order to reduce the number of tokens and thus reduce the computational overhead of the ViT architecture, we use Attn as the pruning metric, take the topk indices as the retained token positions, and discard the remaining tokens:

[0056] idx = argsort(Attn)[0:k];

[0057] X′ img =Selcet(idx,Z′ img )=[t img,idx1 ,t img,idx2 ,...,t img,idxk ,];

[0058] X′ lmk =Selcet(idx,Z′ lmk )=[t lmk,idx1 ,t lmk,idx2 ,...,t lmk,idxk ,];

[0059] Where Selcet(·) is a selection function that selects the token with the specified index idx along the spatial dimension, t ·,i ∈R 1×D A single token at index position i of X′.

[0060] The lightweight facial expression recognition method based on global-local cross attention described in the present invention utilizes the system to perform facial expression recognition.

[0061] Those skilled in the art can make various other corresponding changes and deformations based on the technical solutions and concepts described above, and all of these changes and deformations should fall within the scope of protection of the claims of the present invention.

Claims

1. A lightweight facial expression recognition system based on global-local cross attention, characterized by: When processing facial expression recognition, the following steps are performed: S1. Image preprocessing; S2. Feature pre-extraction: The first three stages of IR-50 pre-trained on Ms-Celeb-1M are used as feature pre-extractors to obtain global features. The pre-trained MobileFaceNet is used as a landmark detector to obtain local features from the original image, and the weights are frozen during the training phase. Define the input image X∈R H×W×C , where H and W are the height and width of the image, respectively, and C is the number of channels; the image feature X is obtained in the feature pre-extractor using IR-50 pre-trained on Ms-Celeb-1M. img ∈R N×D ; Obtain landmark feature X through the trained face landmark detector MobileFaceNet lmk ∈R N×D , where N is the number of tokens and D is the feature dimension; During training, IR-50 is fine-tuned while keeping the weights of MobileFaceNet frozen to ensure correct landmark output; S3. Pruning based on cross attention: For the global branch, the pruning module projects the image features into two feature vectors Q' img ∈R N×D ,K′ img ∈R N×D For the local branch, the landmark feature is projected into 3 feature vectors Q lmk ∈R N×D , K lmk ∈R N×D , V lmk ∈R N×D : Q′ img =X img W Q1 ,K′ img =X img W K1 Q lmk =X lmk W Q2 ,K lmk =X lmk W K2 ,V lmk =X lmk W V2 ; Q lmk ∈R N×D Taking the global average Avgpooling(·) represents global averaging along the spatial dimension; Exchange the Q vectors of the two branches, fuse local and global features across branches, and for the global branch, use For the characteristics of strong perception of local important areas, learn the importance of local areas: where Attn∈R 1×N is the local perception attention, which is the importance measure of each Token. is the normalized scaling factor; The step S1 is specifically as follows: In the data preprocessing stage, face detection and key point alignment are performed on the facial expression images of the dataset, and data enhancement strategies are used for processing; In step S2, at each stage of the global branch, global features are extracted from the original Transformer encoder and the input features are projected through three linear layers: Q=X img W Q ,K=X img W K ,V=X img W V ; Where W Q ,W K ,W V ∈R D×D ; The multi-head attention can then be calculated using the following formula: MHSA(X)[head1;...;head h ]W O ; in is the normalized scaling factor, head i is the self-attention output of each head, and are the Q, K and V projection matrices of the i-th head, W O ∈R D×D is the output weight matrix, SA(·) is the calculation structure of self-attention, and MHSA(·) is the calculation structure of multi-head self-attention; The final output is obtained through the multi-layer perceptron MLP and layer normalization LayerNorm, and the probability distribution is calculated through Cls Token: Z′=LayerNorm(X+MHSA(X)); Z=MLP(Z′)=GELU(Z′W1)W2; Where W head Represents the weight of the classification head. The MLP consists of two fully connected layers with weights W1 and W2 respectively. Cls Token is the output of the last layer Transformer of the global branch. Cls Token is the output of the last layer Transformer of the local branch, and the output is the final classification probability; In step S3, global features are introduced into local branches to perform information compensation: Z lmk =MLP(LayerNorm(Z lmk +MHSA(Q′ img ,K lmk ,V lmk ))); where Z lmk is the output of the cross-fusion Transformer; By leveraging the local characteristics of landmarks to guide the model to prune tokens, unimportant or interfering areas are discarded, FER performance is enhanced, and computational overhead is reduced. In order to reduce the number of tokens and thus the computational overhead of the ViT architecture, Attn is used as the pruning metric, the topk indices are taken as the retained token positions, and the remaining tokens are discarded: idx = argsort(Attn)[0:k]; X′ img =Selcet(idx,Z′ img )=[t img,idx1 ,t img,idx2 ,…,t img,idxk ,]; X′ lmk =Selcet(idx,Z′ lmk )=[t lmk,idx1 ,t lmk,idx2 ,...,t lmk,idxk ,]; Where Selcet(·) is a selection function that selects the token with the specified index idx along the spatial dimension, t img，i ∈R 1×D , t lmk，i ∈R 1×D X′ img and X′ lmk A single token at index position i.

2. A lightweight facial expression recognition method based on global-local cross attention, characterized by: The system as claimed in claim 1 is used to perform facial expression recognition.

Images