Lightweight facial expression recognition method and system based on global and local cross attention

Abstract

The invention discloses a lightweight facial expression recognition method and system based on global and local cross attention, relates to artificial intelligence, and provides the scheme for solving the technical problems in the prior art. When facial expression recognition is processed, the following steps are carried out: S1, image preprocessing; s2, pre-extracting features; and S3, pruning processing based on cross attention. According to the lightweight facial expression recognition method based on global and local cross attention, facial expression recognition is carried out by using the system. The method has the advantages that the Landmark detection is combined with the ViT model, and the performance of facial expression recognition is remarkably enhanced through a cross attention mechanism of global and local characteristics. The method can focus on the information of the local key area while capturing the global facial features, thereby improving the recognition accuracy and robustness. According to the method, the token is pruned by utilizing the local characteristic guide model of the Landmark, so that unimportant or interfering areas can be effectively abandoned, the FER performance is enhanced, and the calculation overhead is reduced.

Description

Technical Field

[0001] The present invention relates to artificial intelligence, and in particular to a lightweight facial expression recognition method and system based on global-local cross attention. Background Art

[0002] Facial expression is one of the most powerful, natural and universal signals for humans to express their emotional state and intention. Understanding human emotional state is a basic prerequisite for many computer vision tasks. Due to the importance of automatic facial expression analysis in social robots, medical treatment, driver fatigue monitoring and many other human-computer interaction systems, people have conducted a lot of research on the facial expression recognition (FER) task. Traditional FER methods mainly rely on manual feature extraction (such as local binary pattern (LBP), histogram of oriented gradients (HOG), scale-invariant feature transform (SIFT) and non-negative matrix factorization (NMF)). However, these methods lack generalization when subjected to strong environmental interference.

[0003] In recent years, with the development of deep learning and visual transformer (ViT) technology, the performance of facial expression recognition has been significantly improved. ViT can capture the global features in expression images by dividing the image into a series of tokens and processing these tokens. However, due to the lack of the inherent inductive bias of convolutional neural networks, it is relatively weak in capturing local details and processing local features. In addition, the ViT model has a large computational overhead when processing high-resolution images, especially in an environment with limited computing resources, which also poses a challenge to the application of real-time expression recognition.

[0004] Landmark detection, as a key feature modeling method in facial expression recognition, is widely used in the field of expression recognition. Landmark detection locates key points of the face (including eyes, nose, mouth, etc.) to enhance the model's focus on local areas, thereby improving sensitivity and robustness to changes in facial details. This technology has been widely studied and has made significant progress in enhancing the performance of facial expression recognition models. Thanks to the excellent local feature capture capability of Landmark detection, the application of Landmark detection in outdoor environments can more accurately extract key features of facial areas and significantly improve the accuracy and stability of expression recognition. Especially in tasks that combine global features and local detail information, Landmark detection plays a vital role in improving the robustness of the model and handling complex expression changes.

[0005] Defects and shortcomings of the existing technology:

[0006] 1. The network architecture based on ViT is relatively weak in capturing local details and processing local features due to the lack of the inherent inductive bias of convolutional neural networks.

[0007] 2. The computational complexity of ViT is related to the square of the number of tokens. The computational overhead is large when processing high-resolution images, especially in an environment with limited computing resources, which also poses a challenge to the application of real-time expression recognition. Summary of the invention

[0008] The purpose of the present invention is to provide a lightweight facial expression recognition method and system based on global-local cross attention to solve the problems existing in the above-mentioned prior art.

[0009] The lightweight facial expression recognition system based on global-local cross attention described in the present invention performs the following steps when processing facial expression recognition:

[0010] S1. Image preprocessing;

[0011] S2. Feature pre-extraction: The first three stages of IR-50 pre-trained on Ms-Celeb-1M are used as feature pre-extractors to obtain global features. The pre-trained MobileFaceNet is used as a landmark detector to obtain local features from the original image, and the weights are frozen during the training phase. Define the input image X∈R H×W×C , where H and W are the height and width of the image, respectively, and C is the number of channels; the image feature X is obtained by using the IR-50 pre-trained on Ms-Celeb-1M in the feature pre-extractor img ∈R N×D ; Obtain landmark feature X through the trained face landmark detector MobileFaceNet lmk ∈R N×D , where N is the number of tokens and D is the feature dimension; during the training process, IR-50 is fine-tuned while keeping the weights of MobileFaceNet frozen to ensure the correct landmark output;

[0012] S3. Pruning based on cross attention: For the global branch, the pruning module projects the image features into two feature vectors Q′ img ∈R N×D ,K′ img ∈R N×D For local branches, the landmark features are projected into three feature vectors Q lmk ∈R N ×D ,K lmk ∈R N×D , K lmk ∈R N×D :

[0013] Q′ img =Ximg W Q1 ,K′ img =X img W K1

[0014] Q lmk =X img W Q2 , K lmk =X img W K2 , V lmk =X lmk W V2

[0015] Q lmk ∈R N×D Taking the global average

[0016]

[0017] Avgpooling(·) means global averaging along the spatial dimension;

[0018] Exchange the Q vectors of the two branches to fuse local and global features across branches. For the global branch, use For the characteristics of strong perception of local important areas, learn the importance of local areas:

[0019]

[0020] where Attn∈R 1×N It is local-aware attention, which is used as the importance measure of each Token.

[0021] The lightweight facial expression recognition method based on global-local cross attention described in the present invention utilizes the system to perform facial expression recognition.

[0022] The lightweight facial expression recognition method and system based on global-local cross attention described in the present invention have the following advantages:

[0023] (1) By combining landmark detection with the ViT model, the performance of facial expression recognition (FER) is significantly enhanced through the cross-attention mechanism of global and local features. It can capture global facial features while focusing on the information of local key areas, thereby improving the accuracy and robustness of recognition.

[0024] (2) By utilizing the local characteristics of landmarks to guide the model to prune tokens, it is possible to effectively discard unimportant or interfering areas, thereby enhancing the performance of FER and reducing computational overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the network architecture of the lightweight facial expression recognition system described in the present invention.

[0026] Figure 2 It is a schematic diagram of the network architecture of the pruning module described in the present invention. DETAILED DESCRIPTION

[0027] like Figure 1 to Figure 2 As shown, the lightweight facial expression recognition system based on global-local cross attention described in the present invention performs the following steps when processing facial expression recognition:

[0028] S1. Image preprocessing: In the data preprocessing stage, face detection and key point alignment are performed on the facial expression images of the dataset to ensure that the model can focus on the features of the face area. In order to enhance the diversity and robustness of the training data, a series of data enhancement strategies are used during training, including random horizontal flipping, random cropping, and random erasing, so as to improve the generalization ability of the model in practical applications.

[0029] S2. Feature pre-extraction: The first three stages of IR-50 pre-trained on Ms-Celeb-1M are used as feature pre-extractors to obtain global features, and the pre-trained MobileFaceNet is used as a landmark detector to obtain local features from the original image, and the weights are frozen during the training phase. Define the input image X∈R H×W×C , where H and W are the height and width of the image respectively, and C is the number of channels. In the feature pre-extractor, the image feature X is obtained by using the IR-50 pre-trained on Ms-Celeb-1M. img ∈R N×D Obtain landmark feature X through the trained face landmark detector MobileFaceNet lmk ∈R N×D , where N is the number of tokens and D is the feature dimension. During the training process, IR-50 is fine-tuned while keeping the weights of MobileFaceNet frozen to ensure the correct landmark output.

[0030] At each stage of the global branch, global features are extracted by the original Transformer encoder and the input features are projected through three linear layers:

[0031] Q=X img W Q , K = X img W K , V = X img W V ;

[0032] Where W Q , W K , W V ∈R D×D .

[0033] Then the multi-head attention can be calculated by the following formula:

[0034]

[0035] MHSA(X)=[head 1 ;...;head h ]W O ;

[0036] in is the normalized scaling factor, head i is the self-attention output of each head, W O ∈R D×D is the output weight matrix, SA(·) is the calculation structure of self-attention, and MHSA(·) is the calculation structure of multi-head self-attention.

[0037] The final output is obtained through the multi-layer perceptron MLP and layer normalization LayerNorm, and the probability distribution is calculated through Cls Token:

[0038] Z′=LayerNorm(X+MHSA(X));

[0039] Z = MLP(Z′) = GELU(Z′W 1 )W 2 ;

[0040]

[0041] Where W head represents the weight of the classification head, ClsToken is the output of the last layer Transformer of the global branch. It is the Cls Token output by the last layer Transformer of the local branch, and the output is the final classification probability.

[0042] S3. Pruning based on cross attention: For the global branch, the pruning module projects the image features into two feature vectors Q′ img ∈R N×D ,K′ img ∈R N×D For local branches, the landmark features are projected into three feature vectors Q lmk ∈R N ×D ,K lmk ∈RN×D , K lmk ∈R N×D :

[0043] Q′ img =X img W Q1 ,K′ img =X img W K1

[0044] Q lmk =X img W Q2 ,K lmk =X img W K2 ,V lmk =X lmk W V2

[0045] Q lmk ∈R N×D Taking the global average

[0046]

[0047] Avgpooling(·) represents global averaging along the spatial dimension.

[0048] Exchange the Q vectors of the two branches to fuse local and global features across branches. For the global branch, use For the characteristics of strong perception of local important areas, learn the importance of local areas:

[0049]

[0050] where Attn∈R 1×N It is local-aware attention, which is used as the importance measure of each Token.

[0051] For local branches, Landmark mainly focuses on the features of local areas such as eyes, nose, and mouth, while ignoring the global nature of facial expressions. When there are large posture changes, lighting, or occlusion, Landmark detection may be inaccurate, so it is necessary to introduce global features for local branches to compensate for information. At the same time, the fusion of global features can guide local branches to introduce long-range dependency characteristics and guide the model to focus on the correlation between local areas. The calculation process is as follows:

[0052]

[0053] Z lmk =MLP(LayerNorm(Z lmk +MHSA(Q′img , K lmk , V lmk )));

[0054] Where Z lmk is the output of the cross-fusion Transformer.

[0055] In order to reduce the number of tokens and thus reduce the computational overhead of the ViT architecture, Attn is used as the pruning metric, the topk indices are taken as the reserved token positions, and the remaining tokens are discarded:

[0056] idx = argsort(Attn)[0:k];

[0057] X′ img =Selcet(idx,Z′ img )=[t img,idx1 ,t img,idx2 ,...,t img,idxk ,];

[0058] X′ lmk =Selcet(idx,Z′ lmk )=[t lmk,idx1 ,t lmk,idx2 ,...,t lmk,idxk ,];

[0059] Where Selcet(·) is a selection function that selects the token with the index specified by idx along the spatial dimension, t ·,i ∈R 1×D A single Token at index position i of X′.

[0060] The lightweight facial expression recognition method based on global-local cross attention described in the present invention utilizes the system to perform facial expression recognition.

[0061] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A lightweight facial expression recognition system based on global-local cross attention, characterized by: When processing facial expression recognition, the following steps are performed: S1. Image preprocessing; S2. Feature pre-extraction: The first three stages of IR-50 pre-trained on Ms-Celeb-1M are used as feature pre-extractors to obtain global features. The pre-trained MobileFaceNet is used as a landmark detector to obtain local features from the original image, and the weights are frozen during the training phase. Define the input image X∈R H×W×C Where H and W are the height and width of the image respectively, and C is the number of channels; In the feature pre-extractor, the image feature X is obtained using IR-50 pre-trained on Ms-Celeb-1M. img ∈R N×D ; Obtain landmark feature X through the trained face landmark detector MobileFaceNet lmk ∈R N×D , where N is the number of tokens and D is the feature dimension; During training, IR-50 is fine-tuned while keeping the weights of MobileFaceNet frozen to ensure correct landmark output; S3. Pruning based on cross attention: For the global branch, the pruning module projects the image features into two feature vectors Q′ img ∈R N×D , K′ img ∈R N×D For local branches, the landmark features are projected into three feature vectors Q lmk ∈R N×D , K lmk ∈R N×D , K lmk ∈R N×D : Q′ img =X img W Q1 ,K′ img =X img W K1 Q lmk =X img W Q2 ,K lmk =X img W K2 ,V lmk =X lmk W V2 Q lmk ∈R N×D Taking the global average Avgpooling(·) means global averaging along the spatial dimension; Exchange the Q vectors of the two branches to fuse local and global features across branches. For the global branch, use For the characteristics of strong perception of local important areas, learn the importance of local areas: where Attn∈R 1×N It is local-aware attention, which is used as the importance measure of each Token.

2. The lightweight facial expression recognition system based on global-local cross attention according to claim 1 is characterized in that: The step S1 is specifically as follows: In the data preprocessing stage, face detection and key point alignment are performed on the facial expression images of the dataset, and data enhancement strategies are used for processing.

3. The lightweight facial expression recognition system based on global-local cross attention according to claim 2 is characterized in that: In step S2, at each stage of the global branch, global features are extracted by the original Transformer encoder and the input features are projected through three linear layers: Q=X img W Q ,K=X img W K ,V=X img W V ; Where W Q , W K , W V ∈R D×D ; Then the multi-head attention can be calculated by the following formula: head i =SA(XW i Q ,XW i K ,XW i V ); MHSA(X)=[head1;...;head h ]W O ; in is the normalized scaling factor, head i is the self-attention output of each head, W O ∈R D×D is the output weight matrix, SA(·) is the calculation structure of self-attention, and MHSA(·) is the calculation structure of multi-head self-attention; The final output is obtained through the multi-layer perceptron MLP and layer normalization LayerNorm, and the probability distribution is calculated through Cls Token: Z′=LayerNorm(X+MHSA(X)); Z=MLP(Z′)=GELU(Z′W1)W2; Where W head represents the weight of the classification head, ClsToken is the output of the last layer Transformer of the global branch. It is the Cls Token output by the last layer Transformer of the local branch, and the output is the final classification probability.

4. The lightweight facial expression recognition system based on global-local cross attention according to claim 3 is characterized in that: In step S3, global features are introduced into local branches for information compensation: Z lmk =MLP(LayerNorm(Z lmk +MHSA(Q′ img ,K lmk ,V lmk ))); Where Z lmk is the output of the cross-fusion Transformer; Using Attn as the pruning metric, take the topk index as the retained Token position, and discard the remaining Tokens: idx = argsort(Attn)[0:k]; X′ img =Selcet(idx,Z′ img )=[t img,idx1 ,t img，idx2 ,...,t img，idxk ,]; X′ imk =Selcet(idx,Z′ lmk )=[t lmk,idx1 ,t lmk,idx2 ,...,t lmk,idxk ,]; Where Selcet(·) is a selection function that selects the token with the index specified by idx along the spatial dimension, t ·，i ∈R 1×D A single Token at index position i of X′.

5. A lightweight facial expression recognition method based on global-local cross attention, characterized in that: Facial expression recognition is performed using the system as described in any one of claims 1 to 4.

Images