Multi-task self-distillation facial expression recognition method and system based on coarse-grained labels

Abstract

The present invention discloses a multi-task self-distillation facial expression recognition method and system based on coarse-grained labels, which involves artificial intelligence and proposes this solution to solve the problems in the prior art. The following steps are performed during expression recognition: S1. Feature pre-extraction; S2. Fine-grained feature extraction; S3. Coarse-grained feature extraction; S4. Self-distillation loss; S5. Feature alignment; S6. Overall training loss. The advantages are: (1) Multi-task learning is used to guide the learning of fine-grained facial expression features through relatively simple coarse-grained classification tasks, thereby reducing the learned non-expression-related redundant features. In addition, self-distillation loss is introduced to further realize the knowledge transfer from coarse-grained to fine-grained branches. (2) Coarse-grained labels are based on facial expression priors and are obtained through manual segmentation, avoiding the need for additional expert labeling. (3) Coarse-grained facial expression features are mapped to fine-grained feature space, further improving the effect of knowledge distillation.

Description

Technical Field

[0001] The present invention relates to artificial intelligence, and in particular to a multi-task self-distillation facial expression recognition method and system based on coarse-grained labels. Background Art

[0002] Facial expressions are a powerful form of nonverbal communication, capable of conveying emotions and intentions. Automatic analysis of facial expressions is of great significance in fields such as healthcare, finance, and human-computer interaction systems. With the development of computer technology, facial expression recognition (FER) methods based on deep learning have been widely studied. In the early stages, traditional FER methods mainly relied on manual feature extraction, such as local binary patterns (LBP), non-negative matrix factorization (NMF), and sparse learning. These methods used artificially predefined features to extract facial expression information. Due to the significant improvement in chip processing power (such as GPUs) and the optimization of network architectures, research has gradually shifted to deep learning methods. Deep learning technology utilizes more abundant expression data to more effectively handle the various challenges of emotion recognition in real-world scenarios, and has achieved significant performance improvements.

[0003] Despite the powerful feature learning capabilities of deep learning, its application to facial expression recognition (FER) remains problematic. In recent years, researchers have collected a large number of facial expression datasets from challenging real-world scenarios, driving the transition of FER from laboratory to outdoor settings. In outdoor settings, various interferences, such as illumination, occlusion, pose variations, and redundant non-expression-related facial information, are key determinants of recognition accuracy. To fully extract facial expression features under these interferences, existing FER methods primarily focus on designing network training methods. Xie et al. (W. Xie, Z. Peng, L. Shen, W. Lu, Y. Zhang, S. Song, Cross-layer contrastive learning of latent semantics for facial expression recognition, IEEE Transactions on Image Processing (2024) 2514–2529) proposed a cross-layer contrastive learning framework that enhances shallow features by aligning shallow and deep features, allowing for the learning of more discriminative features through multi-scale feature fusion. However, this approach does not fully exploit prior knowledge about facial expressions and fails to reduce the learned non-expression-related redundancy. Multi-task learning is a technique for simultaneously learning multiple tasks to enhance network performance through shared representations. Liu et al. (Y.Liu, X.Zhang, J.Kauttonen, G.Zhao, Uncertain facial expression recognition via multi-task assisted correction, IEEE Transactions on Multimedia 26(2024)2531–2543) proposed a multi-task assisted correction framework that highlights reliable samples and suppresses uncertain samples through a confidence estimation module. It also introduces action unit detection and pleasure-arousal measurement as auxiliary tasks to learn semantic distribution and alleviate category imbalance. However, these methods require additional facial labels, which increases the cost of expert annotation.Zhang et al. (X. Zhang, J. Zhu, D. Wang, Y. Wang, T. Liang, H. Wang, Y. Yin, A gradual self distillation network with adaptive channel attention for facial expression recognition, Applied Soft Computing 161 (2024) 111762–111773) proposed a method to gradually extract knowledge from deeper blocks to shallower blocks. However, since each layer extracts different features, this distillation method sometimes leads to learning chaotic features. Considering the knowledge transfer ability of self-distillation, feature self-distillation can guide the learning of complex fine-grained features through coarse-grained intermediate features that are easier to learn.

[0004] Defects and shortcomings of existing technology:

[0005] 1. Existing network training methods do not fully consider the prior information of facial expressions, and usually inevitably learn a large amount of redundant information related to non-expression, which easily interferes with the accuracy of expression recognition and affects the model's discriminative ability.

[0006] 2. Existing multi-task learning methods usually require additional expert annotation, such as identity, key points, valence arousal, and other label information, which significantly increases the cost and complexity of data annotation. Especially when annotating on large-scale datasets, the cost and time investment are extremely huge.

[0007] 3. In the multi-task learning framework, the features learned by different tasks are often located in different latent spaces. Due to the misalignment of feature spaces, existing methods have difficulty in achieving effective knowledge transfer, resulting in the information learned by sub-tasks not being fully utilized by the main task, thereby limiting the overall performance of the model. Summary of the Invention

[0008] The purpose of the present invention is to provide a multi-task self-distillation facial expression recognition method and system based on coarse-grained labels to solve the problems existing in the above-mentioned prior art.

[0009] The multi-task self-distillation facial expression recognition system based on coarse-grained labels described in the present invention includes a feature extractor, a main task branch module, a feature alignment module and a coarse-grained auxiliary module;

[0010] Perform the following steps during expression recognition:

[0011] S1. Feature pre-extraction;

[0012] S2. Fine-grained feature extraction;

[0013] S3. Coarse-grained feature extraction;

[0014] S4. Losses from distillation;

[0015] S5. Feature alignment;

[0016] S6. Overall training loss.

[0017] The multi-task self-distillation facial expression recognition method based on coarse-grained labels described in the present invention utilizes the system to perform facial expression recognition.

[0018] The advantages of the multi-task self-distillation facial expression recognition method and system based on coarse-grained labels described in the present invention are:

[0019] (1) Multi-task learning is used to guide the learning of fine-grained facial expression features through a relatively simple coarse-grained classification task, thereby reducing the learned redundant features that are not related to expression. In addition, self-distillation loss is introduced to further achieve knowledge transfer from coarse-grained to fine-grained branches.

[0020] (2) Coarse-grained labels are based on facial expression priors and obtained through manual segmentation, avoiding the need for additional expert annotation.

[0021] (3) Map the coarse-grained facial expression features to the fine-grained feature space to further improve the effect of knowledge distillation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural diagram of the multi-task self-distillation facial expression recognition system described in the present invention.

[0023] Figure 2 It is a structural diagram of the feature alignment module described in the present invention. DETAILED DESCRIPTION

[0024] The multi-task self-distillation facial expression recognition system based on coarse-grained labels in the present invention includes a feature extractor, a main task branch module, a feature alignment module and a coarse-grained auxiliary module, such as Figure 1 and Figure 2 As shown in the figure, the feature pre-extractor is the first three stages of Tiny Swin Transformer, the Encoder is the fourth stage of Tiny Swin Transformer, AvgPooling represents the global average operation in the spatial dimension, FC represents the fully connected layer, ReLu and Leakyrelu represent the corresponding activation functions, and Dropout is a regularization operation to prevent overfitting. Represents the Hadamard product operation.

[0025] The multi-task self-distillation facial expression recognition system based on coarse-grained labels performs the following steps during expression recognition:

[0026] S1. Feature pre-extraction: The feature pre-extractor is used to obtain shared shallow features. The first three stages of Tiny SwinTransformer are used as feature pre-extractors. The input facial expression image is Where c represents the number of channels, h and w represent the height and width of the facial expression image respectively. The facial expression image is processed by the feature pre-extractor to obtain shallow features. Where t represents the number of tokens in the feature and d is the dimension of each token. These shallow features are used for further feature extraction and knowledge transfer.

[0027] S2. Fine-grained feature extraction: The extracted shallow features are used for further feature extraction and knowledge transfer. In order to learn fine-grained facial expression features for classifying the seven basic expressions of happiness, surprise, sadness, anger, disgust, fear and neutrality, the features x The encoder is input into the main task branch module. The encoder has the same configuration as the fourth stage of Tiny Swin Transformer and is specifically used to extract deep facial expression features. In order to reduce the dimension of the feature space, the output of t tokens is globally averaged and pooled using the average pooling operation to obtain the final fine-grained feature.

[0028] x f =AvgPool(E f (x));

[0029] p f =σ(FC f (x f ));

[0030] FC f (·) represents the fully connected layer operation, and σ(·) is the softmax operation, which is used to transform the output into the final classification probability distribution.

[0031] The cross entropy loss function is used to train the model. The cross entropy loss function is used to measure the difference between the model output and the true label. Its formula is as follows:

[0032]

[0033] Among them, N is the total number of samples, K1 is the number of categories for fine-grained classification, is the kth item in the one-hot encoding of the true label of the i-th sample, is the probability of belonging to the kth class predicted by the model.

[0034] S3. Coarse-grained feature extraction:

[0035] In outdoor environments, fine-grained facial expression recognition tasks are often affected by environmental noise and high inter-class similarity, resulting in redundant learned features and poor discriminability. To address this issue, a coarse-grained FER task based on expression priors is introduced. This task includes a binary classification task for "neutral" and "expression," and a four-class classification task for "positive," "negative," "neutral," and "surprise." These two types of coarse-grained labels provide prior knowledge of facial expressions based on different expression classification benchmarks.

[0036] The coarse-grained auxiliary module performs these relatively simple coarse-grained classification tasks through multi-task learning. The input shallow features x are fed into two auxiliary branches, where they are passed through encoders to extract coarse-grained features. The encoders in the coarse-grained auxiliary module have the same structure as the encoders in the main task module. First, the shallow features x pass through the encoders in the two auxiliary branches to obtain features for the two-class and four-class classification tasks:

[0037] x c1 =AvgPool(E c1 (x)), x c2 =AvgPool(E c2 (x));

[0038] Among them, E c1 is the encoder for the four-category task, E c2 It is an encoder for binary classification tasks. Then, the coarse-grained features are obtained through average pooling operation. and Then the probability distribution P of coarse-grained classification is obtained through the full connection layer and softmax operation c1 and p c2 , use the cross entropy function to calculate the loss of the four-classification task and the two-classification task respectively:

[0039]

[0040] Among them, K2 represents the number of categories in the four-category task, and K3 represents the number of categories in the two-category task. is the one-hot label of the i-th sample in the four-category task, is the one-hot label of the i-th sample in the binary classification task, is the probability that the network prediction of the four-class classification task belongs to the kth class, is the probability that the network prediction for a binary classification task belongs to the kth class.

[0041] S4. Self-distillation loss: In multi-task learning, knowledge distillation can transfer knowledge from subtasks to the main task, thereby enhancing the model's discriminative ability without incurring additional computational overhead. Combining multi-task learning with self-distillation loss, we utilize coarse-grained auxiliary branches as teachers and the main branch as students. By using self-distillation, the main branch can more effectively learn coarse-grained expression priors. The process is expressed as follows:

[0042]

[0043] in, Respectively by Obtained through the softmax function and feature alignment module, KL divergence together with the softmax operation makes distillation more stable.

[0044] S5. Feature Alignment: Facial expression features learned based on labels of different granularity typically reside in different latent feature spaces. Directly transferring knowledge between tasks can lead to performance limitations. Therefore, to effectively transfer knowledge, a feature alignment module is proposed to map coarse-grained features into a fine-grained feature space, thereby guiding the learning of fine-grained tasks through knowledge distillation.

[0045] Coarse-grained features x for the four-category task c1 and the coarse-grained features x for the binary classification task c2 Mapped to the fine-grained feature space through the feature alignment module, the mapped features are represented as x f1 and x f2 The formula is as follows:

[0046] x f1 =SE(x c1 )⊙FAM c1 (x c1 ), x f2 =SE(x c1 )⊙FAM c2 (x c2 );

[0047] Among them, x f1 is the projection of the coarse-grained features of the four categories to the fine-grained feature space, x f2 It is the projection of the coarse-grained features of the two-class classification to the fine-grained feature space, FAM c1 (·) is the feature alignment module for four categories, FAM c2 (·) is the feature alignment module for the binary classification task. Each feature alignment module contains three fully connected layers and LeakyReLU activation function to enhance the expression ability of the module. Figure 2As shown in Figure 3, after feature alignment, the coarse-grained expression features can be further transformed into fine-grained expression features, thereby transferring the knowledge in the auxiliary branch to the main task branch.

[0048] S6. Overall training loss: Combine the loss of multi-task classification task and self-distillation loss to construct the overall loss function. All of them are used as the discriminant loss function for the classification task. In order to simplify the weight setting of these loss items, the total weight of these three losses is set to 1. In addition, Both are used for self-distillation losses. The final loss function is expressed as follows:

[0049]

[0050] Among them, α, β, and γ are trade-off parameters for balancing various supervision items.

[0051] By distilling the main task through the four-classification task and the two-classification task, there is no need to consider the problems of large differences in task difficulty and inefficient information transmission. Therefore, a progressive distillation strategy is adopted: the two-classification branch acts as a teacher to specify the learning of the four-classification branch, and the four-classification branch acts as a teacher to guide the learning of the main task branch, helping the model to gradually build a more detailed feature representation, ensuring that the model can extract effective features layer by layer from simple to complex.

[0052]

[0053] γ1 and γ2 change dynamically with the current epoch number e. The purpose is to make the network tend to let the four-classification branch learn knowledge of different granularities from the two-classification branch in the early stage, and focus on knowledge distillation of the main branch in the later stage.

[0054] The multi-task self-distillation facial expression recognition method based on coarse-grained labels described in the present invention utilizes the system to perform facial expression recognition.

[0055] Those skilled in the art can make various other corresponding changes and deformations based on the technical solutions and concepts described above, and all of these changes and deformations should fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-task self-distillation facial expression recognition system based on coarse-grained labels, characterized by: It includes feature extractor, main task branch module, feature alignment module and coarse-grained auxiliary module; Perform the following steps during expression recognition: S1. Feature pre-extraction; S2. Fine-grained feature extraction; S3. Coarse-grained feature extraction; S4. Losses from distillation; S5. Feature alignment; S6. Overall training loss; The step S1 is specifically as follows: The feature pre-extractor obtains shared shallow features and uses the first three stages of Tiny Swin Transformer as the feature pre-extractor; the input facial expression image is Wherein c represents the number of channels, h and w represent the height and width of the facial expression image respectively; the facial expression image is processed by the feature pre-extractor to obtain shallow features. Where t represents the number of tokens of the feature, and d is the dimension of each token; The step S2 is specifically as follows: The extracted shallow features are used for further feature extraction and knowledge transfer; the feature x is input to the encoder in the main task branch module; the encoder has the same configuration as the fourth stage of Tiny Swin Transformer and is specifically used to extract deep facial expression features; Use the average pooling operation to perform global average pooling on the output of t tokens to obtain the final fine-grained features x f =AvgtPool(E f (x)); p f =σ(FC f (x f )); FC f (·) represents the fully connected layer operation; σ(·) is the softmax operation used to transform the output into the final classification probability distribution; The model is trained using the cross-entropy loss function, which measures the difference between the model's output and the true label. The formula is as follows: Among them, N is the total number of samples, K1 is the number of categories for fine-grained classification, is the kth item in the one-hot encoding of the true label of the i-th sample, is the probability of belonging to the kth class predicted by the model; The step S3 is specifically as follows: A coarse-grained FER task is introduced based on the expression prior, including a two-class classification task for "neutral" and "expression", and a four-class classification task for "positive", "negative", "neutral", and "surprise"; The coarse-grained auxiliary module performs the coarse-grained classification task through multi-task learning. The input shallow features x are fed into two auxiliary branches respectively, and the coarse-grained features are extracted through the encoder. The encoder in the coarse-grained auxiliary module has the same structure as the encoder in the main task module. First, the shallow feature x passes through the encoder in the two auxiliary branches to obtain the features of the two-class and four-class tasks: x c1 =AvgPool(E c1 (x)), x c2 =AvgPool(E c2 (x)); Among them, E c1 is the encoder for the four-category task, E c2 is an encoder for binary classification tasks; Then, the coarse-grained features are obtained through the average pooling operation and Then, the probability distribution p of the coarse-grained classification is obtained through the fully connected layer and softmax operation. c1 and p c2 , use the cross entropy function to calculate the loss of the four-classification task and the two-classification task respectively: Among them, K2 represents the number of categories in the four-category task, and K3 represents the number of categories in the two-category task. is the one-hot label of the i-th sample in the four-category task, is the one-hot label of the i-th sample in the binary classification task, is the probability that the network prediction of the four-class classification task belongs to the g-th class, is the probability that the network prediction for the binary classification task belongs to the hth class; The step S4 is specifically as follows: Combining multi-task learning with self-distillation loss, using coarse-grained auxiliary branches as teachers and main branches as students, we learn coarse-grained expression priors through self-distillation. The process can be expressed as follows: in, Respectively by Obtained through the softmax function and feature alignment module; The step S5 is specifically as follows: The feature alignment module is used to map coarse-grained features into fine-grained feature space; Coarse-grained features x for the four-category task c1 and the coarse-grained features x for the binary classification task c2 Mapped to the fine-grained feature space through the feature alignment module, the mapped features are represented as x f1 and x f2 : Among them, x f1 is the projection of the coarse-grained features of the four categories to the fine-grained feature space, x f2 It is the projection of the coarse-grained features of the two-class classification to the fine-grained feature space, FAM c1 (·) is the feature alignment module for four categories, FAM c2 (·) is the feature alignment module for the binary classification task. Each feature alignment module contains three fully connected layers and a LeakyReLU activation function to enhance the module's expressive power. After feature alignment, the coarse-grained expression features are further converted into fine-grained expression features, thereby transferring knowledge from the auxiliary branch to the main task branch. The step S6 is specifically as follows: Combine the loss of multi-task classification task and self-distillation loss to construct the overall loss function; The total weight of these three losses is set to 1; Both are losses used for self-distillation; the final loss function is expressed as: Among them, α, β, and γ are trade-off parameters for balancing various supervision items.

2. The multi-task self-distillation facial expression recognition system based on coarse-grained labels according to claim 1 is characterized in that In step S6, a progressive distillation strategy is adopted: The two-classification branch serves as a teacher to specify the learning of the four-classification branch, and the four-classification branch serves as a teacher to guide the learning of the main task branch, helping the model to gradually build more detailed feature representations and extract effective features layer by layer. γ1,γ2 change dynamically with the current epoch number e.

3. A multi-task self-distillation facial expression recognition method based on coarse-grained labels, characterized by: Facial expression recognition is performed using the system as described in any one of claims 1-2.

Images