Facial expression recognition method in federated learning scene

Through the global local interference decoupling method, the challenges of heterogeneous expression data and interference in federated learning are solved, and more efficient expression recognition performance is achieved.

CN119091482AActive Publication Date: 2024-12-06XIAMEN UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411051228.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-06
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Federated learning faces the challenge of heterogeneous expression data and heterogeneous interference in expression recognition scenarios, resulting in degradation in the performance of existing methods, especially due to heterogeneous distribution of client data and differences in interference attributes between different clients.

Method used

Using the global local interference decoupling (GLDD) method, the interference recognition model is pre-trained on the PIE expression dataset, and the global and local models are initialized. The client performs expression image sample feature extraction and interference decoupling, and the server aggregates feature center is orthogonal. Decoupling and retrain the global model until converges.

Benefits of technology

It effectively alleviates the influence of heterogeneous interference, reduces the impact of heterogeneous expression data and interference, and improves the model training effect and recognition accuracy of federal expression recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091482A_ABST
    Figure CN119091482A_ABST
Patent Text Reader

Abstract

The invention discloses a facial expression recognition method in a federated learning scene, which ingeniously applies a global local interference decoupling (GLDD) method to federated expression recognition, so that local interference decoupling can relieve the challenge of heterogeneous interference by adaptively considering the influence of interference on a local client in a model training process; meanwhile, global interference decoupling reduces the influence of heterogeneous expression data and heterogeneous interference by aggregating expressions and interference information from clients on a global server.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, face recognition technology and model training technology, and in particular to a facial expression recognition method in a federated learning scenario. Background Art

[0002] In the past few decades, facial expression recognition has attracted extensive attention in the fields of computer vision and pattern recognition due to its wide range of applications, such as human-computer interaction, driving monitoring, and intelligent education.

[0003] With the advancement of deep learning, various facial expression recognition methods have been developed (H. Yang, U. Ciftci, and L. Yin, "Facial expression recognition by deexpression residue learning," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2168–2177) and have demonstrated good performance in unconstrained environments. Existing expression recognition methods aim to extract discriminative expression features. For example, some literatures disclose that some methods design effective loss functions to enhance intra-class compactness and inter-class separability. The above methods implicitly consider the interference effects caused by various interference attributes. There are also some literatures that disclose certain methods that explicitly mitigate these disturbances. For example, Meng et al. (Z. Meng, P. Liu, J. Cai, S. Han, and Y. Tong, "Identity-aware convolutional neural network for facial expression recognition," in Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, 2017, pp. 558–565) mitigate the influence of facial identity by considering expression similarity and identity similarity. Ruan et al. (D. Ruan, Y. Yan, S. Chen, J.-H. Xue, and H. Wang, "Deep disturbance disentangled learning for facial expression recognition," in Proceedings of the ACM International Conference on Multimedia, 2020, pp. 2833–2841.) developed a deep disturbance disentangled learning (DDL) method that uses adversarial transfer learning to decouple disturbances.Later, they (D.Ruan, R.Mo, Y.Yan, S.Chen, J.-H.Xue, and H.Wang, "Adaptive deep disturbance-disentangled learning for facial expression recognition," International Journal of Computer Vision, vol.130, no.2, pp.455–477, 2022.) proposed an adaptive DDL method based on self-attention layer. Mo et al. (R.Mo, Y.Yan, J.-H.Xue, S.Chen, and H.Wang, "D 3net: Dual-branch disturbance disentangling network for facial expression recognition," in Proceedings of the ACM International Conference on Multimedia, 2021, pp.779–787.) proposed D 3-Net to model the potential disturbance attribute distribution based on the prior of the Indian buffet process. Wang et al. (K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6897–6906.) developed a self-healing network (SCN) to address label uncertainty in FER.

[0004] Typically, these methods require large-scale data for centralized model training. However, in many practical applications, it is very difficult to obtain a large amount of facial expression data due to privacy issues and legal restrictions. As a substitute, the training data is stored on the local device and not directly shared between local clients. Therefore, how to train accurate FER models using private decentralized expression data deserves further study.

[0005] Federated learning (FL), as a new decentralized learning paradigm, aims to learn a global model on the server by aggregating local models trained on the client. It provides a distributed learning framework that can jointly train models without sharing client data, alleviating privacy issues. Most existing FL methods follow the process of FedAvg (B.McMahan,E.Moore,D.Ramage,S.Hampson,and BAy Arcas,“Communication-efficient learning of deep networks from decentralized data,”in Proceedingsof the Artificial Intelligence and Statistics,2017,pp.1273–1282). A major problem in FL is the heterogeneity of client data distribution. To address this issue, FedProx (S.Li,W.Deng,andJ.Du,“Reliable crowdsourcing and deep localitypreserving learning forexpression recognition in the wild,”in Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition,2017,pp.2852–2861.) introduces a proximal term. MOON (Q. Li, B. He, and D. Song, "Model-contrastive federated learning," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 713–10 722.) adds model contrastive loss to local model training by using comparative learning.CCVR (M. Luo, F. Chen, D. Hu, Y. Zhang, J. Liang, and J. Feng, “No fear of heterogeneity: Classifier calibration for federated learning with non-IID data,” Advances in Neural Information Processing Systems, pp. 59725984, 2021.) retrains the classifier by sampling virtual features from an approximate Gaussian mixture model. Zhu et al. (Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in Proceedings of the International Conference on Machine Learning, 2021, pp. 12 878–12 889.) use GAN to learn a lightweight generator to aggregate client information on the server in a data-free manner. Recently, knowledge distillation (KD) has been introduced into FL. In addition, some methods have recently studied face recognition (FR) under the FL paradigm. However, federated face recognition is fundamentally different from federated expression recognition. For federated face recognition, the total number of identities from different clients is large. In contrast, federated expression recognition has a limited number of expression categories (7 or 8). In addition, federated face recognition needs to predict unseen identities, while federated expression recognition classifies predefined expression categories. Therefore, federated face recognition focuses on learning discriminative features for unseen identities, while federated expression recognition aims to extract feature representations for predefined expression categories. These differences require the development of different learning strategies for federated expression recognition.

[0006] Unfortunately, federated learning has not been well exploited in the context of expression recognition. Federated FER is a practical but difficult task that faces two challenges. First, similar to general FL tasks (e.g., natural image classification), the distribution of expression data stored on clients may vary significantly, resulting in heterogeneous expression data. Second, the interference attributes (e.g., identity, pose, and illumination) on each client can also vary greatly, resulting in heterogeneous interference. For example, some clients mainly store expression data from a single identity, so they involve interference caused by pose and illumination variations. Other clients store expression data from multiple identities, so they contain interference caused by identity and illumination differences. In other words, each client has some specific interference attributes (which we define as significant interference attributes). These two challenges are closely related to federated expression recognition and lead to a sharp drop in the performance of existing FL methods.

[0007] To address the above challenges, this paper proposes a novel global local interference decoupling (GLDD) method for federated expression recognition. Local interference decoupling mitigates the challenge of heterogeneous interference by adaptively considering the impact of interference on local clients. Meanwhile, global interference decoupling reduces the impact of heterogeneous expression data and heterogeneous interference by aggregating the expression and interference information from clients on the global server. Summary of the invention

[0008] In view of this, the purpose of the present invention is to propose a facial expression recognition method in a federated learning scenario that is reliable in implementation, flexible in application, and capable of reducing the impact of heterogeneous expression data and heterogeneous interference.

[0009] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0010] A method for facial expression recognition in a federated learning scenario comprises the following steps:

[0011] A. Pre-train a distractor recognition model on the PIE expression dataset to encode the prior of distractor attributes, and then use it as a teacher model for guidance;

[0012] B. The server initializes the global model and the local model, and sends the parameters of the global model as initial parameters to the client that loads the local model;

[0013] C. The client inputs the expression image samples of the preset expression data set into the backbone network of the local model, and the local model extracts the common features of the expression image samples, and then inputs the common features into the dual-branch network for expression classification and interference decoupling, so as to realize training and obtain the expression feature center and interference feature center;

[0014] D. After the local model training is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server, and the server aggregates the global model and the global expression feature center and global interference feature center;

[0015] E. The server uses the global expression feature center and the global interference feature center to perform orthogonal decoupling to obtain a new expression feature center, and uses the features of the decoupled expression feature center to retrain the classifier of the global model until convergence;

[0016] F. Send the parameters of the global model trained on the server to the client again, and the client loads the local model for the next round of training; until a convergent global model is obtained, and then the global model is used for facial expression recognition.

[0017] As a possible implementation mode, further, step A of this solution includes:

[0018] A1. Prepare a PIE expression dataset, wherein the expression image samples of the PIE expression dataset all contain three different interference attributes;

[0019] A2. Use the resnet18 network as the initial interference recognition model, load the pre-trained model parameters, and then import the data of the PIE expression dataset for training;

[0020] A3. Train the interference recognition model for at least 40 rounds until the model converges, and obtain the trained interference recognition model to encode the prior of interference attributes, which is then used as a teacher model for guidance.

[0021] As a preferred implementation option, preferably, step B of this solution includes:

[0022] B1. The server initializes the global model parameters w t and the parameters of the local model

[0023] B2. Taking the local model as the local model, the server sends the parameters of the global model as the initial parameters to the client that loads the local model, and the local model loads the parameters of the global model to prepare for training.

[0024] As a preferred implementation option, preferably, step C of this solution includes:

[0025] C1. For each client k, send the i-th expression image sample of the preset expression data set to the backbone network to obtain the universal feature

[0026] C2, sending the obtained universal features to the expression classification branch and interference branch of the dual-branch network for expression classification and interference decoupling;

[0027] C3. Sending common features to expression feature encoder Get facial features Then feed it into the classifier Perform expression classification and use cross entropy loss for optimization:

[0028]

[0029] where f(·,·) represents the backbone network, represents the parameters of the backbone network in the tth round;

[0030] C4, the interference branch of the local model extracts client-specific interference features under the guidance of the interference recognition model; the common features are fed into an encoder To extract client-specific interference signatures The image will also be entered Extract a set of prior-based attribute features through a pre-trained interference recognition model in, represents the attribute feature corresponding to the jth interference attribute of the kth client, D is the number of interference attributes, and each attribute feature encodes the prior semantic information of the interference attribute.

[0031] As a better implementation option, preferably, in step C of this solution, there is a probability that the interference attributes of different clients may be very different, which will lead to heterogeneous interference; therefore, considering the impact of different interference attributes on each client, an entropy-guided feature encoding module including a weight learning block and a feature combination block is designed to promote the knowledge transfer from the pre-trained interference recognition model to the local model; that is, given a batch of images, the prediction result of each interference attribute can be obtained; based on this, the formula for calculating the class entropy of the interference attribute is defined as follows:

[0032]

[0033] in, Indicates the proportion of samples classified as category m in a batch of samples, C j is the number of categories of the jth interference attribute; after obtaining the class entropy of all interference attributes, it is normalized to 1 to obtain the normalized entropy g = Z[h1,…,h D ], where Z is the normalized value; the NE-MSE loss is defined as:

[0034]

[0035] Among them, {α1,…,αD}The learnable weights obtained by the weight learning module, g j is the jth element in g;

[0036] Based on the learned weights, the attributes given a priori are fused by feature combination to obtain interference features based on a priori, which is expressed as Then the pre-trained interference recognition model is transferred to the interference branch through the knowledge transfer of KL divergence. The calculation formula is defined as:

[0037]

[0038] Among them, D KL (·,·) represents KL divergence;

[0039] The joint loss optimization function of the entire dual branch is as follows:

[0040] L local =L exp +γ1L kl +γ2L mse

[0041] Among them, γ1 and γ2 are equilibrium parameters.

[0042] As a preferred implementation option, preferably, step D of this solution includes:

[0043] D1. After the training round of the local model is completed, each client will calculate the local expression feature center and the local interference feature center. The calculation method is as follows:

[0044]

[0045]

[0046] in, Indicates that the cth category is in D k The number of represents the data volume of the cth category of the kth client; |D k | represents the number of k-th client samples, represents the local expression feature center, represents the local interference feature center, represents the feature extraction module, represents the backbone network, represents the interference feature extraction module, C represents the number of expression categories, x i represents the i-th sample.

[0047] D2. After the local model calculation is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server; the server aggregates the information uploaded by each client to obtain the global model and the global expression feature center and the global interference feature center; the aggregation calculation method is as follows:

[0048]

[0049]

[0050]

[0051] Among them, w t represents the parameters of the global model, represents the center of the global expression feature after aggregation, represents the center of the global interference feature after aggregation; |S t | represents the number of clients selected in round t; p k Represents the weight of the k-th client, which is the ratio of the number of client data sets to all data combined.

[0052] As a preferred implementation option, preferably, step E of this solution includes:

[0053] E1. Calculate the projection of the global expression feature center on the global interference feature center. The expression is as follows:

[0054]

[0055] Among them, the projection represents the overlapping part of the global interference center on the global expression feature center. The decoupled expression feature center can be obtained by subtracting the overlapping part; its expression is as follows:

[0056]

[0057] Among them, the decoupled global expression feature center effectively encodes specific expression information while suppressing multiple types of interference attributes;

[0058] E2. Use the decoupled global expression feature centers as training samples to retrain the global model to alleviate the heterogeneity of expression data. The optimization objective of retraining is defined as follows:

[0059]

[0060] where h t is the classifier of the global model. ,y c yes The corresponding expression category label, η is the regularization parameter, L ceis the cross entropy loss, is the retrained classifier.

[0061] Based on the above, the present invention also provides a facial expression recognition method, which is applied to the facial expression recognition method in the above-mentioned federated learning scenario.

[0062] Based on the above, the present invention also provides a facial expression recognition system, which is applied with the facial expression recognition method in the above-mentioned federated learning scenario.

[0063] By adopting the above-mentioned technical scheme, the present invention has the following beneficial effects compared with the prior art: the present scheme cleverly uses the global local interference decoupling (GLDD) method for federated expression recognition, so that during the model training process, its local interference decoupling can adaptively consider the impact of interference on local clients, thereby alleviating the challenge of heterogeneous interference; at the same time, global interference decoupling reduces the impact of heterogeneous expression data and heterogeneous interference by aggregating expressions and interference information from clients on the global server. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 Schematic diagram of the application principle of this method.

[0066] Figure 2 This is the overall flow chart for the implementation of this solution.

[0067] Figure 3 Performance diagram of this method when applied on the RAF-DB dataset and FERPlus dataset. DETAILED DESCRIPTION

[0068] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0069] Combination Figure 1 As shown, this embodiment provides a method for facial expression recognition in a federated learning scenario, which includes the following steps:

[0070] A. Prepare the PIE expression dataset and pre-train an interference recognition model on the PIE expression dataset to encode the prior of interference attributes, which is then used as a teacher model for guidance;

[0071] B. The server initializes the global model and the local model, and sends the parameters of the global model as initial parameters to the client that loads the local model;

[0072] C. The client inputs the expression image samples of the preset expression data set into the backbone network of the local model, and the local model extracts the common features of the expression image samples, and then inputs the common features into the dual-branch network for expression classification and interference decoupling, so as to realize training and obtain the expression feature center and interference feature center;

[0073] D. After the local model training is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server, and the server aggregates the global model and the global expression feature center and global interference feature center;

[0074] E. The server uses the global expression feature center and the global interference feature center to perform orthogonal decoupling to obtain a new expression feature center, and uses the features of the decoupled expression feature center to retrain the classifier of the global model until convergence;

[0075] F. Send the parameters of the global model trained on the server to the client again, and the client loads the local model for the next round of training; until a convergent global model is obtained, and then the global model is used for facial expression recognition.

[0076] In this scheme, the PIE expression dataset can use the RAF-DB and FERPlus datasets. The RAF-DB dataset is a rich and diverse outdoor expression dataset collected from the Internet. It includes 7 basic expression categories and 11 compound expression categories, including 15,339 basic expression images and 3,954 compound expression images; FERPlus is an extended version of FER2013, containing 28,709 training images, 3,589 verification images and 3,589 test images. It contains 8 basic expression categories of human faces.

[0077] The interference recognition model of this scheme is pre-trained on the Multi-PIE database to extract interference attribute features for classifying identity, posture and illumination. The Multi-PIE database contains 755,370 images of 337 subjects under 15 viewing angles and 20 illumination conditions. By pre-training on Multi-PIE, DCM encodes the distribution of various interference attribute features. We repeat the training ten times and report the average recognition accuracy on the test data.

[0078] As a possible implementation, further, step A of this solution includes the following in training the interference model:

[0079] A1. Prepare a PIE expression dataset, wherein the expression image samples of the PIE expression dataset all contain three different interference attributes;

[0080] A2. Based on the RAF-DB and FERPlus datasets, the resnet18 network is used as the initial interference recognition model, and the pre-trained model parameters are loaded, and then the data of the PIE expression dataset is imported for training; each facial image of the sample in the dataset is first resized to 256×256, and then randomly cropped to 224×224. This scheme uses ResNet-18 as the backbone network for all methods, and all experiments are conducted using PyTorch on a single NVIDIAGeForce RTX 3090GPU; by performing 100 communication rounds, active customers are randomly sampled at an active customer ratio of R=50% in each round;

[0081] A3. Train the interference recognition model for at least 40 rounds until the model converges, and obtain the trained interference recognition model to encode the prior of interference attributes, which is then used as a teacher model for guidance.

[0082] As a preferred implementation option, preferably, step B of this solution includes:

[0083] B1. The server initializes the global model parameters w t and the parameters of the local model

[0084] B2. Taking the local model as the local model, the server sends the parameters of the global model as the initial parameters to the client that loads the local model, and the local model loads the parameters of the global model to prepare for training.

[0085] As a preferred implementation option, preferably, step C of this solution includes:

[0086] C1. For each client k, send the i-th expression image sample of the preset expression data set to the backbone network to obtain the universal feature

[0087] C2, sending the obtained universal features to the expression classification branch and interference branch of the dual-branch network for expression classification and interference decoupling;

[0088] C3. Sending common features to expression feature encoder Get facial features Then feed it into the classifier Perform expression classification and use cross entropy loss for optimization:

[0089]

[0090] in represents the backbone network, represents the parameters of the backbone network in the tth round;

[0091] C4, the interference branch of the local model extracts client-specific interference features under the guidance of the interference recognition model; the common features are fed into an encoder To extract client-specific interference signatures The image will also be entered Extract a set of prior-based attribute features through a pre-trained interference recognition model in, represents the attribute feature corresponding to the h-th interference attribute of the k-th client, D is the number of interference attributes, and each attribute feature encodes the prior semantic information of the interference attribute.

[0092] As a better implementation option, preferably, in step C of this solution, there is a probability that the interference attributes of different clients may be very different, which will lead to heterogeneous interference; therefore, considering the impact of different interference attributes on each client, an entropy-guided feature encoding module including a weight learning block and a feature combination block is designed to promote the knowledge transfer from the pre-trained interference recognition model to the local model; that is, given a batch of images, the prediction result of each interference attribute can be obtained; based on this, the formula for calculating the class entropy of the interference attribute is defined as follows:

[0093]

[0094] in, Indicates the proportion of samples classified as category m in a batch of samples, C j is the number of categories of the jth interference attribute; after obtaining the class entropy of all interference attributes, it is normalized to 1 to obtain the normalized entropy g = Z[h1,…,h D ], where Z is the normalized value; the NE-MSE loss is defined as:

[0095]

[0096] Among them, {α1,…,α D}The learnable weights obtained by the weight learning module, g j is the jth element in g;

[0097] Based on the learned weights, the attributes given a priori are fused by feature combination to obtain interference features based on a priori, which is expressed as Then the pre-trained interference recognition model is transferred to the interference branch through the knowledge transfer of KL divergence. The calculation formula is defined as:

[0098]

[0099] in, represents KL divergence;

[0100] The joint loss optimization function of the entire dual branch is as follows:

[0101] L local =L exp +γ1L kl +γ2L mse

[0102] Among them, γ1 and γ2 are equilibrium parameters.

[0103] As a preferred implementation option, preferably, step D of this solution includes:

[0104] D1. After the training round of the local model is completed, each client will calculate the local expression feature center and the local interference feature center. The calculation method is as follows:

[0105]

[0106]

[0107] in, Indicates that the cth category is in D k The number of Indicates the amount of data of the cth category of the kth client. |D k | represents the number of k-th client samples, represents the local expression feature center, represents the local interference feature center, represents the feature extraction module; represents the backbone network, represents the interference feature extraction module, C represents the number of expression categories, x i represents the i-th sample.

[0108] D2. After the local model calculation is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server; the server aggregates the information uploaded by each client to obtain the global model and the global expression feature center and the global interference feature center; the aggregation calculation method is as follows:

[0109]

[0110]

[0111]

[0112] Among them, w t represents the parameters of the global model, represents the center of the global expression feature after aggregation, represents the center of the global interference feature after aggregation; |S t | represents the number of clients selected in round t; p k Represents the weight of the k-th client, which is the ratio of the number of client data sets to all data combined.

[0113] As a preferred implementation option, preferably, step E of this solution includes:

[0114] E1. Calculate the projection of the global expression feature center on the global interference feature center. The expression is as follows:

[0115]

[0116] Among them, the projection represents the overlapping part of the global interference center on the global expression feature center. The decoupled expression feature center can be obtained by subtracting the overlapping part; its expression is as follows:

[0117]

[0118] Among them, the decoupled global expression feature center effectively encodes specific expression information while suppressing multiple types of interference attributes;

[0119] E2. Use the decoupled global expression feature centers as training samples to retrain the global model to alleviate the heterogeneity of expression data. The optimization objective of retraining is defined as follows:

[0120]

[0121] where h t is the classifier of the global model. ,y c yes The corresponding expression category label, η is the regularization parameter, L ceis the cross entropy loss, is the retrained classifier.

[0122] This scheme is similar to Shang et al. (X. Shang, Y. Lu, G. Huang, and H. Wang, “Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features,” in Proceedings of the International Joint Conference on Artificial Intelligence). The database is partitioned into K = 10 clients using a Dirichlet distribution controlled by the α parameter (i.e., Dir(α)). The value of α is set to 0.2, 0.1, or 0.05. Lower values ​​of α indicate a more heterogeneous distribution. For local training, the batch size is set to 32, and the local model is optimized using the SGD algorithm with a learning rate of 0.10. For global training, this scheme uses the SGD algorithm with a learning rate of 0.01 to optimize the global model. In the formula, the values ​​of γ1 and γ2 are set to 1.0 and 0.5, respectively; the number of rounds for retraining the global model is set to 40.

[0123] For the convenience of comparison, this scheme is compared with several existing recognition methods to obtain the accuracy (%) comparison results of this scheme method and other expression recognition methods under federated learning. GLDD is the method of the present invention; among them, all methods are carried out under the same experimental environment and parameters. In the comparative test, all models have undergone 100 rounds of global training, and each round will have 10 rounds of local training. The degree of data heterogeneity is controlled by α. The larger the value, the lower the degree of data heterogeneity, and the smaller the value, the greater the degree of data heterogeneity. The results are as follows:

[0124]

[0125] Among them, FedAvgM corresponds to the method proposed by T.-M et al. (T.-M. Hsu, H. Qi, and M. Brown, "Measuring the effects of non-identical data distribution for federated visual classification," arXiv preprint arXiv:1909.06335, 2019.)

[0126] FedProx corresponds to the method proposed by T. Li et al. (T. Li, AK Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450, 2020.)

[0127] AFL corresponds to the method proposed by M. Mohri, et al. (M. Mohri, G. Sivek, and AT Suresh, "Agnostic federated learning," in Proceedings of the International Conference on Machine Learning, 2019, pp. 4615–4625.)

[0128] MOON corresponds to the method proposed by Q. Li et al. (Q. Li, B. He, and D. Song, "Model-contrastive federated learning," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 713–10 722.)

[0129] FedGen corresponds to the method proposed by Z. Zhu et al. (Z. Zhu, J. Hong, and J. Zhou, "Data-free knowledge distillation for heterogeneous federated learning," in Proceedings of the International Conference on Machine Learning, 2021, pp. 12 878–12 889)

[0130] FedAvg+SCN corresponds to the method proposed by K. Wang et al. (K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, "Suppressing uncertainties for large-scale facial expression recognition," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6897–6906.)

[0131] FedAvg+ADDL corresponds to the method proposed by D.Ruan et al. (D.Ruan, R.Mo, Y.Yan, S.Chen, J.-H.Xue, and H.Wang, "Adaptive deep disturbance-disentangled learning for facial expression recognition," International Journal of Computer Vision, vol.130, no.2, pp.455–477, 2022.)

[0132] The above table compares the accuracy of this solution with several existing recognition methods in face recognition; it can be seen that compared with other methods, this solution method achieves the best results overall.

[0133] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for facial expression recognition in a federated learning scenario, characterized in that: It includes the following steps: A. Pre-train a distractor recognition model on the PIE expression dataset to encode the prior of distractor attributes, and then use it as a teacher model for guidance; B. The server initializes the global model and the local model, and sends the parameters of the global model as initial parameters to the client that loads the local model; C. The client inputs the expression image samples of the preset expression data set into the backbone network of the local model, and the local model extracts the common features of the expression image samples, and then inputs the common features into the dual-branch network for expression classification and interference decoupling, so as to realize training and obtain the expression feature center and interference feature center; D. After the local model training is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server, and the server aggregates the global model and the global expression feature center and global interference feature center; E. The server uses the global expression feature center and the global interference feature center to perform orthogonal decoupling to obtain a new expression feature center, and uses the features of the decoupled expression feature center to retrain the classifier of the global model until convergence; F. Send the parameters of the global model trained on the server to the client again, and the client loads the local model for the next round of training; Until a converged global model is obtained, the global model is then used for facial expression recognition.

2. The method for facial expression recognition in a federated learning scenario as claimed in claim 1, characterized in that: Step A includes: A1. Prepare a PIE expression dataset, wherein the expression image samples of the PIE expression dataset all contain three different interference attributes; A2. Use the resnet18 network as the initial interference recognition model, load the pre-trained model parameters, and then import the data of the PIE expression dataset for training; A3. Train the interference recognition model for at least 40 rounds until the model converges, and obtain the trained interference recognition model to encode the prior of interference attributes, which is then used as a teacher model for guidance.

3. The method for facial expression recognition in a federated learning scenario according to claim 1 or 2, characterized in that: Step B includes: B1. The server initializes the global model parameters w t and the parameters of the local model B2. Taking the local model as the local model, the server sends the parameters of the global model as the initial parameters to the client that loads the local model, and the local model loads the parameters of the global model to prepare for training.

4. The method for facial expression recognition in a federated learning scenario as claimed in claim 3, characterized in that: Step C includes: C1. For each client k, send the i-th expression image sample of the preset expression data set to the backbone network to obtain the universal feature C2, sending the obtained universal features to the expression classification branch and interference branch of the dual-branch network for expression classification and interference decoupling; C3. Sending common features to expression feature encoder Get facial features Then feed it into the classifier Perform expression classification and use cross entropy loss for optimization: in represents the backbone network, represents the parameters of the backbone network in the tth round, C represents the number of expression categories, represents the classification label of the i-th sample of the k-th client; represents the expression feature encoder; represents the expression feature classifier; C4, the interference branch of the local model extracts client-specific interference features under the guidance of the interference recognition model; the common features are fed into an encoder To extract client-specific interference signatures You will also be inputting pictures Extract a set of prior-based attribute features through a pre-trained interference recognition model in, It represents the attribute feature corresponding to the Dth interference attribute of the kth client, where D is the number of interference attributes and each attribute feature encodes the prior semantic information of the interference attribute.

5. The method for facial expression recognition in a federated learning scenario as claimed in claim 4, characterized in that: In step C, there is a probability that the interference attributes of different clients may be very different, which will lead to heterogeneous interference; therefore, considering the impact of different interference attributes on each client, an entropy-guided feature encoding module including a weight learning block and a feature combination block is designed to promote the knowledge transfer from the pre-trained interference recognition model to the local model; that is, given a batch of images, the prediction results of each interference attribute can be obtained; based on this, the formula for calculating the class entropy of the interference attribute is defined as follows: in, Indicates the proportion of samples classified as category m in a batch of samples, C j is the number of categories of the jth interference attribute; after obtaining the class entropy of all interference attributes, it is normalized to 1 to obtain the normalized entropy g = Z[h1,…,h D ], where Z is the normalized value; the NE-MSE loss is defined as: Among them, α j The weight learning module obtains the learnable weight of the jth interference feature, g j is the jth element in g, L mse represents the mse loss, D represents the number of interference attributes; Based on the learned weights, the attributes given a priori are fused by feature combination to obtain interference features based on a priori, which is expressed as Then the pre-trained interference recognition model is transferred to the interference branch through the knowledge transfer of KL divergence. The calculation formula is defined as: in, represents KL divergence; The joint loss optimization function of the entire dual branch is as follows: L local =L exp +γ1L kl +γ2L mse Among them, γ1 and γ2 are equilibrium parameters, L local is the overall loss function, L exp is the loss function of the expression branch, L kl is the loss function of the interference branch to transfer knowledge, L mse is the entropy normalized loss.

6. The method for facial expression recognition in a federated learning scenario as claimed in claim 5, characterized in that: Step D includes: D1. After the training round of the local model is completed, each client will calculate the local expression feature center and the local interference feature center. The calculation method is as follows: in, Indicates that the cth category is in D k the number of represents the data volume of the cth category of the kth client; |D k | represents the number of k-th client samples, represents the local expression feature center, represents the local interference feature center, represents the feature extraction module, represents the backbone network, represents the interference feature extraction module, C represents the number of expression categories, x i represents the i-th sample; D2. After the local model calculation is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server; the server aggregates the information uploaded by each client to obtain the global model and the global expression feature center and the global interference feature center; the aggregation calculation method is as follows: Among them, w t represents the parameters of the global model, represents the local model parameters of the k-th client in the t-th round, represents the center of the global expression feature after aggregation, represents the center of the global interference feature after aggregation; |S t | represents the number of clients selected in round t; p k represents the weight of the k-th client, which is the ratio of the number of client data sets to all data combined; It represents the local interference feature center of the kth client. is the class feature center of the cth class sample of the kth client.

7. The method for facial expression recognition in a federated learning scenario as claimed in claim 6, characterized in that: Step E includes: E1. Calculate the projection of the global expression feature center on the global interference feature center. The expression is as follows: in, represents the center of the global expression feature after aggregation, represents the center of the global interference feature after aggregation, It represents the overlapping part of the global interference center and the global expression feature center. The decoupled expression feature center can be obtained by subtracting the overlapping part. Its expression is as follows: in, represents the center of the global expression feature after aggregation, Represents the overlapping part of the global interference center and the global expression feature center; Represents the decoupled expression feature center; the decoupled global expression feature center effectively encodes specific expression information while suppressing multiple types of interference attributes; E2. Use the decoupled global expression feature centers as training samples to retrain the global model to alleviate the heterogeneity of expression data. The optimization objective of retraining is defined as follows: where h t is the classifier of the global model; y c yes The corresponding expression category label, η is the regularization parameter, L ce is the cross entropy loss, is the retrained classifier.

8. A method for recognizing facial expressions, characterized in that: The invention is applied to the method for facial expression recognition in a federated learning scenario as described in any one of claims 1 to 7.

9. A facial expression recognition system, characterized in that: The invention is applied to the method for facial expression recognition in a federated learning scenario as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Expression recognition method integrating global and local features of human face

    CN112651301A

  • Facial expression recognition method based on double-branch interference separation network

    CN113239833A

  • Facial expression recognition method based on local sensitive feature and global feature fusion

    CN114783034A

  • A Micro-expression Recognition Method Based on Feature Decoupling

    CN114937298A

  • Emotion recognition method based on multi-modal clustering federal learning

    CN115620370A