Facial expression recognition method in federated learning scenario

Abstract

The application discloses a face expression recognition method in a federal learning scene, and ingeniously uses a global-local disturbance decoupling (GLDD) method for federal expression recognition, so that in the model training process, the local disturbance decoupling can consider the influence of the disturbance on the local client through self-adaption, and the challenge of heterogeneous disturbance is relieved; meanwhile, the global disturbance decoupling reduces the influence of heterogeneous expression data and heterogeneous disturbance by aggregating expression and disturbance information from clients on the global server.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology, facial recognition technology, and model training technology, and in particular to a facial expression recognition method in a federated learning scenario. Background Technology

[0002] Over the past few decades, facial expression recognition has attracted widespread attention in the fields of computer vision and pattern recognition due to its extensive applications, such as human-computer interaction, driver monitoring, and smart education.

[0003] With the advancement of deep learning, various facial expression recognition methods have been developed (H. Yang, U. Ciftci, and L. Yin, “Facial expression recognition by deexpression residue learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2168–2177) and have demonstrated good performance in unconstrained environments. Existing expression recognition methods aim to extract discriminative facial expression features. For example, some literature discloses methods that design effective loss functions to enhance intra-class tightness and inter-class separability. However, these methods implicitly consider the interference effects caused by various perturbation attributes. Some literature has also disclosed methods that explicitly mitigate these interferences. For example, Meng et al. (Z. Meng, P. Liu, J. Cai, S. Han, and Y. Tong, “Identity-aware convolutional neural network for facial expression recognition,” in Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, 2017, pp. 558–565) mitigated the influence of facial identity by considering expression similarity and identity similarity. Ruan et al. (D. Ruan, Y. Yan, S. Chen, J.-H. Xue, and H. Wang, “Deep disturbance disentangled learning for facial expression recognition,” in Proceedings of the ACM International Conference on Multimedia, 2020, pp. 2833–2841) developed a deep disturbance decoupling learning (DDL) method that utilizes adversarial transfer learning to decouple interferences.Later, they (D. Ruan, R. Mo, Y. Yan, S. Chen, J.-H. Xue, and H. Wang, “Adaptive deep disturbance-disentangled learning for facial expression recognition,” International Journal of Computer Vision, vol. 130, no. 2, pp. 455–477, 2022.) proposed an adaptive DDL method based on a self-attention layer. Mo et al. (R. Mo, Y. Yan, J.-H. Xue, S. Chen, and H. Wang, “D 3net: Dual-branch disturbance disentangling network for facial expression recognition,” in Proceedings of the ACM International Conference on Multimedia, 2021, pp. 779–787.) proposed D 3-Net to model the distribution of latent disturbance attributes based on prior knowledge of the Indian buffet process. Wang et al. (K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6897–6906.) developed a self-healing network (SCN) to address label uncertainty in FER.

[0004] Typically, these methods require large-scale data for centralized model training. However, in many practical applications, obtaining large amounts of facial expression data is very difficult due to privacy concerns and legal restrictions. As an alternative, training data is stored on local devices and not directly shared between local clients. Therefore, how to train accurate FER models using private, distributed facial expression data warrants further investigation.

[0005] Federated learning (FL), as a novel decentralized learning paradigm, aims to learn a global model on a server by aggregating local models trained on clients. It provides a distributed learning framework that allows models to be trained collaboratively without sharing client data, mitigating privacy concerns. Most existing FL methods follow the process of FedAvg (B. McMahan, E. Moore, D. Ramage, S. Hampson, and B.A. Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proceedings of the Artificial Intelligence and Statistics, 2017, pp. 1273–1282). A major problem in FL is the heterogeneity of client data distribution. To address this, FedProx (S. Li, W. Deng, and J. Du, “Reliable crowdsourcing and deep localitypreserving learning for expression recognition in the wild,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2852–2861) introduced a proximate term. MOON (Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 713–10 722.) adds the model's contrastive loss to the local model training by using comparative learning.CCVR (M. Luo, F. Chen, D. Hu, Y. Zhang, J. Liang, and J. Feng, “No fear of heterogeneity: Classifier calibration for federated learning with non-IID data,” Advances in Neural Information Processing Systems, pp. 5972-5984, 2021.) retrains the classifier by sampling virtual features from an approximate Gaussian mixture model. Zhu et al. (Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in Proceedings of the International Conference on Machine Learning, 2021, pp. 12 878–12 889.) used a GAN to learn a lightweight generator that aggregates client information on the server in a data-free manner. Recently, knowledge distillation (KD) has been introduced into FL. In addition, recently, some methods have studied face recognition (FR) under the FL paradigm. However, federated face recognition is fundamentally different from federated expression recognition. For federated face recognition, the total number of identities from different clients is large. In contrast, federated expression recognition has a limited number of expression categories (7 or 8). Furthermore, federated face recognition needs to predict unseen identities, while federated expression recognition classifies predefined expression categories. Therefore, federated face recognition focuses on learning discriminative features of unseen identities, while federated expression recognition aims to extract feature representations of predefined expression categories. These differences necessitate the development of different learning strategies for federated expression recognition.

[0006] Unfortunately, federated learning has not been well applied in the context of facial expression recognition. Federated FER is a practical but challenging task, facing two main challenges. First, similar to general federated learning tasks (such as natural image classification), the distribution of facial expression data stored on the clients can vary significantly, resulting in heterogeneous facial expression data. Second, the perturbation attributes (such as identity, pose, and illumination) on each client can also differ greatly, leading to heterogeneous perturbation. For example, some clients primarily store facial expression data from a single identity, thus involving perturbation caused by variations in pose and illumination. Other clients store facial expression data from multiple identities, thus containing perturbation caused by differences in identity and illumination. In other words, each client has some specific perturbation attributes (which we define as significant perturbation attributes). These two challenges are closely related to federated facial expression recognition, causing a sharp decline in the performance of existing federated learning methods.

[0007] To address the aforementioned challenges, this paper proposes a novel Global-Local Interference Decoupling (GLDD) method for federated facial expression recognition. Local interference decoupling mitigates the challenge of heterogeneous interference by adaptively considering the impact of interference on local clients. Simultaneously, global interference decoupling reduces the influence of heterogeneous facial expression data and interference by aggregating facial expression and interference information from clients on a global server. Summary of the Invention

[0008] In view of this, the purpose of this invention is to propose a facial expression recognition method in federated learning scenarios that is reliable in implementation, flexible in application, and can reduce the impact of heterogeneous expression data and heterogeneous interference.

[0009] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows:

[0010] A method for facial expression recognition in a federated learning scenario includes the following steps:

[0011] A. Pre-train a disturbance recognition model on the PIE facial expression dataset to encode the prior of the disturbance attributes, and then use it as a teacher model for guidance.

[0012] B. The server initializes the global model and the local model, and sends the parameters of the global model as initial parameters to the client loading the local model.

[0013] C. The client inputs facial image samples from the preset facial expression dataset into the backbone network of the local model. The local model extracts the general features of the facial image samples and then inputs the general features into the dual-branch network for facial expression classification and interference decoupling, so as to achieve training and obtain facial expression feature centers and interference feature centers.

[0014] D. After the local model is trained, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server, and the server aggregates them to obtain the global model and global expression feature center and global interference feature center.

[0015] E. The server uses the global expression feature center and the global interference feature center to perform orthogonal decoupling to obtain a new expression feature center, and uses the features of the decoupled expression feature center to retrain the classifier of the global model until convergence.

[0016] F. Send the parameters of the global model trained on the server back to the client, and the client loads the local model for the next round of training; until a converged global model is obtained, and then the global model is used for facial expression recognition.

[0017] As one possible implementation, step A of this solution further includes:

[0018] A1. Prepare the PIE facial expression dataset, in which the facial expression image samples in the PIE facial expression dataset all contain 3 different interference attributes;

[0019] A2. Use the ResNet18 network as the initial interference recognition model, load the pre-trained model parameters, and then import the data from the PIE facial expression dataset for training.

[0020] A3. Train the interference identification model for at least 40 rounds until the model converges to obtain a trained interference identification model, which is then used to encode the prior of interference attributes and used as a guiding teacher model.

[0021] As a preferred implementation option, step B of this solution preferably includes:

[0022] B1. Server initialization of global model parameters w t Parameters of the local model

[0023] B2. Using the local model as the local model, the server sends the parameters of the global model as initial parameters to the client that loads the local model, and the local model loads the parameters of the global model to prepare for training.

[0024] As a preferred implementation option, step C of this solution preferably includes:

[0025] C1. For each client k, feed the i-th expression image sample from the preset expression dataset into the backbone network to obtain general features.

[0026] C2. The obtained general features are fed into the expression classification branch and the interference branch of the dual-branch network respectively for expression classification and interference decoupling;

[0027] C3. Input general features into the facial expression feature encoder Obtain facial features Then it is fed into the classifier. Perform facial expression classification; and optimize using cross-entropy loss:

[0028]

[0029] Where f(·,·) represents the backbone network, This represents the parameters of the backbone network in round t;

[0030] C4. The interference branch of the local model extracts client-specific interference features under the guidance of the interference recognition model; the general features are then fed into an encoder. To extract client-specific interference features At the same time, images will also be input. A set of prior-based attribute features is extracted using a pre-trained interference recognition model. in, Let D represent the attribute feature corresponding to the j-th interfering attribute of the k-th client, where D is the number of interfering attributes, and each attribute feature encodes the prior semantic information of the interfering attribute.

[0031] As a preferred implementation option, in step C of this scheme, the probability that the interference attributes of different clients may differ significantly can lead to heterogeneous interference. Therefore, considering the impact of different interference attributes on each client, an entropy-guided feature encoding module containing a weight learning block and a feature combination block is designed to facilitate knowledge transfer from the pre-trained interference recognition model to the local model. That is, given a batch of images, the prediction result for each interference attribute can be obtained. Based on this, the formula for calculating the class entropy of the interference attribute is defined as follows:

[0032]

[0033] in, C represents the proportion of samples classified as category m in a batch of samples. j Let g be the number of categories of the j-th interfering attribute. After obtaining the class entropy of all interfering attributes, normalize it to 1 to obtain the normalized entropy g = Z[h1,…,h]. D ], where Z is the normalized value; the NE-MSE loss is defined as:

[0034]

[0035] Where, {α1,…,αD The learnable weights, g, obtained by the weight learning module j It is the j-th element in g;

[0036] Based on the learned weights, feature fusion is performed on the given prior attributes through feature combination to obtain perturbation features based on the prior, which are represented as follows: Then, the pre-trained interference recognition model is transferred to the interference branch through knowledge transfer using KL divergence. The calculation formula is defined as:

[0037]

[0038] Among them, D KL (·,·) represents the KL divergence;

[0039] The joint loss optimization function for the entire bi-branch is as follows:

[0040] L local =L exp +γ1L kl +γ2L mse

[0041] Among them, γ1 and γ2 are equilibrium parameters.

[0042] As a preferred implementation option, step D of this solution preferably includes:

[0043] D1. After the training rounds of the local model are completed, each client will calculate the local facial expression feature center and the local interference feature center. The calculation method is as follows:

[0044]

[0045]

[0046] in, This indicates that the c-th category is in D. k The quantity. |D represents the amount of data in the c-th category for the k-th client; k | represents the number of samples from the k-th client. Indicates the local facial expression feature center. Indicates the local interference feature center. This indicates the feature extraction module. Indicates the backbone network. This represents the interference feature extraction module, where C represents the number of expression categories, and x represents the number of expression categories. i Let i represent the i-th sample.

[0047] D2. After the local model calculation is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server. The server then aggregates the information uploaded by each client to obtain the global model, global expression feature center, and global interference feature center. The aggregation calculation method is as follows:

[0048]

[0049]

[0050]

[0051] Among them, w t The parameters representing the global model, This represents the aggregated global facial expression feature center. Represents the global interference feature center after aggregation; |S t | represents the number of clients selected in round t; p k This represents the weight of the k-th client, which is the ratio of the number of client datasets to the total number of data sets.

[0052] As a preferred implementation option, step E of this solution preferably includes:

[0053] E1. Calculate the projection of the global expression feature center onto the global interference feature center, as shown in the following expression:

[0054]

[0055] Here, projection represents the overlapping portion of the global interference center and the global expression feature center. The decoupled expression feature center can be obtained by subtracting the overlapping portion; its expression is as follows:

[0056]

[0057] Among them, the decoupled global expression feature center effectively encodes specific expression information while suppressing multiple types of interference attributes;

[0058] E2. Use the decoupled global facial expression feature centers as training samples to retrain the global model to reduce the heterogeneity of the expression data. The optimization objective of the retraining is defined as follows:

[0059]

[0060] Where h t It is the classifier for the global model. c yes The corresponding emoji category label, η is the regular expression parameter, L ceIt is cross-entropy loss. It is a retrained classifier.

[0061] Based on the above, the present invention also provides a facial expression recognition method, which is applied to the facial expression recognition method in the federated learning scenario described above.

[0062] Based on the above, the present invention also provides a facial expression recognition system, which applies the facial expression recognition method in the federated learning scenario described above.

[0063] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: The ingenious solution uses the Global Local Interference Decoupling (GLDD) method for federated expression recognition, so that during the model training process, its local interference decoupling can adaptively consider the impact of interference on the local client, thus alleviating the challenge of heterogeneous interference; at the same time, global interference decoupling reduces the impact of heterogeneous expression data and heterogeneous interference by aggregating the expression and interference information from the client on the global server. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a schematic diagram illustrating the application principle of the method in this scheme.

[0066] Figure 2 This is the overall flowchart for implementing the proposed method.

[0067] Figure 3 The graph shows the performance of the proposed method on the RAF-DB and FERPlus datasets. Detailed Implementation

[0068] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Combination Figure 1 As shown in the figure, this embodiment provides a facial expression recognition method in a federated learning scenario, which includes the following steps:

[0070] A. Prepare the PIE emoji dataset, pre-train a disturbance recognition model on the PIE emoji dataset to encode the prior of the disturbance attributes, and then use it as the teacher model for guidance.

[0071] B. The server initializes the global model and the local model, and sends the parameters of the global model as initial parameters to the client loading the local model.

[0072] C. The client inputs facial image samples from the preset facial expression dataset into the backbone network of the local model. The local model extracts the general features of the facial image samples and then inputs the general features into the dual-branch network for facial expression classification and interference decoupling, so as to achieve training and obtain facial expression feature centers and interference feature centers.

[0073] D. After the local model is trained, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server, and the server aggregates them to obtain the global model and global expression feature center and global interference feature center.

[0074] E. The server uses the global expression feature center and the global interference feature center to perform orthogonal decoupling to obtain a new expression feature center, and uses the features of the decoupled expression feature center to retrain the classifier of the global model until convergence.

[0075] F. Send the parameters of the global model trained on the server back to the client, and the client loads the local model for the next round of training; until a converged global model is obtained, and then the global model is used for facial expression recognition.

[0076] In this scheme, the PIE facial expression dataset can use the RAF-DB and FERPlus datasets. The RAF-DB dataset is a rich and diverse outdoor facial expression dataset collected from the Internet. It includes 7 basic facial expression categories and 11 composite facial expression categories, including 15,339 basic facial expression images and 3,954 composite facial expression images. FERPlus is an extended version of FER2013, containing 28,709 training images, 3,589 validation images, and 3,589 test images. It includes 8 basic facial expression categories.

[0077] Our interference recognition model is pre-trained on the Multi-PIE database to extract interference attribute features for classifying identity, pose, and lighting. The Multi-PIE database contains 755,370 images of 337 subjects across 15 viewpoints and 20 lighting conditions. By pre-training on Multi-PIE, the DCM encodes the distribution of various interference attribute features. We repeat the training ten times and report the average recognition accuracy on the test data.

[0078] As one possible implementation, further, step A of this solution, in training the interference model, includes:

[0079] A1. Prepare the PIE facial expression dataset, in which the facial expression image samples in the PIE facial expression dataset all contain 3 different interference attributes;

[0080] A2. Based on the RAF-DB and FERPlus datasets, a ResNet-18 network was used as the initial interference recognition model, and pre-trained model parameters were loaded. Then, data from the PIE facial expression dataset was imported for training. Each facial image in the dataset was first resized to 256×256 and then randomly cropped to 224×224. This scheme uses ResNet-18 as the backbone network for all methods, and all experiments were conducted using PyTorch on a single NVIDIA GeForce RTX 3090 GPU. 100 communication rounds were conducted, with active customers randomly sampled in each round at a ratio of R=50%.

[0081] A3. Train the interference identification model for at least 40 rounds until the model converges to obtain a trained interference identification model, which is then used to encode the prior of interference attributes and used as a guiding teacher model.

[0082] As a preferred implementation option, step B of this solution preferably includes:

[0083] B1. Server initialization of global model parameters w t Parameters of the local model

[0084] B2. Using the local model as the local model, the server sends the parameters of the global model as initial parameters to the client that loads the local model, and the local model loads the parameters of the global model to prepare for training.

[0085] As a preferred implementation option, step C of this solution preferably includes:

[0086] C1. For each client k, feed the i-th expression image sample from the preset expression dataset into the backbone network to obtain general features.

[0087] C2. The obtained general features are fed into the expression classification branch and the interference branch of the dual-branch network respectively for expression classification and interference decoupling;

[0088] C3. Input general features into the facial expression feature encoder Obtain facial features Then it is fed into the classifier. Perform facial expression classification; and optimize using cross-entropy loss:

[0089]

[0090] in Indicates the backbone network. This represents the parameters of the backbone network in round t;

[0091] C4. The interference branch of the local model extracts client-specific interference features under the guidance of the interference recognition model; the general features are then fed into an encoder. To extract client-specific interference features At the same time, images will also be input. A set of prior-based attribute features is extracted using a pre-trained interference recognition model. in, Let D represent the attribute feature corresponding to the h-th interfering attribute of the k-th client, where D is the number of interfering attributes, and each attribute feature encodes the prior semantic information of the interfering attribute.

[0092] As a preferred implementation option, in step C of this scheme, the probability that the interference attributes of different clients may differ significantly can lead to heterogeneous interference. Therefore, considering the impact of different interference attributes on each client, an entropy-guided feature encoding module containing a weight learning block and a feature combination block is designed to facilitate knowledge transfer from the pre-trained interference recognition model to the local model. That is, given a batch of images, the prediction result for each interference attribute can be obtained. Based on this, the formula for calculating the class entropy of the interference attribute is defined as follows:

[0093]

[0094] in, C represents the proportion of samples classified as category m in a batch of samples. j Let g be the number of categories of the j-th interfering attribute. After obtaining the class entropy of all interfering attributes, normalize it to 1 to obtain the normalized entropy g = Z[h1,…,h]. D ], where Z is the normalized value; the NE-MSE loss is defined as:

[0095]

[0096] Where, {α1,…,α D The learnable weights, g, obtained by the weight learning module j It is the j-th element in g;

[0097] Based on the learned weights, feature fusion is performed on the given prior attributes through feature combination to obtain perturbation features based on the prior, which are represented as follows: Then, the pre-trained interference recognition model is transferred to the interference branch through knowledge transfer using KL divergence. The calculation formula is defined as:

[0098]

[0099] in, Indicates KL divergence;

[0100] The joint loss optimization function for the entire bi-branch is as follows:

[0101] L local =L exp +γ1L kl +γ2L mse

[0102] Among them, γ1 and γ2 are equilibrium parameters.

[0103] As a preferred implementation option, step D of this solution preferably includes:

[0104] D1. After the training rounds of the local model are completed, each client will calculate the local facial expression feature center and the local interference feature center. The calculation method is as follows:

[0105]

[0106]

[0107] in, This indicates that the c-th category is in D. k The quantity. This represents the amount of data in the c-th category for the k-th client. |D k | represents the number of samples from the k-th client. Indicates the local facial expression feature center. Indicates the local interference feature center. This indicates the feature extraction module; Indicates the backbone network. This represents the interference feature extraction module, where C represents the number of expression categories, and x represents the number of expression categories. i Let i represent the i-th sample.

[0108] D2. After the local model calculation is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server. The server then aggregates the information uploaded by each client to obtain the global model, global expression feature center, and global interference feature center. The aggregation calculation method is as follows:

[0109]

[0110]

[0111]

[0112] Among them, w t The parameters representing the global model, This represents the aggregated global facial expression feature center. Represents the global interference feature center after aggregation; |S t | represents the number of clients selected in round t; p k This represents the weight of the k-th client, which is the ratio of the number of client datasets to the total number of data sets.

[0113] As a preferred implementation option, step E of this solution preferably includes:

[0114] E1. Calculate the projection of the global expression feature center onto the global interference feature center, as shown in the following expression:

[0115]

[0116] Here, projection represents the overlapping portion of the global interference center and the global expression feature center. The decoupled expression feature center can be obtained by subtracting the overlapping portion; its expression is as follows:

[0117]

[0118] Among them, the decoupled global expression feature center effectively encodes specific expression information while suppressing multiple types of interference attributes;

[0119] E2. Use the decoupled global facial expression feature centers as training samples to retrain the global model to reduce the heterogeneity of the expression data. The optimization objective of the retraining is defined as follows:

[0120]

[0121] Where h t It is the classifier for the global model. c yes The corresponding emoji category label, η is the regular expression parameter, L ceIt is cross-entropy loss. It is a retrained classifier.

[0122] This approach is similar to that of Shang et al. (X. Shang, Y. Lu, G. Huang, and H. Wang, “Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features,” in Proceedings of the International Joint Conference on Artificial Intelligence). The database is partitioned into K = 10 clients using a Dirichlet distribution (Dir(α)) controlled by the α parameter. The value of α is set to 0.2, 0.1, or 0.05. Lower α values ​​indicate a more heterogeneous distribution. For local training, the batch size is set to 32, and the local model is optimized using the SGD algorithm with a learning rate of 0.10. For global training, this approach optimizes the global model using the SGD algorithm with a learning rate of 0.01. In the formula, the values ​​of γ1 and γ2 are set to 1.0 and 0.5, respectively; the number of epochs for retraining the global model is set to 40.

[0123] For ease of comparison, this scheme is compared with several existing recognition methods to obtain the accuracy (%) comparison results of this scheme with other expression recognition methods under federated learning. GLDD is the method of this invention. All methods were conducted under the same experimental environment and parameters. In the comparative test, all models underwent 100 rounds of global training, with 10 rounds of local training per round. The degree of data heterogeneity was controlled by α; a larger value indicates lower data heterogeneity, and a smaller value indicates higher data heterogeneity. The results are as follows:

[0124]

[0125] FedAvgM corresponds to the method proposed by T.-M et al. (T.-MHHsu, H. Qi, and M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification,” arXiv preprint arXiv:1909.06335, 2019.)

[0126] FedProx corresponds to the method proposed by T. Li et al. (T. Li, A.K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450, 2020.)

[0127] AFL corresponds to the method proposed by M. Mohri et al. (M. Mohri, G. Sivek, and ATSuresh, “Agnostic federated learning,” in Proceedings of the International Conference on Machine Learning, 2019, pp. 4615–4625.)

[0128] MOON corresponds to the method proposed by Q. Li et al. (Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021, pp. 10 713–10 722.)

[0129] FedGen corresponds to the method proposed by Z. Zhu et al. (Z. Zhu, J. Hong, and J. Zhou, “Data-free knowledge distillation for heterogeneous federated learning,” in Proceedings of the International Conference on Machine Learning, 2021, pp. 12 878–12 889).

[0130] FedAvg+SCN corresponds to the method proposed by K. Wang et al. (K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing uncertainties for large-scale facial expression recognition,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6897–6906.)

[0131] FedAvg+ADDL corresponds to the method proposed by D.Ruan et al. (D.Ruan, R.Mo, Y.Yan, S.Chen, J.-H.Xue, and H.Wang, "Adaptive deep disturbance-disentangled learning for facial expression recognition," International Journal of Computer Vision, vol.130, no.2, pp.455–477, 2022.)

[0132] The table above compares the accuracy of this scheme with several existing recognition methods in face recognition; it can be seen that, compared with other methods, this scheme achieves the best overall results.

[0133] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A facial expression recognition method in a federated learning scenario, characterized in that, It includes the following steps: A. Pre-train a disturbance recognition model on the PIE facial expression dataset to encode the prior of the disturbance attributes, and then use it as a teacher model for guidance. B. The server initializes the global model and the local model, and sends the parameters of the global model as initial parameters to the client loading the local model. C. The client inputs facial image samples from the preset facial expression dataset into the backbone network of the local model. The local model extracts the general features of the facial image samples and then inputs the general features into the dual-branch network for facial expression classification and interference decoupling, so as to achieve training and obtain facial expression feature centers and interference feature centers. D. After the local model is trained, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server, and the server aggregates them to obtain the global model and global expression feature center and global interference feature center. E. The server uses the global expression feature center and the global interference feature center to perform orthogonal decoupling to obtain a new expression feature center, and uses the features of the decoupled expression feature center to retrain the classifier of the global model until convergence. F. Send the parameters of the globally trained model on the server back to the client, so that the client can load the local model for the next round of training; The process continues until a convergent global model is obtained, and then the global model is used for facial expression recognition.

2. The facial expression recognition method in a federated learning scenario as described in claim 1, characterized in that, Step A includes: A1. Prepare the PIE facial expression dataset, in which the facial expression image samples in the PIE facial expression dataset all contain 3 different interference attributes; A2. Use the ResNet18 network as the initial interference recognition model, load the pre-trained model parameters, and then import the data from the PIE facial expression dataset for training. A3. Train the interference identification model for at least 40 rounds until the model converges to obtain a trained interference identification model, which is then used to encode the prior of interference attributes and used as a guiding teacher model.

3. The facial expression recognition method in a federated learning scenario as described in claim 1 or 2, characterized in that, Step B includes: B1. Server initialization of global model parameters w t Parameters of the local model B2. Using the local model as the local model, the server sends the parameters of the global model as initial parameters to the client that loads the local model, and the local model loads the parameters of the global model to prepare for training.

4. The facial expression recognition method in a federated learning scenario as described in claim 3, characterized in that, Step C includes: C1. For each client k, the i-th expression image sample from the preset expression dataset is fed into the backbone network to obtain general features. C2. The obtained general features are fed into the expression classification branch and the interference branch of the dual-branch network respectively for expression classification and interference decoupling; C3. Input general features into the facial expression feature encoder Obtain facial features Then it is fed into the classifier. Perform facial expression classification; and optimize using cross-entropy loss: in Indicates the backbone network. Let represent the parameters of the backbone network in the t-th round, and C represent the number of expression categories. This represents the classification label of the i-th sample from the k-th client; This represents an facial expression feature encoder; This represents an expression feature classifier; C4. The local model's interference branch extracts client-specific interference features under the guidance of the interference recognition model; the general features are then fed into an encoder. To extract client-specific interference features At the same time, images will also be input. A set of prior-based attribute features is extracted using a pre-trained interference recognition model. in, This represents the attribute feature corresponding to the Dth interfering attribute of the kth client, where D is the number of interfering attributes, and each attribute feature encodes the prior semantic information of the interfering attribute.

5. The facial expression recognition method in a federated learning scenario as described in claim 4, characterized in that, In step C, the probability of significant differences in the interference attributes of different clients can lead to heterogeneous interference. Therefore, considering the impact of different interference attributes on each client, an entropy-guided feature encoding module containing weight learning blocks and feature combination blocks is designed to facilitate knowledge transfer from the pre-trained interference recognition model to the local model. That is, given a batch of images, the prediction result for each interference attribute can be obtained. Based on this, the formula for calculating the class entropy of the interference attribute is defined as follows: in, C represents the proportion of samples classified as category m in a batch of samples. j Let g be the number of categories of the j-th interfering attribute. After obtaining the class entropy of all interfering attributes, normalize it to 1 to obtain the normalized entropy g = Z[h1,…,h]. D ], where Z is the normalized value; the NE-MSE loss is defined as: Where, α j To obtain the learnable weights for the j-th perturbation feature by the weight learning module, g j It is the j-th element in g, L mse This represents the MSE loss, where D represents the number of interfering attributes; Based on the learned weights, feature fusion is performed on the given prior attributes through feature combination to obtain perturbation features based on the prior, which are represented as follows: Then, the pre-trained interference recognition model is transferred to the interference branch through knowledge transfer using KL divergence. The calculation formula is defined as: in, Indicates KL divergence; The joint loss optimization function for the entire bi-branch is as follows: L local =L exp +γ1L kl +γ2L mse Where γ1 and γ2 are equilibrium parameters, L local It is the overall loss function, L exp L is the loss function for the expression branch. kl It is the loss function that interferes with the knowledge of branch transitions, L mse It is the entropy normalization loss.

6. The facial expression recognition method in a federated learning scenario as described in claim 5, characterized in that, Step D includes: D1. After the training rounds of the local model are completed, each client will calculate the local facial expression feature center and the local interference feature center. The calculation method is as follows: in, This indicates that the c-th category is in D. k Quantity; |D represents the amount of data in the c-th category for the k-th client; k | represents the number of samples from the k-th client. Indicates the local facial expression feature center. Indicates the local interference feature center. This indicates the feature extraction module. Indicates the backbone network. This represents the interference feature extraction module, where C represents the number of expression categories, and x represents the number of expression categories. i This represents the i-th sample; D2. After the local model calculation is completed, the local model and its corresponding local expression feature center and local interference feature center are uploaded to the server. The server then aggregates the information uploaded by each client to obtain the global model, global expression feature center, and global interference feature center. The aggregation calculation method is as follows: Among them, w t The parameters representing the global model, This represents the local model parameters of the k-th client in round t. This represents the aggregated global facial expression feature center. Represents the global interference feature center after aggregation; |S t | represents the number of clients selected in round t; p k This represents the weight of the k-th client, which is the ratio of the number of client datasets to the sum of all data. This represents the local interference feature center of the k-th client. It is the class feature center of the c-th class sample of the k-th client.

7. The facial expression recognition method in a federated learning scenario as described in claim 6, characterized in that, Step E includes: E1. Calculate the projection of the global expression feature center onto the global interference feature center, as shown in the following expression: in, This represents the aggregated global facial expression feature center. This represents the global interference feature center after aggregation. The overlapping portion of the global interference center and the global expression feature center can be subtracted to obtain the decoupled expression feature center; its expression is as follows: in, This represents the aggregated global facial expression feature center. This represents the overlapping portion of the global interference center and the global expression feature center; This represents the decoupled facial expression feature center; the decoupled global facial expression feature center effectively encodes specific facial expression information while suppressing various types of interfering attributes; E2. Use the decoupled global facial expression feature centers as training samples to retrain the global model to reduce the heterogeneity of the expression data. The optimization objective of the retraining is defined as follows: Where h t It is the classifier for the global model; y c yes The corresponding emoji category label, η is the regular expression parameter, L ce It is cross-entropy loss. It is a retrained classifier.

8. A method for facial expression recognition, characterized in that, Its application includes the facial expression recognition method in the federated learning scenario as described in any one of claims 1 to 7.

9. A facial expression recognition system, characterized in that, Its application includes the facial expression recognition method in the federated learning scenario as described in any one of claims 1 to 7.

Images