A federated learning method for long-tail heterogeneous data

Abstract

The present invention discloses a federated learning method for long-tail heterogeneous data, comprising the following steps: step 1, a server randomly initializes a global model w and sends model parameters to each client, each client uses the received model parameters to update the model, and uploads the updated model parameters to the server; step 2, the server aggregates the received local model parameters to obtain a teacher model and a student model; step 3, the server calibrates the teacher model obtained in step 2, allows the teacher model to learn on unbiased knowledge, and thus teaches a good student model; step 4, uses knowledge distillation to transfer the unbiased knowledge of the teacher model to the student model, and then sends the student model to each client to start the next round of federated training.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a federated learning method for long-tail heterogeneous data. Background Art

[0002] With the further development of big data, it has become a global trend to attach importance to data privacy and security. At the same time, most industry data presents the phenomenon of data islands. How to conduct cross-organizational data cooperation under the premise of satisfying user privacy protection, data security and regulations is a major problem that troubles artificial intelligence practitioners. "Federated learning" will become a key technology to solve this industry problem.

[0003] Federated learning was first proposed by Google in 2016. Its goal is to achieve joint modeling and improve the effectiveness of artificial intelligence models on the basis of ensuring data privacy, security, and legal compliance. A core challenge of federated learning is the heterogeneity of data distribution among different parties, that is, heterogeneous data, which greatly reduces the performance of federated learning. At the same time, the total data distribution of the parties involved in the training often presents a long-tail distribution rather than a balanced distribution, which will cause the model to perform well on categories with a large number of samples (head classes) and poorly on categories with a small number of samples (tail classes). Therefore, it is of great significance to study the long-tail distribution and heterogeneous data problems in federated learning.

[0004] Existing methods for solving heterogeneous data in federated learning can be divided into two categories: client-side methods and server-side methods. The first type of method is mainly to regularize the client's local model update, using the knowledge of the global model to limit the training direction of the client's local model to prevent those local models that are significantly different from the global model from deviating from the training direction of the entire system, thereby reducing the impact of heterogeneous data. The second type of method is the server-side method, which mainly uses special aggregation strategies to alleviate the negative impact of heterogeneous data.

[0005] The above methods have solved the problem of heterogeneous data in federated learning to a certain extent. However, these methods are implemented under the premise that the global data distribution is balanced. In real scenarios, the global data distribution is almost impossible to be balanced, and its distribution often tends to be a long-tail distribution. The global model trained by the above methods still has poor performance in the tail class and cannot achieve good results.

[0006] Existing methods for studying long-tail data are mainly divided into three categories: resampling, reweighting, and representation learning. The first method mainly repeats the data of categories with small sample sizes and reduces the data of categories with large sample sizes, with the aim of rebuilding a balanced data set. The second method modifies the loss function to make it more beneficial to the training of the tail categories. The third method uses the characteristics of deep learning to focus on representation learning of input data.

[0007] In the above methods to solve the long-tail data distribution, the data is concentrated together for model training, which is not applicable in the real federated learning environment. To solve the data imbalance problem in federated learning, Duan, M et al. proposed the Astraea method, which first samples the data before training the model to build a balanced training data set to alleviate the global imbalance. Then use some Mediators to assign the client group responsible for each Mediator according to the KL divergence between the Mediators, and rearrange the model training of the client. By selectively combining clients with heterogeneous data, a new local balance may be achieved. However, this will cause some clients to never be selected, that is, they will not participate in the federated training process, and the information they have cannot be used. Wang, L et al. proposed the Ratio loss method, which realizes the opaque monitoring of the data imbalance problem in federated learning, and proposes a new loss function RatioLoss to reduce the impact of imbalance. However, the performance of this method drops sharply with the deepening of data heterogeneity. Summary of the invention

[0008] In order to solve the shortcomings of the existing technology and achieve the purpose of improving the model performance under federated learning while meeting the requirements of user privacy protection and data security, thereby improving the efficiency of image recognition, the present invention adopts the following technical solutions:

[0009] A federated learning method for long-tail heterogeneous data includes the following steps:

[0010] S1, the server randomly initializes the global model w and sends the model parameters to each client. Each client uses the received model parameters to update the local model and uploads the updated local model parameters to the server;

[0011] S2, the server aggregates the received local model parameters to obtain the teacher model and the student model;

[0012] S3, the server calibrates the teacher model so that the teacher model learns on unbiased knowledge, thereby teaching a good student model;

[0013] S4, through knowledge distillation, transfers the unbiased knowledge of the teacher model to the student model, and then sends the student model to each client to start the next round of federated training.

[0014] Furthermore, in step S1, the server initializes the global model parameter w, randomly selects the client set S participating in this round of training, and broadcasts the model parameter to the client set S participating in this round of training. Each client in S uses the received global model parameter w and local data to perform stochastic gradient descent (SGD) to update the local model. The local model parameter updated by client k is w k ,After the update, each client sends its updated model parameters back to the server.

[0015] Further, step S2 includes the following steps:

[0016] S21, the server side averages and weights the local model parameters to obtain the student model. The calculation formula is as follows:

[0017]

[0018] φ s (x) = φ w (x) (Formula 2)

[0019] Among them, |D k | represents the amount of image data owned by the kth client, |D| represents the total amount of image data owned by all clients, K represents the number of clients, x represents the input image data, φ w (·) represents the network of the federated average model, φ s (·) represents the network of the student model;

[0020] S22, the server performs weighted aggregation on the local model parameters to obtain the teacher model. The calculation formula is as follows:

[0021]

[0022] Among them, φ t (·) represents the network of the teacher model, e k Represents the weight assigned to client k, indicating The network of the kth client.

[0023] Furthermore, in step S3, since the local models are trained on local data with different distributions, the performance of each local model on the tail class may be different, so we want to assign higher weights to the local models that perform better on the tail class. However, the server does not know which images are tail classes and which client local models perform well on them. Therefore, instead of giving each client a fixed weight, we propose a client-based weight allocation strategy to calculate the weight e of each client local model. k , and finally e kNormalize it so that its sum is equal to 1, which is the final weight, weight e k The calculation formula is as follows:

[0024]

[0025] Among them, a e ∈R c and b e represents the network parameters that can be learned, R c Represents a c-dimensional vector, T is the transposed symbol, and client-based calibration is like the self-attention mechanism. The weights of the local model are calculated based on the original output logits of the model, and then the weights are multiplied back to the original output logits.

[0026] Furthermore, in step S3, if no client local model can handle the tail class well, the teacher model obtained by weighted ensemble is still biased towards the head class. To solve this problem, we propose a class-based original output logits calibration strategy to further improve the performance of the model in the tail class. Let the calibrated model output logits be z cl , the calculation formula is as follows:

[0027] z cl =a z ⊙φ t (x)+b z (Formula 5)

[0028] Among them, a z and b z represents the network parameters that can be learned, and ⊙ represents the Hadamard product.

[0029] Furthermore, in step S3, the above logits calibration strategy is effective only if the local model extracts good enough representation information from the input image data. If the feature extraction of the input image data by the client local model is seriously affected by the long-tail distribution, then it is not enough to calibrate the output logits. Therefore, we need to update the feature extractor to further improve the model performance. We use additional balanced labeled images on the server to form a balanced labeled dataset. Fine-tune on the global model w to obtain the fine-tuned model because The data distribution is balanced, so fine-tuning the model We can obtain an unbiased feature extractor, and then we can obtain the output logits of the fine-tuned model for the input image data x as where z ft represents the output of the fine-tuned model for x, Represents the network of the fine-tuned model.

[0030] Further, fine-tuning the model Among them, η represents the learning rate, represents the loss function, Indicates derivation.

[0031] Further, in step S3, z cl and z ft It is to calibrate the teacher model from two different levels. cl The output logits level of the teacher model is calibrated, and the feature extractor of the model is fixed, while z ft is the result of fine-tuning the feature extractor to improve the model's feature extraction capability. To fully combine the advantages of the two, we propose to use a calibrated gating network to calibrate z cl and z ft To make a trade-off, the calibrated gating network takes the integrated features as input and outputs weights through a nonlinear layer, so that each sample obtains different weights according to its own characteristics. The weight calculation formula is as follows:

[0032] σ=sigmoid(u T v) (Formula 6)

[0033] in, represents the integrated feature, represents the feature extractor of the kth client, u∈R d represents the network parameters that can be learned, R d represents a d-dimensional vector, so the final calibration model output logits through the calibration gating network is z′, calculated as follows:

[0034] z′=σz cl +(1-σ)z ft (Formula 7)

[0035] where σ∈(0,1) is used to weigh z cl and z ft Both models output logits.

[0036] Furthermore, during the entire process of integrated calibration, all learnable parameters are The cross entropy loss on is updated, and the loss function is as follows:

[0037]

[0038] Among them, C represents the number of categories, y j represents the true label of the input image data, j represents the value of the jth dimension in y, exp(·) represents the exponential function with the natural constant e as the base, z′ j represents the value of the jth dimension in the final calibration z′, z′i represents the value of the i-th dimension in the final calibration z′, z′ is a c-dimensional vector, z′ j and z′ i Represents the value of one dimension respectively.

[0039] Furthermore, in step S4, the unbiased knowledge of the teacher model is transferred to the student model through knowledge distillation. Specifically, in order to better teach the unbiased knowledge of the teacher model to the student model, we use a combination of labeled data training and unlabeled data distillation to train the student model, and its loss function is as follows:

[0040] L′=(1-λ)L CE +λL KL (Formula 9)

[0041] Among them, L CE represents the cross entropy loss between the model output logits of the student model and the image true label ground-truth, L KL Represents the relative entropy (KL, Kullback-Leibler) divergence of the model output logits between the teacher model and the student model, by balancing the labeled data set Calculate L CE , and use another unlabeled dataset consisting of unlabeled images Calculate L KL To further improve the knowledge distillation performance, λ∈[0,1] represents a hyperparameter for L CE and L KL Make trade-offs.

[0042] The advantages and beneficial effects of the present invention are:

[0043] This paper studies the joint problem of heterogeneous data and long-tail distribution in federated learning, fully utilizes the diversity of client local models to deal with heterogeneous data problems, and proposes a new model calibration strategy and gating network to effectively solve the long-tail problem, further improving the model performance under federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the method of the present invention.

[0045] Figure 2 It is a client data distribution diagram in the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0047] like Figure 1 As shown in FIG. 1 , a federated learning method for long-tail heterogeneous data includes the following steps:

[0048] Step 1: Prepare the data set, initialize the network, distribute the data set to each client, and update the model.

[0049] Step 1.1: The datasets used are CIFAR-10, CIFAR-100, and ImageNet_LT.

[0050] The CIFAR-10 dataset contains 60,000 color images of 32*32 size, divided into 10 categories: airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck. There are 6,000 images in each category, 50,000 of which are used for training, forming 5 training batches, each with 10,000 images; the other 10,000 are used for testing, forming a separate batch. In the test batch, 1,000 images are randomly selected from each of the 10 categories. The remaining images are randomly arranged to form a training batch. Note that the number of images of each category in a training batch is not necessarily the same. In general, each category has 5,000 images. In addition, 100 images are randomly selected from each category in CIFAR-10 to form an additional balanced dataset. Use CIFAR-100 as unlabeled data for knowledge distillation.

[0051] CIFAR-100 has 100 classes, each containing 600 images. Each class has 500 training images and 100 test images. The 100 classes in CIFAR-100 are divided into 20 superclasses. Each image comes with a "fine" label (the class it belongs to) and a "coarse" label (the superclass it belongs to). In CIFAR-100, 10 images per class are randomly selected to form an additional balanced dataset. Use downsampled ImageNet (image size 32*32) as unlabeled data for knowledge distillation.

[0052] ImageNet-LT is a large image classification dataset, a long-tail version of ImageNet, which is obtained by sampling a subset of Pareto distribution according to α=6. It contains 115,800 images in 1000 categories, with the largest and smallest categories containing 1280 and 5 images respectively. We obtain the dataset from the balanced validation data Oversampled CIFAR100 (image size 224*224) is used as unlabeled data for knowledge distillation.

[0053] The above three data sets are distributed to different clients according to the heterogeneity degree η = 0.1 in the Dirichlet distribution as their local data. The data distribution diagram on CIFAR-10 is as follows: Figure 2 shown.

[0054] Step 1.2: Build a federated learning environment and initialize the network.

[0055] ResNet-8 networks are used for training on CIFAR-10-LT and CIFAR-100-LT, and ResNet-50 networks are used for training on ImageNet-LT. All our experiments are run by PyTorch on two NVIDIA GeForce RTX 3080 GPUs. In general, we design 20 clients for a total of 200 rounds of training, and select 40% of the clients to participate in federated training in each round. For client training, the batch size is set to 128, the learning rate is 0.1, and SGD is used as the optimizer. For global model training on the server side, we set the calibration epoch to 100, the distillation epoch to 100, and use the Adam optimizer with a learning rate of 0.001 for knowledge distillation.

[0056] Step 1.3: Client model update.

[0057] The server initializes the global model parameters w, randomly selects the client set S participating in this round of training, and broadcasts the model parameters to the client set S participating in this round of training. Each client in S uses the received global model parameters w and local data to perform stochastic gradient descent (SGD) to update their models. Let the updated model parameters of client k be w k After the update, each client sends its updated model parameters back to the server.

[0058] Step 2: The server obtains the teacher model and the student model. The specific process includes the following sub-steps:

[0059] Step 2.1: First, the server performs average weighting on the received model parameters to obtain the student model. The calculation formula is as follows:

[0060]

[0061] φ s (x) = φ w (x) (Formula 2)

[0062] Step 2.2, the server then performs weighted aggregation on the model parameters obtained from the client to obtain the teacher model. The calculation formula is as follows:

[0063]

[0064] Among them, e k is the weight assigned to client k.

[0065] Step 3: The server calibrates the teacher model obtained in step D.

[0066] Since the global data presents a long-tail distribution, the models obtained by each client will also be biased towards the head class and perform poorly on the tail class. Therefore, the teacher model obtained by weighting the models of each client is also biased towards the head class and ignores the tail class. Since the teacher model is biased towards the head class, the knowledge it contains is extremely biased, which will cause the student model taught to be biased, which will seriously damage the model performance. Therefore, we need to calibrate the teacher model to obtain an unbiased teacher model, and then teach the unbiased knowledge to the student model. The specific method includes the following sub-steps:

[0067] Step 3.1. Since local models are trained on local data with different distributions, each local model may perform differently on the tail classes, so we want to assign higher weights to local models that perform better on the tail classes. However, the server does not know which classes are tail classes and which local models perform well on them, so instead of giving each client a fixed weight, we propose a client-based weight assignment strategy to calculate the weight e for each client's local model. k , and e k Normalize it so that its sum is equal to 1, which is the final weight. The calculation formula is as follows:

[0068]

[0069] Among them, a e ∈R C and b e is a learnable parameter. Client-side calibration works just like the self-attention mechanism, where the weights are calculated for the local model based on the raw logits and then multiplied back to the raw logits.

[0070] Step 3.2: If no local model can handle the tail class well, the teacher model obtained by weighted ensemble may still be biased towards the head class. To solve this problem, we propose a class-based logits calibration strategy to further improve the performance of the model in the tail class. Let the output logits of the calibrated model be z cl , the calculation formula is as follows:

[0071] z cl =a z ⊙φ t (x)+b z (Formula 5)

[0072] Among them, a z and b z is a network parameter that can be learned, and ⊙ represents the Hadamard product.

[0073] Step 3.3: The above logits calibration strategy is effective only if the local model extracts good enough representation information for the input data. If the local model's feature extraction of the data is seriously affected by the long-tail distribution, then simply calibrating the logits is not enough. Therefore, we need to update the feature extractor to further improve the model performance. We use additional balanced labeled data on the server side. Fine-tune on the global model w to obtain the fine-tuned model because is balanced, so fine-tune the model An unbiased feature extractor can be obtained. Then, we can obtain the fine-tuned logits for the input x as

[0074] Step 3.4: Through the above steps, we can see that z cl and z ft It is to calibrate the teacher model from two different levels. cl The output logits level of the teacher model is calibrated, and the feature extractor of the model is fixed, while z ft It is the result of fine-tuning the feature extractor to improve the model’s feature extraction capability. To fully combine the advantages of both, we propose a calibration gating network to cl and z ft The gating network takes the integrated features as input and outputs weights through a nonlinear layer, so that each sample obtains different weights according to its own features. The weight calculation formula is as follows:

[0075] σ=sigmoid(u T v) (Formula 6)

[0076] in, is the integrated feature, u∈R d is a network parameter that can be learned. Therefore, the final calibrated logits of the calibrated gating network are z′, calculated as follows:

[0077] z′=σz cl +(1-σ)z ft (Formula 7)

[0078] where σ∈(0,1) is used to weigh the two logits.

[0079] Step 3.5: During the whole process of integrated calibration, all learnable parameters are calculated by The cross entropy loss on is updated, and the loss function is as follows:

[0080]

[0081] Step 4: Use knowledge distillation to transfer the unbiased knowledge of the teacher model to the student model.

[0082] In order to better teach the unbiased knowledge of the teacher model (i.e., the calibrated ensemble model) to the student model (i.e., the global model), we train the student model by combining labeled data training and unlabeled data distillation. The loss function consists of two parts: (1) L CE is the cross entropy loss between the logits of the student model and the ground-truth; (2) L KL is the Kullback-Leibler (KL) divergence between the logits of the teacher model and the student model. We use To calculate L CE , and use another unlabeled dataset To calculate L KL To further improve the knowledge distillation performance. The final loss function is weighed by the hyperparameter λ∈[0,1]:

[0083] L′=(1-λ)L CE +λL KL (Formula 9)

[0084] Table 1 shows the accuracy (%) comparison results of the present invention and several other federated learning methods on the CIFAR-10-LT and CIFAR-100-LT datasets with imbalance ratios of 100, 50 and 10. The bold results in the table are the optimal results of each indicator.

[0085] It can be seen from the results in Table 1 that the method of the present invention can solve the joint problem of long-tail distribution and heterogeneous data in federated learning, and the method of the present invention achieves the highest test accuracy at all imbalance levels.

[0086] Table 1

[0087]

[0088] Table 2 shows the accuracy (%) comparison results of the present invention and several federated learning methods on the ImageNet-LT dataset. The bold results in the table are the optimal results of each indicator.

[0089] Table 2 compares the accuracy of several methods on three categories: head class (sample number exceeds 100), middle class (sample number between 20 and 100) and tail class (sample number less than 20). Compared with other methods, the method of the present invention achieved the best results. At the same time, the accuracy of the method of the present invention on the tail class reached 15.91%. The method of the present invention solves the joint problem of long-tail distribution and heterogeneous data in federated learning, and greatly improves the performance of the model on the tail class while improving the overall performance of the model.

[0090] Table 2

[0091]

[0092]

[0093] In Table 1 and Table 2:

[0094] FedAvg corresponds to the method proposed by McMahan, B et al. (McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; and y Arcas, BA 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, 1273–1282.);

[0095] FedAvgM corresponds to the method proposed by Hsu, T.-MH et al. (Hsu, T.-MH; Qi, H.; and Brown, M. 2019. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335.);

[0096] FedProx corresponds to the method proposed by Li, T. et al. (Li, T.; Sahu, AK; Zaheer, M.; Sanjabi, M.; Talwalkar, A.; and Smith, V. 2020b. Federated optimization in heterogeneous networks. In Machine Learning and Systems, 429–450.). By adding a proximal term correction term to the client-updated loss function, it improves the model accuracy while also improving the stability of convergence.

[0097] FedNova corresponds to the method proposed by Wang, J. et al. (Wang, J.; Liu, Q.; Liang, H.; Joshi, G.; and Poor, HV2020b. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems, 7611–7623.);

[0098] FedDF corresponds to the method proposed by Lin, T. et al. (Lin, T.; Kong, L.; Stich, SU; and Jaggi, M. 2020. Ensemble distillation for robust model fusion in federated learning. In Advances in Neural Information Processing Systems, 2351–2363.), which applies ensemble distillation to the model fusion of federated learning, and can perform model fusion through unlabeled data or data generated by the generator;

[0099] FedBE corresponds to the method proposed by Chen, H.-Y. et al. (Chen, H.-Y.; and Chao, W.-L. 2021. FedBE: Making bayesian model ensemble applicable to federated learning. In International Conference on Learning Representations.), which achieves powerful aggregation from the perspective of Bayesian reasoning by sampling high-quality global models and combining them through a Bayesian model;

[0100] Fed-Focal Loss corresponds to the method proposed by Sarkar, D et al. (Sarkar, D.; Narang, A.; and Rai, S. 2020. Fed-focal loss for imbalanced data classification in federated learning. arXiv preprint arXiv: 2011.06283.);

[0101] Ratio Loss corresponds to the method proposed by Wang, L et al. (Wang, L.; Xu, S.; Wang, X.; and Zhu, Q. 2021a. Addressing class imbalance in federated learning. In AAAI Conference on Artificial Intelligence, 10165–10173.), which realizes the opaque monitoring of data imbalance in federated learning and proposes a new loss function Ratio Loss to reduce the impact of imbalance;

[0102] cRT, τ-norm, and LWS correspond to the method proposed by Kang, B et al. (Kang, B.; Xie, S.; Rohrbach, M.; Yan, Z.; Gordo, A.; Feng, J.; and Kalantidis, Y. 2020. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations.), which shows that data imbalance does not affect the high-quality representation of learning input data. The authors show that only adjusting the classifier can also achieve powerful long-tail recognition capabilities.

[0103] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning method for long-tail heterogeneous data, characterized by The steps include: S1, the server randomly initializes the global model w and sends the model parameters to the client. The client uses the received model parameters to update the local model and uploads the updated local model parameters to the server; S2, the server aggregates the local model parameters to obtain the teacher model and the student model; including the following steps: S21, the server side averages and weights the local model parameters to obtain the student model. The calculation formula is as follows: φ s (x) = φ w (x) (Formula 2) Where |D k | represents the amount of data owned by the kth client, |D| represents the total amount of data owned by all clients, K represents the number of clients, x represents the input image data, φ w (·) represents the network of the federated average model, φ s (·) represents the network of the student model, and the local model parameter updated by client k is w k ; S22, the server performs weighted aggregation on the local model parameters to obtain the teacher model. The calculation formula is as follows: Among them, φ t (·) represents the network of the teacher model, e k represents the weight of client k, The network of the kth client; S3, the server calibrates the teacher model so that the teacher model can learn on unbiased knowledge; a client-based weight distribution strategy is proposed to calculate the weight e of each client local model. k , and finally e k Normalize it so that its sum is equal to 1, which is the final weight, weight e k The calculation formula is as follows: Among them, a e ∈R c and b e represents the network parameters that can be learned, R c represents a c-dimensional vector, T is the transpose symbol, the weights of the local model are calculated based on the original output of the model, and then the weights are multiplied back to the original output; A class-based raw output calibration strategy is proposed, and the calibrated model output is z cl , the calculation formula is as follows: z cl =a z ⊙φ t (x)+b z (Formula 5) Where a z and b z represents the network parameters that can be learned, ⊙ represents the Hadamard product; S4, through knowledge distillation, transfers the unbiased knowledge of the teacher model to the student model, and then sends the student model to the client to start the next round of federated training.

2. A federated learning method for long-tail heterogeneous data according to claim 1, characterized in that In step S1, the server initializes the global model parameter w, randomly selects the client set S participating in this round of training, and broadcasts the model parameter to the clients in the client set S participating in this round of training. Using the received global model parameter w and local data, stochastic gradient descent is performed to update the local model. The local model parameter updated by client k is w k ,After the update, the client sends the updated model parameters back to the server.

3. A federated learning method for long-tail heterogeneous data according to claim 1, characterized in that In step S3, an additional balanced labeled dataset is used on the server side. Fine-tune on the global model w to obtain the fine-tuned model The output of the fine-tuned model for input data x is where z ft represents the output of the fine-tuned model for x, Represents the network of the fine-tuned model.

4. A federated learning method for long-tail heterogeneous data according to claim 3, characterized in that The fine-tuned model Among them, η represents the learning rate, represents the loss function, Indicates derivative.

5. A federated learning method for long-tail heterogeneous data according to claim 3, characterized in that In step S3, the gating network is calibrated to z cl and z ft To make a trade-off, the calibrated gating network takes the integrated features as input and outputs the weights through the nonlinear layer. The weight calculation formula is as follows: σ = sigmoid (u T v) (formula 6) in, represents the integrated feature, represents the feature extractor of the kth client, u∈R d represents the network parameters that can be learned, R d Represents a d-dimensional vector, and the final calibration model output of the calibration gating network is z′, which is calculated as follows: z′ = σz cl +(1 - σ)z ft (Equation 7) where σ∈(0,1) is used to weigh z cl and z ft Two model outputs.

6. A federated learning method for long-tail heterogeneous data according to claim 5, characterized in that During the whole process of integrated calibration, the parameters that can be learned are The cross entropy loss on is updated, and the loss function is as follows: Among them, C represents the number of categories, y j represents the true label of the input data, j represents the value of the jth dimension in y, exp(·) represents the exponential function, and z j ′ represents the value of the jth dimension in the final calibration z′, z i ′ represents the value of the i-th dimension in the final calibration z′.

7. A federated learning method for long-tail heterogeneous data according to claim 1, characterized in that In step S4, the unbiased knowledge of the teacher model is transferred to the student model through knowledge distillation. Specifically, the student model is trained by combining labeled data training and unlabeled data distillation, and its loss function is as follows: L′ = (1 - λ)L CE + λL KL (Formula 9) Among them, L CE represents the cross entropy loss between the model output of the student model and the true label, L KL Represents the relative entropy divergence of the model output between the teacher model and the student model, by balancing the labeled dataset Calculate L CE , and use the unlabeled dataset Calculate L KL , λ∈[0,1] represents the hyperparameter, for L CE and L KL Make trade-offs.

Images