Federal long-tail calculation optimization method of distribution scene based on adaptive gradient balance

Abstract

In the real world, data privacy and long-tail distribution are common problems. The invention concerns a federated long-tail learning task in which each client holds a local heterogeneous data set, which together presents a long-tail distribution if these data sets are summarized over the global range. In this case, the existing federal optimization and centralized long tail learning method may not be applicable because the characteristics of the overall long tail distribution need to be described under the privacy constraint and the local learning strategy needs to be adjusted to cope with the long tail imbalance. In order to deal with this problem, the present invention designs a self-adjusting gradient balancer module that re-weights the gradient of the client in a closed loop manner based on global long-tail priori feedback derived by a direct priori analyzer module. Thus, the client can effectively relieve distribution drift caused by data isomerism in the model training process, and the global model with better performance on minority classes is obtained while the performance of the majority classes is kept. The method is applied to the fields where data are often distributed in a long-tail mode, such as the medical field and the economic field.

Description

Technical Field

[0001] The present invention belongs to the field of federated deep learning, and specifically relates to a federated long-tail computing optimization method for distributed scenarios based on adaptive gradient balance. Background Art

[0002] Long-tail data, a common phenomenon in data sets, reveals the imbalance of data distribution. Specifically, a few categories occupy the vast majority of samples in the data set, forming the so-called "head" data, while in contrast, there are "tail" data with a large number of samples but a small number of samples in each category. This data form is reflected in many fields, including recommendation systems, image recognition, natural language processing, etc. The existence of long-tail data poses a considerable challenge to the training of machine learning models. Because traditional machine learning models are often more likely to overfit the head categories with a large number of samples during training, resulting in a significant reduction in the prediction performance on the tail categories. When we delve into the characteristics of long-tail data, we will find that there are dual challenges of diversity and sparsity behind it. Diversity is reflected in the tail categories that are small in number but diverse in variety. These categories are difficult to be fully learned by the model due to insufficient number of samples. Sparsity is reflected in the proportion of tail data in the entire data set. This sparsity makes it difficult for the model to capture the characteristics of tail data during training, further exacerbating the performance degradation of the model on tail categories.

[0003] Federated learning, as an emerging distributed machine learning method, has attracted widespread attention for its unique training method. It allows global models to be trained by sharing model parameters while keeping data localized. This method not only effectively protects the privacy and security of users, but also enables collaborative training between multiple decentralized data sources, making full use of the advantages of distributed data. Especially when dealing with distributed data scenarios across devices or institutions, federated learning has shown its unique charm and potential. However, although federated learning has achieved remarkable results in distributed data processing, its model performance is often difficult to achieve the expected results when facing long-tail data.

[0004] At present, although some federated learning methods for imbalanced data and distillation-based federated learning methods have been proposed, their performance is still unsatisfactory. Recent studies have suggested using client selection strategies to match clients with complementarity and using ratio loss to estimate the global imbalance. This method aims to enhance local optimization, but it may also cause clients that are not matched with other clients to miss the opportunity to participate in server model aggregation. In addition, as the degree of data incompleteness increases, the performance of ratio loss will gradually decrease. Summary of the invention

[0005] In view of the shortcomings of the existing technical methods, the purpose of the present invention is to provide a federated long-tail computing optimization method for distributed scenarios based on adaptive gradient balancing, which reweights the gradient of the client in a closed-loop manner. It can effectively alleviate the distribution drift caused by data heterogeneity during model training, while maintaining the performance of the majority class, obtain a global model with better performance on the minority class, and improve the robustness and generalization ability of the global model. The technical solution of the present invention is as follows:

[0006] A method for optimizing federated long-tail computing in a distributed scenario based on adaptive gradient balance includes the following steps:

[0007] 1) First, the server obtains the prior vector P of the overall data class distribution through privacy calculation r . The server then sends the collected prior vectors to each client;

[0008] 2) Consider the difference Δ between the accumulated positive and negative gradients j (t), where This captures the overall imbalance during the training process;

[0009] 3) Use closed-loop self-adjustment to adjust the tail class’s Δ j (t) is always aligned to 0. Specifically, we implement a proportional-integral-derivative controller to continuously update the gradient reweighting coefficients with respect to the logarithm in a class-wise manner, and the reweighting process is adaptively self-adjusted based on the error feedback until Δ j (t) achieving predetermined balance objectives;

[0010] 4) Set the error feedback to e j (t) = Δ j (t)-z j (t), represents the current state Δ during training j (t) and target z j (t) The distance between

[0011] 5) Given e j (t), set the adaptive gradient balance output With K P The proportional term indicates that the controller will give a larger re-weighting coefficient β1 to compensate for the imbalance of the current large error. I The integral term looks back at past errors and reduces steady-state excursions that the proportional term cannot handle.

[0013] In addition, since only the proportional and integral loops are used to calculate β j It is difficult to achieve stable and smooth reweighting, so we introduce a DA third differential term to control for fluctuations based on future expectations;

[0014] 6) Combines the principles of momentum optimization and adaptive learning rate. By calculating the first and second moment estimates of the gradient, the learning rate is dynamically adjusted and the model parameters are updated;

[0015] 7) Use a simple activation function to calculate and And calculate the second moment estimate To control and adjust the learning rate to update the model parameters;

[0017] 8) The client and server repeat training steps 1) to 7) in each round until the model reaches a fitting state.

[0018] Compared with the prior art, the present invention has the following technical features and effects:

[0019] (1) Privacy computing technology plays a vital role in protecting data security, but many existing technologies often only focus on data security and ignore how to maximize the value and utilization of data while ensuring data privacy. This method introduces a privacy protection mechanism to ensure that under the federated learning framework, not only can data privacy be effectively protected, but also the prior knowledge contained in the data can be fully utilized. This method that combines privacy protection and data utilization can not only meet the high requirements of federated learning for data security, but also significantly improve the performance of the model.

[0020] (2) In order to effectively solve the performance degradation problem caused by uneven data distribution during model training, this method adopts a closed-loop weighting mechanism to reweight the client's gradient. This mechanism can significantly alleviate the distribution drift problem caused by data heterogeneity, so that the global model can significantly improve the performance on minority data while maintaining the performance of majority class data. In this way, the global model can better adapt to different data distributions, thereby showing higher accuracy and robustness in practical applications.

[0021] (3) In each round of model training, this method ensures that the positive and negative gradients of the tail data are always zero. This design effectively improves the performance of the model on the tail data. Through this sophisticated gradient control mechanism, the model can better capture and learn the characteristics of the tail data during training, thereby improving the overall performance. This performance improvement is not only reflected in the accuracy of the model, but also in the generalization ability of the model and its adaptability to various data distributions, so that the model can still maintain efficient and stable performance when facing diverse data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1This is the data distribution diagram of all clients in the present invention

[0023] Figure 2 The workflow diagram of the client adaptive gradient balance of the present invention

[0024] Figure 3 This is a curve diagram of the training results of different long-tail degrees of the present invention

[0025] Figure 4 This is a curve diagram of the training results of different isomerism levels of the present invention. DETAILED DESCRIPTION

[0026] The present invention is described below in conjunction with the accompanying drawings and embodiments.

[0027] In this paper, we choose ResNet-18 as our network training model, generated on 20 clients using the CIFAR-10-LT dataset. The problem clients with missing categories mainly appear in the tail categories, which is consistent with the real scene. In order to better simulate the real-world scene, we conduct 200 rounds of training, 50% of which are active clients in each round. In local training, we use a batch size of 128, a learning rate of 0.12, and SGD as the optimizer. For server training, we set the calibration step size (I) to 100 and use Adam as the optimizer.

[0028] The specific parameter selection of the model involved in the present invention is shown in Table 1 as an example.

[0029] Table 1 Model parameters

[0030]

[0031]

[0032] When conducting experiments, the present invention specifically sets the distribution of client data, such as Figure 1As shown. It can be clearly observed from the figure that the head class data has more samples, while the tail class data presents the characteristic of fewer samples. This imbalance in data distribution makes the client face greater challenges during the training process. In addition, the client does not have samples of all data types, which means that the data set of each client is a subset of the overall data set, and there is a certain difference between the long-tail distribution of these subsets and the long-tail distribution of the overall data. More specifically, the distribution of client data is not exactly the same as that of the overall data, which makes it impossible for each client to accurately infer the distribution characteristics of the overall data based on its own data alone. The complexity and heterogeneity of this data distribution not only increase the difficulty of model training, but also require the model to fully consider and adapt to this diverse data distribution during the training process. Therefore, when conducting experiments, the present invention fully considers the particularity of this data distribution and proposes a series of effective solutions to improve the performance and robustness of the model when facing different data distributions.

[0033] The federated learning model structure adopted by the present invention is composed of 20 clients and a central server. This distributed learning architecture can collaboratively train models on multiple independent data sources without the need to aggregate the original data of each client to the central server, thereby improving the overall performance of the model while protecting data privacy. In each round of training, the client will first adjust the training model based on prior knowledge to achieve adaptive gradient balance. This adaptive adjustment can effectively solve the gradient imbalance problem caused by uneven data distribution, so that the model can achieve better training results on different data sets.

[0034] After completing the adjustment of the local model, the client will update the local model c Upload to the central server. Subsequently, the server will receive and aggregate all local models from the clients. In this invention, in order to optimize the overall performance and convergence speed of the model, we use the FedAvg algorithm. The FedAvg algorithm generates a global model by weighted averaging the parameters of each client model, thereby achieving effective integration of all client data.

[0035] Finally, the server aggregates the global model ω j After being transmitted to each client, the client will start the next round of training process after receiving the updated global model. This cyclic iterative approach not only ensures the continuous optimization of the model in multiple rounds of training, but also enables the global model to gradually adapt and improve its adaptability to the data distribution of each client. Through this efficient federated learning architecture and careful aggregation strategy, the present invention can significantly improve the performance and robustness of the model in a diverse data environment while protecting data privacy.

[0036] The present invention starts with the client model training process, and uses adaptive gradient balancing to ensure that the client local model has gradient balance on the tail class, thereby improving the performance of the local model on the tail class. Figure 2 As shown in the figure, it is divided into the training phase and the gradient balancing phase. The client's gradient is reweighted in a closed-loop manner. It can effectively alleviate the distribution drift caused by data heterogeneity during model training, and obtain a global model with better performance on the minority class while maintaining the performance of the majority class.

[0037] The model first uses privacy computing to introduce appropriate noise and randomize the data distribution knowledge in each calculation step in a circular manner. The client randomly perturbs its own data and sends the sum of the perturbed data of the previous client and its own perturbed data to the next client, while also uploading the random perturbation of this client to the server. When the data reaches the last client, it is uploaded to the server, which eliminates all perturbations and obtains the global class distribution.

[0038] During the model training process, the collector collects the gradient of the model and then considers the difference Δ between the accumulated positive gradient and the negative gradient j (t), where This captures the overall imbalance of the entire training process. Then, the tail class Δ j (t) is always aligned to 0. Specifically, we implement a proportional-integral-derivative controller to continuously update the gradient reweighting coefficients with respect to the logarithm in a class-wise manner, and the reweighting process is adaptively self-adjusted based on the error feedback until Δ j (t) Achieve predetermined balance objectives.

[0039] By setting the error feedback to e j (t) = Δ j (t)-z j (t), represents the current state Δ during training j (t) and target z j (t) to conduct system adaptive workflow, given e j (t), set the adaptive gradient balance output:

[0040]

[0041] With K P The proportional term indicates that the controller will give a larger re-weighting coefficient β1 to compensate for the imbalance of the current large error. IThe integral term can look back at past errors and reduce the steady-state offset that the proportional term cannot handle. In addition, since only the proportional and integral loops are used to calculate β j It is difficult to achieve stable and smooth reweighting, so we introduce a D The third differential term is used to control for fluctuations based on future expectations.

[0042] It combines the principles of momentum optimization and adaptive learning rate. By calculating the first and second moment estimates of the gradient, the learning rate is dynamically adjusted and the model parameters are updated. A simple activation function is used to calculate and And calculate the second moment estimate To control and adjust the learning rate to update the model parameters.

[0043] In order to further demonstrate the technical effect of the present invention, the training results of the present invention and the reference [Y.Zhou,Q.Ye,and J.Lv,“Communication-efficient federated learning with compensated overlap-fedavg,”IEEE Transactions on Parallel and Distributed Systems,vol.33,no.1,pp.192–205,2022.

[0044] T.-MHHsu, H.Qi, and M.Brown, “Measuring the effects of nonidenticaldata distribution for federated visual classification,” arXiv preprint arXiv:1909.06335.

[0045] T.Li, AKSahu, M.Zaheer, M.Sanjabi, A.Talwalkar, and V.Smith, "Federated optimization in heterogeneous networks," Proceedings of Machine learning and systems, vol.2, pp.429–450, 2020.

[0046] J.Wang,Q.Liu,H.Liang,G.Joshi,and H.V.Poor,“Tackling the objectiveinconsistency problemin heterogeneous federated optimization,”Advances inneural information processing systems,vol.33,pp.7611–7623,2020.

[0047] T.Lin,L.Kong,S.U.Stich,and M.Jaggi,“Ensembledistillation for robustmodel fusion in federated learning,”Advances in neural information processingsystems,vol.33,pp.2351–2363,2020.

[0048] H.-Y.Chen and W.-L.Chao,“Fedbe:Making bayesian model ensembleapplicable to federated learning,”in ICLR,2021,pp.1–19.

[0049] D.Sarkar,A.Narang,and S.Rai,“Fed-focal loss for imbalanced dataclassification in federated learning,”2020.

[0050] L.Wang,S.Xu,X.Wang,and Q.Zhu,“Addressing class imbalance in federatedlearning,”in Proceedings of the AAAI Conference on Artificial Intelligence,vol.35,no.11,2021,pp.10 165–10 173.

[0051] Several traditional federated learning methods proposed in B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, “Decoupling representation and classifier for long-tailed recognition,” in ICLR, 2020, pp. 1–16] are compared, and the comparison results are shown in Table 2.

[0052] Table 2 Comparison of experimental results

[0053]

[0054]

[0055] The compared federated learning methods in Table 2 are mainly divided into traditional federated learning methods: FedAvg, FedAvgM, FedProx and FedNova; distillation-based federated learning methods: FedDF and FedBE; imbalance-oriented federated learning methods: FedFocal Loss, Ratio Loss, FedAvg+CRT, FedAvg+τ-norm and FEDIC. It can be seen from the results that the model trained by the method proposed in the present invention is superior to these traditional federated learning methods in terms of performance regardless of the degree of long tail. And it can still maintain a high performance when the degree of long tail is relatively large, which shows that it has strong generalization ability, especially when the class distribution has a significant long tail. This performance is attributed to the processing of data non-iid and global class distribution imbalance, while traditional methods mainly focus on non-iid, resulting in suboptimal robustness.

Claims

1. A method for optimizing federated long-tail computing in distributed scenarios based on adaptive gradient balancing. This method starts from the client and reweights the client's gradient in a closed-loop manner. It can effectively alleviate the distribution drift caused by data heterogeneity during model training, and obtain a global model with better performance on the minority class while maintaining the performance of the majority class. It includes the following steps: 1) First, the server obtains the prior vector P of the overall data class distribution through privacy calculation r . The server then sends the collected prior vectors to each client; 2) Consider the difference Δ between the accumulated positive and negative gradients j (t), where This captures the overall imbalance during the training process; 3) Use closed-loop self-adjustment to adjust the tail class’s Δ j (t) is always aligned to 0. Specifically, we implement a proportional-integral-derivative controller to continuously update the gradient reweighting coefficients with respect to the logarithm in a class-wise manner, and the reweighting process is adaptively self-adjusted based on the error feedback until Δ j (t) achieving predetermined balance objectives; 4) Set the error feedback to e j (t) = Δ j (t)-z j (t), represents the current state Δ during training j (t) and target z j (t) The distance between 5) Given e j (t), set the adaptive gradient balance output With K P The proportional term indicates that the controller will give a larger re-weighting coefficient β1 to compensate for the imbalance of the current large error. I The integral term can look back at past errors and reduce the steady-state offset that the proportional term cannot handle. In addition, since only the proportional and integral loops are used to calculate β j It is difficult to achieve stable and smooth reweighting, so we introduce a D A third differential term to control for fluctuations based on future expectations; 6) Combines the principles of momentum optimization and adaptive learning rate. By calculating the first and second moment estimates of the gradient, the learning rate is dynamically adjusted and the model parameters are updated; 7) Use a simple activation function to calculate and And calculate the second moment estimate To control and adjust the learning rate to update the model parameters; 8) The client and server repeat training steps 1) to 7) in each round until the model reaches a fitting state.

Images