A federated learning information sharing method based on two-stage feature distillation

Abstract

The purpose of the present application is to propose a two-stage feature distillation-based federated learning information sharing method to solve the data heterogeneity and model accuracy problems of federated learning. Through two-stage feature distillation, the present application finally divides out the performance-sensitive features for global sharing, and trains a classifier to predict the target. Compared with the original federated learning method, the present application can alleviate the heterogeneous data problem of federated learning, at the same time, since each client only needs to give the server non-full features, the cost of network communication can be reduced, in addition, since the performance robust features are saved locally, the amount of information sharing is reduced, so the present application can accelerate the convergence speed. In addition, since the information-sensitive features are shared, the model convergence trend can be reflected to the greatest extent, and the accuracy of the model can be guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information sharing, in particular to alleviate data heterogeneity and guarantee model accuracy in federated learning information sharing, and specifically to a federated learning information sharing method based on two-stage feature distillation. BACKGROUND

[0002] In the current era of big data, the phenomenon of "data silos" is very common. Mobile phones and tablets have gradually become the main computing devices for many people. These devices are equipped with powerful sensors such as cameras, microphones, and GPS, and because of their portability, they can obtain a large amount of unique data, and most of this data is private. Models trained based on this data have higher availability than single-client models. However, due to the sensitivity of the data, its centralized storage can pose risks and liability issues.

[0003] Federated learning is a distributed training method in which clients can train a global model by sharing gradients under the collaboration of a central server. The data used for training is retained locally on the client and is not shared, thereby alleviating data privacy issues. FedAvg, a pioneering work, introduces model aggregation of local models, which addresses the diversity of computation and communication and provides a strategy for distributed training. However, a notorious problem in federated learning is the use of non-identically distributed data for training. Due to different user behaviors, the local data of different clients can have significant heterogeneity, which can lead to unstable training and slow convergence, and result in suboptimal or even harmful model performance. Therefore, how to address this problem and propose more effective solutions to this problem is of great value and significance in today's era of explosive data growth, but with privacy concerns about data sharing.

[0004] There have been a large number of works exploring solutions to federated learning with non-identically distributed data. They can be roughly divided into four categories: 1) Client drift mitigation, which modifies the local objective of the client to make the local model consistent with the global model to some extent; 2) Aggregation scheme, which improves the model fusion mechanism on the server side; 3) Data sharing, which introduces a public dataset or synthetic data to help build a more balanced data distribution on the client or server; 4) Personalized federated learning, which aims to train personalized models for individual clients rather than a shared global model.

[0005] Among the above explorations, sharing client information is considered a direct and promising approach to alleviating data heterogeneity. However, the most important problem when sharing client information is protecting data privacy and improving model performance, which hinders the effectiveness of information sharing strategies. SUMMARY

[0006] The purpose of the present application is to propose a two-stage feature distillation-based federated learning information sharing method to solve the data heterogeneity and model accuracy problems of federated learning, which can effectively improve the convergence speed and accuracy of federated learning data sharing.

[0007] The purpose of the present application is achieved as follows:

[0008] Firstly, the first stage feature distillation divides the data into two parts by the IB (information bottleneck) strategy, one part is performance sensitive data, and the other part is performance robust data. Then, the second stage feature distillation is performed, and we regard the performance contribution of FL features to the model as a sequential cooperative game, and calculate the performance contribution of a certain feature in the performance sensitive feature to the model performance according to the Shapley algorithm. Finally, we set a threshold according to the contribution of the feature to the model to further remove the features with low contribution.

[0009] The specific way is as follows:

[0010] A two-stage feature distillation-based federated learning information sharing method comprises the following steps:

[0011] Step 1, training by the method of federated learning, first, the global server and each client need to be initialized, for the server, we initialize the global model φ 0 , and the current global training round T r . Then the server sends the global parameters φ 0 to all clients. For each client, the obtained input has the current epoch, the local data set D k of the client, and the learning rate η k .

[0012] Step 2, considering the relationship between the federated learning client and the server, the client returns the performance sensitive feature to the server by the training method based on two-stage feature distillation, and the server integrates the global parameters, and the result can be predicted by the classifier.

[0013] Further, according to the two-stage feature distillation-based federated learning information sharing method of claim 1, for the first distillation in step 2, according to the information bottleneck method, Z represents the information extracted from the input X and its label Y, I(·) indicates mutual information, and the target of the first feature distillation is:

[0014]

[0015] Further, according to the federated learning information sharing method based on two-stage feature distillation in claim 1, the step 2 attempts to construct a loss function to solve this optimization problem. Wherein θ is to generate sensitive features z(·; θ), (x, y) obeys the distribution P(X k ,Y k ) of the kth client. p(y|·) is the likelihood of Y=y. So, the expression of optimization is as follows:

[0016]

[0017] Further, according to the federated learning information sharing method based on two-stage feature distillation in claim 1, since the above formula cannot be well used to divide features, it cannot guarantee that the performance robust features decomposed are preserved in form or characteristics with the original features. Therefore, a competitive mechanism is introduced to explicitly model two types of features. First, the performance robust features q(x; θ) are explicitly modeled to ensure that these features are part of the original feature information. Then the performance sensitive features z(x; θ) are implicitly modeled by the formula z(x; θ)=x-q(x; θ). In other words, the performance sensitive features are obtained by subtracting the performance robust features from the original features. The principle formula after conversion is as follows:

[0018]

[0019] Further, according to the federated learning information sharing method based on two-stage feature distillation in claim 1, for the calculated performance sensitive features z(x; θ)=x-q(x; θ), a local classifier f(·; w k ) can be introduced, and the parameters of the classifier are w k , and its task is to predict the label y according to the input performance sensitive features z(x; θ). In other words, the classifier uses the generated performance sensitive features to complete the prediction task, and the generation model extracts important features related to prediction. The cross-entropy loss function l(·) is used to measure the difference between the predicted label and the true label. Therefore, the local classifier will predict the label y according to the performance sensitive features z(x; θ) and calculate the corresponding loss l((f(z(x; θ); w k ), y). The complete optimization target calculation formula is as follows:

[0020]

[0021] Further, according to the two-stage feature distillation-based federated learning information sharing method in claim 1, the parameters θ of the generated model are optimized in step 2, so that the classifier f (·; w k ) based on the performance-sensitive feature z(x; θ) = x-q(x; θ) can minimize the cross-entropy loss (i.e. the more accurate the prediction is, the better), while ensuring that the norm of the performance-sensitive feature does not exceed the preset threshold ρ to control the size of the feature.

[0022] Further, according to the two-stage feature distillation-based federated learning information sharing method in claim 1, after the first-stage feature distillation in step 2, the second-stage feature distillation is started. The second-stage feature distillation mainly performs further purification operation on the performance-sensitive features screened out in the first stage. Since IB is only based on the information bottleneck theory for division, it ignores the independent contribution of features, which may have the risk of information loss, so we consider the interaction effect between features, and use Shapley algorithm to perform further feature distillation, which can provide more accurate feature evaluation and selection, and help to overcome the possible feature selection deficiency when using IB method alone, thereby improving the performance and robustness of the federated learning model. This combined method makes the feature selection process more comprehensive and adaptable, and can effectively cope with the challenges brought by data heterogeneity. Shapley value is derived from game theory, which calculates the marginal contribution of a feature by considering all possible feature combinations. Its advantage is that it can fairly and accurately reflect the role of each feature in different combinations, so it is suitable for feature selection problems in federated learning. In this invention, its main role is to evaluate the contribution of each feature. Its principle is as follows:

[0023]

[0024] where Ф i (v) is the Shapley value of feature i, S is a subset of features F that does not contain feature i. v(S) is the accuracy of the model composed of feature set S. v(S∪{i})-v(S) is the marginal gain brought by feature i after joining the feature set S. is the weight of each subset, which ensures that all possible feature arrangement combinations are considered fairly.

[0025] Further, according to the two-stage feature distillation-based federated learning information sharing method in claim 1, the Shapley algorithm can help determine which features are more important and which are secondary for the final prediction y in step 2. The advantage is that since the performance-sensitive features are to be shared, the network communication overhead can be minimized after the local secondary feature distillation, improving the training efficiency.

[0026] Further, according to the two-stage feature distillation-based federated learning information sharing method in claim 1, the specific implementation in step 2 is: for each client, the Shapley value of the performance-sensitive feature is calculated by constructing a basic classifier. However, since the calculation of the Shapley value needs to evaluate the marginal contribution of a certain feature in all possible feature combinations. If there are N features, the number of all possible subsets of the features is 2 N , which means that the calculation cost increases exponentially with the number of features. Even for a moderate number of features, directly evaluating all feature combinations will bring huge computational overhead. Therefore, the Shapley value is approximated by randomly selecting a part of the subsets through Monte Carlo sampling. Through this step, the performance-sensitive features that need to be shared are screened out and used for global sharing.

[0027] Further, according to the two-stage feature distillation-based federated learning information sharing method in claim 1, in order to protect privacy, Gaussian noise, Laplace noise, etc. can be added to the globally shared performance-sensitive features.

[0028] Further, according to the two-stage feature distillation-based federated learning information sharing method in claim 1, after obtaining the results returned by the clients, the server aggregates the obtained parameters according to the strategy (FedAvg, etc.) to obtain an updated classifier, and prediction can be performed according to the final classifier.

[0029] Compared with the prior art, the beneficial effects of the present application are:

[0030] By two-stage feature distillation, the present application finally divides out the performance-sensitive features for global sharing, and trains a classifier to predict the target. Compared with the original federated learning method, the present application can alleviate the heterogeneous data problem of federated learning. At the same time, since each client only needs to give the server non-full-amount features, the cost of network communication can be reduced. In addition, since the performance-robust features are saved locally, the amount of information sharing is reduced, so the present application can accelerate the convergence speed. In addition, since the performance-sensitive features are shared, the model convergence trend can be reflected to the greatest extent, and the accuracy of the model can be guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 It is a two-stage feature distillation-based federated learning information sharing method flowchart.

[0032] Figure 2 It is a Shapley algorithm flowchart optimized in the second-stage feature distillation. DETAILED DESCRIPTION

[0033] The application will be described in further detail below with reference to the drawings and specific embodiments. However, it should be understood that the above-mentioned subject matter of the application is not limited to the following embodiments, and any technology implemented based on the content of the application falls within the scope of the application.

[0034] Embodiment 1

[0035] As Figure 1 shown, a federated learning information sharing method based on two-stage feature distillation The application first requires each client to divide a part of the data as global shared data, and each client sends the shared data to the Server. Then, the Server initializes the global parameters and distributes the parameters to each client. The client performs federated learning based on two-stage feature distillation through local data, returns the performance-sensitive features divided by training to the Server for global sharing to alleviate data heterogeneity, improves the training efficiency of federated learning, and gives the Server the parameters of local training, and the Server aggregates to obtain the global model parameters, and finally uses the global model obtained by aggregation for label prediction. Specifically, it includes the following steps:

[0036] Step 1, training by the method of federated learning, first need to initialize the global server and each client, for Server, initialize the global model φ 0 , and the current global training round T r Then the Server sends the global parameters φ 0 to all clients. For each client, the obtained input has the current Epoch, the client's local data set D k , and the learning rate η k .

[0037] Step 2, considering the relationship between the federated learning client and the Server, the client returns the performance-sensitive features to the Server through the training method based on two-stage feature distillation, and the Server integrates the global parameters, and the classifier can predict the result.

[0038] Embodiment 2

[0039] This embodiment is a specific implementation of the federated learning information sharing method based on two-stage feature distillation described in Embodiment 1, which includes the following steps:

[0040] S1: First, the global server and each client need to be initialized, for Server, the global model φ 0 is initialized, and the current global training round T r Then the Server sends the global parameters φ 0is sent to all clients. For each client, the resulting input has the current Epoch, the client's local dataset D k , and the learning rate η k . The client needs to share part of the dataset to the server.

[0041] S2: Considering the relationship between the federated learning client and the server, the client returns the performance-sensitive feature to the server through the training method based on two-stage feature distillation, and the server integrates the global parameters to predict the results through the classifier. It includes the following specific steps:

[0042] S21: After starting the training, each client first performs the first-stage feature distillation according to IB, divides the data features into performance-sensitive features and performance-robust features, and constructs a classifier that can predict the label.

[0043] Further, the first-stage feature distillation method described in this embodiment and the principle of constructing the classifier are as follows:

[0044] Let Z represent the information extracted from the input X and its label Y, and I(·) represent mutual information. The first step of feature distillation aims to:

[0045]

[0046] Then try to construct a loss function to solve this optimization problem. Where θ is to generate sensitive features z(·; θ), (x, y) obeys the distribution P(X k , Y k ) of the kth client. p(y|·) is the probability of Y=y. Then the optimization objective expression is as follows:

[0047]

[0048] Since the above formula cannot be well used to divide the features, it cannot guarantee that the performance-robust features decomposed from the original features are preserved in form or characteristics, so a competitive mechanism is introduced to explicitly model the two types of features. First, the performance-robust feature q(x; θ) is explicitly modeled to ensure that these features are part of the original feature information. Then the performance-sensitive feature z(x; θ) is implicitly modeled by the formula z(x; θ) = x-q(x; θ). In other words, the performance-sensitive feature is obtained by subtracting the performance-robust feature from the original feature. Our transformed principle formula is as follows:

[0049]

[0050] S22: Then, a local classifier f(·; w k), the parameter of this classifier is w k , which aims to predict the label y given the input performance-sensitive feature z(x; 0). In other words, the classifier utilizes the generated performance-sensitive feature to accomplish the prediction task, while the generative model extracts the important features related to the prediction. The cross-entropy loss function l(·) is used to measure the difference between the predicted label and the true label. Therefore, the local classifier will predict the label y according to the performance-sensitive feature z(x; 0) and calculate the corresponding loss l((f(z(x; 0); w k ). The complete optimization objective is calculated as follows:

[0051]

[0052] S23: We first divide the first step performance-sensitive features according to IB. Then, since IB only divides based on the information bottleneck theory, it ignores the independent contribution of features, which may have the risk of information loss. Therefore, we consider the interaction effect between features and use the Shapley algorithm for further feature distillation, which can provide more accurate feature evaluation and selection, helping to overcome the possible feature selection deficiency when using IB alone, thereby improving the performance and robustness of the federated learning model. This combined method makes the feature selection process more comprehensive and adaptable, effectively addressing the challenges brought by data heterogeneity. The process of using the optimized Shapley algorithm for the second stage of feature distillation is shown in Figure 2 .

[0053] Further, the reason for using the optimized Shapley algorithm in S23 is that the Shapley value needs to evaluate the marginal contribution of a feature in all possible feature combinations. If there are N features, the number of all possible subsets of features is 2 N , which means that the computational cost increases exponentially with the number of features. Even for a moderate number of features, directly evaluating all feature combinations will bring huge computational overhead. Therefore, by Monte Carlo sampling, we approximate the Shapley value by randomly selecting a subset of features. Through this step, we filter out the performance-sensitive features that need to be shared, and to protect privacy, we add Laplace or Gaussian noise to the performance-sensitive features, and finally generate the features returned to the server for global sharing.

[0054] S24: After client training, the client returns the model parameters and performance-sensitive features of this round of training to the server. The server aggregates according to the corresponding federated learning algorithm (such as FedAvg, FedNova, etc.). After all clients complete one round of training, this round of training is completed. After global convergence, the final global model is obtained.

Claims

1. A federated learning information sharing method based on two-stage feature distillation, characterized in that: The method comprises the following steps: Step 1, training by the method of federated learning, first need to initialize the global server and each client, for Server, we initialize the global model φ 0 , and the current global training round T r ; then Server sends the global parameters φ 0 to all clients; for each client, the input obtained has the current Epoch, the client's local data set D k , and the learning rate η k ; Step 2, considering the relationship between the federal learning client and the server, the client returns the performance-sensitive features to the server through the training method based on two-stage feature distillation, the server integrates the global parameters, and the result can be predicted through the classifier; In step 2, for the first step of distillation, according to the information bottleneck method, Z represents the information extracted from the input X and its label Y, I(·) indicates mutual information, and the target of the first step of feature distillation is: Then, construct the loss function to solve this optimization problem; where θ is to generate sensitive feature z(·; θ), (x, y) is subject to the distribution P(X k ,Y k ) of the kth client; p(y|·) is the likelihood of Y = y; therefore, the expression of optimization is as follows: Since the above formula cannot be well used for dividing features and cannot guarantee that the performance robust features decomposed are reserved in form or characteristics with the original features, a competitive mechanism is introduced to model two types of features explicitly. First, the performance robust feature q(x; θ) is modeled explicitly to ensure that these features are part of the original feature information; then the performance-sensitive feature z(x; θ) is implicitly modeled by the formula z(x; θ) = x-q(x; θ); In other words, the performance-sensitive feature is obtained by subtracting the performance-robust feature from the original feature; the principle of our conversion is as follows: For the computed performance-sensitive feature z(x; θ) = x - q(x; θ), we can introduce a local classifier f(·; w k ) whose parameters are w k , and its task is to predict the label y from the input performance-sensitive feature z(x; θ); in other words, the classifier uses the generated performance-sensitive feature to complete the prediction task, while the generation model extracts important features related to prediction; the cross-entropy loss function l(·) is used to measure the difference between the predicted label and the true label; therefore, the local classifier will predict the label y from the performance-sensitive feature z(x; θ) and calculate the corresponding loss l((f(z(x; θ); w k ), y); the complete optimization objective is calculated as follows: After the first-stage feature distillation, the second-stage feature distillation begins; the second-stage feature distillation mainly performs further purification operation on the performance-sensitive features screened out in the first stage; since IB only divides based on information bottleneck theory, it ignores the independent contribution of features, which may have the risk of information loss, so we consider the interaction effect between features and use Shapley algorithm to evaluate the contribution of each feature. The principle is as follows: where Ф i (v) is the Shapley value of feature i, S is a subset of features F that does not include feature i; v(S) is the accuracy of the model consisting of the feature set S; v(S∪{i})-v(S) is the marginal gain brought by feature i after it is added to the feature set S; is the weight of each subset, ensuring that all possible feature permutations are considered fairly; In step 2, in order to protect privacy, Gaussian noise or Laplace noise can be added to the globally shared performance-sensitive features.

2. The federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: The parameters θ of the generative model in step 2 are optimized so that the classifier f(·; w k ) according to the performance-sensitive feature z(x; θ) = x - q(x; θ) can minimize the cross-entropy loss, while it is necessary to ensure that the norm of the performance-sensitive feature does not exceed the preset threshold ρ to control the size of the feature.

3. The federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: The specific method in step 2 is: for each client, calculate the Shapley value of the performance-sensitive feature by constructing a basic classifier; approximate Shapley value is calculated by randomly selecting a subset through Monte Carlo sampling; through this step, the performance-sensitive features that need to be shared are screened out and used for global sharing.

4. The federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: The approximate Shapley algorithm used in step 2 has the following steps: 1) set the sampling round; 2) randomly generate a feature subset; 3) evaluate model performance; 4) calculate marginal contribution; 5) accumulate and average; in this way, we can approximate the Shapley value of each feature without traversing all possible feature combinations; then we can screen out the performance-sensitive features that need to be shared for the final second-stage feature distillation according to the specified threshold.

5. The federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: After obtaining the results returned by the client in step 2, the server aggregates the parameters obtained according to the strategy to obtain an updated classifier, and the final classifier can be used for prediction.

Images