Federal learning information sharing method based on two-stage characteristic distillation

Abstract

The invention aims to provide a federated learning information sharing method based on two-stage characteristic distillation to solve the problems of data isomerism and model precision of federated learning. According to the method, through two-stage feature distillation, features with sensitive performance are finally segmented for global sharing, a classifier is trained to predict a target, compared with an original federated learning method, the problem of heterogeneous data of federated learning can be relieved, meanwhile, due to the fact that all clients only give non-full-amount features to the server, the efficiency is greatly improved, and the cost is reduced. In addition, due to the fact that the characteristic of the performance robustness is stored locally, the number of shared information is reduced, and the convergence speed can be increased. In addition, information sensitive features are shared, so that the convergence trend of the model can be reflected to the greatest extent, and the precision of the model can be ensured.

Description

Technical Field

[0001] The present invention relates to the field of information sharing, and in particular to alleviating data heterogeneity and ensuring model accuracy in federated learning information sharing, and specifically is a federated learning information sharing method based on two-stage feature distillation. Background Art

[0002] In the current era of big data, the phenomenon of "data islands" is very common. Mobile phones and tablets have gradually become the main computing devices for many people. These devices are equipped with powerful sensors such as cameras, microphones, and GPS, and due to their portability, they can obtain a large amount of unique data, most of which are private. Models trained based on this data have higher availability than models of a single client. However, due to the sensitivity of the data, storing it centrally may bring risks and liability issues.

[0003] Federated learning is a distributed training method in which clients can train global models in collaboration with a central server by sharing gradients. At the same time, the trained data is retained on the local client and not shared, thereby alleviating data privacy issues. The pioneering work of FedAvg introduced model aggregation of local models, which addresses the diversity of computation and communication and provides a strategy for distributed training. However, a notorious problem in federated learning is the use of non-independent and identically distributed data for training. Due to different user behaviors, the local data of different clients may have large heterogeneity, which can lead to unstable training and slow convergence, and result in unsatisfactory or even harmful model performance. Therefore, how to deal with this problem and propose a more effective solution to the problem is of great value and significance in today's era when the amount of data is exploding, but all parties have privacy concerns about data sharing, and data sharing is urgently needed.

[0004] There has been a lot of work exploring solutions for federated learning of non-IID data. They can be roughly divided into four categories: 1) Client drift mitigation, which modifies the local objectives of the client to make the local model consistent with the global model to a certain extent; 2) Aggregation schemes, which improve the model fusion mechanism on the server side; 3) Data sharing, which introduces public datasets or synthetic data to help build a more balanced data distribution on the client or server; 4) Personalized federated learning, which aims to train personalized models for individual clients instead of shared global models.

[0005] In the above explorations, sharing client information is considered a direct and promising way to alleviate data heterogeneity. However, when we share customer information, one of the most important issues is to protect data privacy and improve model performance, which hinders the effectiveness of information sharing strategies. Summary of the invention

[0006] The purpose of this invention is to propose a federated learning information sharing method based on two-stage feature distillation to solve the data heterogeneity and model accuracy problems of federated learning, which can effectively improve the convergence speed and accuracy of federated learning data sharing.

[0007] The object of the present invention is achieved in this way:

[0008] First, the first stage of feature distillation is to divide the data into two parts through the IB (information bottleneck) strategy, one part is performance-sensitive data, and the other part is performance-robust data. Then, in the second stage of feature distillation, we regard the contribution of FL features to the performance of the model as a sequential cooperation game, and calculate the contribution of a certain feature among the performance-sensitive features to the model performance according to the Shapley algorithm. Finally, we set a threshold according to the contribution of the feature to the model to further eliminate the features with lower contribution.

[0009] The specific method is as follows:

[0010] A federated learning information sharing method based on two-stage feature distillation includes the following steps:

[0011] Step 1: Training through federated learning method, first we need to initialize the global server and each client. For the server, we initialize the global model φ 0 , and the number of rounds of current global training T r Then the server sets the global parameter φ 0 Sent to all clients. For each client, the input includes the current Epoch, the client’s local dataset D k , and the learning rate η k .

[0012] Step 2: Consider the relationship between the federated learning client and server. The client returns performance-sensitive features to the server through a training method based on two-stage feature distillation. The server integrates global parameters and predicts the results through a classifier.

[0013] Further, according to the federated learning information sharing method based on two-stage feature distillation described in claim 1, in step 2, for the first step of distillation, according to the information bottleneck method, Z represents the information extracted from the input X and its label Y, I(·) refers to the mutual information, and the goal of the first step of feature distillation is:

[0014]

[0015] Further, according to the method for sharing information of federated learning based on two-stage feature distillation described in claim 1, the step 2 attempts to construct a loss function to solve this optimization problem. Where θ is used to generate sensitive features z(·;θ), (x, y) obeys the distribution P(X) of the kth client. k ,Y k ). p(y|·) is the probability that Y = y. Therefore, the optimized expression is as follows:

[0016]

[0017] Furthermore, according to the federated learning information sharing method based on two-stage feature distillation described in claim 1, in step 2, since the above formula cannot be used well to divide features, and it cannot be guaranteed that the decomposed performance-robust features and the original features are retained in form or characteristics, a competition mechanism is introduced to explicitly model two types of features. First, the performance-robust features q(x; θ) are explicitly modeled to ensure that these features are part of the original feature information. Then the performance-sensitive features z(x; θ) are implicitly modeled by the formula z(x; θ) = xq(x; θ). In other words, the performance-sensitive features are obtained by subtracting the performance-robust features from the original features. The principle formula after our conversion is as follows:

[0018]

[0019] Further, according to the method for sharing information in federated learning based on two-stage feature distillation as described in claim 1, in step 2, for the calculated performance-sensitive feature z(x;θ)=xq(x;θ), we can introduce a local classifier f(·; w k ), the parameter of this classifier is w k , its task is to predict the label y based on the input performance-sensitive features z(x; θ). In other words, the classifier uses the generated performance-sensitive features to complete the prediction task, while the generative model extracts important features related to the prediction. The cross-entropy loss function l(·) is used to measure the difference between the predicted label and the true label. Therefore, the local classifier predicts the label y based on the performance-sensitive features z(x; θ) and calculates the corresponding loss l((f(z(x; θ); w k ),y). The calculation formula of its complete optimization objective is:

[0020]

[0021] Further, according to the method for sharing information in federated learning based on two-stage feature distillation as described in claim 1, in step 2, the parameter θ of the generative model is optimized so that the classifier f(·; w) according to the performance sensitive feature z(x; θ) = xq(x; θ) is k ) can minimize the cross entropy loss (i.e., the more accurate the prediction, the better), while ensuring the norm of performance-sensitive features Do not exceed the preset threshold ρ to control the size of the feature.

[0022] Further, according to a federated learning information sharing method based on two-stage feature distillation as described in claim 1, in step 2, after the first stage feature distillation, the second stage feature distillation is started. The feature distillation in the second stage mainly performs further purification operations on the performance-sensitive features screened out in the first stage. Since IB is only divided based on the information bottleneck theory, it ignores the independent contribution of features and may have the risk of information loss. Therefore, we consider the interaction effect between features and use the Shapley algorithm to perform further feature distillation, which can provide more accurate feature evaluation and selection, and help overcome the insufficient feature selection that may occur when the IB method is used alone, thereby improving the performance and robustness of the federated learning model. This combined method makes the feature selection process more comprehensive and more adaptable, and can effectively cope with the challenges brought by data heterogeneity. The Shapley value originates from game theory, which calculates the marginal contribution of a feature by considering all possible feature combinations. Its advantage is that it can fairly and accurately reflect the role of each feature in different combinations, and is therefore suitable for feature selection problems in federated learning. The role in the present invention is mainly to evaluate the contribution of each feature. Its principle formula is as follows:

[0023]

[0024] Among them Ф i (v) is the Shapley value of feature i, S is a subset of feature F, excluding feature i. v(S) is the accuracy of the model consisting of feature set S. v(S∪{i})-v(S) is the marginal gain of adding feature i to feature set S. is the weight of each subset, ensuring that all possible feature permutations are considered fairly.

[0025] Further, according to the method for sharing information of federated learning based on two-stage feature distillation described in claim 1, the Shapley algorithm in step 2 can help determine which features are more important for the final prediction y and which are less important. The advantage is that since performance-sensitive features are to be shared, for federated learning, after local secondary feature distillation, the overhead of network communication can be minimized and the training efficiency can be improved.

[0026] Further, according to the method for sharing information in federated learning based on two-stage feature distillation described in claim 1, the specific method in step 2 is: for each client, a basic classifier is constructed to calculate the Shapley value of the performance-sensitive feature. However, since the calculation of the Shapley value requires evaluating the marginal contribution of a certain feature in all possible feature combinations. If there are N features, then the number of all possible subsets of the features is 2 N , which means that the computational cost grows exponentially with the number of features. Even for a moderate number of features, directly evaluating all feature combinations will incur huge computational overhead. Therefore, Monte Carlo sampling is used to approximate the Shapley value by randomly selecting a subset. This step selects performance-sensitive features that need to be shared for global sharing.

[0027] Furthermore, according to the federated learning information sharing method based on two-stage feature distillation described in claim 1, in step 2, in order to protect privacy, Gaussian noise, Laplace noise, etc. can be added to the globally shared performance-sensitive features.

[0028] Furthermore, according to the federated learning information sharing method based on two-stage feature distillation described in claim 1, in step 2, after obtaining the result returned by the client, the server aggregates the parameters obtained according to the strategy (FedAvg, etc.) to obtain an updated classifier, and prediction can be performed based on the final classifier.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention uses two-stage feature distillation to finally segment performance-sensitive features for global sharing, and trains a classifier to predict the target. Compared with the original federated learning method, the present invention can alleviate the heterogeneous data problem of federated learning. At the same time, since each client only uses a partial amount of features to the server, the cost of network communication can be reduced. In addition, since the robust features are saved locally, the amount of information sharing is reduced, so the present invention can accelerate the convergence speed. In addition, since the information-sensitive features are shared, the convergence trend of the model can be reflected to the greatest extent, and the accuracy of the model can also be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flowchart of a federated learning information sharing method based on two-stage feature distillation.

[0032] Figure 2 This is the flowchart of the optimized Shapley algorithm during the second stage of feature distillation. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. However, it should not be understood that the scope of the above subject matter of the present invention is limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0034] Example 1

[0035] like Figure 1 As shown, a method for sharing information in federated learning based on two-stage feature distillation. First, each client needs to divide a part of the data as global shared data, and each client sends the shared data to the server. Subsequently, the server initializes the global parameters and distributes the parameters to each client. The client performs federated learning based on two-stage feature distillation through local data, returns the performance-sensitive features segmented by training to the server for global sharing to reduce data heterogeneity and improve the training efficiency of federated learning, and sends the locally trained parameters to the server. The server aggregates the global model parameters, and finally uses the aggregated global model for label prediction. Specifically, it includes the following steps:

[0036] Step 1: Training through federated learning method, first you need to initialize the global server and each client. For the server, initialize the global model φ 0 , and the number of rounds of current global training T r Then the server sets the global parameter φ 0 Sent to all clients. For each client, the input includes the current Epoch, the client’s local dataset D k , and the learning rate η k .

[0037] Step 2: Consider the relationship between the federated learning client and server. The client returns performance-sensitive features to the server through a training method based on two-stage feature distillation. The server integrates global parameters and predicts the results through a classifier.

[0038] Example 2

[0039] This embodiment is a specific implementation of the federated learning information sharing method based on two-stage feature distillation described in Embodiment 1, and includes the following steps:

[0040] S1: First, you need to initialize the global server and each client. For the server, initialize the global model φ 0 , and the number of rounds of current global training T r Then the server sets the global parameter φ 0Sent to all clients. For each client, the input includes the current Epoch, the client’s local dataset D k , and the learning rate η k , the client needs to share part of the data set with the server.

[0041] S2: Considering the relationship between the client and server of federated learning, the client returns performance-sensitive features to the server through a training method based on two-stage feature distillation. The server integrates global parameters and predicts the results through the classifier. The specific steps are as follows:

[0042] S21: After starting training, each client first performs the first stage feature distillation according to IB, divides the data features into performance-sensitive features and performance-robust features, and builds a classifier that can predict labels.

[0043] Furthermore, the first-stage feature distillation method and the construction principle of the classifier described in this embodiment are:

[0044] Let Z represent the information extracted from the input X and its label Y, and I(·) refers to the mutual information. The goal of the first step of feature distillation is:

[0045]

[0046] Next, we try to construct a loss function to solve this optimization problem. Here, θ is used to generate sensitive features z(·;θ), (x, y) follows the distribution P(X k ,Y k ). p(y|·) is the probability that Y = y. Then the optimization objective expression is as follows:

[0047]

[0048] Since the above formula cannot be used to divide features well, and cannot guarantee that the decomposed performance-robust features are retained in form or characteristics with the original features, a competition mechanism is introduced to explicitly model the two types of features. First, the performance-robust features q(x; θ) are explicitly modeled to ensure that these features are part of the original feature information. Then the performance-sensitive features z(x; θ) are implicitly modeled by the formula z(x; θ) = xq(x; θ). In other words, the performance-sensitive features are obtained by subtracting the performance-robust features from the original features. The principle formula after our conversion is as follows:

[0049]

[0050] S22: Then, we can introduce a local classifier f(·; w k), the parameter of this classifier is w k , its task is to predict the label y based on the input performance-sensitive features z(x; θ). In other words, the classifier uses the generated performance-sensitive features to complete the prediction task, while the generative model extracts important features related to the prediction. The cross-entropy loss function l(·) is used to measure the difference between the predicted label and the true label. Therefore, the local classifier predicts the label y based on the performance-sensitive features z(x; θ) and calculates the corresponding loss l((f(z(x; θ); w k ),y). Our complete optimization objective is calculated as:

[0051]

[0052] S23: We first divided the first step of performance-sensitive features based on IB. Next, because IB is only based on the information bottleneck theory for division, it ignores the independent contribution of features and may have the risk of information loss. Therefore, we consider the interaction effect between features and use the Shapley algorithm to perform further feature distillation. This can provide more accurate feature evaluation and selection, which helps to overcome the insufficient feature selection that may occur when using the IB method alone, thereby improving the performance and robustness of the federated learning model. This combined method makes the feature selection process more comprehensive and more adaptable, and can effectively cope with the challenges brought by data heterogeneity. The process of the second stage feature distillation using the optimized Shapley algorithm is as follows: Figure 2 shown.

[0053] Furthermore, for an optimized version of the Shapley algorithm described in S23, the reason for using it is that the Shapley value needs to evaluate the marginal contribution of a feature in all possible feature combinations. If there are N features, then the number of all possible subsets of the features is 2 N , which means that the computational cost grows exponentially with the number of features. Even for a moderate number of features, directly evaluating all feature combinations will incur huge computational overhead. Therefore, Monte Carlo sampling is used to approximate the Shapley value by randomly selecting a subset. Through this step, performance-sensitive features that need to be shared are screened out, and, in order to protect privacy, we add Laplace or Gaussian noise to performance-sensitive features, and finally generate features returned to the server for global sharing.

[0054] S24: After the client training, the client returns the model parameters and performance-sensitive features of this round of training to the server. The server aggregates according to the corresponding federated learning algorithm (such as FedAvg, FedNova, etc.). After all clients complete a round of training, this round of training ends. After global convergence, the final global model is obtained.

Claims

1. A federated learning information sharing method based on two-stage feature distillation, characterized by: The method comprises the following steps: Step 1: Training through federated learning method, first we need to initialize the global server and each client. For the server, we initialize the global model φ 0 , and the number of rounds of current global training T r Then the server sets the global parameter φ 0 Sent to all clients; for each client, the input includes the current Epoch, the client’s local dataset D k , and the learning rate η k ; Step 2: Consider the relationship between the federated learning client and server. The client returns performance-sensitive features to the server through a training method based on two-stage feature distillation. The server integrates global parameters and predicts the results through a classifier.

2. A federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: In step 2, for the first step of distillation, according to the information bottleneck method, Z is used to represent the information extracted from the input X and its label Y, and I(·) refers to the mutual information. Then the goal of the first step of feature distillation is: Then, we try to construct a loss function to solve this optimization problem; where θ is used to generate sensitive features z(·;θ), (x, y) follows the distribution P(X k ,Y k ); p(y|·) is the probability that Y=y; therefore, the optimized expression is as follows: Since the above formula cannot be used to divide features well, and it cannot be guaranteed that the decomposed performance-robust features are retained in form or characteristics with the original features, a competition mechanism is introduced to explicitly model two types of features; first, the performance-robust features q(x; θ) are explicitly modeled to ensure that these features are the part that retains the original feature information; then the performance-sensitive features z(x; θ) are implicitly modeled by the formula z(x; θ) = xq(x; θ); in other words, the performance-sensitive features are obtained by subtracting the performance-robust features from the original features; the principle formula after our conversion is as follows:

3. A federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: For the performance-sensitive feature z(x;θ)=xq(x;θ) calculated in step 2, we can introduce a local classifier f(·;w k ), the parameter of this classifier is w k , its task is to predict the label y based on the input performance-sensitive features z(x; θ); in other words, the classifier uses the generated performance-sensitive features to complete the prediction task, while the generative model extracts important features related to the prediction; the cross-entropy loss function l(·) is used to measure the difference between the predicted label and the true label; therefore, the local classifier predicts the label y based on the performance-sensitive features z(x; θ) and calculates the corresponding loss l((f(z(x; θ); w k ),y); the calculation formula of its complete optimization objective is:

4. A federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: In step 2, the parameter θ of the generative model is optimized so that the classifier f(·; w k ) can minimize the cross entropy loss (i.e., the more accurate the prediction, the better), while ensuring the norm of performance-sensitive features Do not exceed the preset threshold ρ to control the size of the feature.

5. A federated learning information sharing method based on two-stage feature distillation according to claim 1, characterized in that: In the step 2, after the first stage of feature distillation, the second stage of feature distillation begins; the second stage of feature distillation mainly performs further purification operations on the performance-sensitive features screened out in the first stage; since IB is only based on the information bottleneck theory for division, it ignores the independent contribution of features and may have the risk of information loss, so we consider the interaction effect between features and use the Shapley algorithm to perform further feature distillation, which can provide more accurate feature evaluation and selection, help overcome the insufficient feature selection that may occur when using the IB method alone, thereby improving the performance and robustness of the federated learning model; this combined method makes the feature selection process more comprehensive and more adaptable, and can effectively cope with the challenges brought by data heterogeneity; the Shapley value originates from game theory, which calculates the marginal contribution of a feature by considering all possible feature combinations; Its advantage is that it can fairly and accurately reflect the role of each feature in different combinations, so it is suitable for the feature selection problem in federated learning; its role in the present invention is mainly to evaluate the contribution of each feature; The principle formula is as follows: Among them Ф i (v) is the Shapley value of feature i, S is a subset of feature F, excluding feature i; v(S) is the accuracy of the model composed of feature set S; v(S∪{i})-v(S) is the marginal gain brought by adding feature i to feature set S; is the weight of each subset, ensuring that all possible feature permutations are considered fairly.

6. According to the method for sharing information of federated learning based on two-stage feature distillation as described in claim 1, the Shapley algorithm in step 2 can help determine which features are more important for the final prediction y and which are less important; the advantage is that, since performance-sensitive features are to be shared; for federated learning, after local secondary feature distillation, the overhead of network communication can be minimized and training efficiency can be improved.

7. A method for sharing information in federated learning based on two-stage feature distillation according to claim 1, characterized in that: The specific method in step 2 is as follows: for each client, a basic classifier is constructed to calculate the Shapley value of the performance-sensitive feature; however, since the calculation of the Shapley value requires evaluating the marginal contribution of a certain feature in all possible feature combinations; if there are N features, then the number of all possible subsets of the features is 2 N , which means that the computational cost grows exponentially with the number of features; even for a moderate number of features, directly evaluating all feature combinations will incur huge computational overhead; so Monte Carlo sampling is used to approximate the Shapley value by randomly selecting a subset; this step screens out performance-sensitive features that need to be shared for global sharing.

8. A method for sharing information in federated learning based on two-stage feature distillation according to claim 1, characterized in that: The approximate Shapley algorithm used in step 2 comprises the following steps: 1) setting the sampling round; 2) randomly generating a feature subset; 3) Evaluate model performance; 4) Calculate marginal contribution; 5) Accumulate and average; In this way, we can approximate the Shapley value of each feature without traversing all possible feature combinations; Then we can filter out performance-sensitive features that need to be shared by the final second-stage feature distillation according to the specified threshold.

9. A method for sharing information in federated learning based on two-stage feature distillation according to claim 1, characterized in that: In the step 2, in order to protect privacy, Gaussian noise, Laplace noise, etc. may be added to the globally shared performance-sensitive features.

10. A method for sharing information in federated learning based on two-stage feature distillation according to claim 1, characterized in that: In step 2, after obtaining the result returned by the client, the server aggregates the obtained parameters according to the strategy (FedAvg, etc.) to obtain an updated classifier, and prediction can be performed based on the final classifier.

Images