A Heterogeneous Federated Learning Method and System Based on Personalized Information Separation

Abstract

The present invention discloses a heterogeneous federated learning method and system based on personalized information separation. The method includes the following steps: The client uses a feature extractor to extract the feature information of the original data set, separates the personalized information in the feature information, and sends it to the server; The server uses the personalized information to train a generator and broadcasts the trained generator to the client; The client uses the received generator to generate enhanced samples, which together with the original data set form an extended data set; The server and the client use the federated learning method to train the model, where the client uses the extended data set to train the local model, and the server updates the global model by aggregating the local models of each client. The present invention trains a generator by separating the personalized information, and then extends the data set through the generator, solving the data heterogeneity problem and improving the communication efficiency at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular, to a heterogeneous federated learning method and system based on personalized information separation. Background Art

[0002] Federated Learning (FL) is a distributed model training framework that allows various clients to collaboratively train a global model in an edge computing environment without disclosing their local data. However, FL usually faces data heterogeneity, that is, distribution shift between clients. Sharing client information has shown great potential in alleviating data heterogeneity, but it encounters a dilemma in protecting privacy and improving model performance. To address the heterogeneous challenge, the pioneering work FedAvg (Federated Averaging) solved the diversity of computation and communication by introducing model aggregation of local training models, but still faced the problem of client drift caused by data heterogeneity. To alleviate this dilemma, a method of federated feature distillation was proposed. It discovers information from the data and divides the data into performance-sensitive features (performance-sensitive features containing very little information can make the model generalize well) and performance-robust features (containing almost all the information in the data). In this way, almost all the information in the data can be saved locally, and only the performance-sensitive features need to be shared. Therefore, if the data features can be perfectly divided into these two types of features, the performance-privacy dilemma can be well solved. This method finds some natural internal connections in the information bottleneck theory. Ideally, the information bottleneck aims to eliminate redundant information in the data and retain important information that helps to make correct predictions. Then, the redundant information discarded by the information bottleneck becomes the information that does not need to be shared. Therefore, inspired by the information bottleneck, redundant information can be "discarded" as much as possible locally, and only a very small amount of information is sent to solve the data heterogeneity problem. This method believes that data is the source of information, but it does not consider that not all clients will have enough data. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems in the prior art and propose a method for training a generator by separating personalized information and then expanding the dataset through the generator to solve the data heterogeneity problem.

[0004] The technical solution adopted by the present invention to solve its technical problems is: to provide a heterogeneous federated learning method based on personalized information separation, including the following steps:

[0005] The client uses a feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information, and sends it to the server;

[0006] The server trains a generator using personalized information and broadcasts the trained generator to the clients;

[0007] The client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset;

[0008] The server and the clients train a model using federated learning. Among them, the clients use the extended dataset to train local models, and the server updates the global model by aggregating the local models of each client.

[0009] Preferably, the client uses a feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information. Different clients select their respective feature types according to their application scenarios and data types.

[0010] Preferably, the client uses a feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information, including the following steps:

[0011] After preprocessing the input data, extract the feature information through the VAE model;

[0012] Based on a partitioning strategy, divide the feature information into global information and personalized information.

[0013] Preferably, the partitioning strategy for dividing the feature information into global information and personalized information includes based on Information Bottleneck (IB), and the IB objective function is expressed as:

[0014] ;

[0015] Among them, is the likelihood term, indicating maximizing , is the KL divergence constraint of the distribution, indicating minimizing ; is a Lagrange multiplier; and both represent variational distributions; represents the conditional probability distribution of given under the condition of

[0016] Preferably, before sending the personalized information to the server, the client encrypts the personalized information using differential privacy, specifically including the following steps:

[0017] Add noise to the personalized information, expressed as:

[0018] ;

[0019] Among them, represents the encrypted personalized information, represents the personalized information, represents the noise added using differential privacy, represents the sensitivity, represents the privacy budget;

[0020] Track and limit the total amount of added noise, expressed as:

[0021] ;

[0022] Among them, and respectively represent two adjacent data sets, and there is only one sample difference between the two data sets; represents a random algorithm, represents a random algorithm a possible output of; represents when the input data set is when, the random algorithm output probability; represents when the input data set is when, the random algorithm output probability.

[0023] Preferably, the server uses the personalized information to train a generator, specifically: the server uses the personalized information as training data to train the generator, enabling the generator to learn and simulate data samples with similar personalized characteristics; the training process of the generator uses the GAN model.

[0024] Preferably, the client uses the received generator to generate enhanced samples, including the following steps:

[0025] Use the trained generator model to generate new data samples, and at the same time sample the target labels to generate corresponding feature representations;

[0026] Integrate the generated feature representations with the original data set to form an extended data set;

[0027] Normalize the extended data set.

[0028] Preferably, the server and the client use the method of federated learning to train the model, including the following steps:

[0029] Initialization step, the server initializes a global model and its parameters and sends them to the client;

[0030] Local update step: After each client receives the global model and its parameters, it uses the extended dataset for training, extracts the parameters of the trained model, and uploads them to the server.

[0031] Global update step: The server collects the model parameters from all clients, aggregates all the model parameters as the new global model parameters; updates the global model with the new global model parameters and distributes them to the clients.

[0032] Iterative step: Repeat the local update step and the global update step until the global model reaches the preset convergence condition or reaches the preset number of training rounds to obtain the trained global model.

[0033] Preferably, for aggregating all the model parameters, the FedAvg algorithm is used to calculate the average value of all local model parameters in a weighted average manner as the new global model parameters.

[0034] The present invention also provides a heterogeneous federated learning system based on personalized information separation for implementing the heterogeneous federated learning method based on personalized information separation described in any one of the above, including:

[0035] Information separation module: The client uses the feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information, and sends it to the server.

[0036] Generator training module: The server uses the personalized information to train the generator and broadcasts the trained generator to the clients.

[0037] Data expansion module: The client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset.

[0038] Global model training module: The server and the clients use the federated learning method to train the model, where the clients use the extended dataset to train the local models, and the server updates the global model by aggregating the local models of each client.

[0039] The present invention has the following beneficial effects:

[0040] (1) The present invention extracts information from user data to train the generator without any external data; by separating the personalized information and training the generator, corresponding data samples can be generated for different types of heterogeneous data, and this method helps to alleviate the challenges brought by data heterogeneity because the generated data can simulate various heterogeneous scenarios in the real world.

[0041] (2) Traditional federated learning requires participating devices to communicate frequently during the training process to share model updates. However, in the present invention, by using a generator to expand the dataset, the amount of training data can be increased without adding an extra communication burden, which helps to reduce the number of communications and the amount of communication data, thereby improving communication efficiency.

[0042] (3) The generator trained in the present invention can generate high-quality and diverse data samples. These samples not only have statistical characteristics similar to the original data but also contain rich personalized information.

[0043] (4) In the past, to solve the heterogeneity problem, the integrated knowledge from local models was used to enrich the global model. However, this cannot completely solve the inherent heterogeneity between user models. In the present invention, the generator is trained with the integrated information of personalized information from local models, which can be used to alleviate the heterogeneity problem of local client models.

[0044] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Description of the Drawings

[0045] Figure 1 It is a method step diagram of an embodiment of the present invention;

[0046] Figure 2 It is a process schematic diagram of an embodiment of the present invention;

[0047] Figure 3 It is a schematic diagram of personalized information extraction of an embodiment of the present invention;

[0048] Figure 4 It is a system structure diagram of an embodiment of the present invention. Detailed Embodiments

[0049] Refer to Figure 1 and Figure 2 As shown, it is a method step diagram and a process schematic diagram of an embodiment of the present invention, including the following steps:

[0050] S101, The client uses a feature extractor to extract the feature information of the original dataset and separate the personalized information in the feature information, and sends it to the server;

[0051] S102, The server uses the personalized information to train the generator and broadcasts the trained generator to the client;

[0052] S103, The client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset;

[0053] S104, the server and the client train a model using federated learning, where the client uses an extended dataset to train a local model, and the server updates the global model by aggregating the local models of each client.

[0054] Specifically, in S101, the input data (original dataset) is subjected to necessary preprocessing to ensure data consistency and model training efficiency, and then key features are extracted from the preprocessed data. The method of feature selection and extraction depends on the specific application scenario and data type, as shown in the following table:

[0055]

[0056] Specifically, in S101, the key lies in separating personalized information and global information. Inspired by the division of data content and style in the embodiments of the present invention, the dilemma between information sharing and performance improvement is studied from the perspective of feature division. By introducing an appropriate division, performance-sensitive features can be shared while maintaining the localization of performance-robust features, that is: the performance-sensitive features in the data are the key to alleviating data heterogeneity. First, an exact definition of an effective division is given, which captures the desired properties when dividing features. Subsequently, these two types of features are formally defined following the rules of the effective division.

[0057] The partitioning strategy refers to dividing variables into two parts within the same metric space, that is . The premise for the effectiveness of this partitioning strategy is that it satisfies the following conditions:

[0058] (i) ;

[0059] (ii) ;

[0060] (iii) ;

[0061] where represents the information entropy, and represents the mutual information.

[0062] Assume is an effective partitioning strategy. If it satisfies , where is the label of , then is personalized information, and is global information.

[0063] The partitioning strategy adopted in the embodiments of the present invention is based on Information Bottleneck (IB). In the traditional Information Bottleneck framework, the core objective is to capture the information related to the output labels while maximizing the compression of the input . Here, represents the latent embedding, which is a compressed and information-rich representation of , retaining the key information in while minimizing redundancy. This objective can be formalized as follows:

[0064] ;

[0065] where represents the constraint on the mutual information between and , represents the constraint on the mutual information between and , and is a Lagrange multiplier. The core of the Information Bottleneck is achieved through the latent variable , and the main objectives are: minimizing , compressing the redundant information irrelevant to and . Maximizing , retaining the discriminative information related to the compression of and , which is equivalent to solving an optimization problem in the information theory framework:

[0066]

[0067] where represents the conditional probability distribution of given .

[0068] It is difficult to directly calculate the mutual information. In practice, it is approximated by the variational distributions and :

[0069]

[0070] where the first term is the likelihood term: forcing to predict , maximizing , and the second term is the KL divergence constraint on the distribution of .

[0071] Data heterogeneity essentially stems from the differences in data distribution among clients. Aggregating models without accessing the data will inevitably lead to performance degradation; while sharing client data can greatly improve model performance, it essentially violates privacy; applying differential privacy (DP) to protect shared data seems feasible, but it comes at a cost as it results in a loss of model accuracy. Therefore, it is necessary to track and limit the total amount of noise while maintaining confidentiality.

[0072] Specifically, the embodiments of the present invention will use differential privacy for protection throughout the entire process of uploading personalized information.

[0073] Differential privacy is a mathematical framework for protecting personal privacy. It ensures that the change of any single data point will not have a significant impact on the statistical results by adding noise to the data. In federated learning, differential privacy can be used to further protect the privacy of clients, including the following steps:

[0074] Noise addition. Before sending the model update to the server, the client can add an appropriate amount of noise to its update.

[0075] Privacy budget. Track and limit the total amount of noise added throughout the learning process to ensure the level of privacy protection, expressed as:

[0076] ;

[0077] Where, and represent two adjacent data sets, and the two data sets differ by only one sample; represents a random algorithm, represents a possible output of the random algorithm ; represents when the input data set is the probability that the random algorithm outputs ; represents when the input data set is the probability that the random algorithm outputs ; represents the privacy budget (or privacy loss parameter), which is a non - negative real number used to control the degree of privacy protection; The smaller

[0078] When the server receives the personalized information sent by the client, it can start training a generator. The isolated personalized information is used as training data to train the generator model. The goal is to enable the generator to learn and simulate data samples with similar personalized characteristics. This training process uses Generative Adversarial Networks (GANs). The GAN model is a deep learning model that generates new data similar to the training data through adversarial training of two neural networks. These two neural networks are the Generator and the Discriminator respectively. The Generator is responsible for generating seemingly real samples. It takes a random noise as input and tries to generate data similar to that in the dataset. The Discriminator is responsible for distinguishing real samples from generated samples. It takes a sample (which may be a real sample or a generated sample) as input and outputs the probability that the sample is a real sample. During the training process, the Generator and the Discriminator compete with each other. The Generator continuously improves the quality of the generated samples in order to deceive the Discriminator; the Discriminator continuously improves its discrimination ability to better distinguish real samples from generated samples. This process eventually reaches a balance, where the Generator can generate high-quality samples and the Discriminator's judgment ability reaches its best. During the training process, the Generator and the Discriminator compete with each other. The Generator continuously improves the quality of the generated samples in order to deceive the Discriminator; the Discriminator continuously improves its discrimination ability to better distinguish real samples from generated samples. This process eventually reaches a balance, where the Generator can generate high-quality samples and the Discriminator's judgment ability reaches its best.

[0079] Specifically, in S103, each client generates an extended dataset, which specifically includes the following steps:

[0080] S1021, Generate data; Use the trained generator model to generate new data samples. These samples should retain the personalized characteristics of the original data while increasing the diversity and scale of the data.

[0081] S1022, Data integration; Integrate the generated data with the original dataset to form an extended dataset. Ensure that the generated data is of comparable quality to the original data and avoid introducing too much noise or bias.

[0082] S1023, Standardization processing; At the same time, perform standardization processing on the extended dataset to ensure that data from different sources or different users is consistent in terms of format, dimension, etc. Use machine learning techniques (such as feature selection, dimensionality reduction, etc.) to fuse the features of different sources or users to reduce the impact of data heterogeneity on model training.

[0083] Specifically, the federated learning training in S104 includes the following steps:

[0084] S1041, Initialize the model; The central server first initializes a global model and its parameters. This global model is the starting point for all subsequent local model training.

[0085] S1042, Distribute the model; The server distributes the global model and its parameters to all selected clients (user devices).

[0086] S1043, Local model training; After each client receives the global model, it uses local raw data and augmented samples (i.e., extended datasets) to train the global model and obtains the trained local model.

[0087] S1044, Upload local model parameters; Each client extracts the parameters of the trained local model. To protect privacy, the client usually encrypts the model parameters before uploading them. Then, the encrypted model parameters are uploaded to the server.

[0088] S1045, Server aggregates model parameters; The server collects the encrypted model parameters from all selected clients. The server decrypts the collected encrypted model parameters and then aggregates all the model parameters using an aggregation algorithm. The FedAvg algorithm calculates the average value of all local model parameters by weighted average as the new global model parameters. The server updates the global model according to the aggregated model parameters and prepares to distribute it to the clients again for the next round of training.

[0089] S1046, Repeat training and aggregation; The above process (from model distribution in S1042 to parameter aggregation in S1045) is repeated for multiple rounds until the global model reaches the preset convergence condition or the preset number of training rounds. In each round of training, the server may further optimize and adjust the global model according to the performance of the aggregated model.

[0090] To verify the effectiveness of the embodiments of the present invention, a variety of heterogeneous partitioning experiments were conducted on MNIST, FMNIST, CIFAF-10, and CIFAR-100. The experimental environment used the PyTorch framework, and the hardware parameters included an NVIDIA GTX ForceRTX 4090 GPU, an Intel (R) Core (TM) 390 i7-10700 CPU@2.90GHz, 64 GB of RAM, and a Windows 10 64-bit operating system based on an x64 processor system. For client-side local training, the random seed was set to seed = 0. To simulate real-world applications, the Dirichlet distribution was mainly used to generate non-independent and identically distributed partitions for clients. The experiments show that applying the personalized feature information extraction implemented in the present invention as a plug-in method can improve the ACC value on different algorithms and different datasets. On the CIFAR-100 dataset, the improvement is even as high as 18.02%, and the communication rounds are significantly reduced. This shows that the embodiments of the present invention can reduce the dimension of data by extracting personalized information from the data itself, effectively alleviate data heterogeneity, and reduce communication overhead without revealing privacy.

[0091] See Figure 4 As shown, it is the system structure diagram of the embodiments of the present invention, including:

[0092] An information separation module 401, where the client uses a feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information, and sends it to the server;

[0093] A generator training module 402, where the server uses the personalized information to train the generator and broadcasts the trained generator to the clients;

[0094] A data expansion module 403, where the client uses the received generator to generate enhanced samples, which together with the original dataset form an expanded dataset;

[0095] A global model training module 404, where the server and the clients use federated learning to train the model. Among them, the clients use the expanded dataset to train local models, and the server updates the global model by aggregating the local models of each client.

[0096] It can be seen that the present invention can solve the data heterogeneity problem and improve communication efficiency at the same time by separating personalized information to train a generator and then expanding the dataset through the generator.

[0097] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc., made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A heterogeneous federated learning method based on personalized information separation, characterized in that, It includes the following steps: The client uses a feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information, and sends it to the server; The server uses the personalized information to train a generator, and broadcasts the trained generator to the client; The client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset; The server and the client use the federated learning method to train the model. Among them, the client uses the extended dataset to train the local model, and the server updates the global model by aggregating the local models of each client; The client uses a feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information, including the following steps: After preprocessing the input data, the feature information is extracted through a VAE model; Based on a partitioning strategy, the feature information is partitioned into global information and personalized information; The partitioning strategy for partitioning the feature information into global information and personalized information adopts a partitioning strategy based on Information Bottleneck (IB), and the IB objective function is expressed as: Among them, is the likelihood term, representing maximizing I(Z; Y), D KL (p(z|x)‖r(z)) is the KL divergence constraint on the distribution of Z, representing minimizing I(Z; X); β is a Lagrange multiplier; q(y|z) and r(z) both represent variational distributions; p(z|x) represents the conditional probability distribution of Z given X; Before the client sends the personalized information to the server, differential privacy is used to encrypt the personalized information, which specifically includes the following steps: Adding noise to the personalized information, expressed as: Among them, represents the encrypted personalized information, x i represents the personalized information, Noise() represents the noise added using differential privacy, Δf represents the sensitivity, and ∈ represents the privacy budget; Tracking and restricting the total amount of added noise, expressed as: Pr[M(D)=S]≤exp(ε)·Pr[M(D′)=S]; Where D and D' respectively represent two adjacent datasets, and the two datasets differ by only one sample; M represents a random algorithm, S represents a possible output of the random algorithm M; Pr[M(D)=S] represents the probability that the random algorithm M outputs S when the input dataset is D; Pr[M(D')=S] represents the probability that the random algorithm M outputs S when the input dataset is D'; The server and the client use the federated learning method to train the model, including the following steps: Initialization step, the server initializes a global model and its parameters, and sends them to the client; Local update step, after each client receives the global model and its parameters, it uses the extended dataset for training, extracts the parameters of the trained model, and uploads them to the server; Global update step, the server collects the model parameters from all clients, aggregates all the model parameters as the new global model parameters; uses the new global model parameters to update the global model, and distributes them to the client; Iteration step, repeat the local update step and the global update step until the global model reaches the preset convergence condition or reaches the preset number of training rounds, and obtain the trained global model.

2. The heterogeneous federated learning method based on personalized information separation according to claim 1, wherein The client uses a feature extractor to extract the feature information of the original dataset and separates the personalized information in the feature information. Different clients select their respective feature types according to their application scenarios and data types.

3. The heterogeneous federated learning method based on personalized information separation according to claim 1, wherein The server uses the personalized information to train the generator. Specifically, the server uses the personalized information as training data to train the generator, so that the generator learns and simulates data samples with similar personalized characteristics; the training process of the generator uses a GAN model.

4. The heterogeneous federated learning method based on personalized information separation according to claim 1, wherein The client uses the received generator to generate enhanced samples, including the following steps: Use the trained generator model to generate new data samples, and at the same time sample target labels to generate corresponding feature representations; Integrate the generated enhanced data samples with the original dataset to form an extended dataset; Perform normalization processing on the extended dataset.

5. The heterogeneous federated learning method based on personalized information separation according to claim 1, characterized in that For aggregating all model parameters, the FedAvg algorithm is used to calculate the average value of all local model parameters by weighted average as the new global model parameters.

6. A heterogeneous federated learning system based on personalized information separation, characterized in that, For implementing the heterogeneous federated learning method based on personalized information separation according to any one of claims 1 to 5, including: An information separation module, where the client uses a feature extractor to extract the feature information of the original dataset and separate the personalized information in the feature information, and sends it to the server; A generator training module, where the server uses the personalized information to train the generator and broadcasts the trained generator to the client; A data extension module, where the client uses the received generator to generate enhanced samples and forms an extended dataset together with the original dataset; A global model training module, where the server and the client use federated learning to train the model, where the client uses the extended dataset to train the local model, and the server updates the global model by aggregating the local models of each client.

Images