Heterogeneous federal learning method and system based on personalized information separation

Abstract

The invention discloses a heterogeneous federal learning method and system based on personalized information separation, and the method comprises the following steps: a client extracts feature information of an original data set through a feature extractor, separates personalized information in the feature information, and transmits the personalized information to a server; the server trains the generator by using the personalized information, and broadcasts the trained generator to the client; the client generates an enhanced sample by using the received generator, and forms an extended data set together with the original data set; the server and the clients use a federated learning mode to train models, the clients use an extended data set to train local models, and the server updates a global model by aggregating the local models of the clients. According to the method, the generator is trained by separating the personalized information, and the data set is expanded through the generator, so that the problem of data isomerism is solved, and the communication efficiency is improved at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to a heterogeneous federated learning method and system based on personalized information separation. Background Art

[0002] Federated Learning (FL) is a distributed model training framework that allows individual clients to collaboratively train global models in an edge computing environment without exposing their local data. However, FL often faces data heterogeneity, that is, distribution transfer between clients. Sharing customer information shows great potential in alleviating data heterogeneity, but it encounters a dilemma in protecting privacy and improving model performance. In order to solve the heterogeneity challenge, the pioneering work FedAvg (Federated Averaging) introduced model aggregation of local training models. It solved the diversity of computing and communication, but still faced the problem of client drift caused by data heterogeneity. In order to alleviate this dilemma, some people proposed the method of federated feature distillation, which divides the data into performance-sensitive features (performance-sensitive features containing very little information can make the model generalize well) and performance-robust features (containing almost all the information in the data) by mining information from the data. In this way, almost all the information in the data can be saved locally, and only performance-sensitive features need to be shared. Therefore, if the data features can be perfectly divided into these two types of features, the performance-privacy dilemma can be well solved. This method seeks some natural intrinsic connections in the information bottleneck theory. Ideally, the information bottleneck aims to remove redundant information from the data and retain important information that helps to make correct predictions. Then, the redundant information lost by the information bottleneck becomes information that does not need to be shared. Therefore, inspired by the information bottleneck, redundant information can be "discarded" locally as much as possible, and only a very small amount of information can be sent to solve the problem of data heterogeneity. This method believes that data is the source of information, but does not take into account that not all clients will have enough data. Summary of the invention

[0003] The purpose of the present invention is to solve the problems in the prior art and to propose a method for solving the problem of data heterogeneity by separating personalized information to train a generator and then expanding the data set through the generator.

[0004] The technical solution adopted by the present invention to solve the technical problem is: to provide a heterogeneous federated learning method based on personalized information separation, comprising the following steps:

[0005] The client uses the feature extractor to extract the feature information of the original data set and separates the personalized information from the feature information and sends it to the server;

[0006] The server trains the generator using personalized information and broadcasts the trained generator to the client;

[0007] The client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset;

[0008] The server and client use federated learning to train the model, where the client uses an extended dataset to train the local model, and the server updates the global model by aggregating the local models of each client.

[0009] Preferably, the client uses a feature extractor to extract feature information of the original data set and separates personalized information from the feature information, and different clients select their own feature types according to their application scenarios and data types.

[0010] Preferably, the client uses a feature extractor to extract feature information of the original data set and separates personalized information from the feature information, including the following steps:

[0011] After preprocessing the input data, feature information is extracted through the VAE model;

[0012] The feature information is divided into global information and personalized information based on the partitioning strategy.

[0013] Preferably, the feature information is divided into global information and personalized information based on the division strategy, and the adopted division strategy includes information bottleneck IB, and the IB objective function is expressed as:

[0014] ;

[0015] in, is the likelihood term, which means maximizing , is the KL divergence constraint The distribution of ; is a Lagrange multiplier; and Both represent variational distribution; Indicates that in a given In the case The conditional probability distribution of .

[0016] Preferably, before the client sends the personalized information to the server, differential privacy is used to encrypt the personalized information, which specifically includes the following steps:

[0017] Add noise to the personalized information, expressed as:

[0018] ;

[0019] in, Indicates the encrypted personalized information. Represents personalized information. represents the noise added using differential privacy, Indicates sensitivity, represents the privacy budget;

[0020] Track and limit the total amount of noise added, expressed as:

[0021] ;

[0022] in, and Represent two adjacent data sets, with only one sample difference between the two data sets; represents a random algorithm, Represents a random algorithm A possible output of It means that when the input data set is When the random algorithm Output The probability of It means that when the input data set is When the random algorithm Output probability.

[0023] Preferably, the server uses personalized information to train the generator, specifically: the server uses the personalized information as training data to train the generator, allowing the generator to learn and simulate data samples with similar personalized features; the training process of the generator uses a GAN model.

[0024] Preferably, the client generates an enhanced sample using the received generator, comprising the following steps:

[0025] Use the trained generator model to generate new data samples, sample target labels, and generate corresponding feature representations;

[0026] Integrate the generated feature representation with the original dataset to form an extended dataset;

[0027] The expanded dataset is standardized.

[0028] Preferably, the server and the client train the model using a federated learning method, including the following steps:

[0029] Initialization step: the server initializes a global model and its parameters and sends them to the client;

[0030] In the local update step, after each client receives the global model and its parameters, it uses the extended data set for training, extracts the parameters of the trained model, and uploads them to the server;

[0031] In the global update step, the server collects model parameters from all clients, aggregates all model parameters as new global model parameters, updates the global model with the new global model parameters, and distributes them to the clients.

[0032] Iteration step, repeat the local update step and the global update step until the global model reaches the preset convergence condition or reaches the preset number of training rounds, and obtains the trained global model.

[0033] Preferably, all model parameters are aggregated, and the average values ​​of all local model parameters are calculated by weighted averaging using the FedAvg algorithm as new global model parameters.

[0034] The present invention also provides a heterogeneous federated learning system based on personalized information separation, which is used to implement any of the above-mentioned heterogeneous federated learning methods based on personalized information separation, including:

[0035] Information separation module: the client uses the feature extractor to extract the feature information of the original data set and separates the personalized information from the feature information and sends it to the server;

[0036] Generator training module, the server uses personalized information to train the generator and broadcasts the trained generator to the client;

[0037] Data extension module, the client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset;

[0038] In the global model training module, the server and the client use federated learning to train the model, where the client uses an extended data set to train the local model, and the server updates the global model by aggregating the local models of each client.

[0039] The present invention has the following beneficial effects:

[0040] (1) The present invention extracts information from user data to train the generator without any external data. By separating personalized information and training the generator, corresponding data samples can be generated for different types of heterogeneous data. This method helps alleviate the challenges brought by data heterogeneity because the generated data can simulate various heterogeneous scenarios in the real world.

[0041] (2) Traditional federated learning requires participating devices to communicate frequently during the training process to share model updates. However, the present invention expands the data set through the generator, which can increase the amount of training data without adding additional communication burden, helping to reduce the number of communications and the amount of communication data, thereby improving communication efficiency;

[0042] (3) The generator trained by the present invention can generate high-quality and diverse data samples, which not only have statistical properties similar to the original data, but also contain rich personalized information;

[0043] (4) In the past, solutions to the heterogeneity problem have all been to enrich the global model with integrated knowledge from local models. However, this cannot completely solve the inherent heterogeneity between user models. The present invention uses integrated information from personalized information from local models to train the generator, which can be used to alleviate the heterogeneity problem of local client models.

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A method step diagram of an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of extracting personalized information according to an embodiment of the present invention;

[0048] Figure 4 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] See also Figure 1 and Figure 2 The method step diagram and flow chart of an embodiment of the present invention are shown, which include the following steps:

[0050] S101, the client uses a feature extractor to extract feature information of the original data set and separates personalized information from the feature information, and sends it to the server;

[0051] S102, the server trains the generator using the personalized information and broadcasts the trained generator to the client;

[0052] S103, the client generates enhanced samples using the received generator, and forms an extended dataset together with the original dataset;

[0053] S104, the server and the client train the model using federated learning, wherein the client trains the local model using the extended data set, and the server updates the global model by aggregating the local models of each client.

[0054] Specifically, in S101, the input data (original data set) is preprocessed as necessary to ensure data consistency and model training efficiency, and then key features are extracted from the preprocessed data. The selection and extraction of features depends on the specific application scenario and data type, as shown in the following table:

[0055]

[0056] Specifically, in S101, the focus is on separating personalized information from global information. The embodiment of the present invention is inspired by the division of data content and style, and studies the dilemma between information sharing and performance improvement from the perspective of feature division. By introducing appropriate division, performance-sensitive features can be shared while maintaining the localization of performance-robust features, that is, performance-sensitive features in the data are the key to alleviating data heterogeneity. First, a precise definition of effective division is given, which captures the desired properties when dividing features. Subsequently, the two types of features are formally defined following the rules of effective division.

[0057] The partitioning strategy is to divide the variables into Divided into two parts, namely The partitioning strategy is effective only if it satisfies the following conditions:

[0058] (i) ;

[0059] (ii) ;

[0060] (iii) ;

[0061] in represents information entropy, represents mutual information.

[0062] Assumptions is an effective partitioning strategy if ,in yes , then It is personalized information. It is global information.

[0063] The partitioning strategy adopted in the embodiment of the present invention is based on the information bottleneck (IB). In the traditional information bottleneck framework, the core goal is to capture and output labels. Related Information While maximizing input The degree of compression here represents the latent embedding, which is A compressed and information-rich representation of This goal can be formalized as follows:

[0064] ;

[0065] in, express and The mutual information constraints between express and The mutual information constraints between is a Lagrange multiplier. The core of the information bottleneck is through hidden variables The main goal is to minimize ,compression and Irrelevant redundant information. Maximize , retain compression and This is equivalent to solving the optimization problem in the information theory framework:

[0066]

[0067] in, Indicates that in a given In the case of The conditional probability distribution of .

[0068] It is difficult to directly calculate mutual information. In practice, we use variational distribution and approximate:

[0069]

[0070] The first term is the likelihood term: predict ,maximize , the second term is the KL divergence constraint The distribution of .

[0071] Data heterogeneity essentially stems from differences in data distribution between clients. Aggregating models without accessing data will inevitably lead to performance degradation; while sharing client data can greatly improve model performance, it essentially violates privacy; applying differential privacy (DP) to protect shared data seems feasible, but it is not without cost because it will lead to loss of model accuracy. Therefore, it is necessary to track and limit the total amount of noise while maintaining confidentiality.

[0072] Specifically, the embodiment of the present invention uses differential privacy for protection throughout the entire process of uploading personalized information.

[0073] Differential privacy is a mathematical framework for protecting individual privacy by adding noise to the data to ensure that changes in any single data point do not significantly affect the statistical results. In federated learning, differential privacy can be used to further protect the privacy of the client, including the following steps:

[0074] Noise Addition,Before sending model updates to the server, clients can add an appropriate amount of noise to their updates.

[0075] The privacy budget, which tracks and limits the total amount of noise added throughout the learning process to ensure a level of privacy protection, is expressed as:

[0076] ;

[0077] in, and Represent two adjacent data sets, with only one sample difference between the two data sets; represents a random algorithm, Represents a random algorithm A possible output of It means that when the input data set is When the random algorithm Output probability; It means that when the input data set is When the random algorithm Output probability. represents the privacy budget (or privacy loss parameter), which is a non-negative real number used to control the degree of privacy protection; The smaller it is, the stronger the privacy protection is, but the more noise may be introduced.

[0078] When the server receives the personalized information from the client, it can start learning a generator. The separated personalized information is used as training data to train the generator model. The goal is to enable the generator to learn and simulate data samples with similar personalized features. This training process uses Generative Adversarial Networks (GAN). The GAN model is a deep learning model that generates new data similar to the training data through adversarial training of two neural networks. The two neural networks are the generator and the discriminator. The generator is responsible for generating samples that look real. It receives a random noise as input and tries to generate data similar to the data set. The discriminator is responsible for distinguishing between real samples and generated samples. It receives a sample (which may be a real sample or a generated sample) as input and outputs the probability that the sample is a real sample. During the training process, the generator and the discriminator compete with each other. The generator continuously improves the quality of the generated samples in order to deceive the discriminator; the discriminator continuously improves its discrimination ability to better distinguish between real samples and generated samples. This process eventually reaches a balance, the generator can generate high-quality samples, and the discriminator's judgment ability is also optimal. During the training process, the generator and the discriminator compete with each other. The generator continuously improves the quality of the generated samples in order to deceive the discriminator; the discriminator continuously improves its discrimination ability to better distinguish between real samples and generated samples. This process eventually reaches a balance, the generator can generate high-quality samples, and the discriminator's judgment ability is also optimal.

[0079] Specifically, in S103, each client generates an extended data set, which specifically includes the following steps:

[0080] S1021, Generate data: Generate new data samples using the trained generator model. These samples should retain the personalized characteristics of the original data while increasing the diversity and scale of the data.

[0081] S1022, Data Integration: Integrate the generated data with the original data set to form an extended data set. Ensure that the generated data is of comparable quality to the original data and avoid introducing too much noise or bias.

[0082] S1023, standardization processing; at the same time, the expanded data set is standardized to ensure that the data from different sources or different users are consistent in format, dimension, etc. Machine learning techniques (such as feature selection, dimensionality reduction, etc.) are used to fuse the features of different sources or users to reduce the impact of data heterogeneity on model training.

[0083] Specifically, the federated learning training in S104 includes the following steps:

[0084] S1041, Initialize the model: The central server first initializes a global model and its parameters. This global model is the starting point for all subsequent local model training.

[0085] S1042, distribute the model; the server distributes the global model and its parameters to all selected clients (user devices).

[0086] S1043, local model training; after each client receives the global model, it trains the global model using local original data and enhanced samples (i.e., extended data set) to obtain a trained local model.

[0087] S1044, local model parameter upload: each client extracts the parameters of the trained local model. To protect privacy, the client usually encrypts the model parameters before uploading them. Then, the encrypted model parameters are uploaded to the server.

[0088] S1045, the server aggregates model parameters; the server collects the encrypted model parameters from all selected clients. The server decrypts the collected encrypted model parameters and then aggregates all model parameters using the aggregation algorithm. The FedAvg algorithm calculates the average value of all local model parameters by weighted average as the new global model parameter. The server updates the global model based on the aggregated model parameters and prepares to distribute it to the client again for the next round of training.

[0089] S1046, repeat training and aggregation; the above process (from model distribution in S1042 to parameter aggregation in S1045) will be repeated for multiple rounds until the global model reaches the preset convergence condition or reaches the preset number of training rounds. In each round of training, the server may further optimize and adjust the global model based on the performance of the aggregated model.

[0090] In order to verify the effectiveness of the embodiment of the present invention, a variety of heterogeneous partitioning experiments were conducted on MNIST, FMNIST, CIFAF-10 and CIFAR-100. The experimental environment uses the PyTorch framework, and the hardware parameters include NVIDIA GTX ForceRTX 4090 GPU, Intel (R) Core (TM) 390 i7-10700 CPU @ 2.90GHz, 64 GB RAM and Windows 10 64-bit operating system based on x64 processor system. For local training on the client, the random seed is set to seed = 0. In order to simulate real-world applications, the Dirichle distribution is mainly used to generate client non-independent and identically distributed partitions. The experiment shows that the personalized feature information extraction implemented by the present invention as a plug-in method has improved the ACC value on different algorithms and different data sets, and even improved by up to 18.02% on the CIFAR-100 data set, and the communication rounds are significantly reduced. This shows that the embodiment of the present invention can achieve data dimensionality reduction by extracting personalized information from the data itself, effectively alleviate data heterogeneity and reduce communication overhead without leaking privacy.

[0091] See also Figure 4 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0092] Information separation module 401, the client uses a feature extractor to extract feature information of the original data set and separates personalized information from the feature information and sends it to the server;

[0093] Generator training module 402, the server trains the generator using personalized information and broadcasts the trained generator to the client;

[0094] Data extension module 403, the client generates enhanced samples using the received generator, and forms an extended data set together with the original data set;

[0095] In the global model training module 404, the server and the client train the model using a federated learning method, wherein the client trains the local model using an extended data set, and the server updates the global model by aggregating the local models of each client.

[0096] It can be seen that the present invention can solve the data heterogeneity problem and improve communication efficiency at the same time by separating personalized information to train a generator and then expanding the data set through the generator.

[0097] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A heterogeneous federated learning method based on personalized information separation, characterized in that: The following steps are involved: The client uses the feature extractor to extract the feature information of the original data set and separates the personalized information from the feature information and sends it to the server; The server trains the generator using personalized information and broadcasts the trained generator to the client; The client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset; The server and client use federated learning to train the model, where the client uses an extended dataset to train the local model, and the server updates the global model by aggregating the local models of each client.

2. The heterogeneous federated learning method based on personalized information separation according to claim 1 is characterized in that: The client uses a feature extractor to extract feature information of the original data set and separates personalized information from the feature information. Different clients select their own feature types according to their application scenarios and data types.

3. The heterogeneous federated learning method based on personalized information separation according to claim 1 is characterized in that: The client uses a feature extractor to extract feature information of the original data set and separates personalized information from the feature information, including the following steps: After preprocessing the input data, feature information is extracted through the VAE model; The feature information is divided into global information and personalized information based on the partitioning strategy.

4. The heterogeneous federated learning method based on personalized information separation according to claim 3 is characterized in that: The feature information is divided into global information and personalized information based on the partitioning strategy. The partitioning strategy adopted includes information bottleneck IB. The IB objective function is expressed as: ； in, is the likelihood term, which means maximizing , is the KL divergence constraint The distribution of ; is a Lagrange multiplier; and Both represent variational distribution; Indicates that in a given In the case The conditional probability distribution of .

5. The heterogeneous federated learning method based on personalized information separation according to claim 1 is characterized in that: Before the client sends the personalized information to the server, the personalized information is encrypted using differential privacy, which specifically includes the following steps: Add noise to the personalized information, expressed as: ； in, Indicates the encrypted personalized information. Represents personalized information. represents the noise added using differential privacy, Indicates sensitivity, represents the privacy budget; Track and limit the total amount of noise added, expressed as: ； in, and Represent two adjacent data sets, with only one sample difference between the two data sets; represents a random algorithm, Represents a random algorithm A possible output of It means that when the input data set is When the random algorithm Output probability; It means that when the input data set is When the random algorithm Output probability.

6. The heterogeneous federated learning method based on personalized information separation according to claim 1 is characterized in that: The server uses personalized information to train the generator, specifically: the server uses the personalized information as training data to train the generator, allowing the generator to learn and simulate data samples with similar personalized features; the training process of the generator uses a GAN model.

7. The heterogeneous federated learning method based on personalized information separation according to claim 1 is characterized in that: The client generates an enhanced sample using the received generator, comprising the following steps: Use the trained generator model to generate new data samples, sample target labels, and generate corresponding feature representations; Integrate the generated enhanced data samples with the original dataset to form an extended dataset; The expanded dataset is standardized.

8. The heterogeneous federated learning method based on personalized information separation according to claim 1 is characterized in that: The server and the client train the model using federated learning, including the following steps: Initialization step: the server initializes a global model and its parameters and sends them to the client; In the local update step, after each client receives the global model and its parameters, it uses the extended data set for training, extracts the parameters of the trained model, and uploads them to the server; In the global update step, the server collects model parameters from all clients, aggregates all model parameters as new global model parameters, updates the global model with the new global model parameters, and distributes them to the clients. Iteration step, repeat the local update step and the global update step until the global model reaches the preset convergence condition or reaches the preset number of training rounds, and obtains the trained global model.

9. The heterogeneous federated learning method based on personalized information separation according to claim 8 is characterized in that: All model parameters are aggregated, and the average values ​​of all local model parameters are calculated by weighted averaging using the FedAvg algorithm as new global model parameters.

10. A heterogeneous federated learning system based on personalized information separation, characterized in that: The method for implementing the heterogeneous federated learning based on personalized information separation as described in any one of claims 1 to 9 comprises: Information separation module: the client uses the feature extractor to extract the feature information of the original data set and separates the personalized information from the feature information and sends it to the server; Generator training module, the server uses personalized information to train the generator and broadcasts the trained generator to the client; Data extension module, the client uses the received generator to generate enhanced samples, which together with the original dataset form an extended dataset; In the global model training module, the server and the client use federated learning to train the model, where the client uses an extended data set to train the local model, and the server updates the global model by aggregating the local models of each client.

Images