A federated learning method based on a plug-and-play heterogeneous model
By introducing Plug-and-Play Generative Networks (PPGN) into federated learning, the problem of model and data heterogeneity is solved, training efficiency and communication security are improved, the need for personalized models is realized, and privacy protection capabilities are enhanced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2022-12-21
- Publication Date
- 2026-05-05
AI Technical Summary
Existing federated learning methods struggle to simultaneously optimize for both model heterogeneity and data heterogeneity, and they also fall short in terms of privacy protection and communication efficiency.
By combining Plug-and-Play Generative Networks (PPGN) with federated learning, a generative model is trained on the server side as a sampler. Generated data is used to replace model gradient propagation. By leveraging the replaceability of the generative model, personalized model training can be achieved while protecting privacy.
It improves model training effectiveness and efficiency, reduces communication overhead, achieves privacy protection, supports personalized model needs, and enhances the generalization ability and security of federated learning.
Smart Images
Figure CN115983340B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of federated learning technology, and in particular to a federated learning method based on plug-and-play heterogeneous models. Background Technology
[0002] Federated learning collaboratively trains a global model while ensuring that a group of clients do not upload their local datasets. Each user can only access their own data, thus protecting the privacy of users participating in the training. Due to its privacy-preserving advantages, federated learning has broad application prospects in industries such as medicine, finance, and artificial intelligence, and has become a research hotspot in recent years. However, federated learning focuses on obtaining a high-quality global model by learning from the local data of all participating clients. In real-world scenarios, the data of each client is heterogeneous. When faced with the problem of data heterogeneity, it cannot train a global model applicable to all clients. Meanwhile, privacy protection, communication efficiency constraints, and the need for personalized models have become major research directions in federated learning.
[0003] In recent years, most federated learning research has employed methods such as transfer learning, meta-learning, and reinforcement learning to address the issues of data and model heterogeneity in federated learning. Simultaneously, combining federated learning with methods like differential privacy and homomorphic encryption to protect user privacy is also a mainstream approach. However, most of the methods mentioned above have limitations, including the inability to simultaneously optimize for both model and data heterogeneity, some privacy protection issues, and the inability to address the communication overhead of federated learning. Summary of the Invention
[0004] This application provides a federated learning method based on plug-and-play heterogeneous models, which can be used to solve the technical problem that existing technologies cannot simultaneously optimize for model heterogeneity and data heterogeneity.
[0005] This application provides a federated learning method based on plug-and-play heterogeneous models, the method comprising:
[0006] Step 1: Train the generative model: Initialize the generative model G and the discriminative model C on the server side, and train the generative model G and the discriminative model C using the training method of generative adversarial networks, where G is used to generate the dataset D. * Fake images, where C is used to identify images from the public dataset D. * The real image x,
[0007] Step 2: Initialize the model:
[0008] Step 3: Train the client model M on each client. i ,include:
[0009] Step 4: Server Aggregation: Aggregate sampled data from client i on the server side. And parameters including the amount of data used in training;
[0010] Step 5: Server-side data allocation: Distribute the sampled dataset d ～ The data sets d are redistributed according to the data category distribution weights of each client's private data. i And distribute them to each client;
[0011] Step 6: Model Testing: In each round of communication, the classification model M with updated parameters for this round is tested on all clients. i The test accuracy on the test set is used to calculate the average test accuracy of the model.
[0012] Step 7: The server determines whether to continue the next communication. If it does, it returns to step 3; otherwise, it ends the communication and saves the global network model parameters.
[0013] Optionally, the generative model G includes fully connected layers, batch normalization layers, activation function layers, and deconvolution layers;
[0014] The input to the generative model G is the hidden code h, and the target output is the dataset D from the public dataset. * Image of distribution;
[0015] The discriminant model C consists of fully connected layers, activation function layers, and deconvolution layers;
[0016] Both the generative model G and the discriminative model C use Adam as the optimizer.
[0017] Optional, initialize the model, including:
[0018] Dataset D is divided into private datasets D based on the Dirichlet distribution. i This data is private to each client; different clients possess different amounts of data with varying categories and distributions.
[0019] Initialize the target classification model M, and then propagate the target classification model M to each client as its local classification model M. i The generative model G is shared among all clients, and communication between the client and the server begins.
[0020] Optionally, client-side model M can be trained on each client. i ,include:
[0021] Step 3-1: The client uses private data D i Training client model M i Obtain the loss function of the client model;
[0022] Step 3-2: Starting from the second round of communication, use the image d received from the server. i As an additional dataset for model M i Conduct training;
[0023] Step 3-3: Starting from a randomly generated hidden code h, define a Markov chain Monte Carlo sampler using the Metropolitan Langevin algorithm;
[0024] The sampler uses client model M i G is a conditional network, and G is a generator network, which can maximize activation to specify the class y. ～ ; to generate an image based on conditions, thereby updating the encoding h, and obtaining y ～ Image x for category ～ , take image x ～ As data to be passed to the server y ～ For dataset D i Category labels for the data;
[0025] Steps 3-4: Sample data Send to the server.
[0026] Optionally, the sampling rules in step 3-3 are as follows:
[0027]
[0028] Where p() represents the probability of the corresponding distribution occurring. Let represent random noise that follows a normal distribution, and ∈1, ∈2, ∈3 represent the posterior loss, prior loss, and noise parameter of the image, respectively.
[0029] Optionally, the model architecture for the classification model includes LeNet, CNN, and MobileNet_v3, where the classification model M is used. i The output dimensions of the hidden layer and the last layer are initialized with the length of the hidden encoding h and the number of classes in the target dataset D, respectively.
[0030] Optionally, the sampler has two operating modes:
[0031] For each client, the sampler samples the same number of generated samples for different categories of the private dataset D;
[0032] For each client sampling, the generated samples for different target categories are based on the current client's private dataset D. i The number of training samples for each category is weighted and sampled.
[0033] This invention provides a method for implementing federated learning, which has the following advantages compared to existing technologies: 1. It constructs a generative model trained on the server side as a sampler. Utilizing the replaceability of plug-and-play generative models, the server model is used as a conditional network. The sampled generative data replaces the model gradient in the traditional federated averaging algorithm for transmission between the client and server. Compared to gradient transmission in previous methods, image transmission has controllable and lower transmission overhead, improving the effectiveness and efficiency of model training while saving communication overhead; 2. PPGN can extract model features from the client's local classification model based on a shared dataset. Because no local private data is involved in the sampling process, the sampled data does not expose details of the user's local data, realizing the privacy protection principle in federated learning and being more secure than other federated learning methods based on generative or distillation networks; 3. Since there is no need to transmit model parameters, personalized models with different structures can be trained on the client side, fulfilling the personalized model requirements of federated learning. Attached Figure Description
[0034] Figure 1 A flowchart illustrating an embodiment of this application;
[0035] Figure 2 This is a schematic diagram of a model provided for an embodiment of this application;
[0036] Figure 3 A comparison image between the image generated by the MCMC sampler provided in this embodiment and the original dataset image. Detailed Implementation
[0037] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0038] Therefore, this method proposes combining Plug-and-Play Generative Networks (PPGN) with federated learning to address the heterogeneity problem in federated learning. This method also offers higher security than other generative network-based federated learning methods. The core idea of PPGN is to maximize the activation of one or more neurons in a single classifier network by performing gradient ascent in the latent space of the generator network. In simpler terms, PPGN consists of a generator network G capable of generating a wide range of image types and a replaceable "conditional" network C that tells the generator what to generate. Data privacy is a crucial constraint and principle for model training in federated learning. Since PPGN can generate data that activates the corresponding categories in the target dataset without requiring training on the client's private dataset, it can be combined with federated learning to improve the generalization ability and training and communication efficiency of federated learning. The most useful feature is its "plug-and-play" capability—allowing users to enter a replaceable conditional network and generate images based on conditions specified at test time. Therefore, this method enables federated multimodal learning and personalized federated model learning.
[0039] The embodiments of this application will now be described in conjunction with the accompanying drawings.
[0040] This application provides a federated learning method based on plug-and-play heterogeneous models, including:
[0041] Step 1: Train the generative model: Initialize the generative model G and the discriminative model C on the server side, and train the generative model G and the discriminative model C using the training method of generative adversarial networks, where G is used to generate the dataset D. * Fake images, where C is used to identify images from the public dataset D. * The real image x.
[0042] The generative model G includes fully connected layers, batch normalization layers, activation function layers, and deconvolution layers.
[0043] The input to the generative model G is the hidden code h, and the target output is the dataset D from the public dataset. * Image of distribution;
[0044] The discriminant model C consists of fully connected layers, activation function layers, and deconvolution layers;
[0045] Both the generative model G and the discriminative model C use Adam as the optimizer.
[0046] Step 2: Initialize the model:
[0047] Dataset D is divided into private datasets D based on the Dirichlet distribution. iThis data is private to each client; different clients possess different amounts of data with varying categories and distributions.
[0048] Initialize the target classification model M, and then propagate the target classification model M to each client as its local classification model M. i The generative model G is shared among all clients, and communication between the client and the server begins.
[0049] The model architectures for classification models include LeNet, CNN, and MobileNet v3, among which the classification model M is used. i The output dimensions of the hidden layer and the last layer are initialized with the length of the hidden encoding h and the number of classes in the target dataset D, respectively.
[0050] Step 3: Train the client model M on each client. i ,include:
[0051] Step 3-1: The client uses private data D i Training client model M i Obtain the loss function of the client model;
[0052] Step 3-2: Starting from the second round of communication, use the image d received from the server. i As an additional dataset for model M i Conduct training;
[0053] Step 3-3: Starting with a randomly generated hidden code h, define a Markov chain Monte Carlo (MCMC) sampler using the Metropolitan Langevin algorithm (MALA);
[0054] The sampler uses client model M i G is a conditional network, and G is a generator network, which can maximize activation to specify the class y. ～ ; to generate an image based on conditions, thereby updating the encoding h, and obtaining y ～ Image x for category ～ , take image x ～ As data to be passed to the server y ～ For dataset D i Category labels for the data.
[0055] The sampling rules in step 3-3 are as follows:
[0056]
[0057] Where p() represents the probability of the corresponding distribution occurring. Let denot be random noise that follows a normal distribution, and let ∈1, ∈2, ∈3 represent the posterior loss of the image (making the image look like a picture of the target category), the prior loss (making the image look like a real picture of the target dataset), and the noise parameter (encouraging the generation of more diverse pictures by adding noise), respectively.
[0058] The sampler has two operating modes:
[0059] Average sampling: For each client, the sampler samples the same number of generated samples for different categories of the private dataset D.
[0060] Weighted sampling involves generating samples for each client based on the current client's private dataset D for different target categories. i The number of training samples for each category is weighted and sampled, with more samples having more weight.
[0061] Steps 3-4: Sample data Send to the server;
[0062] Step 4: Server Aggregation: Aggregate sampled data from client i on the server side. And parameters including the amount of data used in training;
[0063] Step 5: Server-side data allocation: Distribute the sampled dataset d ～ The data sets d are redistributed according to the data category distribution weights of each client's private data. i And distribute them to each client;
[0064] Step 6: Model Testing: In each round of communication, the classification model M with updated parameters for this round is tested on all clients. i The test accuracy on the test set is used to calculate the average test accuracy of the model.
[0065] Step 7: The server determines whether to continue the next communication. If it does, it returns to step 3; otherwise, it ends the communication and saves the global network model parameters.
[0066] The method provided in this application transmits data samples directly instead of parameters during communication between the client i and the server. Therefore, compared with the federated averaging algorithm, each client can use different model structures as its client model based on its local device computing power and computing environment, thus realizing the personalized model requirements in federated learning.
[0067] This invention provides a method for implementing federated learning, which has the following advantages compared to existing technologies: 1. It constructs a generative model trained on the server side as a sampler. Utilizing the replaceability of plug-and-play generative models, the server model is used as a conditional network. The sampled generative data replaces the model gradient in the traditional federated averaging algorithm for transmission between the client and server. Compared to gradient transmission in previous methods, image transmission has controllable and lower transmission overhead, improving the effectiveness and efficiency of model training while saving communication overhead; 2. PPGN can extract model features from the client's local classification model based on a shared dataset. Because no local private data is involved in the sampling process, the sampled data does not expose details of the user's local data, realizing the privacy protection principle in federated learning and being more secure than other federated learning methods based on generative or distillation networks; 3. Since there is no need to transmit model parameters, personalized models with different structures can be trained on the client side, fulfilling the personalized model requirements of federated learning.
[0068] The embodiments described above do not constitute a limitation on the scope of protection of this application.
Claims
1. A federated learning method based on plug-and-play heterogeneous models, characterized in that, The method includes: Step 1: Train the generative model: Initialize the generative model on the server side. and discriminant model Generative models are trained using generative adversarial networks. and discriminant model ,in; Used to generate datasets Fake images, among which Used to distinguish data from public datasets Real images , Step 2: Initialize the model: Step 3: Train the client-side model on each client. ,include: Step 4: Server Aggregation: Aggregate data from clients on the server side. Sampling data , and parameters including the amount of data used in training; Step 5: Server-side data allocation: The data sets were redistributed into subsets based on the weighted distribution of data categories within each client's private data. And distribute them to each client; Step 6: Model Testing: In each round of communication, test the classification model with updated parameters on all clients. The test accuracy on the test set is used to calculate the average test accuracy of the model. Step 7: The server determines whether to continue the next communication. If it does, it returns to step 3; otherwise, it ends the communication and saves the global network model parameters. Train client-side models on various clients. ,include: Step 3-1: The client uses private data Training client model Obtain the loss function of the client model; Step 3-2: Starting from the second round of communication, use the images received from the server. As an additional dataset for the model Conduct training; Step 3-3: From a randomly generated hidden code We begin by defining a Markov chain Monte Carlo sampler using the Metropolitan Langevin algorithm. The sampler uses a client-side model. For conditional networks, To generate a network, activation can be maximized to specify the category. ; Generate images based on conditions, thereby updating the encoding. , to obtain Images by category , the picture As data to be passed to the server ; For dataset Category labels for the data; Steps 3-4: Sample data Send to the server; The sampling rules in step 3-3 are as follows: ； in This represents the probability of the corresponding distribution occurring. This represents random noise that follows a normal distribution. These represent the posterior loss, prior loss, and noise parameter of the image, respectively.
2. The federated learning method based on a plug-and-play heterogeneous model according to claim 1, characterized in that, Generative Models It includes fully connected layers, batch normalization layers, activation function layers, and deconvolution layers; Generative Models The input is the hidden code h, and the target output is from a public dataset. Image of distribution; Discriminant model It includes fully connected layers, activation function layers, and deconvolution layers; Generative Models and discriminant model Adam is used as the optimizer in all of them.
3. The federated learning method based on a plug-and-play heterogeneous model according to claim 1, characterized in that, Initialize the model, including: Dataset Divided into private datasets based on Dirichlet distribution This data is private to each client; different clients possess different amounts of data with varying categories and distributions. Initialize the target classification model M, and then... Distribute to each client as its local classification model Share the generative model with all clients This initiates communication between the client and the server.
4. The federated learning method based on a plug-and-play heterogeneous model according to claim 1, characterized in that, The model architectures for classification models include LeNet, CNN, and MobileNet v3, among which classification models are used. The output dimensions of the hidden layer and the last layer are initialized with the length of the hidden code h and the target dataset, respectively. The number of categories in the data.
5. The federated learning method based on a plug-and-play heterogeneous model according to claim 4, characterized in that, The sampler has two operating modes: For each client, the sampler has a private dataset. The same number of samples were taken from all generated samples of different categories; For each client sampling, the generated samples for different target categories are based on the current client's private dataset. The number of training samples for each category is weighted and sampled.
Citation Information
Patent Citations
Differential privacy federal learning method for resisting member reasoning attack
CN114785559A
Federal continuous learning training method based on memory playback and differential privacy
CN115081532A