Data augmentation based on conditional GAN ​​to improve personalized federated learning performance

Abstract

The present invention discloses a method for improving the performance of personalized federated learning by data augmentation based on conditional GAN, comprising the following steps: establishing a personalized federated learning pFedMe model; establishing a conditional GAN ​​model, and then adding it to the pFdeMe model in a personalized federated learning manner to obtain a pFedMe model based on conditional GAN; obtaining a CIFAR10 data set, and implementing data augmentation through the pFedMe model based on conditional GAN; and obtaining the accuracy of the data augmentation method based on conditional GAN ​​on a test set. The present invention can obtain a method in which conditional GAN ​​effectively improves the performance of personalized federated learning, which has practical value for using conditional GAN ​​to perform data augmentation on a model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for improving personalized federated learning performance by data augmentation based on a conditional generative adversarial network (GAN). Background Art

[0002] Federated learning is a privacy-preserving machine learning technique in which a group of clients collaborate with a server to learn a global model without sharing client data. A core challenge of the federated learning problem is to overcome the performance loss caused by statistical heterogeneity between clients. Studies have shown that this heterogeneity limits the global model from providing good performance on each client's task. The personalized federated learning (pFedMe) algorithm, which uses the Moreau envelope as the client regularization loss function, helps optimize the personalized model.

[0003] Nowadays, the large amount of data generated in a large number of handheld devices has stimulated the development of federated learning. The scenario of federated learning involves a large number of clients connected to a server, and its goal is to build a global model in a privacy-preserving and communication-efficient way. Despite the advantages of data privacy protection and low communication overhead, federated learning faces the following challenges that affect its performance and convergence speed: statistical heterogeneity, which means that the data distribution between clients is different. It is difficult for a global model trained with these non-identical distribution data to have good performance on each client's data. When the statistical heterogeneity increases, the generalization error of the global model to the client's local data also increases significantly. On the other hand, local learning without federated learning (i.e., without client collaboration) will also have a large generalization error due to insufficient data.

[0004] Training deep learning models requires a large number of training samples. Insufficient training data can lead to serious overfitting problems and reduce model accuracy. In fact, collecting a large number of samples to train deep learning models requires time and knowledge of relevant fields, which is both expensive and difficult. Data augmentation is one of the common techniques to solve the above problems. It can increase the relevant data in the data set and allow the model to learn more data-related features. In turn, it can effectively avoid overfitting, make the trained model more robust, and significantly improve the generalization performance. Data augmentation is widely used to improve the performance of image and text classification tasks. Advanced methods such as GAN and conditional GAN ​​have designed and optimized data augmentation schemes for these tasks and achieved good results. However, the impact of data augmentation on personalized federated learning has not been fully studied. Summary of the invention

[0005] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a method for improving the performance of personalized federated learning by data augmentation based on conditional GAN. The invention enables the data augmentation method based on conditional GAN ​​to maximize the performance of the personalized federated learning model.

[0006] Technical solution: In order to achieve the above invention objectives, the present invention provides a method for improving personalized federated learning performance through data augmentation based on conditional GAN, comprising the following steps:

[0007] Step 1: Establish a pFedMe model for personalized federated learning;

[0008] Step 2: Add the conditional GAN ​​model to the pFedMe model in step 1 in a personalized federated learning manner;

[0009] Step 3, obtain the training set of the CIFAR10 dataset and train it through the pFedMe model based on conditional GAN ​​in step 2;

[0010] Step 4: Compare the performance of the test set of the CIFAR10 dataset with the original pFedMe model without data augmentation (i.e., the pFedMe model in step 1), the model trained with repeatedly sampled augmented data, and the pFedMe model trained with the proposed data augmentation in step 3.

[0011] Furthermore, in the present invention: the step 1 also includes the following steps,

[0012] Step 1-1, in traditional federated learning, there are N clients communicating with one server to solve the following problems:

[0013]

[0014] To find a global model ω. Function f i : i=1,...,N represents the expected loss function of the data distribution of client i:

[0015]

[0016] Among them, ξ i is a random data sample drawn from the distribution of client i, and is the loss function corresponding to the sample and ω. In federated learning, the client can have non-independent and non-identical data distributions, that is, ξ i and j The distribution of (j≠i) is different;

[0017] Step 1-2, in the considered federated learning personalization problem, the optimization goal is not to obtain the optimal solution to the problem in step 1-1 above, but to provide a specific model for users with different data distributions. To this end, we use a regularized loss function with an l2 norm for each client, as shown below:

[0018]

[0019] Among them, θ i Represents the personalized model of client i, and λ is the regularization parameter that controls the strength of ω on the personalized model. While larger values ​​of λ can benefit clients with unreliable data from rich data aggregation, smaller values ​​of λ can help clients with sufficient useful data prioritize personalization. Note that λ∈(0,∞) to avoid the extreme cases of λ=0 (i.e., no federated learning) or λ=∞ (i.e., no personalized federated learning). The main idea of ​​the above regularized loss function is to allow clients to constrain their own models from different gradient directions while ensuring that the user's local model does not move far away from the global "reference point" ω. Based on this, the optimization objective of personalized federated learning can be written as the following two-level optimization problem:

[0020] in

[0021] In pFedMe, while the global model ω is found by aggregating data from multiple clients at the outer layer, θ i The data distribution with respect to client i is optimized and internally maintains a bounded distance from ω. i The definition of (ω) is the Moreau envelope method widely used in the optimization field, which is helpful for the design of various learning algorithms. is the best solution to the problem within pFedMe, which is defined as follows:

[0022]

[0023] Steps 1-3 are similar to traditional federated learning algorithms such as FedAvg. In each communication round t, the server broadcasts the latest global model ω to all clients. t Then, after all clients have performed R local updates, the server will update the client from a uniformly sampled subset S t Receive the latest local model for model averaging.

[0024] In the inner optimization, each client i solves the optimal personalization problem in step 1-2 to obtain its personalized model in, represents the local model of client i in global round t and local round r. The purpose of the local model is to help build a global model and reduce the number of communications between the client and the server. Second, in the outer optimization, the local update of client i using gradient descent is about F i (Instead of f i ), as shown below:

[0025]

[0026] Where η is the learning rate, Can be based on Using the current personalization model to calculate;

[0027] For steps 1-4, for the actual algorithm, we use The δ approximation of

[0028] obtained by Usually a gradient is required However, this requires i In practice, by using a small batch of data T i To sample, use the following An unbiased estimate of :

[0029]

[0030] Make Secondly, generally speaking, obtaining The closed-form solution to is not straightforward. Instead, a first-order iterative method is often used to obtain a high-precision approximation. definition:

[0031]

[0032] Assume that λ is chosen so that the loss function is strongly convex, and then gradient descent (Nesterov's accelerated gradient descent) is applied to obtain So that:

[0033]

[0034] In addition, O(·) is a hidden constant. δ can be adjusted by controlling the sampling noise and accuracy level ν using the mini-batch size |T|.

[0035] Furthermore, in the present invention: the step 2 further comprises the following steps,

[0036] Step 2-1, introduce GAN as a framework for training generative models. The GAN generative model can train any kind of generator network. Most other generative models require the generator to have some specific functional form, just like the output must obey a Gaussian distribution. In addition, other generative models, such as variational autoencoders, can learn the distribution of data, but studies have shown that the images generated by such generative models are often blurry. However, GAN can learn a model that only generates sample points close to the real data (neural network layer), that is, the distribution learned by GAN is very close to the real distribution.

[0037] GANs consist of a set of “adversarial” models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample comes from real data rather than the generative model G. Both G and D may be nonlinear mapping functions, such as multilayer perceptrons.

[0038] In order to learn the generator distribution p using data x g , the generator constructs a noise distribution from the prior z The mapping function G(z;o) from (z) to data space g ). Discriminator D(x;o) d ) outputs a scalar indicating that x comes from the real data rather than p g probability.

[0039] Both G and D need to be trained: G’s parameters are adjusted to minimize log(1-D(G(z)) and D’s parameters are adjusted to minimize logD(x), as if they follow a game with value function V(G,D), i.e., the following min-max optimization problem:

[0040]

[0041] Step 2-2, in an unconditional generative model, there is no way to control the class of the data being generated. However, by adjusting the model based on additional information, the data generation process can be guided. This condition can be based on class labels, some prior information, or even data from different modalities.

[0042] GAN can be extended to a conditional model, i.e. conditional GAN, when both the generator and the discriminator are conditioned on some additional information y. y can be any type of auxiliary information, such as class labels or data from other modalities. We can perform conditioning by feeding y as an additional input layer to the discriminator and generator.

[0043] In the generator, the prior input noise p z(z) and y are combined into a joint hidden representation, which provides greater flexibility for the adversarial training framework. In the discriminator, x and y are used as input and discriminant functions. The objective function V(D,G) of the minimum-maximum game between the two is as follows:

[0044]

[0045] In step 2-3, the data of each of the N clients are respectively passed through the trained conditional GAN ​​model, and then the unique local model values ​​of each client are sent to the cloud server, and then the cloud server calculates the aggregated average of these models and sends this value back to the N clients, and each client finally updates its local model value. That is, the trained conditional GAN ​​model is added to the pFedMe model described in step 1 in a personalized federated learning manner, thereby achieving data augmentation and obtaining a pFedMe model based on conditional GAN.

[0046] Furthermore, in the present invention: the step 3 also includes,

[0047] Step 3-1, CIFAR10 is a dataset containing 60,000 color images of size 32*32, and all images belong to 10 different categories. We distribute the complete dataset to N=20 clients;

[0048] Step 3-2: Build a simulation environment that matches the actual environment and generate a data set for personalized federated learning tasks. Each user's data set is randomly sampled to divide 75% and 25% of the data for training and testing, and the data in the training set and the test set do not overlap.

[0049] Step 3-3, first, send the training samples to the pFedMe model based on conditional GAN ​​in step 2, and keep the test set samples the same as in step 3-2, and obtain the accuracy of the data augmentation method of the pFedMe model based on conditional GAN ​​on the CIFAR10 test set;

[0050] In step 3-4, the data is expanded directly by repeatedly sampling and copying the training samples, and then fed into the pFedMe model in step 1, while keeping the CIFAR10 test set samples the same as in step 3-2, and obtaining the accuracy of the repeated sampling data augmentation method on the test set.

[0051] Furthermore, in the present invention: the step 4 also includes,

[0052] Step 4-1, pass the CIFAR10 test set in step 3 directly through the pFedMe model in step 1 without using any data augmentation method;

[0053] In step 4-2, the accuracy in step 4-1 and the accuracy based on the two data augmentation methods in step 3 are compared and analyzed in pairs to obtain a method for improving the performance of personalized federated learning.

[0054] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The present invention obtains a method for improving the performance of personalized federated learning by comparing the effects of not using a data augmentation method and two types of using data augmentation methods on the performance of personalized federated learning;

[0056] (2) The present invention proposes to use a generative model based on conditional GAN ​​to augment the training data set of personalized federated learning. Conditional GAN ​​can be used as a generative model to generate data under fixed conditions. The present invention proposes to add conditional GAN ​​to the personalized federated learning model based on the communication mode of personalized federated learning to achieve data augmentation, rather than simply augmenting the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the overall process of the method for improving personalized federated learning performance by data augmentation based on conditional GAN ​​according to the present invention;

[0058] Figure 2 Schematic diagram of the structure of the pFedMe model based on conditional GAN ​​in the present invention;

[0059] Figure 3 This is a schematic diagram comparing the performance curves of the corresponding personalized federated learning model for the CIFAR10 dataset in the present invention under three conditions: no data augmentation method, repeated sampling data augmentation method, and conditional GAN-based data augmentation method. DETAILED DESCRIPTION

[0060] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0061] like Figure 1 As shown in FIG. 1 , it is a schematic diagram of the overall process of the method for improving the performance of personalized federated learning based on conditional GAN ​​data augmentation proposed in the present invention. The method specifically includes the following steps:

[0062] Step 1, establish a pFedMe model for personalized federated learning with N clients communicating with one server;

[0063] Specifically, step 1 also includes the following steps:

[0064] Step 1-1: Based on traditional federated learning, build a personalized federated learning model pFedMe with better performance;

[0065] Furthermore, the construction of the pFedMe model further includes the following steps:

[0066] In step 1-1-1, N clients communicate with one server to solve the following problems:

[0067]

[0068] To find a global model ω. Function f i : i=1,...,N represents the expected loss function of the data distribution of client i:

[0069]

[0070] Among them, ξ i is a random data sample drawn from the distribution of client i, and is the loss function corresponding to the sample and ω. In federated learning, the client can have non-independent and non-identical data distributions, that is, ξ i and j The distribution of (j≠i) is different;

[0071] In step 1-1-2, a regularized loss function with the l2 norm is used for each client as follows:

[0072]

[0073] Among them, θ i Represents the personalized model of client i, and λ is the regularization parameter that controls the strength of ω on the personalized model. While larger values ​​of λ can benefit clients with unreliable data from rich data aggregation, smaller values ​​of λ can help clients with sufficient useful data prioritize personalization. Note that λ∈(0,∞) to avoid the extreme cases of λ=0 (i.e., no federated learning) or λ=∞ (i.e., no personalized federated learning). The main idea of ​​the above regularized loss function is to allow clients to constrain their own models from different gradient directions while ensuring that the user's local model does not move far away from the global "reference point" ω. Based on this, the optimization objective of personalized federated learning can be written as the following two-level optimization problem:

[0074] in

[0075] Step 1-2, solve the optimal personalized model for each client

[0076] Furthermore, the optimal personalized model The calculation also includes the following steps:

[0077] Step 1-2-1, in pFedMe, although the global model ω is found by aggregating data from multiple clients in the outer layer, θ i The data distribution with respect to client i is optimized and internally maintains a bounded distance from ω. i The definition of (ω) is the Moreau envelope method widely used in the optimization field, which is helpful for the design of various learning algorithms. is the best solution to the problem within pFedMe, which is defined as follows:

[0078]

[0079] Step 1-2-2, for the actual algorithm, we use The δ approximation of

[0080] obtained by Usually a gradient is required However, this requires i In practice, by using a small batch of data T i To sample, use the following An unbiased estimate of :

[0081]

[0082] Make Secondly, generally speaking, obtaining The closed-form solution to is not straightforward. Instead, a first-order iterative method is often used to obtain a high-precision approximation. definition:

[0083]

[0084] Assume that λ is chosen so that the loss function is strongly convex, and then gradient descent (Nesterov's accelerated gradient descent) is applied to obtain So that:

[0085]

[0086] Calculation times Where d is the diameter of the search space, v is the accuracy level, and O(·) is a hidden constant. δ can be tuned by controlling the sampling noise and accuracy level v using the mini-batch size |T|.

[0087] Steps 1-3, solving the global model

[0088] Furthermore, the global model The calculation also includes the following steps:

[0089] Step 1-3-1 is similar to traditional federated learning algorithms such as FedAvg. In each communication round t, the server broadcasts the latest global model ω to all clients. t Then, after all clients have performed R local updates, the server will update the client from a uniformly sampled subset S t Receive the latest local model for model averaging.

[0090] In the inner optimization, each client i solves the optimal personalization problem in step 1-2-1 to obtain its personalized model in, represents the local model of client i in global round t and local round r. The purpose of the local model is to help build a global model and reduce the number of communications between the client and the server. Second, in the outer optimization, the local update of client i using gradient descent is about F i (Instead of f i ), as shown below:

[0091]

[0092] Where η is the learning rate, Can be based on Using the current personalization model to calculate.

[0093] Step 2: Add the conditional GAN ​​model to the pFedMe model in step 1 in a personalized federated learning manner;

[0094] Specifically, step 2 also includes the following steps:

[0095] Step 2-1, introduce GAN as a framework for training generative models. The GAN generative model can train any kind of generator network. Most other generative models require the generator to have some specific functional form, just like the output must obey a Gaussian distribution. In addition, other generative models, such as variational autoencoders, can learn the distribution of data, but studies have shown that the images generated by such generative models are often blurry. However, GAN can learn a model that only generates sample points close to the real data (neural network layer), that is, the distribution learned by GAN is very close to the real distribution.

[0096] GANs consist of a set of “adversarial” models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample comes from real data rather than the generative model G. Both G and D may be nonlinear mapping functions, such as multilayer perceptrons.

[0097] In order to learn the generator distribution p using data x g , the generator constructs a noise distribution from the prior z The mapping function G(z;o) from (z) to data space g ). Discriminator D(x;o) d ) outputs a scalar indicating that x comes from the real data rather than p g probability.

[0098] Both G and D need to be trained: G’s parameters are adjusted to minimize log(1-D(G(z)) and D’s parameters are adjusted to minimize logD(x), as if they follow a game with value function V(G,D), i.e., the following min-max optimization problem:

[0099]

[0100] Step 2-2, in an unconditional generative model, there is no way to control the class of the data being generated. However, by adjusting the model based on additional information, the data generation process can be guided. This condition can be based on class labels, some prior information, or even data from different modalities.

[0101] GAN can be extended to a conditional model, i.e. conditional GAN, when both the generator and the discriminator are conditioned on some additional information y. y can be any type of auxiliary information, such as class labels or data from other modalities. We can perform conditioning by feeding y as an additional input layer to the discriminator and generator.

[0102] In the generator, the prior input noise p z (z) and y are combined into a joint hidden representation, which provides greater flexibility for the adversarial training framework. In the discriminator, x and y are used as input and discriminant functions. The objective function V(D,G) of the minimum-maximum game between the two is as follows:

[0103]

[0104] In step 2-3, the data of each of the N clients are respectively passed through the trained conditional GAN ​​model, and then the unique local model values ​​of each client are sent to the cloud server. The cloud server then calculates the aggregated average of these models and sends this value back to the N clients. Each client finally updates its local model value. The trained conditional GAN ​​model is added to the pFedMe model described in step 1 in a personalized federated learning manner, thereby achieving data augmentation and obtaining a pFedMe model based on conditional GAN, such as Figure 2 shown.

[0105] Furthermore, in this embodiment, the main parameters of the neural network of the conditional GAN ​​in the personalized federated learning task are specifically set as shown in Table 1 below:

[0106] Table 1: Conditional GAN ​​parameter setting table

[0107]

[0108] During the experiment, the ReLu function was used as the activation function of the hidden layer of the conditional GAN ​​neural network; the trained conditional GAN ​​model was Figure 2 It is added to the pFdeMe model to achieve the effect of data augmentation.

[0109] Step 3, train the training set of the CIFAR10 dataset through the pFedMe model based on conditional GAN ​​in step 2; obtain the training set of the CIFAR10 dataset, repeatedly sample this training set to expand the data, and send it to the pFedMe model for training;

[0110] Specifically, step 3 also includes the following steps:

[0111] Step 3-1, CIFAR10 is a dataset containing 60,000 color images of size 32*32, and all photos belong to 10 different categories. Due to the limitation of CIFAR10 data size, we distribute the complete dataset to N=20 clients;

[0112] Step 3-2: Build a simulation environment that matches the actual environment and generate a data set for personalized federated learning tasks. All data sets are randomly split, with 75% and 25% used for training and testing, respectively, and the data in the training set and the test set do not overlap.

[0113] Step 3-3, first, send the training samples to the pFedMe model based on conditional GAN ​​in step 2, and keep the test set samples the same as in step 3-2, and obtain the accuracy of the data augmentation method of the pFedMe model based on conditional GAN ​​on the CIFAR10 test set;

[0114] In step 3-4, the training samples are directly copied to expand the data, and then sent to the pFedMe model in step 1, and the CIFAR10 test set samples are kept the same as in step 3-2, and the accuracy of the data augmentation method of directly copying the data on the test set is obtained.

[0115] like Figure 3As shown in the figure, the total number of global communication rounds is set to 800 during simulation, and the accuracy on the test set is expressed in percentage. From the curve of the data augmentation method based on conditional GAN ​​in the figure, it can be seen that the accuracy of the test set increases with the increase of the number of global communication rounds, and finally reaches 65%; from the test set accuracy curve under the data augmentation method based on repeated sampling in the figure, it can be seen that when the number of global communications is small, there is a large fluctuation. With the increase of the number of communication rounds, the accuracy of the test set shows a smaller functional growth trend, and finally reaches 55% (10% smaller than the data augmentation method based on conditional GAN).

[0116] Step 4, compare and analyze the accuracy of the test set of the CIFAR10 dataset on the original pFedMe model without data augmentation (i.e., the pFedMe model in step 1) and the two data augmented pFedMe models in step 3;

[0117] Specifically, step 4 also includes the following steps:

[0118] Step 4-1, pass the CIFAR10 test set in step 3 directly through the pFedMe model in step 1 without using any data augmentation method;

[0119] In step 4-2, the accuracy in step 4-1 and the accuracy based on the two data augmentation methods in step 3 are compared and analyzed in pairs to obtain a method for improving the performance of personalized federated learning.

[0120] contrast Figure 3 From the test set accuracy curve in , we can see that when the number of global communication rounds is greater than about 180 times, the accuracy of the data augmentation method based on conditional GAN ​​on the test set is significantly higher than the accuracy of the method of directly copying the data to expand the data set, that is, the data augmentation method based on conditional GAN ​​is more effective in improving the performance of personalized federated learning. It can also be seen that the accuracy of the test set without using the data augmentation method and the method of directly copying the data to expand the data set is basically the same, except when the number of global communication rounds is small (about ≤20), the method of directly copying the data to expand the data set even reduces the performance of the original model. It can be seen that the method of directly copying the data to expand the data set cannot achieve real data augmentation, let alone improve model performance. In contrast, we believe that the use of the data augmentation method based on conditional GAN ​​can greatly improve the performance of personalized federated learning.

[0121] The present invention realizes a solution for improving the performance of personalized federated learning by data augmentation based on deep conditional GAN ​​for image processing problems. The method of adding conditional GAN ​​to the model in the form of personalized federated learning for data augmentation improves the performance of personalized federated learning, which has important guiding significance for using conditional GAN ​​for data augmentation.

[0122] It should be noted that the above-described embodiments only express some implementation methods of the present invention, and the description thereof cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the concept of the present invention, and these should all fall within the scope of protection of the present invention.

Claims

1. A method for improving personalized federated learning performance through data augmentation based on conditional GAN, characterized by: The following steps are included: Step 1: Establish a pFedMe model for personalized federated learning; Step 2: Add the conditional GAN ​​model to the pFedMe model in step 1 in a personalized federated learning manner; Step 3, obtain the training set of the CIFAR10 dataset and train it through the pFedMe model based on conditional GAN ​​in step 2; Step 4, the test set of the CIFAR10 dataset is used to compare the performance of the original pFedMe model without data augmentation method, that is, the pFedMe model in step 1, the model trained by repeated sampling augmented data, and the pFedMe model trained by the proposed data augmentation in step 3; The step 1 also includes the following steps: Step 1-1, in traditional federated learning, there are N clients communicating with one server to solve the following problems: To find a global model ω; function f i : Represents the expected loss function of the data distribution of client i: Among them, ξ i is a random data sample drawn from the distribution of client i, and is the loss function corresponding to the sample and ω; in federated learning, the client has non-independent and non-identical data distribution, that is, ξ i and j The distribution of is different, where j≠i; Step 1-2, in the considered federated learning personalization problem, the optimization goal is not to obtain the optimal solution to the problem in step 1-1 above, but to provide a specific model for users with different data distributions; for this purpose, a regularized loss function with an l2 norm is used for each client, as shown below: Among them, θ i represents the personalized model of client i, λ is the regularization parameter that controls the strength of ω on the personalized model; although a larger value of λ benefits clients with unreliable data from rich data aggregation, a smaller value of λ helps clients with sufficient useful data to prioritize personalization; note that λ∈(0,∞) to avoid the extreme cases of λ=0 or λ=∞; the above regularized loss function allows clients to constrain their own models from different gradient directions while ensuring that the user's local model does not move far away from the global "reference point" ω; based on this, the optimization objective of personalized federated learning is written as the following two-level optimization problem: in In pFedMe, while the global model ω is found by aggregating data from multiple clients at the outer layer, θ i The data distribution relative to client i is optimized and internally maintains a bounded distance from ω; F i The definition of (ω) is the Moreau envelope method widely used in the optimization field, which is helpful for the design of various learning algorithms; the optimal personalized model is the best solution to the problem within pFedMe, which is defined as follows: Steps 1-3 are similar to the traditional federated learning algorithm of FedAvg. In each communication round t, the server broadcasts the latest global model ω to all clients. t ; Then, after all clients have performed R local updates, the server will update the client from a uniformly sampled subset S t Receive the latest local model for model averaging; In the inner optimization, each client i solves the optimal personalization problem in step 1-2 to obtain its personalized model in, represents the local model of client i in global round t and local round r; the purpose of the local model is to help build a global model and reduce the number of communications between the client and the server; secondly, in the outer optimization, the local update of client i using gradient descent is about F i As shown below: Where η is the learning rate, according to Using the current personalization model to calculate; Get it Usually a gradient is required However, this requires i The distribution of small batch data T i To sample, use the following An unbiased estimate of : Make Second, a first-order iterative method is used to obtain a high-precision approximation definition: Assume that λ is chosen so that the loss function is strongly convex, and then gradient descent is applied to obtain So that: Calculation times where d is the diameter of the search space, v is the accuracy level, and Ο(·) is a hidden constant; δ is adjusted by controlling the sampling noise and accuracy level v using the mini-batch size |T|.

2. The method for improving personalized federated learning performance by data augmentation based on conditional GAN ​​according to claim 1, characterized in that: The step 2 also includes the following steps: Step 2-1, introduce GAN as a framework for training generative models; GAN generative models train any generator network; in addition, GAN learns a model that only generates sample points close to real data, that is, the distribution learned by GAN is very close to the real distribution; GAN consists of a set of "adversarial" models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that the sample comes from the real data rather than the generative model G; both G and D are nonlinear mapping functions; In order to learn the generator distribution p using data x g , the generator constructs a noise distribution from the prior z The mapping function G(z;o) from (z) to data space g ); Discriminator D(x;o d ) outputs a scalar indicating that x comes from the real data rather than p g probability; Both G and D need to be trained: G’s parameters are adjusted to minimize log(1-D(G(z)) and D’s parameters are adjusted to minimize log D(x), as if they follow a game with value function V(G,D), i.e., the following min-max optimization problem: Step 2-2, in unconditional generative models, there is no way to control the class of the data being generated; however, the data generation process is guided by adjusting the model based on additional information; this condition is based on class labels, certain prior information, or even data from different modalities; GANs extend to conditional models, i.e. conditional GANs, when both the generator and the discriminator are conditioned on some additional information y; y is any type of auxiliary information; we perform conditioning by feeding y into both the discriminator and the generator as an additional input layer; In the generator, the prior input noise p z (z) and y are combined into a joint hidden representation, which provides greater flexibility for the adversarial training framework; in the discriminator, x and y are used as input and discriminant functions; the objective function V(D,G) of the minimum-maximum game between the two is as follows: In step 2-3, the data of each of the N clients are respectively passed through the trained conditional GAN ​​model, and then their unique local model values ​​are sent to the cloud server. The cloud server then calculates the aggregated average of these models and sends this value back to the N clients. Each client finally updates its local model value; that is, the trained conditional GAN ​​model is added to the pFedMe model described in step 1 in a personalized federated learning manner, thereby achieving data augmentation and obtaining a pFedMe model based on conditional GAN.

3. The method for improving personalized federated learning performance by data augmentation based on conditional GAN ​​as claimed in claim 2, characterized in that: The step 3 also includes: Step 3-1, CIFAR10 is a dataset containing 60,000 color images of size 32*32, and all images belong to 10 different categories; distribute the complete dataset to N=20 clients; Step 3-2: Build a simulation environment that matches the actual environment and generate a data set for personalized federated learning tasks. Each user's data set is randomly sampled to divide 75% and 25% of the data for training and testing, and the data in the training set and the test set do not overlap. Step 3-3, first, send the training samples to the pFedMe model based on conditional GAN ​​in step 2, and keep the test set samples the same as in step 3-2, and obtain the accuracy of the data augmentation method of the pFedMe model based on conditional GAN ​​on the CIFAR10 test set; In step 3-4, the data is expanded directly by repeatedly sampling and copying the training samples, and then fed into the pFedMe model in step 1, while keeping the CIFAR10 test set samples the same as in step 3-2, and obtaining the accuracy of the repeated sampling data augmentation method on the test set.

4. The method for improving personalized federated learning performance by data augmentation based on conditional GAN ​​as claimed in claim 3, characterized in that: The step 4 also includes, Step 4-1, pass the CIFAR10 test set in step 3 directly through the pFedMe model in step 1 without using any data augmentation method; In step 4-2, the accuracy in step 4-1 and the accuracy based on the two data augmentation methods in step 3 are compared and analyzed in pairs to obtain a method for improving the performance of personalized federated learning.

Images