Longitudinal federal learning data enhancement method based on data isomerism

Abstract

The invention belongs to the field of federated learning, and particularly relates to a data isomerism-based longitudinal federated learning data enhancement method, which comprises the following steps that: each client locally trains a local condition to generate an adversarial network for generating new training data and uploading the new training data to a central server; the central server carries out processing according to the training data uploaded by each client to obtain a global model, and issues the global model to each client; each client takes the global model as a local model, obtains updated local model parameters through training by an index moving average (SGD) method with a double smoothing mechanism, and uploads the updated local model parameters to the central server; the central server calculates the aggregation weight of each client according to the sample proportion and the data distribution; the central server aggregates the local model parameters uploaded by the clients according to the aggregation weights to obtain a new global model and issues the new global model to the clients; repeating the above contents until the global model converges; according to the method, the problem of data enhancement of federal learning in a heterogeneous data environment can be effectively solved.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning, and specifically relates to a vertical federated learning data enhancement method based on data heterogeneity. Background Art

[0002] As a distributed machine learning framework, federated learning can support multiple participants to collaboratively train models without directly sharing data, effectively protecting data privacy and promoting cross-organizational and cross-device learning cooperation. In particular, the vertical federated learning model is good at integrating data features from different participants, making cross-domain data cooperation possible. However, this model also faces the challenges of uneven data volume and differences in data distribution, which makes it extremely difficult to build a global model that is universally applicable to all participants.

[0003] At present, the research on data enhancement in federated learning can be roughly divided into three types. The first type focuses on the application and coordination of data enhancement strategies between different participants; the second type of research direction is to ensure that the data enhancement process does not leak sensitive information; the third type is to dynamically adjust the data enhancement strategy according to the progress of model training. Combined with the current research on data enhancement in federated learning, it is found that many scholars have achieved some results in data enhancement, but there are still some technical problems:

[0004] 1) Client data imbalance problem: In multiple data sources and heterogeneous environments, local data imbalance may cause some participants to contribute less in model training. Using traditional generative adversarial networks for processing cannot achieve true data enhancement. During the training process, it may also cause model crashes, unstable training, and reduced quality of generated data, thereby affecting the performance of the global model.

[0005] 2) Data heterogeneity leads to differences in local data distribution: In a heterogeneous environment, local data distribution may vary greatly. Existing solutions only aggregate parameters based on data distribution differences, without measuring the differences in distribution of each data source. At the same time, the scale of clients participating in federated learning is huge, and the use of traditional aggregation algorithms such as FedAvg will result in training drift, resulting in a decline in the performance of the global model.

[0006] 3) The problem of unstable model parameters caused by data heterogeneity: Due to the influence of data heterogeneity, the data distribution and characteristics of different participants are different. The traditional stochastic gradient descent method may experience large fluctuations when updating model parameters, resulting in fluctuations and inconsistencies in model parameters during the aggregation process, which in turn affects the performance of the global model. Summary of the invention

[0007] To solve the above problems, the present invention provides a vertical federated learning data enhancement method based on data heterogeneity, comprising the following steps:

[0008] S1. Each client trains a local conditional generative adversarial network locally to generate new training data; each client uploads the generated training data to the central server;

[0009] S2. The central server processes the training data uploaded by each client to obtain a global model, and sends the global model to each client;

[0010] S3. Each client uses the global model as the local model, trains the updated local model through the exponential moving average SGD method with a double smoothing mechanism, and uploads the model parameters to the central server;

[0011] S4. The central server calculates the aggregation weight of each client according to the sample ratio and data distribution;

[0012] S5. The central server aggregates the local model parameters uploaded by each client according to the aggregation weight to obtain a new global model;

[0013] S6. Determine whether the global model has converged. If so, end the federated learning. Otherwise, send the new global model to each client and execute step S3.

[0014] Beneficial effects of the present invention:

[0015] The present invention solves the problem of model performance degradation caused by data imbalance among clients through the local generative adversarial network CWGAN-GP technology, and provides more stable gradients, reduces mode collapse and better convergence, further improving the performance and generalization ability of the model; by calculating weights using the sample size ratio of the participants and the distribution difference between the model parameters and the global model parameters, the model parameters are aggregated to optimize the global model; by utilizing the stochastic gradient method and combining the exponential moving average to process the gradient and the parameters successively, the gradient update is smoother and the parameter update is more stable, thereby ensuring the stability of the local model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1 The present invention provides a data enhancement method for vertical federated learning based on data heterogeneity, which specifically includes the following steps:

[0019] S1. Each client trains a local conditional generative adversarial network locally to generate new training data; each client uploads the generated training data to the central server.

[0020] Specifically, in step S1, each client locally trains a local conditional generative adversarial network CWGAN-GP to generate new training data for a small number of clients to achieve data balance; the process of any client locally training a local conditional generative adversarial network includes:

[0021] S11. The client first randomly samples the noise.

[0022] S12. Input the random sampling results into the generator G to generate new data

[0023] S13. Use the discriminator D to verify the probability that the new data is real data, that is, use the discriminator D to verify the generated new data Does it have the same probability distribution as the real data x? The discriminator D evaluates the new data given the conditional information y.

[0024] S14. The client optimizes the following objective function by maximizing the discriminative ability of the discriminator D and minimizing the probability that the new data generated by the generator G is discriminated as generated data:

[0025]

[0026] in, is the actual data distribution and generate data distribution The mixed distribution between is from The interpolation points sampled in .

[0027] In the objective function, the gradient penalty term It ensures that the gradient of the discriminator D has a reasonable norm on the boundary of the real data and the generated data, thus providing better gradient stability.

[0028] S15. Repeatedly iterate the generator and discriminator to minimize the generator loss L(G) and the discriminator loss L(D) until the data generated by the generator is similar to the real data;

[0029] The generator loss is

[0030]

[0031] The discriminator loss is

[0032]

[0033] Among them, G represents the generator, D represents the discriminator, represents the real data distribution, represents the generated data distribution, y represents the conditional information, D(x|y) represents the score of the discriminator on the data x under the given conditional information y, Express expectations, Indicates that the discriminator D generates data under given conditional information y (y is the true label) assessment, It means that under given conditional information y, the discriminator D is Data extracted from Evaluation, ||·||2 represents the data The L2 norm of the gradient vector on the discriminator D, represents the gradient of the discriminator output, and λ represents the gradient penalty coefficient.

[0034] Through the above process, each client can not only generate new training data, but also ensure the stability of the training process through the gradient penalty term, thereby improving the data balance and model performance of the entire system.

[0035] S2. The central server processes the training data uploaded by each client to obtain a global model, and sends the global model to each client.

[0036] Specifically, step S2 specifically includes:

[0037] S21. The central server receives the training data uploaded by each client and normalizes it.

[0038] The normalized processing is expressed as

[0039]

[0040] Among them, X represents a sample in the training data uploaded by each client, X min represents the minimum sample in the training data uploaded by each client, X maxrepresents the maximum sample in the training data uploaded by each client, and X' represents the normalization result.

[0041] Due to the influence of data heterogeneity, the data distribution of each client is quite different. Directly merging the training data uploaded by each client to train the global model may cause the performance of the global model to deteriorate. The global model may not be able to adapt to all different data distributions at the same time, resulting in insufficient generalization ability. Normalizing the data can effectively reduce the differences between distributions.

[0042] S22. Aggregate the normalized training data to form a globally shared synthetic dataset. The synthetic dataset not only retains the unique data characteristics of each client, but also enhances the richness of the dataset through cross-client diversity. The central server can maximize the use of the data characteristics of each client, which helps to improve the generalization performance of the global model.

[0043] S23. The central server uses the stochastic gradient descent algorithm to optimize the synthetic data set, trains a global model, and sends the global model to each client.

[0044] S3. Each client uses the global model as the local model, trains the updated local model through the exponential moving average SGD method with a double smoothing mechanism, and uploads the model parameters to the central server.

[0045] Specifically, any client obtains updated local model parameters through the exponential moving average SGD method with a double smoothing mechanism, including:

[0046] S31. In the tth iteration of client training, a small batch of data is randomly selected from the client local training data set for training, and the gradient information g is calculated based on the training results. t , expressed as

[0047]

[0048] in, represents the mini-batch data extracted at the t-th iteration, Represents the amount of small batch data, It represents the gradient of the loss function of the i-th sample in the mini-batch data to the local model parameters. In a data heterogeneous environment, each mini-batch data may have large differences, and the volatility of the gradient increases, resulting in unstable parameter updates.

[0049] S32. According to the gradient information g t Update the momentum m t , expressed as

[0050] m t =β.m t-1+(1-β)·g t

[0051] Among them, m t-1 represents the momentum of the t-1th iteration update; β represents the attenuation factor of the gradient moving average, ranging from 0 to 1, which determines the influence of the momentum of the previous iteration on the momentum of the current iteration. A larger β value indicates a stronger dependence on historical momentum.

[0052] In the initial stage of the algorithm, momentum m0 is initialized to a zero vector or zero matrix. After the first few iterations, the effect of momentum gradually emerges and then begins to play a smoothing role. Momentum reduces the random noise in each update by accumulating historical gradients. This cumulative effect makes the gradient direction more stable and avoids drastic changes caused by data heterogeneity.

[0053] S33. According to momentum m t Update local model parameters θ t , expressed as

[0054] θ t =θ t-1 -η·m t

[0055] Among them, θ t-1 represents the local model parameters updated at the t-1th iteration, and η is the learning rate;

[0056] S34. According to the local model parameter θ t Calculate the moving average θ MA,t , expressed as

[0057] θ MA,t =h.θ MA,t-1 +(1-h)·θ t

[0058] Among them, θ MA,t-1 represents the moving average of the t-1th iteration update; h represents the attenuation factor of the parameter moving average, which further smoothes the parameter update path and reduces the parameter fluctuation of the model during training. This smoothing effect makes the model more robust.

[0059] S35. When the local model training is completed after the maximum number of iterations, the current moving average θ MA,T Upload to the central server. T represents the maximum number of times.

[0060] By combining momentum and parameter moving average, the convergence speed of the model is improved, and the model's adaptability to data heterogeneity is enhanced. This enables the model to maintain high stability and robustness in the face of complex and changing training environments, and avoid falling into local optimal solutions.

[0061] S4. The central server calculates the aggregation weight of each client based on the sample ratio and data distribution.

[0062] Specifically, step S4 includes:

[0063] S41. The central server calculates the relative contribution value of each participating client, expressed as

[0064]

[0065] Among them, c j represents the relative contribution value of client j, N j Represents the total number of real samples of client j, and N represents the total number of samples of all participating clients; the relative contribution value can ensure that participants with more data have a greater impact on the global model. This helps to prevent participants with less data from having too much influence on the global model.

[0066] S42. The central server calculates the Wasserstein distance between the data distribution of each participating client and the data distribution of the global model, expressed as

[0067]

[0068] Among them, W(P j ,P g ) indicates P j To P g The Wasserstein distance, P j represents the data distribution of participant j, P g represents the data distribution of the global model, represents the infimum of all joint distributions γ, where γ represents the distribution from P j To P g The joint distribution of Represents sample z to sample The distance between P j The sample drawn from Indicates that from P g The sample drawn from

[0069] S43. The central server calculates the data distribution adjustment factor of each participating client, expressed as

[0070] α j =exp(-μ·W(P j ,P g ))

[0071] Among them, α jrepresents the data distribution adjustment factor of client j, and μ is a parameter that controls the influence of the data distribution adjustment factor. The data distribution adjustment factor reflects the difference between the local data distribution of the client and the global data distribution, and quantifies the difference between heterogeneous data distributions. By calculating the data adjustment factor to reduce the difference in data distribution, the model training strategy can be adjusted to improve the performance and generalization ability of the global model on each client.

[0072] S44. Calculate the aggregate weight of each participating client based on the relative contribution value and the data distribution adjustment factor, expressed as

[0073]

[0074] Among them, ε j represents the aggregation weight of client j, and K represents the number of participating clients.

[0075] By performing softmax normalization, the central server can ensure that the contribution weight of each participant is reasonably considered in the global model update, and integrate the data of each client in a more balanced way, thereby improving the robustness and accuracy of the global model.

[0076] S5. The central server aggregates the local model parameters uploaded by each client according to the aggregation weight to obtain a new global model.

[0077] Specifically, step S5 obtains new global model parameters expressed as

[0078]

[0079] in, represents the global model parameters obtained from the lth round of iterative training, Represents the model parameters uploaded by client j. The present invention can measure the differences in local data distribution of multiple clients by calculating the aggregation weight of the client, prevent the occurrence of learning divergence, and enable the global model to better fit different data distributions.

[0080] S6. Determine whether the global model has converged. If so, end the federated learning. Otherwise, send the new global model to each client and execute step S3.

[0081] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0082] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A data enhancement method for vertical federated learning based on data heterogeneity, characterized in that: The following steps are involved: S1. Each client trains a local conditional generative adversarial network locally to generate new training data; Each client uploads the training data it generates to the central server; S2. The central server processes the training data uploaded by each client to obtain a global model, and sends the global model to each client; S3. Each client uses the global model as the local model, trains the updated local model through the exponential moving average SGD method with a double smoothing mechanism, and uploads the model parameters to the central server; S4. The central server calculates the aggregation weight of each client according to the sample ratio and data distribution; S5. The central server aggregates the local model parameters uploaded by each client according to the aggregation weight to obtain a new global model; S6. Determine whether the global model has converged. If so, end the federated learning. Otherwise, send the new global model to each client and execute step S3.

2. According to claim 1, a method for data enhancement based on vertical federated learning of data heterogeneity is characterized in that: Step S1: Each client locally trains a local conditional generative adversarial network, including: Randomly sample the noise, and input the random sampling result into the generator to generate new data; use the discriminator to verify the probability that the new data is real data; Construct an objective function to maximize the discriminative ability of the discriminator and minimize the probability that the new data generated by the generator is discriminated as generated data as the optimization goal; The generator and discriminator are repeatedly iterated to minimize the generator loss and the discriminator loss until the data generated by the generator is similar to the real data.

3. According to the data enhancement method for vertical federated learning based on data heterogeneity according to claim 2, it is characterized in that: The objective function is expressed as The generator loss is The discriminator loss is Among them, G represents the generator, D represents the discriminator, represents the real data distribution, represents the generated data distribution, Represents the actual data distribution and generate data distribution y represents the conditional information, D(x|y) represents the score of the discriminator on the real data x under the given conditional information y, Express expectations, Indicates that the discriminator D generates data under given condition information y assessment, It means that under given conditional information y, the discriminator D is Extracted data , ||·||2 represents the L2 norm, represents the gradient of the discriminator output, and λ represents the gradient penalty coefficient.

4. According to the method for data enhancement based on vertical federated learning of data heterogeneity in claim 1, it is characterized in that: Step S2 specifically includes: S21. The central server receives the training data uploaded by each client and normalizes it; S22. Aggregate the normalized training data to form a globally shared synthetic dataset; S23. The central server uses the stochastic gradient descent algorithm to optimize the synthetic data set, trains a global model, and sends the global model to each client.

5. According to the method for data enhancement based on vertical federated learning of data heterogeneity in claim 1, it is characterized in that: Any client obtains updated local model parameters through the exponential moving average SGD method with double smoothing mechanism, including: S31. In the tth iteration of client training, a small batch of data is randomly selected from the client local training data set for training, and the gradient information g is calculated based on the training results. t , expressed as in, represents the mini-batch data extracted at the t-th iteration, Represents the amount of small batch data, Represents the gradient of the loss function of the i-th sample in the mini-batch data with respect to the local model parameters; S32. According to the gradient information g t Update the momentum m t , expressed as m t =b.m t-1 +(1-β)·g t Among them, β represents the attenuation factor of the gradient moving average, m t-1 represents the momentum of the t-1th iteration update; S33. According to momentum m t Update local model parameters θ t , expressed as i t =θ t-1 -η·m t Among them, θ t-1 represents the local model parameters updated at the t-1th iteration, and η is the learning rate; S34. According to the local model parameter θ t Calculate the moving average θ MA,t , expressed as i MA,t =h.θ MA,t-1 +(1-h)·θ t Among them, h represents the attenuation factor of the parameter moving average, θ MA,t-1 Represents the moving average updated at the t-1th iteration; S35. When the local model training is completed after the maximum number of iterations, the current moving average is uploaded to the central server.

6. According to the method for data enhancement based on vertical federated learning of data heterogeneity in claim 1, it is characterized in that: Step S4 includes: S41. The central server calculates the relative contribution value of each participating client, expressed as Among them, c j represents the relative contribution value of client j, N j represents the total number of true samples of client j, and N represents the total number of samples of all participating clients; S42. The central server calculates the Wasserstein distance between the data distribution of each participating client and the data distribution of the global model, expressed as Among them, W(P j ,P g ) indicates P j To P g The Wasserstein distance, P j represents the data distribution of participant j, P g represents the data distribution of the global model, represents the infimum of the joint distribution γ, where γ represents the j To P g The joint distribution of Represents sample z to sample The distance between P j The sample drawn from Indicates that from P g The sample drawn from S43. The central server calculates the data distribution adjustment factor of each participating client, expressed as a j =exp(-μ·W(P j ,P g )) Among them, α j represents the data distribution adjustment factor of client j, and μ is a parameter that controls the influence of the data distribution adjustment factor; S54. Calculate the aggregate weight of each participating client based on the relative contribution value and the data distribution adjustment factor, expressed as Among them, ε j represents the aggregation weight of client j, and K represents the number of participating clients.

7. The method for data enhancement based on vertical federated learning of data heterogeneity according to claim 1, characterized in that: Step S5: The central server aggregates the local model parameters uploaded by each client according to the aggregation weight to obtain a new global model, which is expressed as in, represents the global model parameters obtained from the lth round of iterative training, ε j represents the aggregation weight of client j, and K represents the number of participating clients; Represents the model parameters uploaded by client j.

Images