Data generation method based on variational autoencoder, unsupervised clustering algorithm and federated learning

Abstract

This paper discloses a data generation method based on a variational autoencoder, an unsupervised clustering algorithm, and federated learning. This method uses a federated learning architecture to jointly train the variational autoencoder of each local client. Furthermore, based on the differences in data domains of different clients, it proposes using an unsupervised clustering algorithm to group the different clients. Federated learning model training is then performed separately within each cluster. This greatly alleviates the harm caused by data domain differences to federated model training, ultimately allowing each group to train a global generative model. The trained global generative model can be used to generate more secure shared data during the prediction phase, providing effective data support for more machine learning and deep learning tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of data privacy security and data generation in deep learning, and in particular relates to a data generation method based on variational autoencoders, unsupervised clustering algorithms and federated learning. Background Art

[0002] Data is the new energy

[0003] With the rapid development of information technologies such as big data, cloud computing, the Internet of Things, and the internet, artificial intelligence technologies, represented by machine learning and deep learning, have entered a period of rapid development, ushering in a new round of technological revolution. Machine learning and deep learning are learning processes that discover patterns in data. Two crucial factors are algorithms and data. Algorithms address the question of "how to learn," while data addresses the question of "where to learn from." Thanks to the rapid development of deep learning, researchers have proposed numerous algorithms to address the "how to learn" problem in various scenarios. However, a common saying in the machine learning community is that "data and features determine the upper limit of machine learning, while models and algorithms only approximate this upper limit." This proves that even the most sophisticated algorithms, without the support of good data, will struggle to achieve the performance needed to solve practical problems. In today's world, data seems ubiquitous. With the rapid development of the internet, massive amounts of data are being continuously produced and stored.

[0004] In the digital economy, data is considered a new energy source, possessing immeasurable value. Compared to oil, data is reusable. While data is undoubtedly abundant today, it is scattered across different companies, individuals, and devices. Data sharing and openness between different systems and organizations are generally low, leading to information silos. Massive amounts of data remain isolated, making it difficult to integrate and unleash their potential. Furthermore, indiscriminate data sharing raises privacy and security concerns.

[0005] Data privacy and security

[0006] In recent years, negative incidents involving user privacy breaches and data leaks have emerged. For example, in 2018, a third-party company collected the personal information of nearly 50 million users through an app, representing a quarter of a country's electorate, a significant number of users. In the same year, a software vulnerability leaked the private photos of 68 million users. This series of incidents has caused users to worry about the privacy and security of their data, and privacy regulators have imposed significant fines. With the public's growing concern for data security and privacy protection, countries have begun to establish data security laws and regulations, enacting data security laws and personal information protection laws, striving to provide legal protection for personal data privacy.

[0007] Federated Learning

[0008] In this context, it's difficult to collect sufficient data for machine learning and deep learning model training. Compared to the traditional method of centrally collecting data from various sources for model training, each party must now train independently. With limited data, training a good model becomes significantly more challenging. Researchers are considering how to effectively integrate and utilize this dispersed data without infringing user privacy. The concept of federated learning was proposed in 2016. Unlike traditional machine learning algorithms, which require all data to be centralized for training, federated learning sends the model to each data owner, where the data is learned locally. The results are then integrated to create the final model. Federated learning allows users to form a federation to train a global model while keeping data local to the client and not sharing it, effectively addressing data privacy and security concerns.

[0009] Federated learning aims to build a federated learning model based on distributed datasets. During model training, model-related information can be exchanged between participants (or in encrypted form), but the original data cannot. This exchange does not expose any private data. Trained federated learning models can be placed on each participant in the federated learning system or shared among multiple parties.

[0010] Horizontal federated learning involves data from different participants with significant feature overlap (horizontally), but with low overlap in data samples (vertically). For example, the participants in federated learning might be two banks serving different regional markets. Their customer bases may differ significantly, but their customer characteristics may overlap significantly due to similar business models.

[0011] Variational Autoencoder

[0012] In the context of massive amounts of data being isolated and forming data silos, we hope to generate more secure and shareable data through a federated learning architecture combined with a generative model called variational autoencoder. Variational autoencoders have a wide range of real-world applications, such as image generation and style transfer. However, like other deep learning models, they require a large amount of data for training, otherwise the generated data quality will be poor.

[0013] With the development of information technologies such as big data, cloud computing, the internet, and the Internet of Things, deep learning-based generative models such as variational autoencoders have demonstrated promising data generation results. However, deep learning typically requires vast amounts of data for model training. While much data exists in real life, due to concerns about data privacy and security, there's no way to directly aggregate this data for centralized model training. For example, after a company reaches a certain stage of development, it inevitably develops multiple business units, each with its own data. However, each unit is like an isolated island, with data from different units disconnected and isolated from each other. This is known as a data silo. Summary of the Invention

[0014] To address these issues, this paper innovatively proposes a data generation method based on variational autoencoders, unsupervised clustering algorithms, and federated learning. Taking into account the differences in data domains across different clients, this method uses an unsupervised clustering algorithm to group different clients. A federated learning model is then trained independently within each cluster, ultimately resulting in a highly effective federated generation model. This federated generation model can be used during the inference phase to generate more secure, shareable data, providing effective data support for a wider range of machine learning and deep learning tasks.

[0015] The specific technical solution for achieving the purpose of the present invention is:

[0016] A data generation method based on a variational autoencoder, an unsupervised clustering algorithm, and federated learning includes the following steps:

[0017] Model training phase

[0018] Step S1: During each round of federated learning communication, the central server randomly selects a proportion of local clients, K1, from all local clients, where K1 can be selected in the range of 10%-50%. The central server then sends the encoder parameters of the central server to the selected local clients to update their encoder parameters.

[0019] Step S2: The selected local client uses the local training set to train the variational autoencoder of the generative model, defines the mean square error loss function and KL divergence as the optimization target, uses gradient descent as the optimization method, and iteratively trains the local model;

[0020] Step S3: After the local training is completed, the selected client uploads the encoder parameters in the local variational decoder to the central server via network communication;

[0021] Step S4: The central server aggregates the encoder parameters uploaded from the local clients and updates the encoder parameters of the central server;

[0022] Step S5: Repeat steps S1 to S4 until all local clients are selected by the central server at least 3-5 times, send the current central server's encoder parameters to all clients, and update the parameters of the local client encoders;

[0023] Step S6: Each local model maps its original data to a low-dimensional space through an encoder, and clusters it using the unsupervised clustering algorithm K-means++ in this low-dimensional space to obtain G1 groups, where G1 can be selected in the range of 3-5. The low-dimensional vectors of each group are averaged and then uploaded to the central server.

[0024] Step S7: After receiving the low-dimensional vectors sent by all local clients, the central server uses the unsupervised clustering algorithm K-means++ to cluster all low-dimensional vectors into G2 groups, where the optional range of G2 is 4-8. The group with the most low-dimensional vectors belonging to a client is used as the group to which the client is assigned.

[0025] Step S8: After the local clients are grouped, the federated learning model training is performed independently in each group;

[0026] Step S9: During each round of communication, the central server of the current group randomly selects local clients with a ratio of K2 from all local clients in the current group, where the optional range of K2 is 40%-80%. The central server of the current group then sends the encoder and decoder parameters of the central server of the current group to the selected local clients to update their encoder and decoder parameters.

[0027] Step S10: The selected clients in each group perform local model training similar to step S2;

[0028] Step S11: The selected client in each group uploads the parameters of the encoder and decoder in the local variational autodecoder to the central server of the current group through network communication transmission;

[0029] Step S12: The central server in each group aggregates the encoder and decoder parameters uploaded from the local clients and updates the encoder and decoder parameters of the central server in the current group;

[0030] Step S13: Repeat steps S9 to S12 until the model of each group converges or reaches a fixed number of communication rounds, stop training, and each group obtains a final global generative model;

[0031] Model prediction stage

[0032] Step S14: Collect N from the standard normal distribution srandom samples, where N s Adjust according to specific business scenarios;

[0033] Step S15: The client of each group uses the decoder of the global generative model to map the random samples into realistic and secure shared data.

[0034] Beneficial effects of the present invention

[0035] (1) The data generation method based on a variational autoencoder, an unsupervised clustering algorithm, and federated learning proposed in this invention, in steps S1-S5, in addition to local model training, the encoder of each client model also utilizes a federated learning architecture. While protecting data privacy and security and without leaving the client's data locally, model-related information can be exchanged between parties without exposing the protected original data. Because this special method can access more data, the encoder of the variational autoencoder can achieve stronger information compression capabilities than local model training alone.

[0036] (2) In the data generation method based on a variational autoencoder, an unsupervised clustering algorithm, and federated learning proposed in the present invention, in step S6, an encoder with strong information compression capabilities is used to map the original data to a low-dimensional space while also carrying noise sampled from a standard normal distribution. This not only obtains a low-dimensional vector reflecting the data information, but also ensures that the central server cannot infer the original data from the low-dimensional vector, thereby effectively protecting the privacy and security of the data.

[0037] (3) The data generation method based on variational autoencoder, unsupervised clustering algorithm and federated learning proposed in the present invention, in the said step S7, considering that the data of different clients are very likely to be non-independent and identically distributed, which will significantly reduce the performance of the entire federated learning model, the present invention proposes to use the information of the low-dimensional vector extracted by the local client and use the unsupervised clustering algorithm K-means++ to group the clients, and divide the clients with similar data distribution into the same group, and divide the clients with large differences in data distribution into different groups.

[0038] (4) The data generation method based on variational autoencoders, unsupervised clustering algorithms, and federated learning proposed by the present invention, in the steps S8-S13, independently performs federated learning model training in each group. Since the clients in each group perform federated learning training under conditions of similar data distribution, this can largely alleviate the harm caused by data distribution differences to federated model training. The encoder parameters and decoder parameters of the variational autoencoder of the generative model in each group can be more effectively optimized. While improving the information compression capability of the encoder, the decoder also utilizes more data information to obtain stronger data generation capabilities. This is conducive to improving the performance of the central server generation model in each group and enhancing generalization capabilities.

[0039] (5) The data generation method based on variational autoencoders, unsupervised clustering algorithms, and federated learning proposed in the present invention can, in steps S14-S15, utilize the data generation capabilities of the decoders of a specific group of central servers according to different common needs to generate a large amount of realistic, secure, and shareable data.

[0040] (6) This paper proposes a data generation method based on a variational autoencoder, an unsupervised clustering algorithm, and federated learning. Previously, no work has combined variational autoencoders and federated learning. The training method proposed in this paper can cleverly utilize data scattered across various locations, effectively improving the variational autoencoder's ability to generate data.

[0041] (7) Currently, due to the generally low degree of openness in data sharing between different systems and organizations, information islands are created. Randomly sharing data for model training raises concerns about user privacy and security, and can seriously violate laws and regulations. To address these issues, the method of the present invention integrates the federated learning architecture into the variational autoencoder model, generating a large amount of secure, shareable, and valuable data, providing effective data support for more machine learning and deep learning tasks.

[0042] (8) The data collected by different devices may have obvious differences. For example, different preferences of the collectors and different geographical locations will result in different styles of photos taken. Federated learning is largely limited by the differences in data distribution between different participants. If the data distribution between the participants is very different, the performance of the federated learning model will be greatly reduced. The present invention cleverly combines the characteristics of variational autoencoders, performs K-means++ clustering on the low-dimensional vectors of the secure and non-disclosed original data extracted by different participants, groups the different participants, and then conducts federated learning training in different groups. This can alleviate the problems caused by inconsistent data distribution to the optimization of federated learning training, and ultimately obtain a federated generation model with good data generation effect. In the inference stage, the federated generation model can be used to generate more secure and shareable data, providing effective data support for more machine learning and deep learning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a model diagram of the variational autoencoder used by the central server and local client of the present invention;

[0044] Figure 2 It is a framework diagram of the present invention;

[0045] Figure 3 It is a flow chart of the prediction stage of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below in conjunction with the following specific implementation process and drawings, including the conditions, processes, training methods, etc. for implementing the present invention. Except for the contents specifically mentioned below, all of them are common knowledge in the field and are not particularly limited by the present invention.

[0047] The present invention proposes a data generation method based on variational autoencoders, unsupervised clustering algorithms, and federated learning, which specifically includes the following steps:

[0048] Assume there are N local clients {P1, P2, ..., P N}, they each have training data {D1,D2,…,D N}, and a central server C is also required in federated learning. The central server C does not have a dataset and is mainly used to collaborate with the client to complete the model training task. At the model level, the participant P i With a variational autoencoder M i , and the central server C has a variational autoencoder M g , all the variational autoencoders involved in this invention have the same structure, which consists of an encoder and a decoder. The encoder is a multi-layer convolutional neural network, and the decoder is also a multi-layer convolutional neural network. Let the parameters of the encoder on the local client be θe , the decoder parameters are θ d , the parameters of the encoder on the central server C are θ ge , the decoder parameters are θ gd In fact, the present invention does not have too many restrictions on the specific network structure of the encoder and decoder, as long as it can meet the model architecture of the variational autoencoder. The specific model structure of the variational autoencoder is as follows: Figure 1 As shown, the framework of the method of the present invention is as follows Figure 2 shown.

[0049] The goal of this invention is to train a federated data generation model that generates data effectively. This model can be used during the prediction phase to generate more secure, shareable data, providing effective data support for more machine learning and deep learning tasks.

[0050] Model training phase

[0051] Randomly initialize the encoder parameters in the central server variational autoencoder to θ ge , the decoder parameters are θ gd .

[0052] Step S1: In each round of communication in federated learning, the central server randomly selects local clients with a proportion of K1 from all local clients, where the optional range of K1 is 10%-50%. Let the selected local client set P = {P1, P5, ..., P N-2}, and the encoder parameters θ of the central server ge Send it to the clients in the set P. After receiving the encoder parameters, the local client will set the encoder parameters θ of its local variational autoencoder e Update to θ ge ;

[0053] Step S2: The clients in the set P use local data to train the variational autoencoder. This training process is a parameter optimization process, with a client P k For example, in its variational autoencoder model, the encoder parameter is θ e and the decoder parameters are θ d On the basis of , we minimize KL divergence and reconstruction loss, and the corresponding optimization objectives are as follows:

[0054]

[0055] Where x is the input of the variational autoencoder, x passes through the encoder to obtain the mean and variance, and then z is sampled through this normal distribution, and z passes through the decoder to obtain the output represents the mean square error loss between input and output, Representation distribution KL divergence distance from the standard normal distribution N(0,I);

[0056] The present invention uses stochastic gradient descent (SGD) to optimize the above objective function, with a learning rate of 0.01, a batch size (batch_size) of 64, and a dimension of the low-dimensional vector z of 32 dimensions. 5 Epochs are trained locally, during which the encoder parameter θ of the variational autoencoder model is e and decoder parameters θ d Continuously updated and optimized, the encoder's information compression capability is improved, and the decoder's data generation capability is improved;

[0057] Step S3: After the local model training in step S2, the clients in the set P transmit the parameters θ of the encoder in the local variational decoder model through network communication transmission. e Upload to central server C;

[0058] Step S4: The central server aggregates the encoder parameters uploaded from the local clients and updates the encoder parameters θ of the central server ge .

[0059]

[0060] in Indicates client s i The encoder parameters of the variational autoencoder are Indicates client s i The number of samples in the dataset;

[0061] Step S5: Repeat steps S1 to S4 until all local clients are selected by the central server at least 3-5 times. Since the clients are randomly selected in a certain proportion in each round, in order to group the N clients more accurately, it is necessary to ensure that each client is selected at least 3-5 times. ge Sent to all clients, the local client's encoder updates the parameters θ e =θ ge ;

[0062] Step S6: Each client uses an encoder to map its local raw data x to a low-dimensional vector z. All the low-dimensional vectors obtained locally are denoted as SetZ and clustered using the unsupervised clustering algorithm K-means++. The process is as follows:

[0063] (1) Randomly select a sample point from the set SetZ as the first initial cluster center;

[0064] (2) Then calculate the shortest distance between each sample and the current cluster center, represented by D(z), and select the sample point corresponding to the maximum D(z) in the set SetZ as the next cluster center;

[0065] (3) Repeat the process of (1) and (2) until G1 cluster centers are selected;

[0066] (4) According to the principle of minimizing the distance to the cluster center, all sample points are divided into the class where each center point is located, and the mean of all sample points in G1 classes is calculated as the G1 center points of the second iteration;

[0067] (5) Repeat step (4) until the center point no longer changes or the specified number of iterations is reached, and the clustering process ends;

[0068] Through K-means++ clustering, SetZ in each client is divided into G1 groups, all low-dimensional vectors of each group are averaged, and then the obtained G1 average low-dimensional vectors are uploaded to the central server, where the optional range of G1 is 3-5.

[0069] Step S7: After receiving the low-dimensional vectors sent by all local clients, the central server uses the unsupervised clustering algorithm K-means++ to cluster all low-dimensional vectors into G2 groups, where the optional range of G2 is 4-8. i The group with the most low-dimensional vectors is considered the client's category. In operations S6-S7, an encoder with strong information compression capabilities is used to map the original data into low-dimensional vectors while also incorporating noise sampled from a standard normal distribution. This not only yields low-dimensional vectors that reflect the data information, but also ensures that the central server cannot infer the original data from the low-dimensional vectors, effectively protecting data privacy and security.

[0070] Step S8: After the local clients are grouped, the federated learning model training is performed independently in each group;

[0071] Step S9: During each round of communication, the central server C of the current group randomly selects local clients with a ratio of K2 from all local clients of the current group, where the optional range of K2 is 40%-80%, and then sets the encoder parameter θ of the variational autoencoder of the central server C of the current group to ge and decoder parameters θ gd Sent to the local client to update the encoder parameters θ of the local client e and decoder parameters θ d ;

[0072] Step S10: The selected clients in each group perform local model training similar to step S2;

[0073] Step S11: The selected client in each group transmits the encoder parameters θ trained in this round in the local variational decoder through network communication. e and decoder parameters θ d Upload to the central server of the current group;

[0074] Step S12: The central server in each group aggregates the encoder parameters θ uploaded from the local clients e and decoder parameters θ d , the parameter aggregation method is the same as step S4, used to update the encoder parameters θ of the central server of the current group ge and decoder parameters θ gd ;

[0075] Step S13: Repeat steps S9-S12 until each group's model converges or reaches a fixed number of communication rounds. Because each client trains the federated model with clients with similar data distributions, this greatly alleviates the harm caused by data domain differences to federated model training. Therefore, each group can train and obtain a final global generative model.

[0076] Model prediction stage

[0077] Step S14: Collect N from the standard normal distribution s random samples, where N s Adjust according to specific business scenarios;

[0078] Step S15: The clients of each group use the global generative model M g The decoder maps the sample set Z into a realistic and secure data set X that can be shared. The prediction process is as follows Figure 3 shown.

Claims

1. A data generation method based on variational autoencoders, unsupervised clustering algorithms, and federated learning, characterized in that: Generate data as follows: Model training phase Step S1: During each round of federated learning, the central server randomly selects a proportion of local clients, K1, from all local clients, and then sends the encoder parameters of the central server to the selected local clients to update their encoder parameters; the selection range of K1 is 10%-50%. Step S2: The selected local client uses the local training set to train the variational autoencoder of the generative model, defines the mean square error loss function and KL divergence as the optimization target, uses gradient descent as the optimization method, and iteratively trains the local model; Step S3: After the local training is completed, the selected client uploads the encoder parameters in the local variational decoder to the central server via network communication; Step S4: The central server aggregates the encoder parameters uploaded from the local clients and updates the encoder parameters of the central server; Step S5: Repeat steps S1 to S4 until all local clients are selected by the central server at least 3-5 times, send the current central server's encoder parameters to all clients, and update the parameters of the local client encoders; Step S6: Each local model maps its original data to a low-dimensional space through an encoder, and clusters it using the unsupervised clustering algorithm K-means++ in this low-dimensional space to obtain G1 groups. The low-dimensional vectors of each group are averaged, and the resulting low-dimensional vectors are uploaded to the central server. G1 is 3-5. Step S7: After receiving the low-dimensional vectors sent by all local clients, the central server uses the unsupervised clustering algorithm K-means++ to cluster all low-dimensional vectors into G2 groups. The group with the most low-dimensional vectors belonging to a client is used as the group to which the client is assigned. G2 is 4-8. Step S8: After the local clients are grouped, the federated learning model training is performed independently in each group; Step S9: During each round of communication, the central server of each group randomly selects local clients with a ratio of K2 from all local clients in the group, and then sends the encoder and decoder parameters of the central server of the group to the selected local clients to update their encoder and decoder parameters; the selection range of K2 is 40%-80%. Step S10: The selected clients in each group perform local model training similar to step S2; Step S11: The selected client in each group uploads the parameters of the encoder and decoder in the local variational autodecoder to the central server of the current group through network communication transmission; Step S12: The central server in each group aggregates the encoder and decoder parameters uploaded from the local clients and updates the encoder and decoder parameters of the central server in the current group; Step S13: Repeat steps S9 to S12 until the model of each group converges or reaches a fixed number of communication rounds, stop training, and each group obtains a final global generative model; Model prediction stage Step S14: Collect N from the standard normal distribution s random samples, where N s Adjust according to specific business scenarios; Step S15: The client of each group uses the decoder of the global generative model to map the random samples into realistic and secure shared data.

2. The data generation method based on variational autoencoder, unsupervised clustering algorithm and federated learning according to claim 1, characterized in that: Step S6 specifically includes: each client maps its local original data x to a low-dimensional vector z through an encoder; all the low-dimensional vectors obtained locally are recorded as SetZ, and clustered using the unsupervised clustering algorithm K-means++, the process is as follows: (1) Randomly select a sample point from the set SetZ as the first initial cluster center; (2) Then calculate the shortest distance between each sample and the current cluster center, represented by D(z), and select the sample point corresponding to the maximum D(z) in the set SetZ as the next cluster center; (3) Repeat the process of (1) and (2) until G1 cluster centers are selected; (4) According to the principle of minimizing the distance to the cluster center, all sample points are divided into the class where each center point is located, and the mean of all sample points in G1 classes is calculated as the G1 center points of the second iteration; (5) Repeat step (4) until the center point no longer changes or the specified number of iterations is reached, and the clustering process ends; Through K-means++ clustering, SetZ in each client is divided into G1 groups, all low-dimensional vectors of each group are averaged, and then the obtained G1 average low-dimensional vectors are uploaded to the central server, where G1 is 3-5.

Images