Federal learning method based on alignment representation

Abstract

The invention belongs to the technical field of data privacy protection, and particularly relates to a federal learning method based on alignment representation. According to the method, a model is divided into a presentation layer and a decision-making layer aiming at the problem of data isomerism of federal learning; the representation layer is responsible for mapping the sample vector into a representation vector, and the decision-making layer is responsible for mapping the representation vector into a probability vector; a local representation vector is defined as a vector obtained by mapping a sample vector through a representation layer of a client model, a global average representation vector is defined as a vector obtained by mapping the sample vector through a local model parameter weighted mean value, and alignment is that the local representation vector and the global average representation vector are matched; enabling the presentation layer of the local model to learn local data and also learn global information; alignment is achieved by reducing the distance between the local representation vector and the average representation vector. In the training process, the distance between the local representation and the global representation is continuously reduced, and through multiple iterations, each client can obtain a model with higher precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of data information privacy protection, and specifically relates to a federated learning method based on alignment representation. Background Art

[0002] With the rapid increase in data volume and the need for privacy protection, federated learning has developed into a very promising direction. Personalized federated learning is a type of federated learning that trains customized models for clients. It is generally composed of many clients that can only access their own private data and a central server that can coordinate the learning process but cannot access any original data. The goal of personalized federated learning is generally to train personalized models for each client by collaboratively utilizing private data stored on multiple clients without explicitly sharing private data. One challenge faced by personalized federated learning is the decrease in the accuracy of the model generated by data heterogeneity. Data heterogeneity means that in federated learning, data comes from different users, enterprises, and scenarios, resulting in a large difference in the distribution of data held by each client. Data heterogeneity results in large differences in the model parameter updates of each client in each round of local training, which will affect the result of the server integrating the model parameters. The client then uses the integrated result to update the personalized model, which in turn affects the accuracy of the personalized model. A possible solution to the challenge of data heterogeneity is to add a regularization term to the local objective function during model training, as shown in the following formula:

[0003]

[0004] Among them, w i * is the result of the model parameters of this round of local training, argmin is to find the minimum value, L is the loss function, w i is a model parameter, λ is a hyperparameter, It is the result of the server integrating model parameters in the previous round. This can reduce the differences in model parameter updates of each client, thereby alleviating the decline in the accuracy of personalized models caused by data heterogeneity. Summary of the invention

[0005] The purpose of the present invention is to propose a federated learning method based on aligned representation, so as to comprehensively consider local data information and global information for federated learning in scenarios where data distribution varies greatly, thereby providing a strong guarantee for mining high-quality data information of the client itself.

[0006] The present invention focuses on the data heterogeneity problem of federated learning. This problem involves a central server and multiple (n) clients; the data stored on each client has a different distribution. In some actual federated learning scenarios, it is necessary to comprehensively consider local data information and global information to train the model. Part of the parameters of the personalized model are responsible for learning shared global feature representations, while the other part learns private local personalized representations. Based on this, the model can be divided into two layers: a representation layer and a decision layer. The representation layer is responsible for mapping sample vectors into representation vectors, and the decision layer is responsible for mapping representation vectors into probability vectors. The sample vector refers to a vector used to store the original data of a specific sample instance; a representation vector refers to a vector that represents a specific sample instance in a low-dimensional space; a probability vector refers to a vector that stores the probability value of a specific sample instance belonging to a certain category.

[0007] The local representation vector refers to the vector obtained by mapping the sample vector through the representation layer of the client model; the global average representation vector refers to the vector obtained by mapping the sample vector through the weighted mean of the local model parameters. "Alignment" means matching the local representation vector with the global average representation vector so that the representation layer of the local model learns not only local data but also global information. The present invention achieves alignment by reducing the distance (such as Euclidean distance) between the local representation vector and the average representation vector.

[0008] Assume there are n clients, and the data on each client are d1, d2, ..., d n ,in:

[0009] d i =(x i ,y i ), (1)

[0010] x i is the sample vector, y i are labels; the models are M1, M2, ..., M n ,in:

[0011] M i =φ i (h i (θ i ), w i ), (2)

[0012] φ i is the model decision layer, h i is the model representation layer, θ i is the representation layer parameter, w i is the decision-making layer parameter;

[0013] The local representation vector is the vector obtained by mapping the sample vector through the representation layer of the client model. The calculation formula is:

[0014] r i =h i (x i , w i ); (3)

[0015] The global average representation vector is the vector obtained by mapping the sample vector through the weighted mean of the local model parameters. The calculation formula is:

[0016]

[0017] in, is the weighted mean of the local model parameters. For example, the weighted mean can be calculated in proportion to the number of samples:

[0018]

[0019] Among them, N is the total number of samples, N i is the number of samples of the ith client, w i * are the local model parameters uploaded by the i-th client.

[0020] Assume that during the training process, all clients synchronously communicate with the server for T rounds; which is divided into the first round of communication training and the second to Tth rounds of communication training:

[0021] (I) In the first round of communication, the training process is:

[0022] (1) Each client uses private data to train a local model, that is, the optimal parameters of the local model for this round are calculated as follows:

[0023]

[0024] Among them, w i * is the result of the model parameters of this round of local training, argmin is to find the minimum value, L is the loss function, φ i is the model decision layer, h i is the model representation layer, θ i is the representation layer parameter, w i is the decision-making layer parameter, x i is the sample vector, y i It’s a label;

[0025] (2) Each client puts its own w i * Upload to the server;

[0026] (3) The server sends a i * Find the weighted mean For example, we can calculate the weighted mean in proportion to the number of samples:

[0027]

[0028] Among them, N is the total number of samples, N i is the number of samples of the ith client, w i * is the local model parameter uploaded by the i-th client;

[0029] (4) The server Return to the client, the client sets the local model parameters to

[0030] (II) In the 2nd to Tth round of communication, the training process is:

[0031] (1) Each client calculates the global average representation vector:

[0032]

[0033] in, is the global average representation vector, h i is the model representation layer, is the weighted mean of the local model parameters; and use private data to train a local model for one round, and use the distance between the local representation vector and the average representation vector (for example, Euclidean distance) as the regular term of the local loss function, so that the purpose of aligning the representation while minimizing the classification loss can be achieved, so that the local model can capture the characteristics of the local data and learn global information; that is, the optimal parameters of the local model in this round are calculated as follows:

[0034]

[0035] Among them, w i * is the result of the model parameters of this round of local training, argmin is to find the minimum value, L is the loss function, φ i is the model decision layer, h i is the model representation layer, θ i is the representation layer parameter, w i is the decision-making layer parameter, x i is the sample vector, y i It's a label. is the global average representation vector, and λ is a hyperparameter used to adjust the degree of learning global information;

[0036] (2) Each client puts its own w i * Upload to the server;

[0037] (3) The server sends a i * Find the weighted mean For example, we can calculate the weighted mean in proportion to the number of samples:

[0038]

[0039] Among them, N is the total number of samples, N i is the number of samples of the ith client, w i * is the local model parameter uploaded by the i-th client;

[0040] (4) The server takes the weighted mean Return to the client, the client sets the local model parameters to

[0041] Repeat the above process, and after T rounds of communication, the federated learning process is completed. Each client outputs the local model as its own personalized model.

[0042] The federated learning method based on aligning local representation and global representation proposed in the present invention allows each client to establish a personalized model and continuously reduce the distance between local representation and global representation during the training process. In this way, after multiple rounds of iterations, each client can obtain a model with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of the federated learning method based on aligned representation.

[0044] Figure 2 This is a sample of image data.

[0045] Figure 3 It is a model example. DETAILED DESCRIPTION

[0046] The federated learning method based on alignment representation provided by the present invention has a process as shown in Figure 1 The specific process is as follows:

[0047] (1) The client trains a local model for one round;

[0048] (2) The client uploads local model parameters;

[0049] (3) Server calculation parameter integration results;

[0050] (4) The server returns the parameter integration result;

[0051] (5) The client calculates the average representation vector;

[0052] (6) The client trains the local model and aligns the representation vectors;

[0053] (7) Repeat (2)-(6) until the stop condition is met.

[0054] The following is a further description of the classification task in conjunction with the MNIST dataset (the MNIST dataset can be obtained from the following website: https: / / yann.lecun.com / exdb / mnist / ); the picture is a sample of a part of the MNIST dataset. Figure 2 As shown; the MNIST dataset includes 10 digital labels, a total of 60,000 training samples and 10,000 test samples; there are n=100 clients, and the 6,000 training samples in the MNIST dataset are randomly distributed on 100 clients with different resources. Each client has 10 categories of pictures, with 60 pictures in each category; the test samples are also randomly distributed on these 100 clients, with 10 categories of pictures on each client, with 10 pictures in each category; it is required to train a classification model for each client in a federated learning manner.

[0055] First, deploy the classification model for 100 clients; Figure 3 As shown in the figure, a three-layer CNN (Convolutional Neural Network) model is set here, where the first two convolutional layers are representation layers, which are used to map samples into representation vectors; the last fully connected layer is the decision layer, which is used to map the representation vector into a probability vector.

[0056] Then, during training, the client randomly reads local image data into the model, reading 50 images each time, calculates the loss and gradient, and updates the model parameters; after training 10*60 / 50=12 times, the client's local sample data is processed and the local model parameters are uploaded to the server.

[0057] Assume that in this round of communication, the server is connected to 100 clients, collects the parameters of 100 models uploaded by the clients, calculates the weighted mean according to the number of samples, and then returns the mean to the connected clients.

[0058] The client uses the mean to update the local model and calculates the average representation vector. Then, during the training process, the classification loss is reduced while the distance between the local representation vector and the average representation vector is reduced.

[0059] Repeat the above training steps, after 50 rounds of communication, the average accuracy of the client model can reach 97.21%. The average accuracy of the client is calculated by averaging the accuracy of the 3-layer CNN model on 100 clients.

Claims

1. A federated learning method based on aligned representation, characterized in that: The data heterogeneity problem of federated learning involves a central server and multiple clients; the data stored on each client has different distributions; in actual federated learning scenarios, it is necessary to comprehensively consider local data information and global information to train the model; part of the parameters of the personalized model is responsible for learning the shared global feature representation, and the other part is responsible for learning the private local personalized representation; based on this, the model is divided into two layers: the representation layer and the decision layer; the representation layer is responsible for mapping the sample vector into the representation vector, and the decision layer is responsible for mapping the representation vector into the probability vector; the sample vector refers to the vector used to store the original data of a specific sample instance; the representation vector refers to the vector that represents a specific sample instance in a low-dimensional space; the probability vector refers to the vector that stores the probability value of a specific sample instance belonging to a certain category; Define the local representation vector as the vector obtained by mapping the sample vector through the representation layer of the client model, and define the global average representation vector as the vector obtained by mapping the sample vector through the weighted mean of the local model parameters. Alignment means matching the local representation vector with the global average representation vector so that the representation layer of the local model learns not only local data but also global information. Alignment is achieved by reducing the distance between the local representation vector and the average representation vector. Specifically: Assume there are n clients, and the data on each client are d1, d2, ..., d n ,in: d i =(x i ,y i ), (1) x i is the sample vector, y i are labels; the models are M1, M2, ..., M n ,in: M i =φ i (h i (i i ),w i ), (2) φ i is the model decision layer, h i is the model representation layer, θ i is the representation layer parameter, w i is the decision-making layer parameter; The calculation formula of the local representation vector is: r i =h i (x i ,w i ); (3) The calculation formula of the global average representation vector is: in, is the weighted mean of the local model parameters, calculated in proportion to the number of samples: Among them, N is the total number of samples, N i is the number of samples of the ith client, w i * are the local model parameters uploaded by the i-th client.

2. The method for federated learning based on alignment representation according to claim 1, characterized in that: Assume that during the training process, all clients synchronously communicate with the server for T rounds; (I) In the first round of communication, the specific process is as follows: (1) Each client uses private data to train a local model, that is, the optimal parameters of the local model for this round are calculated as follows: Among them, w i * is the result of the model parameters of this round of local training, argmin is to find the minimum value, L is the loss function, φ i is the model decision layer, h i is the model representation layer, θ i is the representation layer parameter, w i is the decision-making layer parameter, x i is the sample vector, y i It’s a label; (2) Each client puts its own w i * Upload to the server; (3) The server sends a i * Find the weighted mean Calculate the weighted mean in proportion to the sample size: Among them, N is the total number of samples, N i is the number of samples of the ith client, w i * is the local model parameter uploaded by the i-th client; (4) The server Return to the client, the client sets the local model parameters to (II) In the 2nd to Tth rounds of communication, the specific process is as follows: (1) Each client calculates the global average representation vector: in, is the global average representation vector, h i is the model representation layer, is the weighted mean of the local model parameters; and use private data to train a local model for one round, using the distance between the local representation vector and the average representation vector as the regular term of the local loss function to achieve the purpose of aligning the representation while minimizing the classification loss, so that the local model can capture the characteristics of the local data and learn global information; that is, the optimal parameters of the local model in this round are calculated as follows: Among them, w i * is the result of the model parameters of this round of local training, argmin is to find the minimum value, L is the loss function, φ i is the model decision layer, h i is the model representation layer, θ i is the representation layer parameter, w i is the decision-making layer parameter, x i is the sample vector, y i It's a label. is the global average representation vector, and λ is a hyperparameter used to adjust the degree of learning global information; (2) Each client puts its own w i * Upload to the server; (3) The server sends a i * Find the weighted mean Calculate the weighted mean in proportion to the sample size: Among them, N is the total number of samples, N i is the number of samples of the ith client, w i * is the local model parameter uploaded by the i-th client; (4) The server takes the weighted mean Return to the client, the client sets the local model parameters to Repeat the above process, T rounds of communication are completed, and the federated learning process is completed; each client outputs the local model as its own personalized model.

Images