Personalized federated learning algorithm based on improved K-Means algorithm

Abstract

The invention relates to the field of personalized federated learning, and discloses a personalized federated learning algorithm based on an improved K-Means algorithm, which realizes providing personalized neural network training service for clients, and comprises the following steps: S1, a server cloud collects local models of each client; s2, the cloud model is divided into K clusters under a K-MDM algorithm; s3, iteratively training each cluster, and updating the clustering model; and S4, for the newly added clients, dividing the newly added clients into the nearest cluster. The K-MDM algorithm in the S2 comprises the following steps: S2-1, calculating to obtain an average feature model wavg; s2-2, taking the model farthest from the wavg as a first clustering model; s2-3, taking the existing clustering model as a reference point, and selecting K clustering models in total; and S2-4, dividing all the cloud models into clusters with the closest distances. According to the K-MDM algorithm, the randomness and blindness of a traditional K-Means algorithm are reduced, and the convergence speed is increased. The invention provides an innovative personalized federal learning algorithm based on a K-MDM algorithm.

Description

Technical Field

[0001] The present invention relates to the field of personalized federated learning, and in particular to the innovation of clustering algorithms. It optimizes the selection of the initial clustering model of the K-Means algorithm, reduces the randomness and blindness of the algorithm, and improves the convergence speed. The improved K-Means algorithm is applied to the field of personalized federated learning to provide each client with a personalized neural network model that conforms to the distribution characteristics of its local data. Background Art

[0002] With the rapid development of information digital technology and the popularization of the Internet, network data has become increasingly complex. Enterprises and institutions use deep learning methods to analyze data characteristics in order to provide personalized services to users. Federated learning methods have become a research focus and have made great progress and development.

[0003] Federated Learning (FL) is an innovative distributed machine learning framework. It enables participating clients to collaborate and share training results through server coordination, solving the challenge of device performance differences. At the same time, it does not directly access client local data and protects user privacy. The algorithm that best represents the federation idea is the FedAvg algorithm. In this algorithm, the server samples a batch of clients from all clients and broadcasts the global model to them. The sampled clients iteratively update the parameters of the neural network model through stochastic gradient descent (SGD). After the local update step, the server collects the neural network parameter model and obtains a new global model through weighted averaging. The FedAvg algorithm is widely known as an early achievement in the field of federated learning. However, it is also well-known that the FedAvg algorithm has difficulty in performing satisfactory performance on non-independent and identically distributed (Non-IID) data. Data heterogeneity makes the neural network model produced by local data training have unique personality, and the global model obtained by weighted averaging lacks personalization and cannot be applied to all clients. Therefore, not only can each client not benefit from the federation framework, but the performance of its own local model will be damaged. The FedAvg algorithm, which represents the idea, is trapped by data heterogeneity, and the federation idea itself is therefore hindered in its implementation and application.

[0004] In order to solve the challenge of data heterogeneity, researchers proposed personalized federated learning (pFL). The personalized method attempts to design an optimization method tailored to the data distribution characteristics based on the local data distribution characteristics of the client in the federated learning framework, in order to achieve personalization while retaining the advantages of the federated framework and ultimately reach the global optimal point.

[0005] There are many research directions in the field of pFL, including clustering, regularization, meta-learning, knowledge distillation, etc. This paper focuses on clustering methods and conducts in-depth research on them. Usually, clustering methods presuppose that there is a natural classification of clients in the application environment based on data distribution, so as to avoid the challenge of data heterogeneity. When the client is divided into multiple clusters according to the data distribution characteristics, the client data distribution within the cluster is expressed as IID, and the clustering method performs best at this time. In contrast, when the client cannot be divided into multiple IID sets, the performance of the clustering method will be significantly reduced.

[0006] In order to divide clients into IID sets, various clustering methods have been introduced into the pFL field. The most typical clustering algorithms are the K-Means algorithm and its improved algorithms such as K-Means++, K-modes, and K-prototypes. The pFL algorithm that introduces clustering algorithms often divides the client set into several clusters, applies the FedAvg algorithm within the cluster, and iterates and updates several clustering models. After the algorithm ends, each client will obtain the clustering model of its cluster, which is adapted to the distribution characteristics of the local data set and can be used as a reference for enterprises and institutions to provide personalized services. Summary of the invention

[0007] In the context of introducing clustering algorithms into the pFL field, the classic clustering algorithm K-Means will be applied to the neural network weight parameter model set. Aiming at the randomness and blindness problems of the classic K-Means algorithm in randomly selecting the initial clustering model, the present invention uses the neural network model with mixed characteristics as a reference to find the first clustering model, and then obtains each clustering model in turn based on the idea that the sum of squared errors of each clustering model is the largest and the sum of squared errors of members within the cluster is the smallest. The present invention calls the improved algorithm the maximum distance K-means (K-Maximum Distance Means, K-MDM) algorithm, and applies it to the pFL field, divides the client set into K clusters, applies the FedAvg algorithm to the client set in each cluster, iterates and updates the K cluster models, and provides each client with a personalized neural network model that conforms to its data distribution characteristics.

[0008] To realize the above-mentioned personalized service, the present invention comprises the following steps:

[0009] S1: In the pre-training phase, the server randomly initializes a neural network weight parameter model w initial , and send it to each client m∈Μ, each client will w initial Load it into the local neural network, complete a round of local training, and upload the trained model w to the server cloud to form a cloud model set Φ = {w1,…,w m ,…,w M}.

[0010] S2: Apply the K-MDM algorithm to the cloud model set Φ, divide it and all client sets into K clusters and calculate K cluster models, denoted as Γ={c1,…,c k ,…,c K}, the total number of cloud models and corresponding clients in each cluster is M k .

[0011] S3: After the algorithm officially starts, in each iteration τ = t, the server will cluster the model They are sent to the clients m in their corresponding clusters respectively. Each client loads the cluster model and completes local training. The iterative update formula of the local model is:

[0012]

[0013] The model Upload to the server cloud, perform weighted averaging within the cluster, and iteratively update each cluster model. The formula is:

[0014]

[0015] S4: For a new client joining the training process, the server randomly initializes a model and sends it to the new client. After the new client completes local training, it uploads the model. The server calculates the Euclidean distance between the cloud model and each clustering model, and divides the cloud model and the new client into the cluster corresponding to the clustering model with the smallest Euclidean distance.

[0016] For the K-MDM algorithm in S2 above, it includes the following steps:

[0017] S2-1: Set the number of clusters K and the distance threshold φ. The server randomly selects several cloud models from the model set Φ at a sampling rate μ and calculates their average value. The result is called the average feature model, denoted by w avg .

[0018] S2-2: Server calculation w avg With all models w m The Euclidean distance D of From D avg,m Select the maximum value and use its corresponding model as the first clustering model c k＝1 , and then with c1 as the center, the models with Euclidean distance less than φ are divided into the first cluster.

[0019] S2-3: For the kth clustering model c other than c1 k Calculate the model w that has not been classified into each cluster m Respectively with each clustering model c k Calculate the Euclidean distance, denoted as

[0020]

[0021] Select cloud model from m , which maintains the maximum sum of squared errors (SSE) with all clustering models and satisfies the following formula:

[0022]

[0023] Let w m As the clustering model c k , taking it as the center, the models with Euclidean distance less than φ are classified as the kth cluster. If some models have been classified into other clusters in the previous calculation, they are removed so that there are no duplicate models in all clusters. This step is repeated, and when the number of cluster models k=K, the next step is entered.

[0024] S2-4: Calculate D m,k , all cloud models w m Classify into the clustering model c with the smallest Euclidean distance k The corresponding clusters are then divided into clusters, and the mean of all models in the clusters is calculated. k Iterative update. Repeat this step until the model in each cluster no longer changes and the algorithm ends.

[0025] The neural network weight parameter is a nonlinear mapping of the local data distribution of the client. The characteristics of the data distribution will be mapped to the neural network model, and the weighted combination result of the model can be regarded as a model with mixed data characteristics. Therefore, the present invention attempts to obtain an average feature model with complex data characteristics, and uses it as a reference point to find a suitable initial clustering model. In order to prevent the characteristics from being too complex to lose their reference significance, only some models are selected at a sampling rate μ.

[0026] Each time a cluster model is selected, the models that are closer to it should be classified into the same cluster and cannot become candidates for the next cluster model. Therefore, the present invention sets a distance threshold φ to isolate models that are closer. Under the premise that several cluster models have been obtained, the selection of the kth cluster model should be as far away from the existing cluster as possible. Therefore, the present invention uses the existing cluster model and selects a new cluster model through the maximum SSE.

[0027] Through the above improvements, compared with the classic K-Means algorithm, the present invention optimizes the selection of the initial clustering model and accelerates the convergence speed of the algorithm.

[0028] The present invention applies the K-MDM algorithm to a client set with Non-IID data distribution, divides it into K clusters, and applies the FedAvg algorithm to the clients in the clusters, so as to iteratively update the clustering model generated by the K-MDM algorithm, and finally provide a personalized neural network model that conforms to the data distribution characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flowchart of a personalized federated learning algorithm based on the improved K-Means algorithm.

[0030] Figure 2 Figure 2 is a flow chart of the K-MDM algorithm. DETAILED DESCRIPTION

[0031] The present invention proposes an improved method for the K-Means algorithm, named the K-MDM algorithm, and applies it to the field of personalized federated learning, and constructs a personalized federated learning algorithm based on the improved K-Means algorithm. Based on the unified relationship between the neural network model and the client's local data set in data features, the model with all types of data features is used as a reference point, and the initial clustering model is selected with the help of the maximum SSE. Compared with the traditional K-Means algorithm, the improved K-MDM algorithm optimizes the selection method of the initial clustering center, reduces randomness and blindness, and improves the convergence speed. The K-MDM algorithm is applied to a client set with a Non-IID data distribution, divides it into a specified number of clusters, and iteratively updates the clustering model within the cluster.

[0032] The specific implementation is as follows:

[0033] S1: In the pre-training phase, the server randomly initializes a neural network weight parameter model w initial , and send it to each client m∈Μ, each client will w initial Load it into the local neural network, complete a round of local training, and upload the trained model w to the server cloud to form a cloud model set Φ = {w1,…,wm ,…,w M}.

[0034] S2: Apply the K-EDM algorithm to the cloud model set Φ, divide it and all client sets into k clusters and calculate k cluster models, denoted as Γ={c1,…,c k ,…,c K}, the total number of cloud models and corresponding clients in each cluster is M k .

[0035] S3: After the algorithm officially starts, in each iteration τ = t, the server will cluster the model They are sent to the clients m in their corresponding clusters respectively. Each client loads the cluster model and completes local training. The iterative update formula of the local model is:

[0036]

[0037] The model Upload to the server cloud, perform weighted averaging within the cluster, and iteratively update each cluster model. The formula is:

[0038]

[0039] S4: For a new client joining the training process, the server randomly initializes a model and sends it to the new client. After the new client completes local training, it uploads the model. The server calculates the Euclidean distance between the cloud model and each clustering model, and divides the cloud model and the new client into the cluster corresponding to the clustering model with the smallest Euclidean distance.

[0040] 2. According to claim 1, a personalized federated learning algorithm based on an improved K-Means algorithm is characterized in that: in S2, the K-EDM algorithm comprises the following steps:

[0041] S2-1: Set the number of clusters K and the distance threshold φ. The server randomly selects several cloud models from the model set Φ at a sampling rate μ and calculates their average value. The result is called the average feature model, denoted by w avg .

[0042] S2-2: Server calculation w avg With all models w m The Euclidean distance D of From d avg,m Select the maximum value and use its corresponding model as the first clustering model c k＝1 , and then with c1 as the center, the models with Euclidean distance less than φ are divided into the first cluster.

[0043] S2-3: For the kth clustering model c other than c1 k Calculate the model w that has not been classified into each cluster m Respectively with each clustering model c k Calculate the Euclidean distance, denoted as

[0044]

[0045] Select cloud model from m , which maintains the maximum sum of squared errors (SSE) with all clustering models and satisfies the following formula:

[0046]

[0047] Let w m As the clustering model c k , taking it as the center, the models with Euclidean distance less than φ are classified as the kth cluster. If some models have been classified into other clusters in the previous calculation, they are removed so that there are no duplicate models in all clusters. This step is repeated, and when the number of cluster models k=K, the next step is entered.

[0048] S2-4: Calculate D m,k , all cloud models w m Classify into the clustering model c with the smallest Euclidean distance k The corresponding clusters are then divided into clusters, and the mean of all models in the clusters is calculated. k Iterative update. Repeat this step until the model in each cluster no longer changes and the algorithm ends.

Claims

1. A personalized federated learning algorithm based on an improved K-Means algorithm, characterized by: The method comprises the following steps: S1: In the pre-training phase, the server randomly initializes a neural network weight parameter model w initial , and send it to each client m∈Μ, each client will w initial Load it into the local neural network, complete a round of local training, and upload the trained model w to the server cloud to form a cloud model set Φ = {w1,…,w m ,…,w M }; S2: Apply the K-Maximum Distance Means (K-MDM) algorithm to the cloud model set Φ, divide it and all client sets into K clusters and calculate K cluster models, denoted as Γ = {c1,…,c k ,…,c K }, the total number of cloud models and corresponding clients in each cluster is M k ; S3: After the algorithm officially starts, in each iteration τ = t, the server will cluster the model They are sent to the clients m in their respective clusters. Each client loads the cluster model and completes local training. The iterative update formula of the local model is: The model Upload to the server cloud, perform weighted averaging within the cluster, and iteratively update each cluster model. The formula is: S4: For a new client joining the training process, the server randomly initializes a model and sends it to the new client. After the new client completes local training, it uploads the model. The server calculates the Euclidean distance between the cloud model and each clustering model, and divides the cloud model and the new client into the cluster corresponding to the clustering model with the smallest Euclidean distance.

2. According to claim 1, a personalized federated learning algorithm based on an improved K-Means algorithm is characterized in that: In S2, the K-MDM algorithm includes the following steps: S2-1: Set the number of clusters K and the distance threshold φ. The server randomly selects several cloud models from the model set Φ at a sampling rate μ and calculates their average value. The result is called the average feature model, denoted by w avg ; S2-2: Server calculation w avg With all models w m The Euclidean distance D of From D avg,m Select the maximum value and use its corresponding model as the first clustering model c k＝1 , and then take c1 as the center and classify the models with Euclidean distance less than φ into the first cluster; S2-3: For the kth clustering model c other than c1 k Calculate the model w that has not been classified into each cluster m Respectively with each clustering model c k Calculate the Euclidean distance, denoted as Select cloud model from m , which maintains the maximum sum of squared errors (SSE) with all clustering models and satisfies the following formula: Let w m As a clustering model c k , taking it as the center, the models with Euclidean distance less than φ are classified as the kth cluster. If some models have been classified into other clusters in the previous calculation, they will be removed so that there are no repeated models in all clusters. This step is repeated. When the number of cluster models k=K, go to the next step; S2-4: Calculate D m,k , all cloud models w m Classify into the clustering model c with the smallest Euclidean distance k The corresponding clusters are then divided into clusters, and the mean of all models in the clusters is calculated. k Iterate and update, repeating this step until the models within each cluster no longer change and the algorithm ends.

Images