Personalized federated learning method, device and equipment based on client clustering

Through client clustering and multi-sphere splitting BKM algorithm, the problem that the model cannot adapt to data and business heterogeneity in traditional federated learning is solved, and the efficient training of personalized federated learning in scenarios with a large number of clients is realized, and the improvement of the efficient training of personalized federated learning is improved. Model accuracy and convergence performance are obtained.

CN116258164BActive Publication Date: 2024-11-19STATE GRID JIANGSU ELECTRIC POWER CO LTD NANJING POWER SUPPLY COMPANY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310101277.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-02-03
Publication Date
2024-11-19
Estimated Expiration
2043-02-03

AI Technical Summary

Technical Problem

In the face of data heterogeneity, client heterogeneity and model heterogeneity, traditional federated learning is difficult to adapt to the data distribution and business needs of all clients at the same time, resulting in limited improvement in model accuracy, and existing personalized federated learning methods. When the number of clients is large, the calculation complexity is high and the resource overhead is large, so it cannot effectively improve the model convergence performance and accuracy.

Method used

Through client clustering analysis, the improved BKM algorithm of multi-sphere splitting uses multi-group splitting to cluster clients with similar data distribution and model requirements, and multi-task training is performed using soft parameter sharing method within the cluster until each client obtains personality to reduce algorithm complexity and resource overhead, and improve model convergence efficiency and accuracy.

Benefits of technology

It has achieved a 15.6% to 22.9% improvement in federated learning accuracy in scenarios with large number of clients, and has good training speed and convergence performance, which significantly improves the accuracy and training speed of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116258164B_ABST
    Figure CN116258164B_ABST
Patent Text Reader

Abstract

The present invention discloses a personalized federated learning method, device and equipment based on client clustering, which includes: obtaining a global model, initializing the global model for training, obtaining the gradient generated by each client in the initialization training; performing cluster analysis based on the gradient of the client to determine the cluster to which each client belongs; performing shared training on the first hidden layer set of the deep neural network trained by the client in each cluster, and performing personalized training on the second hidden layer set of the deep neural network trained by the client in each cluster until the local model converges to obtain a personalized model. The method solves the technical problems of traditional federated learning that restrict the convergence performance, accuracy and precision of personalized models.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of personalized federated learning based on client clustering, and in particular to a personalized federated learning method, device and equipment based on client clustering. Background Art

[0002] Traditional federated learning assumes that there is a model that can be applied to the data distribution of all clients at the same time, and obtains a high-quality global model by extracting the common knowledge of all participating clients. However, this model is now facing the following challenges: 1) Data heterogeneity: data on different clients do not meet the independent and identically distributed property; 2) Client heterogeneity: clients have different computing and storage capabilities; 3) Model heterogeneity: different clients require customized models to adapt to their own application environments. In other words, the expressive power of a shared model cannot fit the distribution of all clients at the same time, and it may not be applicable to certain clients with unique performance, which ultimately restricts the improvement of model accuracy.

[0003] For example, in an IoT scenario, the goal of a client is to train an "attractiveness" classifier on a full dataset of face images. Different clients have different definitions of "attractiveness." Client A may think that long hair represents attractiveness, while client B disagrees. In this case, a globally shared model is not applicable to all clients.

[0004] Personalized federated learning can take into account the heterogeneity of client data distribution and business needs while enjoying the advantages of knowledge sharing, and provide clients with personalized models. Federated multi-task learning, as one of the personalized means of federated learning, regards building models for different terminals as multiple tasks and maintains a global model as a reference. Each client trains its own personalized model based on the current reference model and local data by applying regularization terms and other methods. However, this method ignores the similarity relationship between clients and only refers to a global model, which restricts the improvement of convergence and training quality.

[0005] After a certain number of global iterations, the client will show a gradient that is highly correlated with its data distribution and model requirements, so the gradient can be used as a criterion for client similarity clustering. However, the high-dimensional characteristics of the deep neural network gradient have higher requirements for the computational efficiency of the clustering algorithm. At the same time, the client-borne services are dynamic and diverse, and traditional algorithms are sensitive to the initial cluster center.

[0006] To solve the above problems, the patent number CN114492847B, "An Efficient Personalized Federated Learning System and Method", involves an efficient personalized federated learning system and method. The terminal devices in the system include: a terminal device data module, a terminal device model download module, two terminal device model training modules and a terminal device model upload module, and the central server includes a server-side data module, two server-side model integration modules and a server-side model distribution module. The invention is reasonably designed, and it completely places the pruning process and model training on the terminal device, which reduces the burden on the central server and improves the processing efficiency. It also fully considers the differences in data distribution, realizes the personalized function of the model, and can effectively analyze the data collected on different terminal devices, while greatly reducing the communication cost. Ensure the user's privacy information and personalization, as well as the sending of new models when the terminal device data is missing. However, the disadvantage of this method is that it is not suitable for scenarios with a large number of clients.

[0007] The patent number CN112560991A, "Personalized federated learning method based on hybrid expert model", involves a personalized federated learning method based on a hybrid expert model. In order to overcome the defect that private models are difficult to fully train in large-scale stateless mobile federated environments, this invention proposes a personalized federated learning method based on a hybrid expert model: all clients join the federated learning to jointly participate in the training of the global model to obtain the global model parameters θG; each client downloads θG from the server respectively, and uses the parameters to initialize the feature extraction layer and personality classification layer in the client, and uses the fixed base method to personalize to obtain the personality classification layer parameters; at this time, client i has θG including feature extraction layer parameters and global classification layer parameters and personality classification layer parameters, and uses these three to initialize the feature extraction layer, global classification layer and personality classification layer, and jointly trains the gating model to obtain the gating model parameters. Finally, the client obtains the parameters of the feature extraction layer, global classification layer, personality classification layer and gating model to complete personalized federated learning. This patent fine-tunes the parameters of the personality classification layer and the gating model based on the global model parameters obtained through federated learning, and then uses the global model and the personality classification layer to train the gating model separately, realizing the hybrid of the personality classification layer and the global model, which can improve personalization capabilities while retaining global knowledge; this patent uses the personality classification layer and the global model as local and global experts to form a hybrid expert model, and then uses the gating model to combine these experts, and uses the output of the feature extraction layer as the input of the gating model, so that the gating model can more effectively divide the input data. However, its algorithm is complex, brings a lot of resource overhead, and takes a long time to calculate.

[0008] The patent number CN112668726A, "A personalized federated learning method with efficient communication and privacy protection", involves a personalized federated learning method with efficient communication and privacy protection, including the following steps: S1: pull the current global model Wt from the central server to all clients, and initialize the local model of each client; S2: perform E rounds of local training to obtain a new local model; S3: send the model parameters to the central server; S4: aggregate the received model parameters in the central server to obtain the aggregation result Wt+1; S5: update the local models of all clients according to Wt+1; S6: determine whether the predetermined number of iterations has been completed; if so, complete the personalized federated learning; if not, set t=t+1, and return to step S2 for the next round of personalized federated learning. This invention provides a personalized federated learning method with efficient communication and privacy protection, which solves the problem that the existing personalized federated learning method does not achieve a balance between the personalized client local model and the global model. However, it is also not suitable for scenarios with a large number of clients. Summary of the invention

[0009] The present invention provides a personalized federated learning method, device and equipment based on client clustering, which solves the technical problems of traditional federated learning restricting the convergence performance, accuracy and precision of personalized models.

[0010] A personalized federated learning method based on client clustering, comprising:

[0011] Obtain a global model, perform initialization training on the global model, and obtain a gradient generated by each client during the initialization training;

[0012] Performing cluster analysis according to the gradient of the client to determine the cluster to which each client belongs;

[0013] The first hidden layer set of the deep neural network trained by the client in each cluster is shared and trained, and the second hidden layer set of the deep neural network trained by the client in each cluster is personalized and trained until the local model converges to obtain a personalized model.

[0014] In one embodiment of the present invention, the step of obtaining the gradient generated by each client in the initialization training specifically includes: sending the global model to each client to form a local model corresponding to each client; training the local model to determine the loss function of the local model corresponding to each client, wherein the loss function includes the target parameters of the local model; continuously iterating the target parameters n times through a stochastic gradient descent algorithm until the target parameters are determined to minimize the loss function; and obtaining the gradient corresponding to each client.

[0015] In one embodiment of the present invention, cluster analysis is performed according to the gradient of the client to determine the cluster to which each of the client belongs, which specifically includes: determining weighted clients according to the data set of the client and the gradient corresponding to the client; initializing the center of mass and radius of multiple clusters; dividing each of the weighted clients into the cluster where the center of mass is closest to it; determining the neighborhood relationship between any two clusters according to the radius and the center of mass distance between any of the clusters; moving the weighted clients in a cluster to its neighborhood cluster according to the neighborhood relationship and the distance from the weighted client to the center of mass of the cluster to which it belongs and its neighborhood clusters; and continuously iterating the weighted clients in each cluster according to the distance from the weighted client to the center of mass of the cluster to which it belongs and its neighborhood clusters until the weighted clients in each cluster remain unchanged.

[0016] In one embodiment of the present invention, cluster analysis is performed based on the gradient of the client, and also includes: determining the average value of all cluster radii and the incremental step size of cluster splitting; comparing the radius of each cluster with the average value; splitting the clusters whose cluster radius is larger than the average value; and stopping the splitting when it is determined that the number of split clusters is greater than or equal to the incremental step size.

[0017] In one embodiment of the present invention, the weighted clients in each of the clusters are continuously iterated according to the distance from the weighted client to the centroid of the cluster to which it belongs and its neighboring clusters, specifically including: determining the stable domain and active domain of the cluster; if it is determined that the weighted client is in the stable domain, then in the current iteration, the weighted client is not moved to other neighboring clusters of the cluster; the active domain of the cluster is divided into multiple annular areas, and the annular area to which the weighted client belongs is determined; according to the annular area to which the weighted client belongs and the distance from the weighted client to the centroid of the cluster to which it belongs and its neighboring clusters, the weighted client is moved to the corresponding neighboring cluster or its own cluster.

[0018] In one embodiment of the present invention, the neighborhood relationship between any two clusters is determined based on the radius and the center of mass distance between any of the clusters, specifically including: determining the center of mass difference between the first cluster and the second cluster, and determining the norm of the center of mass difference; if half of the norm of the center of mass difference is smaller than the radius of the first cluster, then determining the second cluster as a neighborhood cluster of the first cluster.

[0019] In one embodiment of the present invention, the first hidden layer set of the deep neural network trained by the clients in each cluster is shared and trained individually for the second hidden layer set of the deep neural network trained by the clients in each cluster until the local model converges to obtain a personalized model, specifically including: the cluster to which the clients belong is a ball cluster; determining the first hidden layer set shared by the clients in the ball cluster, and training the shared hidden layer as a global model training task; determining the second hidden layer set of the clients in the ball cluster that is higher than the first hidden layer set, and training the second hidden layer set of each client as a personalized model training task; determining a first objective function for each client when performing the global model training task; and determining a second objective function for each client when performing the personalized model training task based on the first objective function.

[0020] A personalized federated learning device based on client clustering, comprising:

[0021] A gradient acquisition module is used to acquire a global model, perform initialization training on the global model, and acquire the gradient generated by each client in the initialization training;

[0022] A cluster analysis module, used to perform cluster analysis according to the gradient of the client to determine the cluster to which each client belongs;

[0023] The shared training module is used to perform shared training on the first hidden layer set of the deep neural network trained by the client in each cluster, and to perform personalized training on the second hidden layer set of the deep neural network trained by the client in each cluster until the local model converges to obtain a personalized model.

[0024] A personalized federated learning device based on client clustering, comprising:

[0025] at least one processor; and,

[0026] A memory connected to the at least one processor via a bus; wherein,

[0027] The memory stores instructions executable by the at least one processor, and the instructions are executed to implement:

[0028] Obtain a global model, perform initialization training on the global model, and obtain a gradient generated by each client during the initialization training;

[0029] Performing cluster analysis according to the gradient of the client to determine the cluster to which each client belongs;

[0030] The first hidden layer set of the deep neural network trained by the client in each cluster is shared and trained, and the second hidden layer set of the deep neural network trained by the client in each cluster is personalized and trained until the local model converges to obtain a personalized model.

[0031] A non-volatile storage medium stores computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the following steps:

[0032] Obtain a global model, perform initialization training on the global model, and obtain a gradient generated by each client during the initialization training;

[0033] Performing cluster analysis according to the gradient of the client to determine the cluster to which each client belongs;

[0034] The first hidden layer set of the deep neural network trained by the client in each cluster is shared and trained, and the second hidden layer set of the deep neural network trained by the client in each cluster is personalized and trained until the local model converges to obtain a personalized model.

[0035] The present invention provides a personalized federated learning method, device and equipment based on client clustering, which at least includes the following beneficial effects: taking into account the similarity between multiple tasks, clustering is performed based on the improved BKM algorithm based on multi-ball splitting and according to the gradient performance of the client after a certain number of global training times. Then, clients with similar distributions in the cluster use a method based on soft parameter sharing for multi-task training until each client obtains a personalized model, which is not only suitable for scenarios with a large number of clients, but also can reduce algorithm complexity, reduce resource overhead, and reduce calculation time. The model convergence efficiency and accuracy are improved by splitting the centroid of the ball cluster with low complexity. Compared with the above-mentioned patent, the present invention can improve the model accuracy by 15.6% to 22.9%, and has good training speed and convergence performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0037] Figure 1 A schematic diagram of the steps of a personalized federated learning method based on client clustering provided in an embodiment of the present invention;

[0038] Figure 2 A flowchart of federated multi-task learning based on client clustering provided by an embodiment of the present invention;

[0039] Figure 3A clustering time comparison chart of K-Means, BKM algorithm, and improved BKM algorithm provided in an embodiment of the present invention;

[0040] Figure 4 A comparison chart of the accuracy of Fedavg federated learning, federated multi-task learning FMTL, and C federated multi-task learning CFMTL when ratio=0.5 provided in an embodiment of the present invention;

[0041] Figure 5 A comparison chart of loss function value changes of Fedavg federated learning, federated multi-task learning FMTL, and C federated multi-task learning CFMTL when ratio=0.5 provided in an embodiment of the present invention;

[0042] Figure 6 The accuracy change diagram of Fedavg federated learning, federated multi-task learning FMTL, and C federated multi-task learning CFMTL when ratio=0.7 provided in the embodiment of the present invention;

[0043] Figure 7 A comparison chart of loss function value changes of Fedavg federated learning, federated multi-task learning FMTL, and C federated multi-task learning CFMTL when ratio=0.7 provided in an embodiment of the present invention;

[0044] Figure 8 A schematic diagram of a personalized federated learning device based on client clustering provided in an embodiment of the present invention;

[0045] Fig. 9 A schematic diagram of a personalized federated learning device based on client clustering provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be described clearly and completely in conjunction with the specific embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be noted that it is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical terms or scientific terms involved in the present invention should be the usual meanings understood by people with ordinary skills in the technical field to which the present invention belongs. The words "one", "a", "a", "the" and the like involved in the present invention do not indicate a quantitative limitation and may represent the singular or plural. The terms "including", "comprising", "having" and any of their variations involved in the present invention are intended to cover non-exclusive inclusions; the terms "first", "second", "third", etc. involved in the present invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0048] The data distribution and business needs of the clients show obvious heterogeneity. Federated multi-task learning, as a type of personalized federated learning, can effectively take into account the heterogeneity of clients in mobile edge networks and provide personalized models for differentiated clients. However, the traditional federated multi-task learning method treats clients equally and provides a single shared model for personalized reference, which restricts the improvement of the convergence and accuracy of the personalized model. To solve this problem, the present invention provides a federated multi-task learning method based on client clustering. Taking into account the similarity between multiple tasks, an improved BKM algorithm based on multi-sphere splitting is first provided, and clustering is performed according to the gradient performance of the client after a certain number of global training times. Then, clients with similar distributions in the cluster use a method based on soft parameter sharing for multi-task training until each client obtains a personalized model. Compared with the comparison method, the present invention can improve the accuracy of federated learning by 15.6% to 22.9%, and has good training speed and convergence performance.

[0049] The client clustering-based federated multi-task learning architecture proposed in this invention includes a cloud server (CS), a manufacturing execution system (MES), and a client. s∈S, m∈M, and d∈D are used to represent CS, MES, and the client set, respectively. Represents the dataset on client d. CS has powerful computing and communication capabilities to complete the federated learning model aggregation and send the global model Ω g Each MES has a certain computing power and covers several clients through the base stations connected to it. The clients receive the global model Ω through the MES g And perform local training. For a learning task (such as image recognition, path selection), the goal is to obtain the client dataset H = {x i ,y iThe present invention introduces client clustering. After executing the clustering process, clients with similar data distribution and model requirements form a cluster, and then perform personalized learning within the cluster to obtain a dedicated model Ω p .

[0050] Compared with the above patent solution, the present invention can improve the accuracy of federated learning by 15.6% to 22.9%, and has good training speed and convergence performance. The main technical contents of the present invention are as follows:

[0051] (1) Construct a client similarity clustering model based on the local model training gradient. The clustering method based on raw data has the risk of privacy leakage. Considering that the client shows a gradient that is highly correlated with its own heterogeneity after a certain number of global iterative training, the present invention designs a client similarity clustering model based on the model training gradient, which protects user privacy while providing a classification basis for personalized federated learning.

[0052] (2) Design an improved ball K-means (BKM) algorithm based on multi-sphere splitting. The gradient of the deep neural network model belongs to the clustering problem of high-dimensional data. The introduction of the hypersphere model can effectively describe its spatial relationship, but the BKM algorithm has the defect of being sensitive to the initial clustering center. Therefore, the present invention proposes an improved BKM algorithm based on multi-sphere splitting, which uses a low-complexity center increment method to improve clustering efficiency.

[0053] (3) Design a personalized federated learning method within a cluster based on soft parameter sharing. The clients within the cluster implement the soft sharing strategy. The bottom layer of the model uses regularization terms to keep correlation with the global model, while the top layer of the model performs personalized training based on local data distribution and needs until a personalized model is obtained after convergence. Simulations show that the proposed personalized federated learning method based on client clustering has good model performance and convergence speed.

[0054] Combine the following Figures 1 to 9 The present invention will be described in detail.

[0055] Figure 1 A schematic diagram of a personalized federated learning method based on client clustering provided in an embodiment of the present invention may include the following steps:

[0056] S110: Obtain a global model, perform initialization training on the global model, and obtain the gradient generated by each client during the initialization training.

[0057] In one embodiment of the present invention, a global model is sent to each client to form a local model corresponding to each client; the local model is trained to determine the loss function of the local model corresponding to each client, and the loss function includes the target parameters of the local model; the target parameters are iterated n times by a stochastic gradient descent algorithm until the target parameters are determined to minimize the loss function; and the gradient corresponding to each client after n iterations is obtained.

[0058] Specifically, first, initialize the global model and define the total dataset H related to the federated learning task as follows:

[0059]

[0060] Among them, H is the total dataset H and H related to the federated learning task. d Represents a sample dataset on any client d. represents the loss function of local training performed on client d, whose value is equal to the sample dataset H d The difference between the predicted value and the actual value on . Therefore, the total loss function of the dataset on all clients is defined as

[0061]

[0062] Among them, ω represents the weight of the model to be trained, that is, the global model parameter, and |H| represents the total dataset size of the task. The purpose of federated learning is to optimize the global model parameter ω by minimizing the loss function L(ω) of the task. ω is expressed as

[0063] ω=arg min L(ω)

[0064] The parameter updating method of the federated learning of the present invention is stochastic gradient descent (SGD), that is, a piece of data in the data set is randomly selected each time. This method greatly reduces the amount of calculation, but due to its randomness, the local model needs to be trained sufficiently to ensure the quality of the model. The update of the local model parameters is expressed as

[0065]

[0066] Among them, η is the learning rate when the parameters are updated, n∈N represents the number of training iterations, Indicates the nth round of model training parameters on client d, Indicates the model training parameters for the n-1th round on client d. is the gradient of the client after n-1 iterations of global training.

[0067] When the uploaded local models reach a certain number or the number of iterations N, CS will perform global model aggregation on the local models obtained. The specific weight aggregation is expressed as

[0068]

[0069] Among them, |H d | represents the size of the dataset of client d participating in the federated learning task, ω g ′ is the global model parameter before updating, ω g is the updated global model parameter, ω d ′ is the updated local model parameter of the client, ω d is the local model parameter of the client before the update, and |H| represents the total dataset size of the task. It can be seen that the client with a larger dataset gets a larger weight.

[0070] After a certain number of global training iterations, each client has different gradient representations in local model training due to the heterogeneity between clients.

[0071] S120: Perform cluster analysis according to the gradient of the client to determine the cluster to which each client belongs.

[0072] Specifically, gradient-based client clustering enables clients with similarities to form clusters and perform personalized federated learning within the clusters. The BKM algorithm introduces a hypersphere model to describe spatial relationships, which is suitable for clustering problems of high-dimensional data such as deep neural network gradients, but it still has the problem of being too sensitive to the initial cluster centroid.

[0073] In view of the above problems, the present invention provides an improved BKM algorithm based on multi-ball splitting, which uses a low-complexity centroid increment method to improve clustering efficiency. The main steps of the algorithm include ball clustering, neighborhood ball cluster search, stable domain and stable ring, and low-complexity clustering centroid splitting.

[0074] In one embodiment of the present invention, a weighted client is determined according to a data set of the client and a gradient corresponding to the client; the center of mass and radius of multiple clusters are initialized; each weighted client is divided into a cluster with a center of mass closest to it; the neighborhood relationship between any two clusters is determined according to the radius and the center of mass distance between any clusters; the weighted client in a cluster is moved to its neighboring cluster according to the neighborhood relationship and the distance from the weighted client to the center of mass of the cluster to which it belongs and its neighboring clusters; the weighted clients in each cluster are continuously iterated according to the distance from the weighted client to the center of mass of the cluster to which it belongs and its neighboring clusters until the weighted clients in each cluster remain unchanged.

[0075] Specifically, It represents the gradient after client d executes n global trainings, and uses As the standard for client clustering, the definitions of ball clusters and neighborhood ball cluster searches are as follows:

[0076] Definition of a ball cluster: Given a cluster C, if the center of mass c and radius r of C satisfy the following conditions Then C is called a spherical cluster.

[0077] Among them, e d represents the weighted clients (referred to as clients) assigned to C, and |C| represents the number of clients in cluster C.

[0078] The present invention introduces the neighborhood ball cluster search in BKM to simplify the calculation of the confidence between phase ball clusters. i and C j Represents two ball clusters, whose centroids are ci and c respectively j If the two ball clusters satisfy the following conditions:

[0079]

[0080] Then define C j C i It can be concluded that there are three relationships between the two ball clusters:

[0081] 1. The two are each other's neighborhood ball clusters;

[0082] 2.C j C i Neighborhood ball cluster, but C i Not C j The neighborhood ball cluster of ;

[0083] 3. Neither of them is the neighboring sphere cluster of the other.

[0084] Then, we can define the rules for moving clients between two ball clusters. For any e∈C,||ec||≤r,

[0085] 1) If C j Not C i The neighborhood ball cluster has r≤||cc j || / 2, and we can conclude

[0086]

[0087] According to the triangle inequality, we can deduce

[0088]

[0089] Therefore, we can get the ball cluster C i Any e in cannot move to its non-neighboring cluster.

[0090] 2) If Cj is C i 's neighborhood ball cluster, with r > ||c - c j || / 2 and there exists e ∈ C such that ||c - e|| > ||c - c i || / 2. e is a point on the line segment cc i and satisfies ||c - e|| + ||c i - e|| = ||c - c i || and ||c - e|| > ||c - c i || / 2. It can be concluded that

[0091]

[0092]

[0093] According to the triangle inequality, it can be concluded that

[0094]

[0095] Similarly, any e in the to - be - searched ball cluster C i can be moved to its neighborhood ball cluster. Using the neighborhood ball cluster can effectively reduce the distance calculation of clients within the ball cluster in the next iteration, because the client distance calculation is restricted within the neighborhood ball cluster. In the next iteration, the clients in the ball cluster will only be adjusted among adjacent clients, without calculating the distance from the ball cluster centroid to all other clients, thus reducing the calculation time.

[0096] In an embodiment of the present invention, the stable domain and the active domain of the ball cluster are determined; if it is determined that the weighted client is in the stable domain, then the weighted client is not moved to other neighborhood ball clusters of the ball cluster in the current iteration; the active domain of the ball cluster is divided into multiple annular regions, and the annular region to which the weighted client belongs is determined; according to the annular region to which the weighted client belongs and the distance from the weighted client to the centroid of its own ball cluster and its neighborhood ball clusters, the weighted client is moved to the corresponding neighborhood ball cluster or its own ball cluster itself.

[0097] Specifically, each ball cluster is further divided into a stable domain and an active domain inside. The clients in the stable domain do not need to calculate the moving distance in the next iteration, while the clients in the active domain are the opposite.

[0098] C i is the to - be - searched ball cluster, represents the set of centroids of all neighborhood ball clusters of C i . If is non - empty and the centroid of the ball cluster C j then the stable domain of C is defined as the centroid being c i , with a radius of i , radius is If e is at the center of mass c i In the stable domain of

[0099]

[0100]

[0101] It can be concluded that

[0102]

[0103] Therefore, client e in the stable domain cannot move to any neighboring ball cluster in the current iteration. In order to further speed up the calculation, the active domain of the ball cluster is divided into multiple annular regions.

[0104] C is the sphere cluster to be divided with the center of mass c and radius r, {NH C} represents the set of all the centroids of the neighborhood ball clusters of C. Assume that |{NH C}|=k′, where |{NH C}| means {NH C} and k′≠0, c i and c i+1 Respectively represent the centroids of the ith and i+1th neighborhood ball clusters of C, so we use represents the ith ring of C and is defined as follows.

[0105]

[0106] Next, the movement rules of clients between rings are given. C is the ball cluster to be checked with the center of mass c and |{N C = k'. A client in the i-th annular region of cluster C can only move to the i-th nearest neighboring cluster or itself (i = k').

[0107] In one embodiment of the present invention, the centroid difference between the first cluster and the second cluster is determined, and the norm of the centroid difference is determined; if half of the norm of the centroid difference is smaller than the radius of the first cluster, the second cluster is determined to be a neighborhood cluster of the first cluster.

[0108] Specifically, in each iteration of BKM, the distance between the mass centers of the clusters needs to be calculated continuously, which increases the computational complexity. To this end, the present invention defines a new mass center distance calculation method, which utilizes the calculation results in the previous iterations, thereby avoiding redundant calculations. represents the centroid of the ball cluster C in the tth iteration, represents the ball cluster C in the tth iteration i The distance the centroid was moved relative to the t-1th iteration. is the tth iteration and The distance between. Indicates that C i The radius of the spherical cluster C i and C j ,if Then C j Cannot be C i The neighborhood ball clusters of , and the distance calculation between the two can be omitted.

[0109] In one embodiment of the present invention, the average value of all cluster radii and the incremental step size of cluster splitting are determined; the radius of each cluster is compared with the average value; clusters with a radius larger than the average value are split; and when it is determined that the number of split clusters is greater than or equal to the incremental step size, the splitting is stopped.

[0110] Specifically, if the clients within a cluster no longer change, the cluster is called a stable cluster. In the BKM algorithm, a stable cluster does not participate in any distance calculation in the current iteration. As the iteration process progresses, there will be more and more stable clusters, and the acceleration effect of the algorithm based on stable clusters will become more and more obvious.

[0111] The present invention proposes a centroid splitting method based on sorting to overcome the problem of sensitivity to the initial cluster centroid in the BKM algorithm, but the computational complexity of centroid sorting in this method is O(klogk), which restricts the convergence efficiency. Therefore, the present invention proposes a low-complexity centroid splitting method, which first calculates the average value of the radius of all clusters, and then selects clusters larger than the average radius to perform splitting. At the same time, in order to avoid splitting too many clusters each time, the incremental step size is defined as Δk. That is, the number of centroids added each time is between 1 and Δk.

[0112] S130: performing shared training on the first hidden layer set of the deep neural network trained by the client in each cluster, and performing personalized training on the second hidden layer set of the deep neural network trained by the client in each cluster, until the local model converges to obtain a personalized model.

[0113] In the present invention, the clients in each cluster train their own personalized local models based on the global model, and obtain the required model after convergence.

[0114] In one embodiment of the present invention, the cluster to which the client belongs is a ball cluster; a first hidden layer set shared by the clients in the ball cluster is determined, and the shared hidden layer training is used as a global model training task; a second hidden layer set higher than the first hidden layer set is determined for the clients in the ball cluster, and the second hidden layer set training for each client is used as a personalized model training task; a first objective function of each client when performing the global model training task is determined; and a second objective function of each client when performing the personalized model training task is determined based on the first objective function.

[0115] Specifically, the improved BKM algorithm based on multi-sphere splitting classifies clients with the same or similar data distribution into one category. The clients in the cluster further perform a federated multi-task learning process to obtain a personalized model. The deep neural network trained in the client contains multiple hidden layers, which play a role in feature extraction. The hidden layers at the bottom of the shared deep neural network network of the clients in the cluster can learn some common low-level abstract features. The training of the shared hidden layers is called the task of global model training. g Above the shared hidden layer, each client maintains some higher-level hidden layers separately to capture personalized data features and business needs. The training of higher-level hidden layers is called personalized model training tasks. p It can be concluded that after task g and task p After iterations, the clients in the cluster will have the same underlying structure and personalized high-level structures.

[0116] The sharing of the bottom hidden layer is achieved through soft parameter sharing. Soft parameter sharing does not require that the parameters of the bottom hidden layers of different terminal models are exactly the same, but encourages parameter similarity.

[0117] Indicates the nth round of model training parameters on client d, Represents the parameters of the i-th layer. represents the global model parameters of cluster c in the nth round of model aggregation, represents the parameters of its i-th layer. The set of hidden layers participating in soft parameter sharing in a ball cluster is s. d,g (u) is defined as executing task on d g Therefore, the objective of the global model training of cluster c is defined as follows

[0118]

[0119] In the above formula, p d represents the preset weight and ∑ d p d =1, its value reflects the contribution of different clients in model aggregation. d represents the personalized model parameters on client d. Therefore, task p The objective function is defined as follows:

[0120]

[0121]

[0122] Among them, λ iRepresents the similarity parameter between the personalized model and the global model to control the personalization degree of the model.

[0123] like Figure 2 The figure shows a flowchart of federated multi-task learning based on client clustering provided by an embodiment of the present invention.

[0124] Step 1: Initialize global model training to obtain gradients that are highly correlated with the heterogeneity of its clients. Construct a client similarity clustering model based on the model training gradient, and design a client similarity clustering model based on the model training gradient to provide a classification basis for personalized federated learning while protecting user privacy.

[0125] Step 2: Based on the improved BKM algorithm of multi-sphere splitting, clustering is completed according to client similarity. The low-complexity center increment method is used to improve clustering efficiency.

[0126] Step 3: The clients in the cluster execute the soft sharing strategy until the personalized model is obtained after convergence. A personalized federated learning method based on soft parameter sharing is designed. The clients in the cluster execute the soft sharing strategy. The bottom layer of the model uses regularization terms to keep correlation with the global model, while the top layer of the model performs personalized training based on local data distribution and needs until a personalized model is obtained after convergence.

[0127] In one embodiment of the present invention, the experiment is trained on the MNIST and CIFAR 10 data sets. Each data set is divided into equal subsets according to the number of clients, and each subset has a main class, that is, a label with a larger proportion in the subset. The experiment assumes that the main class represents the category of the client. Set ratio to adjust the proportion of the main class in the subset. The larger the ratio, the more extreme the data distribution. The test data is divided into 10 subsets and randomly assigned to the client. The training set and test set in the client have the same main class. The training model uses LeNet, and the model structure is set to 2 convolutional layers and 3 fully connected layers. The learning rate and decay of the Adam optimizer used are set to 0.01 and 0.001, respectively. In the improved BKM algorithm based on multi-ball splitting, k0=5, iter=2. The following two comparison algorithms are selected.

[0128] (1) Federated multi-task learning: All clients share a global reference model to perform federated multi-task learning to obtain a personalized model.

[0129] (2) Fedavg: In each iteration, a client is randomly selected and the global model parameters are sent down. The selected client performs local training and uploads the parameters for model aggregation.

[0130] First, the performance of the improved BKM algorithm based on multi-ball splitting in solving the client's similarity classification is analyzed. The clustering experiment takes the MNIST dataset as an example, and compares and verifies the clustering time with K-Means and the BKM algorithm based on multi-ball splitting under the condition that the same main class is divided into the same cluster.

[0131] First, 20 rounds of global iterative training of federated learning are performed, and the gradient values ​​of 62018 clients are recorded. Figure 3 The changes in clustering time of the three algorithms under fixed gradient dimensions and different numbers of clients are given. It can be seen from the figure that under different numbers, the algorithm proposed in the present invention can always maintain a higher clustering efficiency. Taking the number of 300 as an example, the improved BKM algorithm reduces the delay by 5% and 9% respectively compared with the K-means algorithm and the BKM algorithm. Although the BKM algorithm based on multi-ball splitting designs multi-ball splitting to speed up the calculation, it is still very sensitive to the number of initialized centroids. The improved BKM algorithm based on multi-ball splitting improves the centroid increment calculation and has good clustering efficiency.

[0132] Figure 4 The changes in the accuracy of the three algorithms when ratio = 0.5 under the MNIST data set are given. It can be seen from the figure that when the proportion of the main class in the client is not large, that is, the data distribution is relatively even, C federated multi-task learning is close to convergence in 20 rounds, while federated multi-task learning and Fedavg are close to convergence in 150 rounds. At the same time, C federated multi-task learning has the highest accuracy. Taking the 40th round as an example, the accuracy of C federated multi-task learning reached 96.07%, which is 4.3% and 8.7% higher than that of federated multi-task learning and Fedavg respectively. It can be concluded that the C federated multi-task learning mechanism proposed in the present invention has good convergence ability and accuracy. Figure 5 The figure shows the changes in the loss functions of the three algorithms when ratio = 0.5 under the MNIST dataset. Similar to the convergence of accuracy, the C federated multi-task learning algorithm converges faster than the other two algorithms and has the smallest loss function value.

[0133] Figure 6 The changes in the accuracy of the three algorithms when ratio = 0.7 in the MNIST dataset are given. When the proportion of the main class in the client is large, that is, when the distribution of the main class shows a certain extreme, C federated multi-task learning is close to convergence in 25 rounds, while federated multi-task learning and Fedavg are close to convergence in 200 rounds. At the same time, C federated multi-task learning has the highest accuracy. Taking the 50th round as an example, the accuracy of C federated multi-task learning reached 95.60%, which is 9.4% and 19.5% higher than that of federated multi-task learning and Fedavg respectively. In addition, from Figure 7It can be seen that the C federated multi-task learning algorithm can converge faster than the other two algorithms, and the loss function value is the smallest. It can be concluded that the C federated multi-task learning mechanism can still have good performance when facing clients with certain extreme data distribution.

[0134] The above is a personalized federated learning method based on client clustering provided by an embodiment of the present invention. Based on the same inventive idea, an embodiment of the present invention also provides a corresponding personalized federated learning device based on client clustering, such as Figure 8 shown.

[0135] The gradient acquisition module 801 is used to obtain the global model, perform initialization training on the global model, and obtain the gradient generated by each client in the initialization training; the cluster analysis module 802 is used to perform cluster analysis based on the gradient of the client to determine the cluster to which each client belongs; the shared training module 803 is used to perform shared training on the first hidden layer set of the deep neural network trained on the client in each cluster, and perform personalized training on the second hidden layer set of the deep neural network trained on the client in each cluster, until the local model converges to obtain a personalized model.

[0136] The embodiment of the present invention also provides a corresponding personalized federated learning device based on client clustering, such as Fig. 9 shown.

[0137] This embodiment provides a personalized federated learning device based on client clustering, including:

[0138] At least one processor 901; and a memory 902 communicatively connected to the at least one processor 901 via a bus 903; wherein the memory 902 stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor 901 so that the at least one processor 901 can execute:

[0139] Obtain the global model, perform initialization training on the global model, and obtain the gradient generated by each client in the initialization training; perform cluster analysis based on the client's gradient to determine the cluster to which each client belongs; perform shared training on the first hidden layer set of the deep neural network trained on the clients in each cluster, and perform personalized training on the second hidden layer set of the deep neural network trained on the clients in each cluster until the local model converges to obtain a personalized model.

[0140] Based on the same idea, some embodiments of the present invention also provide a medium corresponding to the above method.

[0141] Some embodiments of the present invention provide a storage medium storing computer executable instructions, wherein the computer executable instructions are executed by a processor to implement the following steps:

[0142] Obtain the global model, perform initialization training on the global model, and obtain the gradient generated by each client in the initialization training; perform cluster analysis based on the client's gradient to determine the cluster to which each client belongs; perform shared training on the first hidden layer set of the deep neural network trained on the clients in each cluster, and perform personalized training on the second hidden layer set of the deep neural network trained on the clients in each cluster until the local model converges to obtain a personalized model.

[0143] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the method and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0144] The method and medium provided in the embodiment of the present invention correspond one-to-one to the method, and therefore, the method and medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the method and medium will not be repeated here.

[0145] It should also be noted that the terms "include", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or method. In the absence of further restrictions, the elements defined by the sentence "including a personalized federated learning based on client clustering" do not exclude the existence of other identical elements in the process, method, commodity or method including the elements.

[0146] The above are only embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

Claims

1. A personalized federated learning method based on client clustering, characterized in that: include: Obtain a global model, perform initialization training on the global model using an image dataset, and obtain a gradient generated by each client during the initialization training; Performing cluster analysis according to the gradient of the client to determine the cluster to which each client belongs, specifically includes: Determine a weighted client according to the data set of the client and the gradient corresponding to the client; Initialize the center of mass and radius of multiple ball clusters; Divide each of the weighted clients into a ball cluster where the centroid closest to the client is located; Determine the neighborhood relationship between any two ball clusters according to the radius and the distance between the centroids of any of the ball clusters; According to the neighborhood relationship and the distance between the weighted client and the centroid of the ball cluster to which it belongs and its neighborhood ball cluster, move the weighted client in the ball cluster to its neighborhood ball cluster; According to the distance between the weighted client and the centroid of the ball cluster to which it belongs and its neighboring ball clusters, the weighted clients in each ball cluster are continuously iterated until the weighted clients in each ball cluster remain unchanged; Performing cluster analysis according to the gradient of the client also includes: Determine the average value of all cluster radii and the incremental step size for cluster splitting; Compare the radius of each cluster with the average value; Splitting the clusters whose cluster radius is larger than the average value; When it is determined that the number of split ball clusters is greater than or equal to the incremental step size, the splitting is stopped; The first hidden layer set of the deep neural network trained by the client in each cluster is shared and trained, and the second hidden layer set of the deep neural network trained by the client in each cluster is personalized and trained until the local model converges to obtain a personalized model.

2. The method according to claim 1, characterized in that: The obtaining of the gradient generated by each client in the initialization training specifically includes: Sending the global model to each client to form a local model corresponding to each client; Training the local model to determine a loss function of the local model corresponding to each of the clients, wherein the loss function includes a target parameter of the local model; Continuously iterating the target parameter n times by using a stochastic gradient descent algorithm until the target parameter is determined to minimize the loss function; Obtain a gradient corresponding to each of the clients.

3. The method according to claim 1, characterized in that The step of continuously iterating the weighted clients in each of the clusters according to the distances from the weighted client to the centroids of the cluster to which it belongs and its neighboring clusters specifically includes: Determine the stable and active domains of the ball cluster; If it is determined that the weighted client is in the stable domain, then in the current iteration, the weighted client is not moved to other neighboring ball clusters of the ball cluster; Dividing the activity area of ​​the ball cluster into a plurality of annular areas, and determining the annular area to which the weighted client belongs; According to the annular area to which the weighted client belongs and the distance from the weighted client to the centroid of the ball cluster to which it belongs and its neighboring ball cluster, the weighted client is moved to the corresponding neighboring ball cluster or the ball cluster to which it belongs.

4. The method according to claim 1, characterized in that The determining the neighborhood relationship between any two ball clusters according to the radius and the distance between the centroids of any of the ball clusters specifically includes: Determine a difference in centroids between the first ball cluster and the second ball cluster, and determine a norm of the centroid difference; If half of the norm of the centroid difference is smaller than the radius of the first cluster, the second cluster is determined to be a neighboring cluster of the first cluster.

5. The method according to claim 1, characterized in that The first hidden layer set of the deep neural network trained by the client in each cluster is shared and trained, and the second hidden layer set of the deep neural network trained by the client in each cluster is personalized and trained until the local model converges to obtain a personalized model, specifically including: The cluster to which the client belongs is a ball cluster; Determine a first hidden layer set shared by clients in the ball cluster, and use the shared hidden layer training as a global model training task; Determine a second hidden layer set for clients in the ball cluster that is higher than the first hidden layer set, and train the second hidden layer set for each client as a personalized model training task; Determine a first objective function for each client when performing the global model training task; Determine a second objective function for each client when performing the personalized model training task based on the first objective function.

6. A personalized federated learning device based on client clustering, characterized in that: include: A gradient acquisition module is used to acquire a global model, perform initialization training on the global model using an image data set, and acquire the gradient generated by each client in the initialization training; A cluster analysis module is used to perform cluster analysis according to the gradient of the client to determine the cluster to which each client belongs, specifically including: Determine a weighted client according to the data set of the client and the gradient corresponding to the client; Initialize the center of mass and radius of multiple ball clusters; Divide each of the weighted clients into a ball cluster where the centroid closest to the client is located; Determine the neighborhood relationship between any two ball clusters according to the radius and the distance between the centroids of any of the ball clusters; According to the neighborhood relationship and the distance between the weighted client and the centroid of the ball cluster to which it belongs and its neighborhood ball cluster, move the weighted client in the ball cluster to its neighborhood ball cluster; According to the distance between the weighted client and the centroid of the ball cluster to which it belongs and its neighboring ball clusters, the weighted clients in each ball cluster are continuously iterated until the weighted clients in each ball cluster remain unchanged; Performing cluster analysis according to the gradient of the client also includes: Determine the average value of all cluster radii and the incremental step size for cluster splitting; Compare the radius of each cluster with the average value; Splitting the clusters whose cluster radius is larger than the average value; When it is determined that the number of split ball clusters is greater than or equal to the incremental step size, the splitting is stopped; The shared training module is used to perform shared training on the first hidden layer set of the deep neural network trained by the client in each cluster, and to perform personalized training on the second hidden layer set of the deep neural network trained by the client in each cluster until the local model converges to obtain a personalized model.

7. A personalized federated learning device based on client clustering, using the method according to any one of claims 1 to 5, characterized in that: include: at least one processor; as well as, A memory connected to the at least one processor via a bus; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed to implement: Obtain a global model, perform initialization training on the global model using an image dataset, and obtain a gradient generated by each client during the initialization training; Performing cluster analysis according to the gradient of the client to determine the cluster to which each client belongs; The first hidden layer set of the deep neural network trained by the client in each cluster is shared and trained, and the second hidden layer set of the deep neural network trained by the client in each cluster is personalized and trained until the local model converges to obtain a personalized model.

8. A non-volatile storage medium storing computer executable instructions, using the method according to any one of claims 1 to 5, characterized in that: The computer executable instructions are executed by a processor to implement the following steps: Obtain a global model, perform initialization training on the global model using an image dataset, and obtain a gradient generated by each client during the initialization training; Performing cluster analysis according to the gradient of the client to determine the cluster to which each client belongs; The first hidden layer set of the deep neural network trained by the client in each cluster is shared and trained, and the second hidden layer set of the deep neural network trained by the client in each cluster is personalized and trained until the local model converges to obtain a personalized model.