A Clustered Federated Learning Method for Heterogeneous Statistics

Abstract

This invention discloses a clustered federated learning method for heterogeneous statistics, comprising: Step 1, constructing an edge node distribution classifier; Step 2, determining the metric for edge node clustering; Step 3, determining the clustering method for node clusters; Step 4, clustering the edge nodes using the clustering method; Step 5, the server initializes the global model and sends the model to the head node of each node cluster; Step 6, after receiving the model, the edge node trains locally on its local dataset and updates the model, then sends the updated model to the next node in the cluster for training, until all nodes in each cluster have completed training and upload the updated model to the server; Step 7, the server receives all the updated models from all clusters, performs a weighted average, and updates the global model; Step 8, repeating steps 6 and 7 until the global model converges. Compared to traditional federated learning methods, this invention is more efficient and has wider applicability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a clustered federated learning method for heterogeneous statistics. Background Technology

[0002] Modern mobile and IoT devices (such as smartphones, smart wearables, and smart home devices) generate massive amounts of data daily, providing opportunities to create complex machine learning (ML) models to solve challenging artificial intelligence tasks. In traditional high-performance computing (HPC), all data is collected and centralized in one place, processed by supercomputers with hundreds to thousands of computing nodes. However, security and privacy concerns have led to new legislation, such as the General Data Protection Regulation (GDPR) and the Health Insurance Portability and Accountability Act (HIPAA), which prevent data transfer to a centralized location, making traditional HPC difficult to apply to collecting and processing distributed data. Federation learning addresses these security and privacy challenges by leveraging distributed data. It trains local models on local data at each client (data party) and uses a central aggregator to accumulate the learning gradients of these local models to train a global model, thus embracing a new paradigm in high-performance computing. While the computing resources of a single client may be far less powerful than the computing nodes of a traditional supercomputer, the computing power from a large number of clients can be aggregated to form a very powerful "distributed virtual supercomputer." Federation learning has already demonstrated its success in a range of applications. From consumer devices like GBoard and keyword discovery to pharmaceuticals, medical research, finance, and manufacturing.

[0003] In federated learning, the data is owned by the clients, and its quantity and content can vary significantly. This leads to severe data heterogeneity, a type of heterogeneity typically absent in data-distributed learning where data distribution is well-controlled. In data-distributed learning, the categories and features of the training data are uniformly distributed across all clients, a pattern known as Independent Identical Distribution (IID). However, in federated learning, the distribution of data categories and features depends on the data owner, resulting in a non-uniform data distribution, termed Non-Independent Identical Distribution (Non-IID Data Heterogeneity). This heterogeneity significantly impacts training time and accuracy, necessitating a technical solution to address this situation. Summary of the Invention

[0004] This invention addresses the problem in federated learning where the distribution of data categories and features depends on the data owner, leading to non-uniform data distribution and significantly impacting training time and accuracy. It proposes a clustered federated learning method oriented towards heterogeneous statistics, applicable to federated learning environments with statistically heterogeneous data, achieving a more efficient and versatile federated learning approach.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A clustered federated learning method for heterogeneous statistics includes:

[0007] Step 1: Construct an edge node distribution classifier;

[0008] Step 2: Determine the metrics for edge node clustering;

[0009] Step 3: Determine the clustering method for the node cluster;

[0010] Step 4: Cluster the edge nodes using a clustering method;

[0011] Step 5: The server initializes the global model and sends the model to the head node of each node cluster;

[0012] Step 6: After receiving the model, the edge node trains and updates the model locally on the local dataset, and sends the updated model to the next node in the cluster for training, until all nodes in the cluster have completed training and upload the updated model to the server.

[0013] Step 7: After receiving the updated models from all clusters, the server performs a weighted average and updates the global model.

[0014] Step 8: Repeat steps 6 and 7 until the global model converges.

[0015] Further, step 1 includes:

[0016] global model f θ Divide into a deep feature extractor and a classifier Where θ=(θ feat ,θ clf ) is the parameter set of the global model;

[0017] Before the federated learning officially begins, a pre-training phase is used to estimate the data distribution on the edge nodes participating in the training. During this phase, each edge node k starts with the same random initialization θ0, trains for e rounds on its local dataset, and updates the model accordingly.

[0018] Based on local classifiers The parameter ψ clf Or its public dataset on the server side On the prediction ψ conf Construct an edge node distribution classifier;

[0019] On the server side, a classifier is used to update the model based on the edge nodes. Obtain an estimate of the data distribution for this node.

[0020] Further, step 2 includes:

[0021] Approximate value of data distribution from edge node k Initially, similar node clusters are established from nodes with different distributions, minimizing the distance between node clusters while maximizing the distance within node clusters;

[0022] Use cosine and Euclidean distances to compare the weights of the customer classifiers. The confidence vector represents the actual probability distribution, and KL divergence is used as a metric.

[0023] Furthermore, the clustering method includes:

[0024] Strategy 1: Customers are randomly assigned to a node cluster until the defined stopping criteria are met;

[0025] Strategy 2: First, use the K-means method to obtain N. S 1. Form homogeneous clusters; then, form all node clusters by iteratively extracting one edge node from each cluster each time, until the number of samples in each node cluster S is equal to the number of samples in the cluster. and edge node K S ≤k S,max ;

[0026] Strategy 3: Randomly select an edge node k i Assign nodes to the current node cluster S, i∈[K]; then select the second edge node k. j , making k i and k j The distance between them reaches its maximum, that is Repeat this process continuously, and finally maximize through iteration. Reaching the predetermined maximum number of edge nodes K S,max and minimum sample size Where τ is a measure of clustering.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] This invention is applicable to federated learning environments with statistically heterogeneous data. It can be easily deployed in a traditional two-tier server-edge node architecture, and can also be expanded to a three-tier cloud-edge server-edge node architecture. Compared to traditional federated learning methods, this invention is more efficient and has wider applicability. Attached Figure Description

[0029] Figure 1 This is one of the flowcharts of a clustered federated learning method for heterogeneous statistics according to an embodiment of the present invention;

[0030] Figure 2 This is the second flowchart of a clustered federated learning method for heterogeneous statistics according to an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0032] The goal of traditional federated learning is to learn a global model. Each edge node k∈[K] can be obtained from the local dataset. Obtain n k For a given number of samples, the mainstream federated learning method, FedAvg, employs an iterative approach based on T communication rounds, aiming to solve... in It is a localized, experiential risk. k It is the cross-entropy loss, n = ∑ k n k This is the total amount of data used in training. In each round t∈[T], the server will... t Send to a randomly selected A portion of the customers. Each customer Using D by minimizing the local objective k Perform local gradient descent, and set θ t Updated to This is then returned to the server. The updated model is then aggregated by the server into a new global model. In, that is However, in real-world scenarios, it cannot be guaranteed that local datasets from different customers are extracted independently from the same underlying distribution.

[0033] In response to the above problems, such as Figure 1 , Figure 2 As shown, this invention proposes a clustered federated learning method for heterogeneous statistics, comprising:

[0034] Step 1: Construct an edge node distribution classifier ψ. This invention uses the global model f... θ Divide into a deep feature extractor and a classifier Where θ=(θ feat ,θ clf () is the parameter set of the global model. The classification output is... Given the following: Before the federated learning officially begins, a pre-training phase is used to estimate the data distribution on the edge nodes participating in the training. During this phase, each edge node k starts from the same randomly initialized θ0, trains for e rounds on its local dataset, and updates the model accordingly. This invention uses two strategies, one based on a local classifier. The parameter ψ clf Or its public dataset on the server side On the prediction ψ conf For strategy one, it is assumed that the weights of the classifier can represent the local distribution of each customer, and are directly fed back to the clustering method φ. (.) For strategy two, within a common "feature set"... Test each in Includes c∈[N C J samples from [the class]. Then, sort them by class. The confidence vector of the k-th customer is defined by averaging the predictions. On the server side, a classifier ψ is used to update the model based on the edge nodes. Obtain an estimate of the data distribution for this node.

[0035] Step two, determine the clustering metric τ. This is based on an approximation of the data distribution at edge node k. Initially, similar node clusters are established from nodes with different distributions, with the aim of minimizing the distance between node clusters while maximizing the distance within node clusters. Given and We need to find a metric to measure the distance between two distribution estimates. Use cosine and Euclidean distances to compare the weights of the customer classifiers. The confidence vector represents the actual probability distribution, and KL divergence is used as a metric.

[0036] Step 3: Determine the clustering method φ for the node cluster. First, define... For customers belonging to node cluster S The set of data. To find the set that satisfies: minimum number of samples. and maximum number of customers K S,max The maximum number of nodes N in the constraint conditions S Given a classifier ψ distributed at an edge node (.)Given the clustering metric τ, this invention introduces three strategies to find approximate values ​​for the maximization problem. The first is φ. rand The first strategy is a simple and practical method where clients are randomly assigned to a node cluster until a defined stopping criterion is met. The second is φ. kmeans Strategy, based on the K-means algorithm: First, use the K-means method to obtain N. S 1. Form homogeneous clusters; then, form all node clusters by iteratively extracting one edge node from each cluster each time, until the number of samples in each node cluster S is equal to the number of samples in the cluster. and edge node K S ≤k S,max Finally, φ geedy The strategy follows a greedy approach to generate the node cluster. Initially, an edge node k is randomly selected. i Assign the node to the current node cluster S, i∈[K]. Then, select the second edge node k. j , making k i and k j The distance between them reaches its maximum, that is Repeat this process continuously, and finally maximize through iteration. Reaching the predetermined maximum number of edge nodes K S,max and minimum sample size

[0037] Step 4: According to the clustering method, after pre-training is completed, all edge nodes participating in training are divided into i node clusters, combining edge nodes with different distributions together, and at the same time, dividing edge nodes with similar distributions.

[0038] Step 5: The server initializes a global model θ. t It communicates with all edge nodes participating in training and sends the model to the entire node cluster S. i , i∈[N S The head node k i,1 .

[0039] Step 6, node k i,1 Upon receiving model θ t Then, in local data E k Each training round updates the model to... Then Send to the next edge node k within this node cluster i,2 Repeat this process until the last client in the current node cluster. Received the model and completed local training. After training is complete, the updated model will be... Send to the head node k in this cluster i,1 .

[0040] Step 7, the head node k in this cluster i,1 Received model Then, based on the training results, determine whether to repeat E. S Step six, if not needed, then the model The data is sent to the server. After the server receives the updated model from all nodes in the cluster, it then...

[0041]

[0042] The model update is averaged.

[0043] Step 8: Repeat steps 6 and 7 until the global model converges.

[0044] It's worth noting that if this method is applied to a three-tier architecture of cloud-edge server-edge node, the edge server at the edge server-edge node level can simply be viewed as the server in the steps described above. At the server-edge server level, the edge server can be considered as the edge node in the steps described above, and sequential training can be performed among the edge servers without further clustering. This is based on the fact that merging models is only useful after the model has been trained on a larger dataset. According to statistics, in N... S After each round, each model may have been trained on the entire dataset, thus the performance of the strategy becomes increasingly close to that of the centralized strategy.

[0045] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A clustered federated learning method for heterogeneous statistics, characterized in that, include: Step 1: Construct an edge node distribution classifier; Step 2: Determine the metrics for edge node clustering; Step 3: Determine the clustering method for the node cluster; Step 4: Cluster the edge nodes using a clustering method; Step 5: The server initializes the global model and sends the model to the head node of each node cluster; Step 6: After receiving the model, the edge node trains and updates the model locally on the local dataset, and sends the updated model to the next node in the cluster for training, until all nodes in the cluster have completed training and upload the updated model to the server. Step 7: After receiving the updated models from all clusters, the server performs a weighted average and updates the global model. Step 8: Repeat steps 6 and 7 until the global model converges; Step 1 includes: The global model f θ Divide into a deep feature extractor and a classifier Where θ=(θ feat ,θ clf ) is the parameter set of the global model; Before the federated learning officially begins, a pre-training phase is used to estimate the data distribution on the edge nodes participating in the training. During this phase, each edge node k starts with the same random initialization θ0, trains for e rounds on its local dataset, and updates the model accordingly. Based on local classifiers The parameter ψ clf Or its public dataset on the server side On the prediction ψ conf Construct an edge node distribution classifier; On the server side, a classifier is used to update the model based on the edge nodes. Obtain an estimate of the data distribution for this node. The clustering method includes: Strategy 1: Customers are randomly assigned to a node cluster until the defined stopping criteria are met; Strategy 2: First, use the K-means method to obtain N. S 1. Form homogeneous clusters; then, form all node clusters by iteratively extracting one edge node from each cluster each time, until the number of samples in each node cluster S is equal to the number of samples in the cluster. and edge node K S ≤k S,max ; Strategy 3: Randomly select an edge node k i Assign nodes to the current node cluster S, i∈[K]; then select the second edge node k. j , making k i and k j The distance between them reaches its maximum, that is Repeat this process continuously, and finally maximize through iteration. Reaching the predetermined maximum number of edge nodes K S,max and minimum sample size Where τ is a measure of clustering.

2. The clustered federated learning method for heterogeneous statistics according to claim 1, characterized in that, Step 2 includes: Approximate value of data distribution from edge node k Initially, similar node clusters are established from nodes with different distributions, minimizing the distance between node clusters while maximizing the distance within node clusters; Use cosine and Euclidean distances to compare the weights of the customer classifiers. The confidence vector represents the actual probability distribution, and KL divergence is used as a metric.

Images