Heterogeneous client-oriented federated learning method based on hierarchical sampling optimization

Abstract

The application discloses a heterogeneous client-oriented joint learning method based on hierarchical sampling optimization, and the available clients are selected from different clusters; a parameter server broadcasts a global model to all clients, and the clients train local data samples to obtain local model parameters; the parameter server collects local model parameter information of each client, and adopts a clustering method to divide the clients into different clusters; in each round of training, the available clients are extracted from each cluster according to sample weights to participate in training and perform gradient aggregation; after receiving the latest global model parameters from the parameter server, the clients participating in training in each round calculate the gradient under the current parameters by using local data, and after iteration, the latest parameters are sent back to the parameter server, and the parameter server performs weighted average on the returned parameters. The application can converge to a global optimal solution under the condition of client heterogeneity.

Description

Technical Field

[0001] This invention belongs to the field of distributed learning technology, and particularly relates to a joint learning method for heterogeneous clients based on hierarchical sampling optimization. Background Technology

[0002] Federated learning is a new distributed learning paradigm. Compared with traditional distributed machine learning methods, it has the following characteristics: (1) there are significant differences in communication, computing power, and storage capabilities among clients (device heterogeneity); (2) there are significant differences in the distribution and quantity of data among clients (data heterogeneity); and (3) high communication consumption. Under client heterogeneity conditions (including device heterogeneity and data heterogeneity), the data distribution among clients varies greatly, which leads to a significant reduction in the model convergence speed. Especially in extreme data heterogeneity cases, traditional federated learning algorithms cannot converge, and the training curve fluctuates significantly with the increase of local iteration rounds.

[0003] Federated learning is a novel distributed machine learning architecture that allows multiple devices (referred to as clients in federated learning) to collaboratively train a global model without needing to upload local data. Compared to traditional distributed machine learning, it differs primarily in the following ways: nodes have independent control over their local devices and data; client nodes are often unreliable (edge ​​nodes in federated learning frequently go offline due to device and communication limitations); communication costs in federated learning exceed computational costs; the data distribution of nodes in federated learning is non-independent and identically distributed (non-IID); and the distribution of local data among clients in federated learning is highly uneven. These new characteristics pose challenges to the design and analysis of federated learning algorithms.

[0004] One major challenge is client heterogeneity, which includes both data heterogeneity and device heterogeneity. Client heterogeneity in federated learning is widespread in real-world conditions. Examples include non-identically distributed client distributions (because each client's data is generated locally, and the sample generation mechanisms may differ between clients, such as different countries or regions); skewed feature distributions (covariate drift), such as in handwriting recognition where even the same character is written differently by different people; skewed label distributions (prior probability drift), such as the predominantly Chinese language user base in China; and imbalanced data sets. Various real-world situations can lead to non-independent and identically distributed data, as well as differences in device computing power and communication capabilities. Traditional machine learning is based on the assumption of independent and identically distributed data, but federated learning differs from centralized machine learning in that, without centralized data, the data at each node is not independently and identically distributed.

[0005] Consider a real-world scenario: when using federated learning to train a mobile input method model, different phones have varying processing speeds, internal data, and network conditions. Newer phones have faster processing and transmission speeds than older models, and phones in towns with better signal strength offer more stable communication than those in rural areas with poor signal. During model training, older phones train slowly and often fail to complete training tasks on time. Phones with poor network conditions are also more prone to signal loss when transmitting the model, causing a discrepancy between the data distribution received by the parameter server and the actual distribution. Furthermore, due to the heterogeneity of the clients, certain types of data participate more frequently in the training process, introducing errors into the training data.

[0006] In recent years, the improvement of machine storage capacity and computing power in the field of big data has greatly promoted the development of large-scale, data center-based distributed machine learning. Traditional distributed machine learning requires the entire dataset to be centralized on a single node or data center for training. However, with the increase in local computing power of mobile devices such as smartphones, smart wearables, and sensors, and the restrictions on user data privacy protection in recent years, training locally on distributed devices and then transmitting the training parameters to a parameter server is more efficient than transmitting data to a central node. This problem is called federated learning, which needs to address the challenges of large-scale training data, privacy protection, heterogeneous data, and devices.

[0007] In 2016, McMahan et al. proposed a deep network federated learning method based on iterative model averaging (FedAvg). Because the learning task is performed through a loose federation of clients coordinated by a central server, these clients resemble a loose federation, hence the name federated learning. Compared to data center-based distributed machine learning, a major advantage of federated learning is that it separates the need for model training from the need for direct access to raw data. This is significant in scenarios with strict data privacy requirements or where data is difficult to centrally share. Furthermore, the FedAvg algorithm uses multiple rounds of local iteration to accelerate learning efficiency, which greatly helps reduce communication overhead.

[0008] Peter Kairouz et al. discussed the latest developments in federated learning and summarized the pressing challenges currently facing it: the problem of non-independent and identically distributed data; the privacy protection of personal data; training under limited communication bandwidth; robustness against malicious nodes and attacks; and fairness in federated learning. The article points out that heterogeneous data and devices on the client side significantly impact learning efficiency in federated learning, which is also one of the pressing challenges currently facing the field.

[0009] To address the issue of non-independent and identically distributed (i.i.d.) data in federated learning, Yue Zhao et al. improved the FedAvg algorithm. They found that applying the FedAvg algorithm resulted in significant accuracy loss when the data was non-independent and identically distributed. They proposed using earthmoving operations to calculate weight divergence, which improved the accuracy of federated learning on non-IID data. Furthermore, they proposed a data-sharing federated learning strategy, creating a small, globally shared dataset across all client devices on a central server to improve training performance on non-IID data. While this approach reduces the impact of data skew, it artificially introduces error, and the data-sharing method fundamentally violates the data privacy principles of federated learning, making implementation extremely difficult.

[0010] Tian Li et al. started with the objective function, adding a constraint to the model's objective function to ensure that the new model doesn't deviate too much from the global model when each client updates using local data, thus reducing the impact of data heterogeneity. Jiang Y et al. took a different approach, arguing that the existence of data heterogeneity prevents us from obtaining a sufficiently accurate global model. Therefore, they personalized the model, using local data to additionally train the global model, resulting in a higher-quality personalized model. Similar to this idea, A. Ghosh et al. and Sattler et al. proposed a method of dividing clients into different clusters and then training a separate global model in each cluster. They used different clustering methods to cluster the client's local empirical loss function or node gradients. Because clients within each cluster have high similarity, the trained intra-cluster global model has high accuracy. However, the model trained using this method has poor generalization ability, violating the principle of co-training, and the clustering methods they used require specifying the number of clusters in advance, which presents difficulties in practical applications. Yikai Yan et al. considered the intermittent availability of clients and believed that different clients participated in training at different times, causing the trained model to be biased towards the data of clients that participated in training more. Therefore, when selecting clients, they prioritized selecting clients that participated in training less often to ensure that each client participated in training at the same number of times as much as possible.

[0011] Under heterogeneous client conditions (including device heterogeneity and data heterogeneity), the data distribution of clients varies greatly, which greatly reduces the convergence speed of the model. Especially in the case of extreme data heterogeneity, traditional federated learning algorithms cannot converge, and the training curve fluctuates significantly with the increase of local iteration rounds. Summary of the Invention

[0012] To address the challenges posed by heterogeneous client architecture to model training in federated learning, this invention proposes FedSSO, a federated learning algorithm optimized by hierarchical sampling. FedSSO employs a density-based clustering method to divide the total number of clients into different clusters, ensuring high similarity among clients within each cluster. Then, usable clients are extracted from different clusters according to sample weights to participate in training. Therefore, data from all types participate proportionally in each training round, accelerating the model's convergence to the global optimum. Simultaneously, a decreasing learning rate and a local iteration round selection mechanism are implemented to guarantee model convergence.

[0013] Specifically, the joint learning method for heterogeneous clients based on hierarchical sampling optimization disclosed in this invention includes the following steps:

[0014] Select available clients from different clusters; the clients receive heterogeneous data.

[0015] The parameter server initializes the global model and then broadcasts the global model to all clients. The clients train their local model parameters based on the received global model and samples of their local data. The parameter server collects the local model parameter information of each client and uses a clustering method to divide the clients into different clusters.

[0016] During each training round, available clients are extracted from each cluster based on sample weights to participate in training, and gradient aggregation is performed to ensure that all types of data participate in each training round and reduce the impact of client heterogeneity; the training objective is a convex function.

[0017] After receiving the latest global model parameters from the parameter server in each round of training, the client uses local data to calculate the gradient under the current parameters. After E iterations of stochastic gradient descent, the client sends the latest parameters back to the parameter server, which then performs a weighted average of the returned parameters.

[0018] Furthermore, the average value of one-dimensional data is learned from N clients, and this objective is transformed into a problem of minimizing the mean square error, as follows:

[0019]

[0020] Where ξ i ~D i It is a sample independently selected from local data, with a mean e. i =E[ξ i ], ρ i ξ is the weight of the i-th client. i For the client's sample, τ i Let be the weight offset of the i-th client.

[0021] Furthermore, when each client contains the same amount of data, the optimal solution to the problem is:

[0022]

[0023] τ i Let be the weight offset of the i-th client. Then the objective function will converge to:

[0024]

[0025] Furthermore, the clustering method is the OPTICS clustering method.

[0026] Furthermore, the formulas for the client-side local model parameters are as follows:

[0027]

[0028] Where η is the learning rate and W0 is the global model. For client-side samples, Let be the loss function for the i-th client, where i is the i-th client.

[0029] Furthermore, the weighted average method is as follows:

[0030]

[0031] K represents the total number of clients drawn in each round, and k represents the kth client.

[0032] Furthermore, T ε The number of iterations required by the algorithm to achieve accuracy ε is the number of communications between the client and the parameter server. for:

[0033]

[0034] G 2 This is the bound of the expected square norm of the stochastic gradient. ρ is the variance bound of the stochastic gradient. k Here, represents the weight of the k-th client, E is the local iteration round, N is the total number of clients, K is the total number of clients extracted in each round, L denotes L-smooth, and μ denotes μ-strong convexity.

[0035] When the degree of data heterogeneity is low, and Γ is close to 0, the larger the local iteration round E is, the better; when the degree of data heterogeneity is high, the smaller Γ is, the smaller the local iteration round E is, the better.

[0036] With all clients participating in training, the algorithm's convergence is as follows:

[0037]

[0038] in, For heterogeneous data parameters.

[0039] With some clients participating in the training, the algorithm's convergence is:

[0040]

[0041] in

[0042] At the same time, the learning rate needs to be set to gradually decrease during training in order to converge to the optimal result and enable the algorithm to achieve O(E) time complexity. 2 The convergence speed of / T).

[0043] The beneficial effects of this invention are as follows:

[0044] Even for optimization problems with convex objective functions, traditional federated learning algorithms fail to converge to the global optimum under heterogeneous client conditions when calculating accurate gradients (non-stochastic gradients). In particular, under highly heterogeneous client conditions, traditional federated learning methods may diverge due to constant learning rates and high local iteration cycles, while this invention overcomes these problems.

[0045] A convergence algorithm is proposed, which employs a stratified sampling method. In each training round, a certain number of available clients are randomly selected from the pre-defined client cluster according to their weights to participate in the training. This algorithm can achieve O(E) convergence. 2 The solution converges to the global optimum at a rate of / T).

[0046] The FedSSO algorithm was evaluated using the standard datasets MNIST, CIFAR-10, and Sentiment140, and compared with FedAvg and FedProx. The evaluation results demonstrate that the FedSSO algorithm achieves higher training accuracy and faster training speed on heterogeneous datasets. Attached Figure Description

[0047] Figure 1 Flowchart of the present invention;

[0048] Figure 2 The results of this invention, along with FedAvg and FedProx, on the MNIST (non-IID) dataset;

[0049] Figure 3 The results of this invention, along with FedAvg and FedProx, on the MNIST (non-IID2) dataset;

[0050] Figure 4The results of this invention, along with FedAvg and FedProx, on the Cifar-10 (non-IID) dataset;

[0051] Figure 5 The results of this invention, along with FedAvg and FedProx, on the Cifar-10 (non-IID2) dataset;

[0052] Figure 6 The results of this invention, along with FedAvg and FedProx, on the Sentiment140 dataset;

[0053] Figure 7 Results of FedAvg under different degrees of heterogeneity in the MNIST(IID) dataset;

[0054] Figure 8 Results of FedAvg under different degrees of heterogeneity in the MNIST (non-IID) dataset;

[0055] Figure 9 Results of FedAvg under different degrees of heterogeneity in the MNIST (non-IID2) dataset;

[0056] Figure 10 Results of FedAvg under different degrees of heterogeneity in the Cifar-10 (IID) dataset;

[0057] Figure 11 Results of FedAvg under different degrees of heterogeneity in the Cifar-10 (non-IID) dataset;

[0058] Figure 12 Results of FedAvg on the Cifar-10 (non-IID2) dataset under different degrees of data distribution heterogeneity;

[0059] Figure 13 Results of this invention under different degrees of heterogeneity in the Sentiment140 dataset;

[0060] Figure 14 The results of this invention under extremely heterogeneous conditions on the MNIST (non-IID) dataset;

[0061] Figure 15 The results of this invention under extremely heterogeneous conditions on the MNIST (non-IID2) dataset;

[0062] Figure 16 The results of this invention under extremely heterogeneous conditions on the Cifar-10 (non-IID) dataset;

[0063] Figure 17Results of the present invention under extreme heterogeneous conditions of the Cifar-10 (non-IID2) dataset;

[0064] Figure 18 Results of the present invention under extreme heterogeneous conditions of the Sentiment140 dataset. Detailed implementation manners

[0065] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any transformation or replacement based on the teachings of the present invention falls within the protection scope of the present invention.

[0066] The optimized model in federated learning is:

[0067]

[0068] where N is the total number of clients, and ρ k is the weight of the k-th client. Assume that the local data distribution of the k-th client is D k , is a sample independently selected from the local data. In the standard FedAvg algorithm, (assuming the t-th round of iteration), first the central parameter server broadcasts the latest global parameter w t to all participating clients for training, and then each participating client performs E rounds of local iteration:

[0069]

[0070] where η t is the learning rate. Assume that K (1 ≤ K < N) clients are selected to participate in training in each round, and the central server aggregates the gradients of the collected clients:

[0071]

[0072] The distribution of the overall data is a mixture of all local data distributions: [[ID=4I]] When the client data belongs to independent and identically distributed, for all k ∈ N, D k = D. However, in real life, the data distributions of different clients are often not the same. Therefore, the present invention is based on the assumption of non-independent and identically distributed data.

[0073] Impact of data heterogeneity Example 1. Consider a distributed optimization problem. Assume that the objective function is a convex function, and the goal is to learn the average value of one-dimensional data from N clients. ξ i ~D i [[ID=5I]]with mean e i = E[ξ i . We transform this goal into a problem of minimizing the mean squared error:

[0074]

[0075] For ease of calculation, assuming that each client contains the same amount of data, the optimal solution can be obtained as follows: Assume τ i Let be the weight offset of the i-th client caused by communication loss, differences in client devices, etc. Then the objective function will converge to:

[0076] The proof, by differentiating the objective function, yields:

[0077]

[0078] Setting the derivative to 0, we get:

[0079]

[0080] Based on the assumption that each client contains the same amount of data, for any ρ i (i∈N), ρ i =1 / N, therefore When calculating the convergence value of the objective function under realistic conditions, ρ i =1 / N+τ i (τi is the aggregate weight offset of client i), If and only if e1 = e2 = ... = e n (Data distribution is IID) or for all i∈{1,2,...,N}, τ i When = 0, x = x * Therefore, traditional federated learning algorithms tend to produce poor results when faced with heterogeneous data.

[0081] The FedSSO algorithm architecture of this invention addresses the issue, as mentioned above, where data heterogeneity and device heterogeneity severely degrade the performance of the FedAvg algorithm. In federated learning, the overall data distribution is a weighted mixture of the local data distributions of each client; in the FedAvg algorithm, this weight is the sample weight. This setting only considers the difference in data volume between clients, neglecting differences in hardware devices and communication. For example, in a classic federated learning scenario, when training a mobile input method model, the latest mobile phones have faster operating and transmission speeds than older models, and phones in towns with better signal coverage have more stable communication transmissions than those in rural areas with signal interference. This causes a discrepancy between the data distribution received by the parameter server and the actual distribution. Due to client heterogeneity, certain types of data participate in the training process more frequently, introducing errors into the training data. To mitigate this error, we consider using all types of data for training in each round, ensuring that the probability of each type of data participating in training is approximately the same. This makes the training data distribution an unbiased mixture of the sample distributions from each client, thus eliminating bias in the training data and establishing convergent results.

[0082] Algorithm 1 provides a detailed explanation of the FedSSO algorithm. The client selection principle of the FedSSO algorithm is to choose available clients from different clusters (lines 2-8). The parameter server first initializes the global model, then broadcasts the model w0 to all clients. Each client, based on the received model, performs a parameter selection on a sample ξ of its local data. i The local model parameters w are obtained through training. i The parameter server collects local model parameter information from each client and uses the OPTICS (Ordering Points To Identify the Clustering Structure) clustering method to divide the clients into different clusters. OPTICS is a density-based clustering algorithm. It defines a cluster as the largest set of points connected by density and divides regions with sufficient density into clusters. Compared to K-means and BIRCH, OPTICS can find clusters of arbitrary shapes in noisy spatial data, while K-means and BIRCH are only suitable for clustering convex sample sets. Compared to the DBSCAN method, OPTICS is insensitive to input parameters, improving clustering stability.

[0083] During each training round, available clients are proportionally selected from each cluster to participate in the training. This ensures that all types of data participate in each training round, reducing the impact of client heterogeneity. After receiving the latest global model parameters from the parameter server in each round, the participating clients use their local data to calculate the gradient under the current parameters. After iterating through E stochastic gradient descent cycles, they send the latest parameters back to the parameter server, which then performs a weighted average of the returned parameters (lines 9-13).

[0084]

[0085]

[0086] This section demonstrates that the FedSSO algorithm achieves O(E) time complexity for strongly convex, smooth functions and heterogeneous data. 2 The algorithm converges to the global optimum. Furthermore, we analyzed the convergence conditions of the algorithm and the necessity of the decreasing learning rate and local iteration round selection mechanism.

[0087] This invention relates to functions F1, F2, ..., F N The following assumptions were made:

[0088] Assume 1. F1, F2, ..., F N It is L-smooth:

[0089] Assume 2. F1, F2, ..., F N It is μ-strong and strongly convex.

[0090] Assumption 3. Assumption This involves uniform random sampling from the local data of the k-th device. The variance of the stochastic gradient is bounded for each client: for k = 1, ..., N.

[0091] Assumption 4. The expected square norm of stochastic gradients is uniformly bounded:

[0092]

[0093] Quantitative indicators of data heterogeneity. Assume F * , The objective functions are F and F, respectively. k The optimal solution, we can obtain This is used to quantify the degree of data heterogeneity. When the client data distribution is IID, Γ = 0; the higher the degree of data heterogeneity, the higher the value of |Γ|.

[0094] Quantitative indicators of equipment heterogeneity. Assume τi Let be the expected difference between the aggregation weight and the parameter weights of the i-th client during model aggregation (this difference is affected by device computing power, communication environment, etc.). Assume... The objective function F k The optimal solution can be obtained by calculation. This is a quantitative indicator for equipment heterogeneity.

[0095] Convergence analysis of Example 1

[0096] First, we prove that FedSSO converges in Example 1, while FedAvg produces a biased result. Based on the configuration, different clients are divided into different clusters {c1,c2,...,c...}. n}, each cluster c i The number of clients is n ci Clients located within the same cluster have similar means, with the mean E(ξ) of each cluster being... ci ) = e ci . For any client k∈{c i}, We can rewrite the objective function as follows:

[0097]

[0098] Because each cluster class extracts only available clients, there are no instances of communication loss or client unavailability. The above formula can be rewritten as:

[0099]

[0100] When the learning rate η ≤ 2 / L, using the gradient descent method, the solution to the above equation is: Known We can obtain:

[0101]

[0102] When clustering is sufficiently accurate, the data in each cluster can be considered to be identically distributed, δ ci →0 The objective function converges to the optimal solution.

[0103] Full Client Participation: This section discusses the convergence of the FedSSO algorithm under full client participation training. In fact, since the FedSSO algorithm targets changes in client selection strategies, it is equivalent to the FedAvg algorithm when all clients participate in training. The convergence of the FedAvg algorithm has been extensively proven, but previous proofs did not consider communication loss and heterogeneous client devices in their presupposition of parameter aggregation weights. In reality, these factors introduce biases into the training objective and the optimal solution. In the proof, variables will be introduced to represent changes in client aggregation weights due to objective reasons.

[0104] Assume the algorithm terminates after T iterations and returns w. T As the solution, E represents the number of local iterations on the client side. We require that T be an integer multiple of E so that w can be output as expected. T .

[0105] Theorem 1. Based on assumptions 1-4, and L,μ,σ k G is as defined in the assumptions, and the learning rate η is assumed to be... t It is decreasing, γ > 0, and for all t ≥ 0, η t ≤2η t+E The FedSSO algorithm with all clients participating satisfies:

[0106]

[0107]

[0108]

[0109]

[0110] Partial Client Participation: This section discusses the convergence of the FedSSO algorithm under partial client participation. Because federated learning suffers severely from the "straggler's effect" (meaning all nodes wait for the slowest node) in a fully client-participatory model, partially client-participatory federated learning has more practical applications. Assume... S is the set of clients participating in training during the k-th iteration. t It is composed of clients randomly selected from various clusters, with a total of K clients selected in each round. Assuming the data volume of each client is balanced, and that all available clients are selected in each training round, it will not be affected by communication loss, therefore ρ1 = ρ2 = ... = ρ N =1 / N, the aggregation steps of FedSSO can be represented as:

[0111]

[0112] Define ρ1 = ρ2 = ... = ρ N =1 / N seems to violate the federated learning assumption about imbalance. We can solve this problem through the following transformation. Assumption This can be viewed as scaling the objective function. Therefore, the global objective function can be transformed into:

[0113]

[0114] Theorem 2: Based on assumptions 1-4, and L,μ,σ k G is as defined in the assumptions, and the learning rate η is assumed to be... t It is decreasing. γ >0, for all t≥0, η t ≤2η t+E B is defined as in Theorem 1. We can obtain:

[0115]

[0116] in, Necessity of Decreasing Learning Rate: This section will prove that choosing a gradually decreasing learning rate is essential for the convergence of federated learning under heterogeneous client conditions. In the previous proof, we obtained:

[0117]

[0118] It is evident that the algorithm's update process is similar to the Markov process, meaning that the next update of the global model is independent of the past and depends only on the current parameters. Since the model update is determined by two parts, we can see that for the above equation, the first term is negative and the second term is positive. Therefore, the choice of learning rate has a significant impact on the algorithm's convergence.

[0119] When the model parameters are close to the optimal solution If the learning rate η t If the first term is a constant, then the first term in the above equation approaches 0, and the second term is a positive constant. In this case, model updates will not reduce the objective function value, but only yield an approximate optimal solution. Therefore, it must be emphasized that this invention requires a decreasing learning rate to achieve convergence to the optimal result.

[0120] Mechanism for selecting the number of local iterations: According to the conclusion of Theorem 2, under appropriate parameter conditions, the dominant term of equation (4) can be obtained as:

[0121]

[0122] Use T εLet represent the number of iterations required for the algorithm to achieve ε precision. To determine the required number of communications, it can be simplified to:

[0123]

[0124] As can be seen from the above formula, a larger local iteration round is not necessarily better; a higher local iteration round may lead to an increase in the number of communications. In fact, for different parameters, there exists an optimal local iteration round E. The calculation of E is related to the model parameters and the degree of data heterogeneity Γ. When the degree of data heterogeneity is low, Γ is close to 0, the latter half of the formula is dominant, and a larger E is better. When the degree of data heterogeneity is high, the smaller Γ is, the more dominant the former half of the formula is, and a smaller E is better.

[0125] We will use different datasets and models to evaluate the FedSSO algorithm, compare it with the FedAvg and FedProx algorithms, and conduct experimental analysis on the local iteration rounds on the client side.

[0126] Datasets: Experiments were conducted using three different standard datasets, which were benchmark datasets summarized from previous work on federated learning. For convex problems, this invention used a multinomial logistic regression (MLP) model to compare the performance of different algorithms on the MNIST dataset. To simulate heterogeneous client environments, the data was distributed across 100 clients, with each client containing only 600 samples. Then, a more complex Cifar-10 dataset was selected. Since the images in this dataset are from common everyday objects such as airplanes and vehicles, it has more errors compared to handwritten font datasets. Therefore, this invention also evenly distributed the overall data across 100 clients, with each client containing data from only one category. To investigate the algorithm's performance on non-convex settings, this invention used an LSTM classifier to perform text sentiment analysis on tweets from the Sentiment140 dataset, where each account corresponds to one device, and the tweets posted by the account are from the local dataset.

[0127] Implementation: This invention selects the FedAvg and FedProx algorithms as baselines, with the parameter mu in FedProx set to 0.2 as per the original paper. To ensure that each sample drawn is an unbiased estimate of the overall sample, a target number of samples are randomly drawn without replacement from the overall sample (FedSSO algorithm draws from each cluster) in each round. This ensures that each sample appears only once in each training round. To simulate different levels of data heterogeneity, diverse sampling strategies are employed. When simulating the case of independent and identically distributed data, the population data is independently sampled without replacement and assigned to each client. This ensures that the data distribution of each client is an unbiased estimate of the population sample. When simulating the case of heterogeneous data, the population data is sorted by label size and then divided into different slices, so that each slice contains data of only one label. The slice data is then randomly assigned to different clients. To simulate different degrees of data heterogeneity, two heterogeneous datasets are designed: non-IID and non-IID2. In non-IID, each client contains data of two classes, while in non-IID2, each client contains data of only one class, simulating the case of extreme data heterogeneity.

[0128] For each dataset, we set the initial learning rate to 0.01, and in the FedSSO algorithm experiments, we set η... t The learning rate decreases by 0.01 / (1+t). The proportion of clients selected in each round to the total number of clients is 0.1, and the local batch size is 10. During clustering, the parameters of the OPTICS clustering method are: density = 2, radius = 0.25.

[0129] First, we tested the experimental results under different data distribution conditions. For the FedAvg and FedProx algorithms, we set the local iteration round E=1. For the FedSSO algorithm, we tested the different results with local iteration rounds of 1 and 5. The results are as follows: Figures 2-6As shown, for the experiments on convex problems, we tested them on the MNIST and Cifar-10 datasets under different heterogeneous data conditions; for the experiments on non-convex problems, we tested them on the Sentiment140 dataset. Since each Twitter user in the dataset is a client, their tweets are their local data, and we only consider a heterogeneous single distribution. It can be observed that the FedSSO algorithm converges in all five experimental settings. Conversely, the training curves of the FedAvg and FedProx algorithms fluctuate significantly, especially when the data distribution is extremely heterogeneous, the models fail to converge, which verifies Theorem 2. Specifically, because the similarity between data in different datasets varies, the MNIST dataset, with its grayscale images of numbers, shows less difference between different data types compared to Cifar-10 (3-channel color images recording common objects in daily life). Therefore, under heterogeneous data distribution conditions, the loss curve of the training model fluctuates less. Experiments on increasing the number of local iterations in the FedSSO algorithm show that when the data distribution is not highly heterogeneous, increasing the number of local iterations can accelerate the convergence of the model. Figure 2 , Figure 4 However, when the data distribution is extremely imbalanced ( Figure 3 , Figure 5 and Figure 6 Increasing the number of local iteration rounds has little effect, and we will discuss this issue in detail in the next section.

[0130] Local Iteration Round Selection Mechanism: This section first demonstrates the convergence of the traditional federated learning method (FedAvg) under different heterogeneous data settings by increasing the number of local iteration rounds. Figures 7-13 The results are shown in the figure. The choice of local iteration rounds must take into account the impact of data heterogeneity. An example is the experiment on the MNIST dataset. Figure 7 , Figure 8 and Figure 9When the data is IID (Independent Identifier), increasing local iterations can accelerate convergence. When the data is non-IID, increasing the number of local iterations can also accelerate convergence, but the improvement is not as significant as in the former case. When the data is non-IID2 (extremely heterogeneous data distribution), increasing the number of local iteration rounds on the client side will slow down the decrease in training loss; when the number of local iteration rounds is too high, the model cannot converge. Simultaneously, we can see that as the degree of data heterogeneity increases, the fluctuation of the model's training loss curve gradually increases. This verifies that setting a decreasing learning rate mechanism is necessary for convergence of federated learning under heterogeneous conditions. When the data distribution of different clients is uneven, increasing local iterations will deepen the differences between model parameters, leading to large fluctuations in the training loss curve. In this case, an excessively high learning rate will cause excessive differences between models on different clients, resulting in failure to converge. Therefore, for traditional federated learning, when the data is independent and uniformly distributed, increasing the number of local iterations can accelerate the model's convergence. When the data distribution is heterogeneous, increasing the number of local iterations may slow down the model's convergence or prevent convergence.

[0131] exist Figures 14-18 This demonstrates the convergence of the FedSSO invention under different local iteration rounds in cases of heterogeneous data distribution. When the degree of data heterogeneity is low (… Figures 14-16 Increasing the number of local iterations can accelerate model convergence. However, as data distribution becomes more heterogeneous, the effect of increasing the number of local iterations on accelerating model convergence gradually decreases, and in some cases, the model may even fail to converge. Figure 17-18 Therefore, the FedSSO algorithm can accelerate convergence by increasing the number of local iteration rounds when the client data is heterogeneous. However, in cases of extreme data heterogeneity, the FedSSO algorithm still cannot guarantee convergence with a high number of local iteration rounds. Therefore, under extremely heterogeneous conditions, choosing a lower number of local iteration rounds is a better option.

[0132] FedSSO employs a density-based clustering method to divide heterogeneous clients into sets of different clusters, ensuring high similarity among clients within each cluster. During each training round, a specified number of clients are proportionally selected from all clusters to participate in training, guaranteeing that all types of data are involved in each training round. Under the standard federated learning assumptions, we provide a convergence proof for the FedSSO algorithm and validate our theory through experiments on standard datasets. Experiments demonstrate that the FedSSO algorithm significantly improves upon FedAvg and FedProx algorithms on heterogeneous datasets. Finally, we analyze the convergence conditions of the algorithm, proving that decreasing the learning rate is crucial for model convergence.

[0133] The beneficial effects of this invention are as follows:

[0134] Even for optimization problems with convex objective functions, traditional federated learning algorithms fail to converge to the global optimum under heterogeneous client conditions when calculating accurate gradients (non-stochastic gradients). In particular, under highly heterogeneous client conditions, traditional federated learning methods may diverge due to constant learning rates and high local iteration cycles, while this invention overcomes these problems.

[0135] A convergence algorithm is proposed, which employs a stratified sampling method. In each training round, a certain number of available clients are randomly selected from the pre-defined client cluster according to their weights to participate in the training. This algorithm can achieve O(E) convergence. 2 The solution converges to the global optimum at a rate of / T).

[0136] The FedSSO algorithm was evaluated using the standard datasets MNIST, CIFAR-10, and Sentiment140, and compared with FedAvg and FedProx. The evaluation results demonstrate that the FedSSO algorithm achieves higher training accuracy and faster training speed on heterogeneous datasets.

[0137] As used herein, the term "preferred" is meant as an example, illustration, or illustration. Any aspect or design described herein as "preferred" need not be construed as being more advantageous than other aspects or designs. Rather, the use of the term "preferred" is intended to present the concept in a specific manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusionary "or." That is, unless otherwise specified or clear from the context, "X uses A or B" naturally includes either of the permutations. That is, if X uses A; X uses B; or X uses both A and B, then "X uses A or B" is satisfied in any of the foregoing examples.

[0138] Furthermore, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the aforementioned components (e.g., elements, etc.), the terminology used to describe such components is intended to correspond to any component (unless otherwise indicated) that performs the specified function of said component (e.g., is functionally equivalent to it), even if structurally not equivalent to the disclosed structure performing the functions in the exemplary implementations of this disclosure shown herein. Moreover, although specific features of this disclosure have been disclosed with respect to only one of several implementations, such features may be combined with one or more features of other implementations that may be desirable and advantageous for a given or particular application. Furthermore, with regard to the use of the terms “comprising,” “having,” “containing,” or variations thereof in the Detailed Description or claims, such terms are intended to be included in a manner similar to the term “including.”

[0139] The functional units in this invention embodiment can be integrated into a processing module, or each unit can exist physically separately, or multiple units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. The aforementioned devices or systems can execute the storage methods in the corresponding method embodiments.

[0140] In summary, the above embodiments are one implementation of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A joint learning method for heterogeneous clients based on hierarchical sampling optimization, applied to... N A client and a parameter server, characterized in that, Includes the following steps: Select available clients from different clusters; the clients receive heterogeneous data. The parameter server initializes the global model and then broadcasts the global model to all clients. The clients train their local model parameters based on the received global model and samples of their local data. The parameter server collects the local model parameter information of each client and uses a clustering method to divide the clients into different clusters. During each training round, available clients are extracted from each cluster based on sample weights to participate in training, and gradient aggregation is performed to ensure that data of all types participate in each training round, reducing the impact of client heterogeneity; the training objective is a convex function; where, To achieve the required accuracy in the algorithm The number of iterations required is the number of communications between the client and the parameter server. for: ； G 2 This is the bound of the expected square norm of the stochastic gradient. This represents the variance bound of the stochastic gradient. It is the first k The weight of each client, E For local iteration rounds, N It is the total number of clients. K The total number of clients drawn in each round. L express L -smooth, express Strong convexity, When the degree of data heterogeneity is low When it is close to 0, the local iteration rounds E The larger the better; when the data is highly heterogeneous. The smaller the number of local iteration rounds E The smaller the better; With all clients participating in the training, the algorithm's convergence is: ； in, , , For heterogeneous data parameters; With some clients participating in the training, the algorithm's convergence is: ； in ; At the same time, the learning rate needs to be set to gradually decrease during training in order to converge to the optimal result and enable the algorithm to achieve optimal performance. The convergence speed; After each round of training, the client receives the latest global model parameters from the parameter server, calculates the gradient under the current parameters using local data, and iterates. E After each stochastic gradient descent iteration, the latest parameters are sent back to the parameter server, which then performs a weighted average of the returned parameters.

2. The joint learning method for heterogeneous clients based on hierarchical sampling optimization according to claim 1, characterized in that, Learn the average value of one-dimensional data from N clients, and transform this objective into a problem of minimizing the mean square error, as follows: ； in It is a sample independently selected from local data, with the mean... , It is the first i The weight of each client, For client-side samples, For the first i Client weight offset.

3. The joint learning method for heterogeneous clients based on hierarchical sampling optimization according to claim 2, characterized in that, When each client contains the same amount of data, the optimal solution to the problem is: ； For the first i If each client has a weight offset, then the objective function will converge to: 。 4. The joint learning method for heterogeneous clients based on hierarchical sampling optimization according to claim 1, characterized in that, The clustering method is the OPTICS clustering method.

5. The joint learning method for heterogeneous clients based on hierarchical sampling optimization according to claim 1, characterized in that, The formulas for the client-side local model parameters are as follows: ； in W is the learning rate. t For global models, For client-side samples, For loss function, i For the first i One client.

6. The joint learning method for heterogeneous clients based on hierarchical sampling optimization according to claim 1, characterized in that, The weighted average method is as follows: ； K The total number of clients drawn in each round. k For the first k One client.

7. The joint learning method for heterogeneous clients based on hierarchical sampling optimization according to claim 6, characterized in that, Based on the following formula: ； Therefore, a decreasing learning rate is required to ensure the algorithm converges to the optimal solution. The decreasing learning rate parameter is: , .

Images