Integrated federated learning optimization method based on clustering weight sampling

Abstract

The invention discloses an integrated federated learning optimization method based on clustering weight sampling, and the method specifically comprises the following steps: a federated learning system comprises a plurality of clients and a server, and the server calculates the similarity between the clients through model updating information uploaded by the clients, clustering the clients by adopting a dynamic clustering method according to the similarity; the server carries out secondary clustering according to a set sampling rule and judges whether a first-stage iteration threshold value is reached, all the clients obtain a latest global model and freeze a model feature recognition layer for fine tuning, the server collects parameters of all the clients after fine tuning, and then the parameters are clustered according to similarity and are subjected to secondary clustering according to the sampling rule; and combining into an enhanced global model through an ensemble learning strategy. The method can be widely applied to data privacy protection scenes in the fields of medical image analysis, financial risk control, intelligent transportation and the like, and a new technical solution is provided for efficient application of federal learning in a heterogeneous environment.

Description

Technical Field

[0001] This invention relates to the field of distributed machine learning and privacy computing, specifically to a federated learning optimization method that utilizes clustering weight sampling and ensemble learning strategies to solve the problem of non-independent and identically distributed (Non-IID) data. Background Technology

[0002] With the increasing severity of privacy and data security issues, Federated Learning (FL), as a distributed machine learning framework, achieves the goal of collaboratively training a global model across multiple participants while ensuring local data storage and privacy security. However, existing federated learning methods face several major challenges in practical applications:

[0003] 1. Data distribution heterogeneity

[0004] The data distribution of different participating clients is often highly inconsistent (i.e., the Non-IID problem), which makes it difficult for the global model to adapt to all clients, resulting in inconsistent optimization objectives and even significant fluctuations in model performance.

[0005] 2. High communication overhead

[0006] Federated learning requires frequent transmission of model parameters. Especially when there are a large number of clients or a high number of training rounds, communication overhead may become a bottleneck, which is particularly noticeable for edge devices or mobile devices with limited bandwidth.

[0007] 3. Differences in client performance

[0008] The computing power and data quality of clients vary greatly, and a uniform training strategy is difficult to meet the needs of all clients, which limits the generalization ability of the global model.

[0009] To address these issues, existing research has attempted to improve federated learning performance through algorithm optimization. For example, FedAvg improves the convergence of the global model by weighted averaging of the client-side local models; FedProx alleviates model drift by introducing regularization terms during optimization. However, these methods still fall short in handling the heterogeneity of client-side data, making it difficult to achieve consistent and efficient performance in practical applications.

[0010] To address this, this invention proposes a federated learning optimization method based on clustering weight sampling and ensemble learning. By using dynamic clustering and secondary sampling, the method enhances the model's adaptability to data heterogeneity and improves global model performance by combining ensemble learning strategies, while reducing communication overhead. Summary of the Invention

[0011] To address the aforementioned issues, this invention provides an ensemble federated learning optimization method based on clustering weight sampling. Its aim is to significantly improve the model's performance and generalization ability in highly heterogeneous data environments while enhancing the global model training efficiency.

[0012] This invention is achieved through the following technical solution:

[0013] An ensemble federated learning optimization method based on cluster weight sampling specifically includes the following steps:

[0014] S1. Construct a federated learning system that includes multiple clients and one server, with the server responsible for initializing global model parameters;

[0015] S2. In each round of training, the server calculates the similarity between clients using the model update information uploaded by the clients;

[0016] S3. Cluster the clients using a dynamic clustering method based on similarity;

[0017] S4. For each group of clients, the server performs secondary clustering according to the set sampling rules, selects one client from each cluster for model training, and generates the corresponding model update;

[0018] S5. Determine whether the first stage iteration threshold has been reached. If yes, proceed to step S6; otherwise, return to step S2.

[0019] S6. All clients obtain the latest global model and freeze the model feature recognition layer for fine-tuning;

[0020] S7. After collecting all the fine-tuned parameters from the clients, the server performs clustering based on similarity and then performs secondary clustering based on the sampling rules;

[0021] S8. Randomly select a client model from each cluster and combine them into an enhanced global model through an ensemble learning strategy;

[0022] S9. Select participating clients according to the intra-cluster client selection strategy and distribute the updated global model parameters;

[0023] S10. Determine whether the second stage iteration threshold has been reached. If yes, proceed to step S11; otherwise, return to step S8.

[0024] S11. Once the set training termination condition is met, the optimized global model is finally generated.

[0025] In a preferred embodiment, the dynamic clustering method used in step S3 to cluster clients based on similarity is achieved by measuring the similarity between the client and the global model to realize dynamic client selection. The similarity calculation is based on two methods: cosine similarity based only on model updates and a combination of cosine similarity based on student models.

[0026] As a preferred embodiment, the use of only cosine similarity based on model updates for the local model w of client i i and global model w g First, calculate the difference in model updates:

[0027] Δw i =w i -w g ,

[0028] Where, Δw i This represents the difference between the local model and the global model of client i;

[0029] Then, the difference between client i and client j is calculated, and cosine similarity is used as a measure. The similarity S between the clients is then calculated. ij Calculated using the following formula:

[0030]

[0031] Where, ||Δw i ||and||Δw j || is the L2 norm of the model update vector, representing the size of the update for each client.

[0032] In a preferred embodiment, the synthesis is based on the cosine similarity of the student model, and the client i is based on the local model w. i Distillation yields student model The server calculates cosine similarity for student models across all clients:

[0033]

[0034] Then, by combining the similarity from the local model updates, S is obtained. ij :

[0035]

[0036] Here, α is a manually set parameter that represents the proportion of the student model's similarity in the overall similarity.

[0037] As a preferred embodiment, for the sampling rule in step S4, before each round of training, the server will cluster according to the similarity between the client model and the global model, and based on the clustering, assign data weights to each client to construct the client selection matrix M. M is a two-dimensional matrix, where each row represents a sampling, each column represents a client, the number of columns in the matrix is equal to the total number of clients, and the number of rows is the number of samples to be taken. Specifically, the value of M[i, j] represents the probability that client j is selected in the i-th sampling in the current round. In the final result, the sum of the elements in each row cannot exceed the total number of samples D. Assuming there are a total of n clients and m samples are taken in each round, the sampling matrix M is constructed as follows:

[0038] M = [M1 M2... M m T , M i = [0 0... 0] 1×n .

[0039] As a preferred embodiment, during the calculation, first, calculate the client similarity. In the first few rounds of training, only consider the update similarity S ij of the local model. After training for a certain number of rounds, the local model has certain global information while retaining local features. At this time, the client will obtain the student model by distillation and synthesize the new similarity S ij based on the student model. Then, use the Ward method to perform hierarchical clustering on the calculated similarity matrix, determine the number of clusters according to the given threshold (ε / m, ε = 1e-10), and clients with similarities less than the threshold are grouped together. Assume that the clients are divided into K clustering clusters B1, B2,..., B K . If K < m, then m in this round of training is changed to K. Then use a voting mechanism. For each client i, the number of votes obtained is T i = m × D i , and the total number of votes of the cluster j where it is located is Then sort the clusters in ascending order according to the total number of votes to form new clusters B1, B2,..., B K . Assume that for B m it starts to be allocated from M j . Each time a client i is taken out, fill the number of votes of i into M j [i]. If the number of votes of the current client i can be fully allocated to M j , directly allocate; if the demand of i exceeds the current remaining capacity, first fill M j , and fill the remaining part into M j+1 [i]. If M j is the last one, then the remaining part is allocated starting from M1. The algorithm first starts from B K-m+1 : B​K-m+1 Starting with M1, B K-m+2 The allocation starts from M2 and ends at B. K From M m Begin the allocation, then for B1, B2, ..., B K-m The allocation starts from M1. After all allocations are completed, the matrix is ​​reversed so that M = [M1, M2, M3, M4, M5, M6, M7, M8, M9, M10, M11, M21, M32, M42, M62, M72, M82, M92, M11, M12, M21, M32, M42, M62, M72, M82, M92, M11, m M m-1 ... M m1 ] T The update is completed in the form of [format missing].

[0040] In a preferred embodiment, for each sampling, if there is a client that has already been taken in this round of training, then the votes of that client are cleared, and the remaining votes are normalized to obtain the probability of the client selected in this sampling.

[0041] In a preferred embodiment, step S8 combines the data into an enhanced global model through an ensemble learning strategy. The server collects and randomly selects the classification layers of a client from each cluster to form a large classification layer in parallel, which is then connected to the feature recognition layer of the global model. The probability vectors output by each classification layer (assuming m dimensions) are merged into a k×m dimension vector after a Softmax operation and sent to the last fully connected layer for result prediction.

[0042] In a preferred embodiment, the intra-cluster client selection strategy in step S9 calculates the client's local model w for each client i. i and global model w g Cosine similarity S between ig Then calculate the accuracy (acc) for each client on the validation set. i Calculate the score for each client. i :

[0043] score i =acc i ×2+(S ig +1)

[0044] Where S ig +1 ensures that the similarity score is a positive number, acc i ×2 ensures that the accuracy score and similarity score have the same range, meaning their weights are consistent. Clients that did not participate in the training round retain their previous scores. To accommodate clients that have not been selected for a long time, a penalty mechanism based on the number of selections is introduced to balance the client's selection probability. The modified scores are as follows:

[0045]

[0046] Where, Nselected,i It represents the number of times client i was selected. A logarithmic function is used to smooth out the impact of the number of selections and prevent excessive compensation. The +2 is used to avoid the denominator being zero.

[0047] In a preferred embodiment, the calculated score for each client needs to be normalized so that the selection probability of each client is determined based on the relative weight.

[0048] Beneficial effects: This invention can be widely applied to data privacy protection scenarios in various fields such as medical image analysis, financial risk control, and intelligent transportation. It provides a new technical solution for the efficient application of federated learning in heterogeneous environments, specifically including the following aspects:

[0049] (1) Effectively addressing data heterogeneity:

[0050] This invention utilizes a dynamic clustering method to group and sample clients based on the similarity of their model updates, ensuring the consistency of the model update direction and thus significantly mitigating the negative impact of data distribution heterogeneity on model training.

[0051] (2) Reduce communication overhead:

[0052] In the later stages of global training, only the optimized classification layer parameters are transmitted, avoiding frequent transmission of all model parameters, which greatly reduces communication costs and improves the applicability of the algorithm in bandwidth-constrained environments.

[0053] (3) Enhance global model performance:

[0054] By combining the classification layers of each group into an enhanced global model through an ensemble learning strategy, the prediction accuracy and generalization ability of the model are improved, especially in scenarios with high data heterogeneity.

[0055] (4) Support client-side personalization requirements:

[0056] The client fine-tunes the classification layer based on the global model to adapt to the local data distribution and generate a personalized model that better meets the client's needs, without affecting the overall performance of the global model. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall method framework provided for an embodiment of the present invention.

[0058] Figure 2 The overall method flowchart provided for embodiments of the present invention. Detailed Implementation

[0059] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0060] This embodiment proposes a method combining cluster sampling and ensemble learning to address the non-independent, identically distributed (non-IID) data problem in Federated Learning (FL), thereby improving model performance and reducing communication overhead. This method samples data by analyzing client-side data heterogeneity and volume, and applies ensemble learning techniques to enhance the global model.

[0061] S1. First, at the start of each training round, the server initializes the global model parameters w0. Subsequently, the system enters two main phases: dynamic client-selective training (Phase 1) and local fine-tuning and ensemble learning (Phase 2).

[0062] Dynamic client selection training (Phase 1):

[0063] S2. In the first stage, the server dynamically selects clients to participate in training based on the client model update history and similarity. Specifically, the server calculates the similarity between each client model and its global model. Similarity calculation is based on two methods: one is to use only cosine similarity to measure model update differences; the other is to combine the similarity after student model distillation. For the update record Δw of client i... i To calculate the cosine similarity S between client i and client j ij The formula is as follows:

[0064]

[0065] In addition, client i will also base its model w on the local model. i Distillation yields student model The server calculates cosine similarity for student models across all clients:

[0066]

[0067] Taking into account both the model update differences and the student model similarity, the final similarity matrix S is obtained, as shown in the following formula:

[0068]

[0069] Here, θ is a manually set parameter used to adjust the weight of student model similarity in the overall similarity score.

[0070] S3. Next, the server performs hierarchical clustering using the Ward method, dividing the clients into several clusters based on the calculated similarity matrix S. The number of clusters K is determined according to a given threshold (∈ / m, ∈=1e-10), where m is the number of samples in each training round.

[0071] S4-1. To ensure that small clusters also receive sufficient attention, the server uses a voting mechanism to allocate samples. Specifically, assuming that m samples need to be selected in this round of training, if the number of clusters K is less than m, then m is adjusted to K. Then, the voting mechanism is used to allocate samples, update the sampling matrix M, and select the client for the next round of training.

[0072] S4-3. The selected client receives the latest global model, trains it on its local data, and calculates the updated model parameters. The client updates the data and uploads it to the server. The server, based on the received updates, first calculates the client's update record. For client i's update record Δw... i The calculation formula is:

[0073]

[0074] Where w g This is a global model.

[0075] S4-4. Update the global model using a weighted average, where SC is the set of clients selected in this round:

[0076]

[0077] S5. Determine whether the first stage iteration threshold has been reached. If yes, proceed to step S6; otherwise, return to step S2.

[0078] Local fine-tuning and ensemble learning (Phase 2):

[0079] S6. In the second stage, the server sends the updated global model to all clients. The clients freeze the feature extraction layer F(·) of the global model and fine-tune the classifier module C(·) on their local data. To prevent overfitting, a small learning rate is used for fine-tuning. Specifically, the client optimizes only its classifier module, maintaining shared feature representations.

[0080] S7. After fine-tuning, the server collects and updates the selection matrix M. This time, each row of M can be regarded as a cluster.

[0081] S8. Then, randomly select a client's classifier C from each cluster. k (·), forming a new large classifier BC(·), and combined with the global feature extraction layer F g(·) Parallel connection. Each classifier C(·) outputs a probability vector, which are combined into a k×m vector after a Softmax operation and then passed to the final fully connected layer for prediction.

[0082] S9. At the end of each training round, the server calculates the cosine similarity S between the client's local model and the global model. ig and accuracy on the validation set. i This is used to derive the overall score. i The formula is as follows:

[0083] score i =acc i ×2+(S ig +1)

[0084] Among them, S ig +1 ensures a positive similarity value, acc i ×2 normalizes accuracy and similarity to the same range, giving them equal weight. To prevent some clients from not being selected for extended periods, a penalty factor based on selection frequency is introduced, as shown in the following formula:

[0085]

[0086] Where, N selected，i This represents the number of times client i was selected. The log function smooths out the impact of selection frequency, preventing overcompensation. The +2 is added to prevent the denominator from being zero. Ultimately, these scores will be standardized and used as the basis for the next round of selection.

[0087] S10. Determine whether the second-stage iteration threshold has been reached. If yes, proceed to step S11; otherwise, return to step S8.

[0088] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. An ensemble federated learning optimization method based on clustering weight sampling, characterized in that, Specifically, the following steps are included: S1. Construct a federated learning system that includes multiple clients and one server, with the server responsible for initializing global model parameters; S2. In each round of training, the server calculates the similarity between clients using the model update information uploaded by the clients; S3. Cluster the clients using a dynamic clustering method based on similarity; S4. For each group of clients, the server performs secondary clustering according to the set sampling rules, selects one client from each cluster for model training, and generates the corresponding model update; S5. Determine whether the first stage iteration threshold has been reached. If yes, proceed to step S6; otherwise, return to step S2. S6. All clients obtain the latest global model and freeze the model feature recognition layer for fine-tuning; S7. After collecting all the fine-tuned parameters from the clients, the server performs clustering based on similarity and then performs secondary clustering based on the sampling rules; S8. Randomly select a client model from each cluster and combine them into an enhanced global model through an ensemble learning strategy; S9. Select participating clients according to the intra-cluster client selection strategy and distribute the updated global model parameters; S10. Determine whether the second stage iteration threshold has been reached. If yes, proceed to step S11; otherwise, return to step S8. S11. Once the set training termination condition is met, the optimized global model is finally generated.

2. The ensemble federated learning optimization method based on clustering weight sampling according to claim 1, characterized in that, In step S3, the dynamic clustering method based on similarity is used to cluster clients. This is achieved by measuring the similarity between the client and the global model to realize dynamic client selection. The similarity calculation is based on two methods: cosine similarity based on model update and cosine similarity based on student model.

3. The ensemble federated learning optimization method based on clustering weight sampling according to claim 2, characterized in that, The method uses only cosine similarity based on model updates for the local model w of client i. i and global model w g First, calculate the difference in model updates: Δw i =in i -In g , Where, Δw i This represents the difference between the local model and the global model of client i; Then, the difference between client i and client j is calculated, and cosine similarity is used as a measure. The similarity S between the clients is then calculated. ij Calculated using the following formula: Where, ||Δw i ||and||Δw j || is the L2 norm of the model update vector, representing the size of the update for each client.

4. The ensemble federated learning optimization method based on clustering weight sampling according to claim 2, characterized in that, The synthesis is based on cosine similarity of the student model, while client i is based on the local model w. i Distillation yields student model The server calculates cosine similarity for student models across all clients: Then, by combining the similarity from the local model updates, S is obtained. ij : Here, α is a manually set parameter that represents the proportion of the student model's similarity in the overall similarity.

5. The ensemble federated learning optimization method based on clustering weight sampling according to claim 1, characterized in that, The sampling rule in step S4 involves the server performing clustering based on the similarity between the client model and the global model before each round of training. Based on this clustering, data weights are assigned to each client to construct a client selection matrix M. M is a two-dimensional matrix where each row represents a sampling, and each column represents a client. The number of columns equals the total number of clients, and the number of rows represents the number of samples to be taken. Specifically, the value of M[i, j] represents the probability that client j is selected in the i-th sampling in the current round. In the final result, the sum of the elements in each row cannot exceed the total number of samples D. Assuming there are a total of n clients, and m clients are sampled in each round, the sampling matrix M is constructed in the following form: M=[M1 M2 ... M m ] T ,M i =[0 0 ... 0] 1×n 。 6. The ensemble federated learning optimization method based on clustering weight sampling according to claim 1, characterized in that, During computation, firstly, client-side similarity is calculated. In the first few rounds of training, only the update similarity S of the local model is considered. ij After training for a certain number of rounds, the local model retains local features while also possessing some global information. At this point, the client uses distillation to obtain the student model and combines the student model to obtain a new similarity S. ij Then, the Ward method is used to perform hierarchical clustering on the calculated similarity matrix. The number of clusters is determined according to a given threshold (ε / m, ε = 1e-10). Clients with similarity values ​​less than the threshold are grouped together. Assume that the clients are divided into K clusters B1, B2, ..., B K If K < m, m is changed to K for this round of training. Then a voting mechanism is used, and for each client i, the number of votes obtained is T. i =m×D i The total number of votes for cluster j is Then the clusters are arranged in ascending order of total votes, forming new clusters B1, B2, ..., B K Assuming for B m To get from M j The allocation begins by taking out one client i at a time and filling i's vote count into M. j At position [i], if the votes of current client i can be fully allocated to M. j Direct allocation; if the demand of i exceeds the current remaining capacity, M will be filled first. j The remainder is filled into M. j+1 At [i], if M j If it's the last one, then the remaining parts are allocated starting from M1. The algorithm first starts from B. K-m+1 Beginning: B K-m+1 Starting with M1, B K-m+2 The allocation starts from M2 and ends at B. K From M m Begin the allocation, then for B1, B2, ..., B K-m The allocation starts from M1. After all allocations are completed, the matrix is ​​reversed so that M = [M1, M2, M3, M4, M5, M6, M7, M8, M9, M10, M11, M21, M32, M42, M62, M72, M82, M92, M11, M12, M21, M32, M42, M62, M72, M82, M92, M11, m M m-1 ... M m1 ] T The update is completed in the form of [format missing].

7. The ensemble federated learning optimization method based on clustering weight sampling according to claim 1, characterized in that, For each sampling, if there is a client that has already been taken in this round of training, then the votes of that client are cleared, and the remaining votes are normalized to obtain the probability of the client selected in this sampling.

8. The ensemble federated learning optimization method based on clustering weight sampling according to claim 1, characterized in that, In step S8, the ensemble learning strategy is used to combine the data into an enhanced global model. The server collects and randomly selects the classification layers of one client from each cluster to form a large classification layer in parallel, which is then connected to the feature recognition layer of the global model. The probability vectors output by each classification layer (assuming m dimensions) are merged into a k×m dimension vector after a Softmax operation and sent to the last fully connected layer for result prediction.

9. The ensemble federated learning optimization method based on clustering weight sampling according to claim 1, characterized in that, The intra-cluster client selection strategy in step S9 involves calculating the client's local model w for each client i. i and global model w g Cosine similarity S between ig Then calculate the accuracy (acc) for each client on the validation set. i Get the score for each client. i : score i =acc i ×2+(S ig +1) Where S ig +1 ensures that the similarity score is a positive number, acc i ×2 ensures that the accuracy score and similarity score have the same range, meaning their weights are consistent. Clients that did not participate in the training round retain their previous scores. To accommodate clients that have not been selected for a long time, a penalty mechanism based on the number of selections is introduced to balance the client's selection probability. The modified scores are as follows: Where, N selected,i It represents the number of times client i is selected. A logarithmic function is used to smooth out the impact of the number of selections and prevent excessive compensation. The +2 is used to avoid the denominator being zero.

10. The ensemble federated learning optimization method based on clustering weight sampling according to claim 1, characterized in that, The calculated score for each client needs to be normalized so that the probability of each client's selection is determined based on the relative weight.

Images