Clustering federal learning method based on neural network model structure
Through the clustered federated learning method based on neural network model structure, the problems of data heterogeneity, system heterogeneity and data drift in federated learning are solved, and more efficient model training and more stable performance are achieved.
Patent Information
- Application Number
- CN202510599128.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-10
- Publication Date
- 2025-08-15
AI Technical Summary
Existing federated learning techniques have shortcomings in dealing with data heterogeneity, system heterogeneity, and data drift, resulting in reduced model performance and inefficiency.
The clustering federated learning method based on neural network model structure is adopted, and the initial model is distributed through the server and pruned, data drift is detected, the model structure similarity calculation algorithm is used for clustering, and the client cluster is dynamically adjusted, which is compatible with multiple model pruning and clustering algorithms.
It improves the accuracy of the model under non-independent and homogeneous data, reduces the calculation and communication overhead, maintains the stability and efficiency of model performance, and adapts to data distribution changes.
Smart Images
Figure CN120494048A_ABST
Abstract
Description
Technical Field
[0001] This note relates to the fields of information security and artificial intelligence, and in particular to a clustering federated learning method based on a neural network model structure. Background Art
[0002] In recent years, federated learning, as an emerging privacy-preserving distributed machine learning technology, has garnered widespread attention. Federated learning allows multiple clients to collaboratively train models without sharing private data. Each client only uploads model parameters, not the data itself, effectively protecting data privacy. However, federated learning faces numerous challenges in practical applications, with data heterogeneity and system heterogeneity being two key issues. Data heterogeneity refers to significant differences in data distribution across clients. This discrepancy can cause divergence in the optimization paths of client-local models, impacting the convergence of the global model. System heterogeneity manifests itself in varying computational and communication capabilities across clients, which can lead to performance bottlenecks during training. For example, some clients may be unable to complete training tasks in a timely manner due to hardware limitations, slowing down the entire federated learning process. To address this system heterogeneity, researchers have proposed model pruning techniques. By generating models tailored to each client's training capabilities, they ensure similar training costs across clients, thereby reducing training latency.
[0003] Federated learning also faces the challenge of data drift. Data drift refers to the change in the distribution of local client data over time, which can lead to degraded model performance. For example, in the financial sector, user consumption behavior may change over time, resulting in reduced model prediction accuracy. To address data drift, researchers have proposed a dynamic clustering strategy that monitors changes in model structure and promptly re-clusters clients to adapt to changes in data distribution.
[0004] Although some methods have been proposed to address data heterogeneity and system heterogeneity, these methods still have shortcomings in practical applications, especially in dealing with data heterogeneity, system heterogeneity, and data drift simultaneously. Therefore, developing a federated learning method that can effectively address these challenges is of great practical significance. This paper proposes a clustering federated learning method based on model structure. By integrating model pruning and clustering federated learning, it simultaneously addresses data heterogeneity, system heterogeneity, and data drift, thereby improving the performance and efficiency of federated learning. Summary of the Invention
[0005] The technical problem to be solved by the present invention is: in response to the data heterogeneity, system heterogeneity and data drift problems existing in existing federated learning technology, the present invention provides a safe and reliable clustering federated learning method based on a neural network model structure.
[0006] To solve the above technical problems, the present invention proposes a technical solution: a clustering federated learning method that overcomes system heterogeneity and data drift, including a server distribution model and a clustering process, which includes the following steps:
[0007] S1. Initialization phase: The server initializes the same initial model ω0 for each client and sets hyperparameters. In each round of iterative training, the server first assigns a clustering model to each client.
[0008] S2. The client receives the model assigned by the server Pre-training E pre After the round, the model pruning technology is used to prune it and generate the pruned model and model structure mask
[0009] S3. The client continues to train the pruned model E after wheel, and local model and mask Upload to the server;
[0010] S4. The server receives the model and mask uploaded by the client, detects client data drift, and determines whether re-clustering is required based on the consistency of the model structure;
[0011] S5. If re-clustering is required, the server uses a similarity calculation algorithm based on the model structure to cluster the clients. Otherwise, it continues to use the previous clustering result.
[0012] S6. The server aggregates the models belonging to the same cluster, generates a new cluster model, and sends it back to the corresponding client.
[0013] Furthermore, in a comprehensive cluster federated learning approach to overcome system heterogeneity and data drift, the model pruning techniques used by the client when pruning the model include, but are not limited to, hybrid pruning. Furthermore, different clients can prune their models to varying degrees based on their storage capacity, network bandwidth, and training capabilities to address system heterogeneity.
[0014] Furthermore, a consistency check is also included to detect whether data offset occurs on the client. This check is used by the server to determine whether the client needs to be re-clustered. The specific steps include:
[0015] S4.1. The server calculates two consecutive rounds of model masks and The Hamming distance of
[0016] S4.2. If the Hamming distance exceeds the consistency threshold τ, the data distribution is considered to have changed and the server needs to re-cluster the clients.
[0017] S4.3. If the Hamming distance does not exceed the consistency threshold τ, the data distribution change is considered acceptable and the server does not re-cluster to reduce computational overhead.
[0018] Furthermore, in the cluster migration strategy for overcoming data drift, the consistency threshold τ is determined by multiplying the Hamming distance of the previous round by a dynamic factor.
[0019] Furthermore, a detailed clustering calculation algorithm based on the model structure is also included. The algorithm process will be used by the server to calculate the clustering results of the client, specifically including the following steps:
[0020] S5.1. Initialize the similarity matrix Sim to a zero matrix and calculate the retention ratio RR = 1-PR based on the pruning ratio PR;
[0021] S5.2. For each pair of model masks m i and m j , calculate the similarity based on the pruning ratio RR and the retention ratio RR;
[0022] If PR is smaller, then calculate the similarity of the pruned structure; otherwise, calculate the similarity of the retained structure.
[0023] S5.4. Compute similarity using the COUNTTARGET function, which returns the number of objects in the same coordinates between two binary masks, averaged by r percentage of the mask size;
[0024] S5.5. Cluster according to the similarity matrix Sim using any clustering method (hierarchical clustering is the default).
[0025] Furthermore, the clustering method technology is compatible with a variety of clustering algorithms, including but not limited to a hierarchical clustering algorithm, a k-means algorithm, etc., and is used to use the calculated similarity as an input of a function to obtain a final clustering result.
[0026] Furthermore, a specific COUNTTARGET function is also included, which is used by the server to calculate the similarity between two clients, specifically including the following steps: receiving a binary mask m i and m j , and the ratio r and target value target (0 or 1); compare m i and m j At each coordinate value, count the number of coordinates that are equal to the target value at the same time; calculate the total number, that is, m in all coordinates i and m jAt the same time, it is equal to the sum of the number of target values; the total is divided by r percentage of the mask size to obtain the average; the average is returned as the similarity measure of the two binary masks on the target value.
[0027] Compared with the prior art, the advantages of the present invention are:
[0028] 1. The present invention's clustering federated learning method based on a neural network structure offers a more accurate clustering algorithm. Existing clustering algorithms may fail when processing heterogeneous models, resulting in inaccurate clustering results. Furthermore, these algorithms incur high costs in terms of computational complexity and communication overhead. This invention proposes a model structure-based similarity calculation algorithm that can efficiently handle the clustering problem of heterogeneous models. This algorithm achieves high clustering accuracy and speed by calculating the similarity of model structure masks.
[0029] 2. The clustering federated learning method based on the neural network structure of the present invention has a more efficient dynamic cluster migration strategy. Existing data drift processing methods usually require frequent re-clustering, which will bring additional computational overhead. In addition, these methods may not be able to detect changes in data distribution in a timely manner in practical applications. A dynamic cluster migration strategy proposed in this invention monitors changes in the model structure and only re-clusters when the data distribution changes significantly. This method effectively reduces computational overhead, while being able to respond to data drift in a timely manner and maintain stable model performance.
[0030] 3. The neural network-based clustering federated learning framework of the present invention is compatible with a variety of model pruning and clustering algorithms. Most model pruning techniques are incompatible with existing clustering federated learning methods, making it difficult to simultaneously leverage the advantages of both techniques in practical applications. However, compatibility with multiple model pruning techniques and clustering algorithms enables the present invention to be flexibly applied to different federated learning scenarios, fully leveraging the advantages of model pruning and clustering federated learning.
[0031] 4. The present invention's clustering federated learning method, based on a neural network structure, offers superior computational efficiency. Existing methods can significantly reduce computational and communication efficiency when dealing with data drift and system heterogeneity, especially when frequently re-clustering. This invention significantly improves computational efficiency while maintaining model performance through a dynamic cluster migration strategy and an efficient similarity calculation algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 Schematic diagram of the working process of a specific embodiment of the present invention.
[0033] Figure 2This figure compares the accuracy of the specific embodiment of the present invention with the One-Shot, FL+HC, IFCA, and FlexCFL models on the FMNIST dataset.
[0034] Figure 3 This figure compares the accuracy of the specific embodiment of the present invention with the One-Shot, FL+HC, IFCA, and FlexCFL models on the SVHN dataset.
[0035] Figure 4 This figure compares the accuracy of a specific embodiment of the present invention with the One-Shot, FL+HC, IFCA, and FlexCFL models on the CIFAR10 dataset.
[0036] Figure 5 This figure compares the accuracy of a specific embodiment of the present invention with the One-Shot, FL+HC, IFCA, and FlexCFL models on the CIFAR100 dataset. DETAILED DESCRIPTION
[0037] The following is in conjunction with the instructions Figure 1 The present invention is further described with reference to the following specific preferred embodiments, but the protection scope of the present invention is not limited thereby.
[0038] The framework of the clustering federated learning method based on the neural network structure in this example includes multiple clients and a central server. The work of the client includes two main steps: local model training and model pruning. The process specifically includes S1. The client first receives the global model from the server and uses the model for local training. In each round of training, the client executes E pre After the training is completed, the client will prune the model according to the specified pruning ratio PR, generate the pruned local model, and calculate the corresponding model structure mask m. i In this process, the goal of model pruning is to remove unimportant parts of the model and retain important structures that have a greater impact on model performance. S3. After pruning, the client continues to perform subsequent training (E after cycles) to further optimize the pruned model. After the training is completed, the client will and mask m i Upload to the server and prepare for cluster operation.
[0039] The server's work mainly includes three main steps: consistency check, similarity calculation and clustering, and aggregation of model parameters. The process specifically includes S1. After receiving the model and mask uploaded by the client, the server first checks the consistency of the model structure. If the change in the model structure exceeds the preset threshold τ, a re-clustering operation is triggered. S2. The server calculates the similarity between clients based on the model structure mask and the pruning ratio, and determines the similarity between clients by comparing the retained and pruned parts of the model structure. The server then clusters the clients using a hierarchical clustering algorithm to generate a new clustering result. S3. The server aggregates the models in each cluster, generates a new clustering model, and sends it back to the corresponding client.
[0040] In this example, during the system initialization phase, the server is responsible for initializing the initial model of each client and setting hyperparameters. Specifically, the server initializes the same initial model ω0 for each client and sets the number of local training rounds E pre and E after , and the consistency threshold τ. The settings of these hyperparameters are crucial for the subsequent training process, ensuring the efficiency and accuracy of model training. The server is also responsible for maintaining the model and clustering results for each cluster and updating these models in each communication round.
[0041] In this example, the client receives the model parameters sent by the server After that, some model training parameters will be determined, including learning rate η, momentum, batch size, and the client will first perform E on the complete model. pre The purpose of pre-training is to allow the model to perform preliminary learning on local data so that the subsequent pruning process can more effectively retain important structures. After that, the model is pruned. The client needs to set the pruning ratio PR to determine the degree of pruning. The pruning process of the pruning client usually adapts to the client's computing power by retaining or pruning the most important structures in the model. For example, for a model with 1000 parameters, the client can choose to retain the 500 most important parameters based on its computing power and prune the remaining 500 parameters. At this time, the pruning ratio PR is set to 0.5. This method will selectively retain or prune certain structures of the model based on the client's computing power and data distribution to adapt to the needs of heterogeneous systems. At the same time, in order to meet the heterogeneity of the system, all clients must have different pruning ratios. In this environment, the pruning ratio of the client cluster is limited to between 0.2 and 0.7, while following a Gaussian distribution. The pruned model continues to undergo subsequent training (E after rounds) to adapt to the client's data distribution. After training is completed, the client will and model structure mask m tUpload to the server for further processing.
[0042] In this example, after receiving the models and masks from all clients, the server first performs a consistency check and calculates the current round m t And the model structure mask m of the previous round t-1 The server uses the Hamming distance between the client and the client to determine whether re-clustering is necessary. If the Hamming distance exceeds the consistency threshold τ, re-clustering is triggered. Otherwise, the server continues to use the clustering results of the previous round to reduce computational overhead. When data drift is detected, the server recalculates the client's cluster identity based on the model structure of the current round and updates the clustering results.
[0043] In this example, the Hamming distance is calculated as follows:
[0044]
[0045] Among them, m i,k and m j,k Represents the value of the kth bit of the two masks, and n represents the length of the mask. Cluster migration is performed in the following way: for each client's model mask, calculate its hash distance with the previous round model, and determine whether the cluster needs to be migrated using the following formula:
[0046] HammingDistance(m t ,m t-1 )>τ
[0047] If the hash distance is greater than the threshold τ, the client's cluster needs to be updated. Otherwise, the original cluster remains unchanged.
[0048] This method effectively avoids frequent cluster reorganization, reduces the performance degradation caused by data drift, and ensures the stability and convergence speed of the model.
[0049] In this example, based on the client model structure mask m i Calculate the similarity between clients. Specifically, the server calculates the similarity between each client model and selects clients with similar data distribution to cluster. The server will first initialize the similarity matrix Sim to a zero matrix and calculate the corresponding retention ratio RR = 1-PR based on the pruning ratio PR of all clients. For each pair of model masks m i and m j, the similarity is calculated based on the pruning ratio PR and the retention ratio RR. Specifically, if PR is a smaller value, the similarity of the pruned structure is calculated, otherwise the similarity of the retained structure is calculated. The server will use the COUNTTARGET function to calculate the similarity, which returns the number of targets in the same coordinates of the two binary masks, averaged by the r percentage of the mask size, and finally use any clustering method (such as the hierarchical clustering method) to perform clustering according to the similarity matrix Sim.
[0050] The similarity calculation formula of the model structure is as follows:
[0051] Sim=COUNTTARGET(m i ,m j ,r,target)
[0052] Among them, m i and m j They represent the model structure masks of the two clients respectively, r represents the pruning ratio or the retention ratio. If PR is a smaller value, enter the pruning ratio. If RR is a smaller value, enter the retention ratio. The value of target can be 0 or 1. If r selects the pruning ratio, target corresponds to 0. If it is the retention ratio, target corresponds to 1.
[0053] In this example, the COUNTTARGET function is used by the server to calculate the similarity between two clients. It includes the following steps: First, receive the binary mask m i and m j , as well as the ratio r and target value target (0 or 1); then compare m i and m j At each coordinate value, count the number of coordinates that are equal to the target value at the same time; calculate the total number, that is, m in all coordinates i and m j At the same time, it is equal to the sum of the target values; the total is divided by r percentage of the mask size to obtain the average; finally, the average is returned as the similarity measure of the two binary masks on the target value.
[0054] In this example, after the server determines the clustering result, it performs weighted averaging on all client models in each cluster to generate a cluster model, and distributes the clustering result obtained by the clustering algorithm to the clients belonging to each cluster.
[0055] To verify the effectiveness of this invention, we conducted experiments on four different datasets (FMNIST, SVHN, CIFAR-10, and CIFAR-100). In the experiments, we used a customized convolutional neural network model to evaluate these datasets. For the setting of hyperparameters, this example uses the SGD optimizer to train the local model, with the learning rate adjusted to 0.001, the momentum is 0.5, and the batch size is 32. For the server, the number of local training rounds E pre and E after The consistency threshold τ is the Hamming distance from the previous round multiplied by a dynamic factor, which is 1.1. The pruning ratio of the client cluster in a heterogeneous environment follows a Gaussian distribution with a mean of 0.4 and a variance of 0.25. This example tests the performance of the invention in an environment with 10 clients and 1 server.
[0056] In the experiment of this example, we compared the method of the present invention with four different clustering federated learning methods (One-Shot, FL+HC, IFCA and FlexCFL). These methods respectively adopt different clustering algorithms and model update strategies to deal with the problems of data heterogeneity and system heterogeneity. In order to simulate non-independent and identically distributed (Non-IID) data, we considered the scenario where the labels are not independent and identically distributed. In the label non-independent and identically distributed scenario, the data is divided into 5 groups, and each group is assigned two unique labels, which are distributed according to the non-independent and identically distributed rate β. A smaller β value means that the group is more likely to receive labels other than those specified. These settings are intended to simulate the diversity and complexity of data distribution in the real world to verify the performance of the present invention in different scenarios.
[0057] Figure 2 、 Figure 3 、 Figure 4 and Figure 5 The results show the change in the average accuracy of the global model of this example and the comparison scheme over time. Experimental results show that this scheme outperforms traditional clustering federated learning methods in terms of model accuracy and convergence speed. In the labeling scenario, the peak accuracy of this example is improved by 18.2%, 18.43%, 10.24%, and 11.66% on different datasets compared to the baseline method, respectively. These results fully demonstrate the advantages of this example in handling data heterogeneity and system heterogeneity. In addition, the experiments also show that this example exhibits better stability and robustness in the face of data drift, and can promptly detect and adapt to changes in data distribution, thereby maintaining continuous improvement in model performance.
[0058] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above with reference to the preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiment that do not depart from the technical solution of the present invention and are based on the technical essence of the present invention shall fall within the scope of protection of the technical solution of the present invention.
Claims
1. A clustering federated learning method based on a neural network structure, characterized by: The following steps are involved: S1. Initialization phase: The server initializes the same initial model ω0 for each client and sets hyperparameters. In each round of iterative training, the server first assigns a clustering model to each client. S2. The client receives the model assigned by the server Pre-training E pre After the round, the model pruning technology is used to prune it and generate the pruned model and model structure mask S3. The client continues to train the pruned model E after wheel, and local model and mask Upload to the server; S4. The server receives the model and mask uploaded by the client, detects client data drift, and determines whether re-clustering is required based on the consistency of the model structure; S5. If re-clustering is required, the server uses a similarity calculation algorithm based on the model structure to cluster the clients. Otherwise, it continues to use the previous clustering result. S6. The server aggregates the models belonging to the same cluster, generates a new cluster model, and sends it back to the corresponding client.
2. The clustering federated learning method based on a neural network structure according to claim 1, characterized in that: The model pruning technology in step S2 includes but is not limited to hybrid pruning, which is used to generate a model that adapts to the training capabilities of each client to solve the problem of system heterogeneity.
3. The clustering federated learning method based on a neural network structure according to claim 1, characterized in that: In step S4, the server detects client data drift according to the mask, including the following steps: S4.
1. The server calculates the model masks for all clients for two consecutive rounds respectively and The Hamming distance of S4.
2. If the Hamming distance exceeds the consistency threshold τ, the data distribution is considered to have changed and the server needs to re-cluster the clients. S4.
3. If the Hamming distance does not exceed the consistency threshold τ, the data distribution change is considered acceptable and the server does not re-cluster to reduce computational overhead.
4. The clustering federated learning method based on a neural network structure according to claim 1, characterized in that: In step S4, the consistency threshold τ is determined by multiplying the Hamming distance of the previous round by a dynamic factor.
5. The clustering federated learning method based on a neural network structure according to claim 1, characterized in that: In step 5, the server performs similarity calculation based on the model structure, including the following steps: S5.
1. The server initializes the similarity matrix Sim to a zero matrix and calculates the retention ratio RR = 1-PR for all clients based on their respective pruning ratios PR. S5.
2. For each pair of model masks m on the client i and m j , calculate the similarity between two clients according to the pruning ratio PR and the retention ratio RR; If PR is smaller, then the similarity of the client-side pruned structure is calculated; otherwise, the similarity of the client-side preserved structure is calculated. S5.
4. Compute similarity using the COUNTTARGET function, which returns the number of objects in the same coordinates between two binary masks, averaged by r percentage of the mask size; S5.
5. Perform clustering using any clustering method (hierarchical clustering is the default) based on the similarity matrix Sim as input.
6. The clustering federated learning method based on a neural network structure according to claim 1, characterized in that: The COUNTTARGET function performed in step 5 includes the following steps: receiving a binary mask m i and m j , and the ratio r and target value target (0 or 1); compare m i and m j At each coordinate value, count the number of coordinates that are equal to the target value at the same time; calculate the total number, that is, m in all coordinates i and m j At the same time, it is equal to the sum of the number of target values; the total is divided by r percentage of the mask size to obtain the average; the average is returned as the similarity measure of the two binary masks on the target value.
7. The clustering federated learning method based on a neural network structure according to claim 1, characterized in that: The clustering method technology performed in step 5 is compatible with a variety of clustering algorithms, including but not limited to a hierarchical clustering algorithm, a k-means algorithm, etc., and is used to use the calculated similarity as an input of a function to obtain a final clustering result.