Federated clustering method and system based on time-asynchronous similarity clustering

Abstract

This paper discloses a federated clustering method based on time-asynchronous similarity clustering. By randomly selecting some clients for training, the communication cost of global training is reduced. Due to the random selectivity of the clients, time differences exist between the client local models, resulting in errors in similarity calculation. To ensure the accuracy of similarity calculation, a federated clustering algorithm with time-weighted random client selection and time-asynchronous similarity calculation is proposed. Time-weighted random client selection dynamically adjusts the probability of client selection based on the client's waiting time, making client selection more uniform and effectively reducing time differences. Time-asynchronous similarity calculation uses the similarity between cluster models to correct the local model similarity calculation method (TAS), which is used to improve the accuracy of model similarity calculation with time differences. Ultimately, this improves the communication efficiency of federated learning under non-IID data.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning technology, and more specifically, the present invention relates to a federated clustering method and system based on time-asynchronous similarity clustering. Background Art

[0002] Federated Learning (FL) uses a shared model to refine the characteristics of a client's local data, allowing the client's data to remain locally and thus protecting the client's data privacy. Specifically, FL's central server initializes the global model; during the training phase, the server sends the global model to each client via the network, and the client uses local data to train the global model to obtain local model updates; after training, each client uploads the local model update to the server; the server aggregates the received local models to obtain a new global model. In real-world environments, the client's local data often exhibits non-independent and identically distributed (Non-IID) properties. Affected by Non-IID data, the client's local model updates may have significant differences, making the convergence of the aggregated global model unstable or inconsistent.

[0003] Various methods have been proposed to address the challenges of federated learning with non-IID data. To directly address the heterogeneous data distribution problem, data sharing methods involve servers storing a uniformly distributed shared dataset for warming up the global model and sharing it with clients to optimize local models. In data augmentation research, the server calculates the amount of data augmentation based on the data distribution of each client, and clients use this data augmentation to perform data augmentation. Direct solutions to the heterogeneous data distribution problem often involve the sharing of private data, which in turn creates the risk of privacy leakage. In model-based research, the Federated Averaging (FedAvg) algorithm weightedly aggregates local models to obtain a global model, mitigating the impact of non-IID data. Personalized models divide the trained model into a base layer and a personalized layer, performing global aggregation and local optimization tasks, respectively, to achieve personalized training for specific data distributions. However, model performance varies significantly under varying degrees of data heterogeneity. Knowledge distillation uses aggregated knowledge from heterogeneous clients to improve the server model, rather than directly aggregating local model parameters.

[0004] Federated learning methods train a unified global model to fit all data. However, under non-IID data distributions, the unified global model training effect is not ideal. Although non-IID has a negative impact on federated learning, the different data distributions of each client reflect the individuality of the client. Clustering Federated Learning (CFL) leverages the individual characteristics of data distribution to group clients with the same data distribution into the same cluster. The global model is trained on a cluster-by-cluster basis, minimizing the differences between the participating local models.

[0005] The cluster federated learning algorithm, MTCFL, leverages the differences in local model convergence directions and timing to design a phased clustering approach. When the average update value of participating client models is less than a given threshold and the maximum value is greater than a given threshold, a clustering process is performed using a binary search method. As a post-processing method, MTCFL achieves optimal clustering after a certain number of training rounds. However, global model convergence is unstable in the early and middle stages of training, and each global round requires all clients to participate in training, significantly increasing the communication burden on clients.

[0006] The IFCA algorithm exploits the fact that local losses vary across data distributions and sends all cluster global models to participating clients. Clients train each cluster's global model to obtain local losses and then determine their cluster by comparing the local losses. IFCA requires participating clients to receive all cluster models and train each cluster model simultaneously to compare losses and determine their cluster. This increases the communication burden on client devices and also pre-sets the number of clusters, which is not practical in real-world environments.

[0007] Existing federated clusters can meet the challenges of federated learning under non-IID data, but current federated clustering methods often use high communication costs in exchange for global model performance, which increases the computing and communication burden for edge devices and mobile devices with poor hardware computing capabilities and high communication costs. Summary of the Invention

[0008] The present invention provides a federated clustering method based on time-asynchronous similarity clustering, aiming to improve the above problems.

[0009] The present invention is implemented as follows: a federated clustering method based on time-asynchronous similarity clustering, the method comprising the following steps:

[0010] (1) Traverse all clients k in the client set K and use the current waiting time Calculate the weight of client k Update the weight list Q r ;

[0011] (2) Using the weight list Q r Randomly select |K|·μ clients from all clients to get the random client list K for round r r ;

[0012] (3) Traverse the random client list K r Client k in traverses the cluster set C of r rounds r If client k belongs to cluster c, then the cluster model of cluster c Send to client k so that client k can use the cluster model based on local data Update to the new local model And the local model and the amount of local data D k Send to the server;

[0013] (4) Receive the local model sent by client k The client list K r Record the client list T that has participated in the training r And the server sends the cluster model of client k and the local model of client k Save as history Traverse the client list T r Clients in, use the history list to calculate the client list T r The model similarity between any two clients is used to obtain the client similarity matrix Matrix of the rth round r ,in,

[0014] (5) Based on the similarity matrix r Perform hierarchical clustering to divide the new cluster set C r+1 ;

[0015] (6) Traverse the cluster set C r+1 The cluster c in the random client list K r And the model of client k that belongs to cluster c Aggregate to form a cluster model of r+1 rounds

[0016] (7) Check whether r+1 reaches the global training number R. If the test result is yes, output the cluster model list If the test result is no, then execute step (1).

[0017] Furthermore, the similarity matrix Matrix r The similarity matrix between client i and client j i,j The calculation process is as follows:

[0018] Detect whether client i and client j belong to the same cluster. If they do, calculate the similarity matrix of client i and client j in asynchronous time based on historical records. i,j If they do not belong to the same cluster, the cosine similarity Matrix of client i and client j is calculated based on historical records. i,j ;

[0019] The historical record is in After local training, it is updated to That is, the local model uploaded by the client to the server

[0020] Furthermore, the similarity matrix between client i and client j in asynchronous time is i,j The calculation formula is as follows:

[0021]

[0022] in, and They represent the cluster models sent by the server to client i and client j respectively, S init Represents the server's initialization global model, and Client i and client j receive and The local model after local training is performed, and cos represents the cosine similarity.

[0023] Furthermore, the similarity matrix of client i and client j that do not belong to the same cluster is i,j The calculation formula is as follows:

[0024]

[0025] Among them, S init Represents the server's initialization global model, and Client i and client j receive and The local model after local training is performed, and cos represents the cosine similarity.

[0026] Furthermore, the weight of client k in round r The calculation formula is as follows:

[0027]

[0028] in, is the waiting time of client k in round r.

[0029] Furthermore, the waiting time of client k in round r+1 is The specific determination method is as follows:

[0030] If client k is in the random client list K in round r r Then reset the waiting time of client k in round r+1 is 1, that is Otherwise, the waiting time of client k in round r+1 is

[0031] Furthermore, the model of cluster c The specific polymerization process is as follows:

[0032]

[0033] in, D k are the local model and local data volume uploaded by client k in step (3), respectively.

[0034] The present invention is implemented as follows: a federated clustering method based on time-asynchronous similarity clustering, the method is specifically as follows:

[0035] Random client list K r Client k in the update uses the gradient algorithm based on local data to update the local model Get the new local model And the local model and the amount of local data D k Sent to the server.

[0036] The present invention is implemented as follows: a federated clustering system based on time-asynchronous similarity clustering, the system comprising:

[0037] Server and client, multiple client-server communication connections;

[0038] Based on the aforementioned federated clustering method based on time-asynchronous similarity clustering, the server calculates the local model similarity of the local models uploaded by randomly selected clients, divides them into new cluster sets, aggregates the local models sent by the clients in the new cluster, obtains the integrated cluster model, and sends the cluster model to the corresponding clients.

[0039] The client updates the local model based on the above-mentioned federated clustering method based on time-asynchronous similarity clustering, and sends the updated local model and local data volume to the server.

[0040] This paper proposes a Federated Average Clustering Learning (FACL) algorithm, which reduces the communication cost of global training by randomly selecting some clients for training. Since the random selectivity of clients causes time differences between their local models, resulting in errors in similarity calculation, to ensure the accuracy of similarity calculation, a federated clustering algorithm with time-weighted random client selection and time-asynchronous similarity calculation is proposed. Time-weighted random client selection dynamically adjusts the probability of client selection based on the client's waiting time, making client selection more uniform and effectively reducing time differences. Time-asynchronous similarity calculation uses the similarity between cluster models to correct the local model similarity calculation method TAS, which is used to improve the accuracy of model similarity calculation with time differences. Ultimately, this improves the communication efficiency of federated learning for non-IID data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A framework diagram of a federated clustering process based on time-asynchronous similarity clustering provided by an embodiment of the present invention;

[0042] Figure 2 A comparison chart of MNIST similarity calculation results provided by an embodiment of the present invention;

[0043] Figure 3 A comparison chart of CIFAR10 similarity calculation results provided by an embodiment of the present invention;

[0044] Figure 4 This is a maximum time difference distribution diagram generated by weighted randomness and pure randomness provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The specific implementation methods of the present invention will be further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0046] This paper proposes a federated clustering method based on time-asynchronous similarity clustering, or FACL. Existing federated clustering methods require all clients to participate in each training round. This ensures that the global model updates received by each client are similar or even identical. Local training using the same model results in smaller similarity errors for local model updates, thereby improving clustering accuracy. However, this incurs a significant communication overhead. Therefore, FACL uses a federated learning method with random client selection to reduce overall communication efficiency. In federated learning with random client selection, the global model sent to the client in each training round is the latest. However, due to multiple local updates and cluster aggregation of the global model, time differences exist between global models from different training rounds. To ensure the accuracy of similarity calculations for client local model updates, FACL uses a similarity calculation method based on cluster model error correction to calculate the similarity of client model parameters. Furthermore, because random client selection introduces uncertainty in the size of the time difference, resulting in large time differences, FACL uses a time-weighted random client selection method to optimize the scheduling of random clients and limit the time difference to a limited range.

[0047] Figure 1 This is an architectural diagram of a federated clustering process based on time-asynchronous similarity clustering provided by an embodiment of the present invention. The method specifically includes the following steps:

[0048] During the initialization phase of FACL, the general parameters of federated learning are set: the local model learning rate η, the set of all clients participating in training K = {0, 1, ..., k}, the number of which is |K|, the percentage of clients randomly selected in each global training round μ, the number of global training rounds R, where r represents the current global training round; the weighted random selection related parameter settings: the time each client k in the current training round r waits to be randomly selected Composition of waiting time list Calculate the weight of each client k using the natural logarithm e exponentially through the waiting time list Composition weight list Federation cluster related settings: C r Represents the cluster set divided by the rth global round. Clients that have participated in training will be divided into the specified cluster c, c∈C r , the latest cluster model for each cluster Use T r Record all clients that have participated in training. Only those that have participated in training will generate local models and participate in cluster division. The only initial global model S init .

[0049] In the rth global training phase of FACL, the server uses the waiting time p of each client r Calculate the weight list Q r , according to the weight list Q r Weighted random selection of |K|×μ clients to obtain the client list K for round r r , the server will set the waiting time p of these selected clients r Reset to 1, the waiting time p of the client that is not selected r Increase by 1, and at the same time K r The client records in T r In, it is expressed as After that, the server determines K r The cluster to which each client belongs in , if client k belongs to cluster c, then the server will cluster c's latest cluster model Send to client k. It is important to note that when the client is selected to participate in training for the first time, the server will send the initial global model S init The cluster model is sent to the client. After receiving the cluster model, client k uses its own local data to train the cluster model to obtain a new local model. And the local model and the amount of local data D k Send to the server. The server receives K r After the local models of all clients are collected, the cluster model sent to the client and the received local model are combined into a model pair, and the model history list is updated. Used to calculate the similarity matrix, use formula (2) to calculate the client local model similarity matrix Matrix under asynchronous time r , use the similarity matrix to perform hierarchical clustering to obtain a new cluster set C r+1 , T r Each client in C is divided into r+1 The cluster in which the client list K of the current round is also included r Finally, for C r+1 Each cluster in the cluster calculates the latest cluster model, and we use the current round client list K r The trained local model is aggregated to obtain a cluster model. Specifically, for C r+1 Client k in cluster c in K r Only when it is in the model aggregation, the aggregation formula is:

[0050]

[0051] After completing R times of global training, the final cluster set CR , and the final cluster model for each cluster Finally, the performance of each cluster model will be calculated and compared with the model performance of the existing federated clustering algorithm to verify the advantages of the federated average clustering algorithm proposed in this paper in communication efficiency.

[0052] For the server:

[0053] (1) Traverse all clients k in the client set K and use the waiting time Recalculate the weight of client k Update the weight list Q r ;

[0054] (2) Using the weight list Q r Randomly select |K|·μ clients from all clients to get the random client list K for round r r ,The larger the weight value, the greater the probability that the corresponding client will be randomly selected;

[0055] (3) Update the waiting time of all clients to form the waiting time of the client in the r+1th round;

[0056] For simple random sampling, each sample in the overall sample set is completely independent and has the same probability of being selected. In federated learning, pure random sampling makes it impossible to predict the time interval between the selection of the same client, resulting in a large time difference. In order to control the generation of large time differences, the present invention recalculates the weighted random selection value of each client based on the length of time the client has not been selected. The waiting time of client k in the r+1th round is The specific determination method is as follows:

[0057] If client k is in the random client list K in round r r Then reset the waiting time of client k in round r+1 is 1, that is Otherwise, the waiting time of client k in round r+1 is

[0058] The waiting time of a client indicates how long it has not been selected. The longer the waiting time, the higher the probability that the client will be selected in the next round. Therefore, the waiting time is associated with the weight of the weighted random selection, and the weight value is defined as the exponential of the client k in the global training round r with the natural constant e as the base.

[0059]

[0060] (4) The random client list K of round r r Add the client in the list Tr In the ,clients are marked as those that have participated in the training;

[0061] (5) Traverse the random client list K r Client k in traverses the cluster set C of r rounds r If client k belongs to cluster c, then the cluster model of cluster c Sent to client k, based on the cluster model Update the history of client k middle: After local training, it is updated to That is, the local model uploaded by the client to the server

[0062] For the client:

[0063] (6) Random client list K r Client k in the update uses the gradient algorithm to update the local model Get the new local model And the local model and the amount of local data D k Sent to the server.

[0064] For the server:

[0065] (7) Receive the local model sent by client k The history of client k Updated to Traverse the client list T that has participated in training r Client k in the client list T is calculated based on the historical records r The local model similarity between any two clients in the , get the similarity matrix Matrix of the rth round r , similarity matrix Matrix r Client list T r The similarity value between any two clients in , where

[0066] In the embodiment of the present invention, the similarity matrix Matrix r The similarity matrix between client i and client j i,j The calculation process is as follows:

[0067] Detect whether client i and client j belong to the same cluster. If they do, calculate the similarity matrix of client i and client j based on formula (2): i,jIf they do not belong to the same cluster, the similarity matrix between client i and client j is calculated based on formula (3): i,j .

[0068] Since the selection of clients is random, the global models trained by different clients in the same cluster may have different training rounds, which makes the local models trained by cluster models with different rounds have errors when comparing similarities. In order to reduce the error of similarity calculation, a time-asynchronous similarity calculation with error correction is proposed. Similarity Matrix i,j The calculation formula is as follows:

[0069]

[0070] in, and They represent the cluster models sent by the server to client i and client j respectively, S init Represents the server's initialization global model, and Client i and client j receive and The local model after local training is performed, and cos represents the cosine similarity.

[0071] TAS represents different local models with the initial model S init The smaller the TAS, the higher the similarity of the local models, and the more likely they are to belong to the same cluster. Since the local models W of different clients may be obtained by updating the cluster models S in different rounds, and the vectors of cluster models in different rounds have similarity differences, that is, Therefore, in order to more accurately calculate the similarity between local models, TAS needs to eliminate the similarity differences of cluster models based on the current local model similarity.

[0072] In this embodiment of the present invention, the similarity matrix of client i and client j that do not belong to the same cluster is i,j The calculation formula is as follows:

[0073]

[0074] (8) Based on the similarity matrix M r Perform hierarchical clustering to divide the new cluster set C r+1 ;

[0075] (9) Traverse the cluster set C r+1 The cluster c in the random client list K r And the model of client k that belongs to cluster c Aggregate to form a cluster model of r+1 rounds The specific polymerization process is as follows:

[0076]

[0077] (10) Check whether r+1 reaches the global training number R. If the test result is yes, output the cluster model list If the test result is no, then execute step (1).

[0078] The present invention also provides a federated cluster system based on time asynchronous similarity clustering, the system comprising: a server and a client, a plurality of client-server communication connections;

[0079] Based on the above-mentioned federated clustering method based on time-asynchronous similarity clustering, the server calculates the local model similarity and divides the local models uploaded by randomly selected clients into new cluster sets, aggregates the local models sent by the clients in the new cluster, obtains the integrated cluster model, and sends the cluster model to the corresponding clients; the client updates the local model based on the above-mentioned federated clustering method based on time-asynchronous similarity clustering, and sends the updated local model and local data volume to the server.

[0080] The present invention denotes the amount of data sent from one end to the other end during a network transmission of the model as θ. The communication volume of each client participating in the training in each global round is 2θ. Therefore, the total communication volume of FACL in the entire federated training process is Θ. FACL for:

[0081] Θ FACL =2θμR|K|

[0082] Generally, in the comparison of federated clustering algorithms, in order to control variables, the values of R and |K| are often the same. Therefore, the difference in communication volume between different federated clustering algorithms is related to the model communication volume of a global round and the number of participating clients in a global round. The FACL algorithm in this paper controls Θ by controlling the value of μ. FACL In particular, when μ = 1, Θ FACL The value of is the same as the total amount of communication of the federated cluster algorithm with all clients participating.

[0083] The present invention compares the FACL algorithm with the following three baseline algorithms:

[0084] (1) FedAvg: A classic algorithm for federated learning that uses stochastic gradient descent to train local models and obtains a global model through weighted aggregation of models.

[0085] (2) MTCFL: The converged model is divided into clusters using the cosine similarity as the criterion, and the clients with the highest similarity are finally clustered into the same cluster.

[0086] (3) IFCA: Each round of global training requires all clients to participate. By training multiple cluster models locally, the loss values of the models are compared to determine the cluster to which the client belongs.

[0087] The data sets used in the experiments of this invention are the following two:

[0088] The first type: the MNIST dataset. It consists of 60,000 training samples and 10,000 test samples. Each sample is a 28*28 pixel grayscale image representing a handwritten digit. Each digit is an integer from 0 to 9, with a total of 10 label types.

[0089] The second type is the CIFAR-10 dataset. The dataset contains color images from 10 different categories, with 6,000 images in each category, for a total of 60,000 images. These categories include airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The size of each image is 3*32*32 pixels.

[0090] This paper addresses the issue of label distribution skew in Non-IID data by partitioning the data. This means that different clients have different label distributions P(y), while data with the same label distribution have the same feature distribution P(x|y). This paper categorizes label distribution skew into three scenarios: partial intersection, true subset, and no intersection. The MNIST and CIFAR-10 datasets are repartitioned according to these three scenarios:

[0091] Distribution-Overlapping: Set five different distributions and assign four labels to each client's local data. That is, 100 clients are assigned labels according to the following distributions: {0, 1, 2, 3}, {2, 3, 4, 5}, {4, 5, 6, 7}, {6, 7, 8, 9}, and {0, 1, 8, 9}. Each distribution has 20 clients, and this distribution causes label overlap between different clients.

[0092] Distribution-Cover: Set four different distributions and assign 100 clients according to the label distributions of {0, 1, 2, 3, 4}, {1, 2, 3}, {5, 6, 7, 8, 9}, {6, 7, 8}, and {0, 4, 5, 9}. That is, there are 25 clients in each distribution. This distribution ensures that the labels of individual clients are completely covered.

[0093] Distribution-Parallel: Set five different distributions and assign two types of labels to the local data of each client. That is, 100 clients are assigned labels according to the distributions {0, 1}, {2, 3}, {4, 5}, {6, 7}, and {8, 9}, with 20 clients in each distribution. This distribution ensures that the labels of different clients do not overlap.

[0094] The federated learning parameters of the FACL algorithm are set as follows: global iteration round T is 2000, local iteration round E is 2, and the percentage of randomly selected clients μ is 10%. In order to verify the advantages of the FACL algorithm in model performance and communication efficiency,

[0095] This method controls the global iteration rate of all algorithms, maintaining consistent communication traffic consumption while ensuring final model convergence. Ultimately, the performance of each algorithm is compared using the maximum model accuracy at training completion. The FACL algorithm's optimization of model convergence speed and communication efficiency is verified by measuring the communication traffic consumed by each algorithm when achieving a specified model accuracy.

[0096] In order to verify the effectiveness of TAS in calculating model similarity with time differences, this paper compares TAS and cosine similarity of clients with the same data distribution with different time differences, where the local models of clients i and j are and The data distribution p(i) = p(j), and the time difference TD (Time Difference) is |mn|. To verify the effectiveness of the wait-time-based weighted random selection method, this paper simulates the client selection process using both pure randomness and weighted randomness, and records the maximum time difference TD.

[0097] The two evaluation metrics used in the present invention's experiments are the model's prediction accuracy on the test set data and the amount of communication required to achieve a specific accuracy. The baseline algorithm, FedAvg, trains a global model; therefore, its accuracy is the prediction accuracy of a single global model on the test set. For federated clustering algorithms, the labels for each cluster's test set data match those for the corresponding cluster's training set data. Because multiple cluster models exist, the accuracy of each federated clustering algorithm is expressed as the average of all clusters' accuracy.

[0098] This paper experiments with FACL on two public datasets, MNIST and CIFAR10, across different data distributions. The paper also compares model accuracy and training communication efficiency with FedAvg and existing federated clustering algorithms. To ensure FACL's effectiveness, the paper divides the training data into three different distributions based on label type: partially overlapping, fully overlapping, and completely non-overlapping.

[0099] Table 1 Model accuracy of each algorithm on the MNIST dataset under the same communication volume

[0100]

[0101] Table 2 Model accuracy of each algorithm on the CIFAR10 dataset under the same communication volume

[0102]

[0103] This paper designs two deep neural network training models for the MNIST and CIFAR10 datasets, respectively. The layer structure of the model for the MNIST dataset is as follows: a convolutional layer with 1 input channel, 20 output channels, a kernel size of 5, and a stride of 1; a convolutional layer with 20 input channels, 50 output channels, a kernel size of 5, and a stride of 1; and a linear layer with output feature sizes of 500, 84, 20, and 10, respectively. The layer structure of the model for the CIFAR10 dataset is as follows: a convolutional layer with 3 input channels, 64 output channels, a kernel size of 5, and a stride of 1; a max pooling layer with a kernel size of 3 and a stride of 2; a convolutional layer with 64 input channels, 64 output channels, a kernel size of 5; and a linear layer with output feature sizes of 384, 192, and 10, respectively.

[0104] In experiments verifying model performance, it is necessary to ensure that the maximum model accuracy is achieved while consuming the same amount of traffic. Therefore, the performance comparison experiments were conducted under the same communication volume. The experimental results are shown in Tables 1 and 2. We can see that, under the same communication volume, the FACL model accuracy outperforms the FedAvg baseline algorithm, as a single global model struggles to fit data with diverse distributions. It also slightly outperforms existing federated clustering algorithms because, at the same communication cost, the FACL clustering model updates more frequently, resulting in faster convergence. It can be seen that under the Non-IID data distribution, multiple clustering models outperform a single global model, validating the superior performance of our proposed federated clustering algorithm.

[0105] Table 3 Communication volume of each algorithm with specified accuracy in MNIST dataset

[0106]

[0107] Table 4 Communication volume of each algorithm with specified accuracy on CIFAR10 dataset

[0108]

[0109] To compare the communication efficiency of each algorithm, this paper records the amount of communication consumed by each algorithm when the model trained by the algorithm achieves a specified accuracy. To ensure the validity of the traffic comparison, the specified model accuracy is a value adjacent to the maximum model accuracy. Therefore, when a poorly performing model fails to achieve the specified model accuracy, the traffic statistics are not counted. The final experimental results are shown in Tables 3 and 4. Compared with FedAvg, the communication cost of FACL proposed in this paper is only approximately 30% of that of FedAvg when the model trained by FedAvg achieves the specified accuracy. Because randomly selecting a subset of participating clients reduces the number of participating clients while ensuring model convergence, FACL can effectively reduce the overall communication volume of the federated cluster.

[0110] The present invention uses the above experimental data to verify the effectiveness of the TAS similarity calculation method. In the FACL training environment, two clients with the same data distribution are selected for training tracking. The relevant model data after each client is selected for training is recorded. After the training is completed, the saved model data is used to perform a comparative experiment on TAS and cosine similarity. The experimental results are shown in Figure 2. Figure 2 、 3 This article uses a variation of cosine similarity, with values ranging from [0, 2]. Smaller values indicate higher similarity. The results show that using TAS with time-asynchronous computation yields better local model similarity. Specifically, the greater the time difference, the lower the model similarity. In particular, when the time difference is 0, representing two clients using the same cluster model for local training in the same global round, the similarity is highest.

[0111] The present invention experimentally verifies the effectiveness of using the time-weighted random selection method in reducing time differences. Regarding the experimental parameter settings, 10 clients are randomly selected from 100 clients in each round of global training, the global round is set to 1000, and the maximum time difference of the entire training process is recorded. In order to ensure the stability and repeatability of the above experimental process, the above process is repeated 1000 times, and the maximum time difference obtained each time is recorded. The experimental results are shown in Figure 2. Figure 4 The horizontal axis represents the maximum time difference produced by each experimental result, and the vertical axis represents the frequency of time difference in 1000 experiments. Figure 4The results show that using time-weighted randomization can reduce the maximum time difference, while purely randomization results in larger and unevenly distributed time differences. TAS validation experiments show that the smaller the time difference, the smaller the error in similarity calculation. Therefore, time-weighted randomization can improve the accuracy of similarity calculation by reducing time differences.

[0112] Existing research on federated clustering algorithms often focuses on the cluster division process. Most federated clustering algorithms require all clients to participate in the training throughout the entire process, or increase the client's communication cost to improve the accuracy of cluster division. The present invention studies the communication efficiency problem of federated clusters and proposes a federated clustering algorithm FACL based on FedAvg's client random selection strategy and the model's weighted average aggregation strategy. In this paper, we found that due to the random selectivity of the client, the client's model has differences in training rounds, and the convergence direction of the model in different rounds is different, which makes it difficult to calculate the similarity directly based on the model uploaded by the client. In order to solve the above problem, this paper analyzes the similarity difference of the client model, which comes from the difference in cluster models in different rounds. Finally, the TAS similarity calculation method is proposed to eliminate the similarity difference of cluster models in different rounds and the time-weighted random selection method, thereby improving the stability of the similarity calculation. Compared with the existing federated clustering algorithm, the FACL algorithm can achieve the specified model performance with less communication cost.

[0113] This paper conducts extensive experiments using the MNIST and CIFAR-10 datasets in a non-IID data environment to evaluate the model performance and global communication volume of the federated average clustering algorithm. The experimental results demonstrate that the proposed federated average clustering algorithm can achieve the same or even better model accuracy than existing federated clustering algorithms. In the communication volume evaluation experiment, the federated average clustering algorithm achieves the specified model accuracy while using 5% to 30% of the communication volume of existing federated clustering algorithms.

[0114] The present invention has been described exemplarily. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the scope of protection of the present invention.

Claims

1. A federated clustering method based on time-asynchronous similarity clustering, characterized in that: The method comprises the following steps: (1) Traverse all clients k in the client set K and use the current waiting time Calculate the weight of client k Update the weight list Q r ; (2) Using the weight list Q r Randomly select |K|·μ clients from all clients to get the random client list K for round r r ; (3) Traverse the random client list K r Client k in traverses the cluster set C of r rounds r If client k belongs to cluster c, then the cluster model of cluster c Send to client k so that client k can use the cluster model based on local data Update to the new local model And the local model and the amount of local data D k Send to the server; (4) Receive the local model sent by client k Calculate the client list T r The local model similarity between any two clients in the , get the similarity matrix Matrix of the rth round r ,in, (5) Based on the similarity matrix r Perform hierarchical clustering to divide the new cluster set C r+1 ; (6) Traverse the cluster set C r+1 The cluster c in the random client list K r And the model of client k that belongs to cluster c Aggregate to form a cluster model of r+1 rounds (7) Check whether r+1 reaches the global training number R. If the test result is yes, output the cluster model list If the test result is no, then execute step (1).

2. The federated clustering method based on time-asynchronous similarity clustering according to claim 1, characterized in that: Similarity Matrix r The similarity matrix between client i and client j i,j The calculation process is as follows: Detect whether client i and client j belong to the same cluster. If they do, calculate the similarity matrix of client i and client j in asynchronous time based on historical records. i,j If they do not belong to the same cluster, the cosine similarity Matrix of client i and client j is calculated based on historical records. i,j ; The historical record is in After local training, it is updated to That is, the local model uploaded by the client to the server 3. The federated clustering method based on time-asynchronous similarity clustering according to claim 2, characterized in that: Similarity Matrix between client i and client j in asynchronous time i,j The calculation formula is as follows: in, and They represent the cluster models sent by the server to client i and client j respectively, S init Represents the server's initialization global model, and Client i and client j receive and The local model after local training is performed, and cos represents the cosine similarity.

4. The federated clustering method based on time-asynchronous similarity clustering according to claim 2, characterized in that: Similarity Matrix between client i and client j that do not belong to the same cluster i,j The calculation formula is as follows: Among them, S init Represents the server's initialization global model, and Client i and client j receive and The local model after local training is performed, and cos represents the cosine similarity.

5. The federated clustering method based on time-asynchronous similarity clustering according to claim 1, characterized in that: The weight of client k in round r The calculation formula is as follows: in, is the waiting time of client k in round r.

6. The federated clustering method based on time-asynchronous similarity clustering according to claim 5, characterized in that: The waiting time of client k in round r+1 The specific determination method is as follows: If client k is in the random client list K in round r r Then reset the waiting time of client k in round r+1 is 1, that is Otherwise, the waiting time of client k in round r+1 is 7. The federated clustering method based on time-asynchronous similarity clustering according to claim 1, characterized in that: Cluster Model The specific polymerization process is as follows: in, D k are the local model and local data volume uploaded by client k in step (3), respectively.

8. The federated clustering method based on time-asynchronous similarity clustering according to claim 1, characterized in that: Random client list K r Client k in the update uses the gradient algorithm based on local data to update the local model Get the new local model And the local model and the amount of local data D k Sent to the server.

9. A federated clustering system based on time-asynchronous similarity clustering, characterized in that: The system comprises: Server and client, multiple client-server communication connections; The server calculates local model similarities and divides new cluster sets for local models uploaded by randomly selected clients based on the federated clustering method based on time-asynchronous similarity clustering as described in any one of claims 1 to 7, aggregates the local models sent by the clients in the new cluster, obtains an integrated cluster model, and sends the cluster model to the corresponding clients; The client updates the local model based on the federated clustering method based on time-asynchronous similarity clustering described in claim 8, and sends the updated local model and local data volume to the server.

Images