Clustering-based semi-asynchronous federated learning method and system

Abstract

The invention relates to the technical field of machine learning, in particular to a clustering-based semi-asynchronous federated learning method and system, and the system comprises the steps: a clustering module obtains model training results and training processing time of local training of a plurality of client devices, carries out the clustering according to the training processing time, and obtains a plurality of groups of device clusters, selecting a coordinator in each group of equipment cluster for coordinating other client equipment in the cluster to keep synchronous communication and synchronous aggregation, and performing asynchronous communication with a server outside the cluster to keep asynchronous updating; the model aggregation module receives the training result uploaded by each equipment cluster by using a server, performs asynchronous updating between clusters based on a delay compensation mechanism to generate a new global model, and sends the new global model back to a coordinator to complete a round of federal learning training; and continuously iterating until the global model is converged. Synchronization and asynchronization methods are fused, and the accuracy is kept while the federal learning convergence efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a clustering-based semi-asynchronous federated learning method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Most of the data generated by IoT devices need to be centrally stored in high-performance servers for modeling and analysis. These data are usually highly privacy-sensitive (such as personal health records, household electricity consumption data, etc.). Federated learning (FL) can avoid the transmission of user privacy data between multiple IoT devices by coordinating multiple IoT devices for AI training without sharing original data.

[0004] Most federated learning methods assume that they are running in a synchronous communication environment, and that clients can upload local models to the server at the same time, implicitly assuming that all selected clients can complete local training at a similar time. However, this assumption is often not true in actual scenarios, as client devices usually have different computing capabilities and network resources, resulting in local updates received by the server not being synchronized. In this case, the client that completes local training and uploads model training results the latest is called a "straggle device." The training time for these devices is long, affecting the training efficiency of the entire federated learning system.

[0005] To address the above issues, most federated learning methods adopt the strategy of letting faster clients wait for lagging devices to complete training before converging. Although this method can balance the progress of each device to a certain extent, it significantly reduces the overall convergence speed and efficiency of the system and causes a waste of computing resources for devices with faster training speeds.

[0006] In order to deal with the problem caused by device lag in synchronous federated learning (FL), the existing technology proposes an asynchronous federated learning method to improve convergence performance. In the asynchronous scheme, once the server receives a local model update from any device, it will immediately update the global model without waiting for all devices to complete the task. This method is simple to manage and has low configuration costs, which helps to improve system efficiency. However, since each global model update is based on only one local model, training bias may be introduced, thereby affecting the performance of the global model. In order to achieve the same training effect as the synchronous scheme (such as the same accuracy), the asynchronous scheme usually requires more training rounds and time, which will cause a significant increase in the communication frequency between the server and the client, resulting in a significant increase in bandwidth consumption. Summary of the invention

[0007] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a clustering-based semi-asynchronous federated learning method and system, which combines the advantages of synchronous and asynchronous methods, and can maintain relatively stable accuracy while improving the convergence efficiency of federated learning.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: A first aspect of the present invention provides a clustering-based semi-asynchronous federated learning system, comprising: The clustering module is configured to: obtain model training results and training processing time of local training of multiple client devices, and cluster them according to the training processing time to obtain multiple device clusters; select a coordinator in each device cluster to coordinate other client devices in the cluster to maintain synchronous communication and synchronous aggregation, and to communicate asynchronously with servers outside the cluster to maintain asynchronous updates; The model aggregation module is configured to: use the server to receive the training results uploaded by each device cluster, perform asynchronous updates between clusters based on the delay compensation mechanism, generate a new global model, and send the new global model back to the coordinator to complete a round of federated learning training; through continuous iterations, until the global model converges.

[0009] Furthermore, the coordinator in each device cluster is selected, specifically: the client device with the shortest processing time in each device cluster is selected as the coordinator with the strongest processing capability in the cluster.

[0010] Furthermore, the processing time includes the sum of the local training computation time and the communication time for the local model update.

[0011] Furthermore, the processing time is + ， , ; in, Compute time for local training, Communication time for local model updates, is the CPU frequency of the client device, The CPU cycles for training one data sample, is the dataset size, is the size of the uploaded data and the transmission rate , is the transmission bandwidth, is the channel gain, is the transmission power, is the background noise power.

[0012] Furthermore, the goal of clustering is as follows: minimize ; ， > 0; in, Represents the weight index, as the variance increases with the increase of .

[0013] Furthermore, cluster The time mean and variance of a round of intra-cluster aggregation are shown in the following formula: ; ; in, is the mean, is the variance, For clusters, For processing time, Represents a client device Is it a cluster? .

[0014] Furthermore, Represents a client device Is it a cluster? , as shown below: ; in, For clusters, is the weight, For processing time.

[0015] Furthermore, the weight The calculation and update method of is as shown below: ; , 0< <1 ; in, is the variance, For clusters, is the weight; p is a parameter, ranging from 0 to 1; is a constant that controls the influence of the previous weight index on the current update.

[0016] Furthermore, the delay compensation mechanism is as follows: ; ; in, is the global round number on the server, Cluster The round number of the uploaded cluster model. Every time the server receives an update, the global round count increases by 1. At the same time, the updated round count is sent back to the coordinator. When the uploaded cluster model is not stale, , if the cluster model sent to the server is stale, then The value is between 0 and 1.

[0017] A second aspect of the present invention provides a clustering-based semi-asynchronous federated learning method, comprising the following steps: Obtain model training results and training processing time for local training of multiple client devices, and cluster them according to the training processing time to obtain multiple device clusters, each of which has training results of multiple client devices; select a coordinator in each device cluster to coordinate other client devices in the cluster to maintain synchronous communication and synchronous aggregation, and to communicate asynchronously with servers outside the cluster to maintain asynchronous updates; The server receives the training results uploaded by each device cluster, performs asynchronous updates between clusters based on the delay compensation mechanism, generates a new global model, and completes a round of federated learning training by sending the new global model back to the coordinator; through continuous iterations, the global model converges.

[0018] Compared with the prior art, one or more of the above technical solutions have the following beneficial effects: Combining the advantages of synchronous and asynchronous FL, the training results of client devices with similar training completion time are clustered together first, and a cluster model with higher accuracy is obtained through synchronous aggregation. Then asynchronous updates are performed between clusters, and the impact of delays is alleviated through a delay compensation mechanism, and then global updates are performed. Compared with traditional federated learning synchronization solutions, this can improve the efficiency of federated learning while maintaining high model accuracy, ultimately improving the global model performance of the federated learning system, saving training completion time, and solving the problem of device heterogeneity. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0020] Figure 1 It is a schematic diagram of the architecture of a clustering-based semi-asynchronous federated learning system provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0023] Terminology explanation: Clustering: Divide samples into several groups (clusters) based on the similarity of certain features, so that the similarity of samples in the same group is high, while the similarity of samples between different groups is low.

[0024] Semi-asynchronous federated learning: Based on traditional federated learning, some clients are allowed to upload updates asynchronously, while there is also a certain amount of synchronous update processing to improve the training efficiency of federated learning.

[0025] Mobile and IoT devices under the federated learning (FL) framework usually have different computing capabilities and are deployed in different network resource environments. The synchronous training method of traditional federated learning assumes that the client can complete local training and upload the model synchronously, but due to the difference in computing power between devices and the dynamic changes in deployed network resources, the update process is often not synchronized. Although the synchronous FL solution can achieve better performance in terms of accuracy, it will produce a "laggard effect" and reduce the training efficiency of the entire system.

[0026] Some solutions propose asynchronous FL solutions to solve the above problems, but asynchronous solutions may cause training deviations, affect model performance, and require more training rounds and more frequent communications, increasing bandwidth consumption. Therefore, in order to balance efficiency and accuracy, this embodiment proposes a clustering-based semi-asynchronous federated learning method and system, which integrates the advantages of synchronization and asynchrony, improves the convergence efficiency of federated learning while maintaining relatively stable model performance, and ultimately improves the overall performance of the system and solves the problem of device heterogeneity in federated learning.

[0027] Embodiment 1: Clustering-based semi-asynchronous federated learning system, including: The clustering module is configured to: obtain model training results and training processing time of multiple client devices for local training, and cluster them according to the training processing time to obtain multiple device clusters, each of which has training results of multiple client devices; select a coordinator in each device cluster to coordinate other client devices in the cluster to maintain synchronous communication and synchronous aggregation, and to communicate asynchronously with servers outside the cluster to maintain asynchronous updates; The model aggregation module is configured to: use the server to receive the training results uploaded by each device cluster, perform asynchronous updates between clusters based on the delay compensation mechanism, generate a new global model, and send the new global model back to the coordinator to complete a round of federated learning training; through continuous iterations, until the global model converges.

[0028] The federated learning system in this embodiment assumes that A federated learning client device and a federated learning server Each client With size , The local data set represents the data on the current device. Each client In its dataset Perform local model training on Yes Equipment The data samples, then the device The local loss function on can be expressed as: ; The ultimate goal of federated learning is to maximize the performance of the global model (i.e., minimize the global model loss function), namely: ; in, represents the best global model, All in the system The total number of data samples for each device.

[0029] The federated learning method proposed in this embodiment includes two main modules: a clustering module and a model aggregation module. The clustering module uses the MinMax K-Means algorithm to cluster devices and perform synchronous aggregation within each device cluster; the model aggregation module is responsible for collecting the model parameters uploaded by each device cluster asynchronously, and updates the global model of FL based on these model parameters, thereby generating new global model parameters.

[0030] Clustering module: The performance of client devices is inherently related to available resources, such as CPU, memory, and network bandwidth. It is not enough to evaluate the performance of active devices based solely on the hardware characteristics of the device, because environmental factors such as temperature and voltage may also have a significant impact on the device. In addition, network resource conditions play a key role in determining the efficiency of the training process because multiple rounds of communication occur between the server and the client. Therefore, this embodiment chooses processing time as the key indicator for measuring the performance of edge devices.

[0031] Processing time in this embodiment Defined as client device The sum of the computation time required to perform its local training task and one round of communication time required to update its local model.

[0032] Local training computing time By client The computing power (i.e. CPU frequency) , CPU cycles to train a data sample And the dataset size is calculated as: .

[0033] Communication time for local model update By transmission rate and the size of the uploaded data express: .in, represents the transmission bandwidth, represents the channel gain, Indicates the transmit power, Represents the background noise power.

[0034] So, the client The processing time is + .

[0035] The goal of the clustering module is to minimize the weighted variance within the largest cluster during a round of intra-cluster aggregation time, that is: minimize ; ， > 0; in, Represents the weight index, as the variance increases with the increase of .

[0036] This embodiment uses and Respectively represent clusters The temporal mean and variance of the intra-cluster aggregation during a round.

[0037] Client will be independently assigned to the nearest cluster, i.e. to the client With Center The cluster with the smallest weighted distance between them. Represents the client Is it a cluster? , as shown below: ; in, For clusters, is the weight, For processing time.

[0038] About weight The calculation and update method of is shown in the following formula: ; , 0< < 1; in, p is a parameter ranging from 0 to 1. is a constant that controls the influence of the previous weight index on the current update.

[0039] The clustering module aims to minimize the weighted variance within the largest cluster. It prevents some clusters from being too dense or sparse by controlling the weight exponent of the weighted variance within the cluster, making the size and variance of each cluster more balanced, thereby enhancing the robustness of the clustering results.

[0040] In addition, after clustering, each cluster elects a device with the most powerful processing power as the coordinator of the cluster. In addition to updating and training locally, the coordinator needs to implement two more functions: ① Act as a proxy server in the cluster, coordinating other client devices in the cluster to maintain synchronous communication and synchronous aggregation; ② As the manager of the client devices in this cluster, it is responsible for asynchronous communication with the FL server outside the cluster and maintaining asynchronous updates.

[0041] Model aggregation module: In this embodiment, clusters are updated asynchronously, and the update method of the global model follows the following formula: ; in, are the parameters of the cluster model uploaded by the coordinator, is the update parameter of the global model, , is a hyperparameter used to reduce the staleness of cluster model updates. Generally speaking, asynchronous update methods will have model staleness, that is, the number of times the coordinator updates the model does not match the number of times the global model is aggregated and updated.

[0042] The number of updates of cluster models uploaded by some clusters is less than the number of updates of the current global model aggregation. Such cluster models will reduce the performance of the global model during global aggregation. Therefore, this embodiment adopts a delay compensation mechanism to solve this problem and sets an obsolete function: ; in, Represents the global round number on the server, Representative cluster The round number of the cluster model upload. Specifically, each time the server receives an update, the global round count is increased by 1.

[0043] At the same time, the updated round count will be sent back to the coordinator. as follows: ; When the uploaded cluster model is not stale , which means that the parameter is just a fixed value, such as 0.5. On the contrary, if the cluster model sent to the server is outdated, The value of is between 0 and 1. The greater the staleness, The closer to 0, the The smaller it is, the less impact it has on the global model.

[0044] Figure 1 The main components of this embodiment are shown, including synchronous communication (represented by solid arrows) and asynchronous communication (represented by dashed arrows), where: The server first initializes the global model and broadcasts the model to each client; In each round, the coordinator in the cluster initiates the training by broadcasting the global model to the rest of the clients in the cluster; Each device (including the coordinator itself) performs local training for multiple epochs and then sends the model parameters to the coordinator; After the coordinator receives the model parameters from all clients in the cluster, it performs synchronized average aggregation to update the cluster-based global model; When the cluster-based global model is ready, the coordinator asynchronously uploads the updated model parameters to the server; The server uses the received model parameters to perform the final global update; The server sends the new global model back to the coordinator to complete a round of FL training; Through continuous iterations, the global model converges.

[0045] This embodiment combines the advantages of synchronous and asynchronous FL, significantly reducing the total idle time on the device and alleviating the problem of falling behind in the system. Compared with the traditional federated learning synchronous solution, it can improve the training efficiency of the system, save training completion time, and solve the problem of device heterogeneity; compared with the traditional asynchronous federated learning solution, it can reduce the global model deviation and improve the overall system performance.

[0046] By adopting clustering and semi-asynchronous methods, clients with similar training completion times are clustered together first, and a cluster model with higher accuracy is obtained through synchronous aggregation. Then, asynchronous updates are performed between clusters, and the impact of delays is alleviated through a delay compensation mechanism, and global updates are performed to improve the efficiency of federated learning while maintaining high model accuracy. Ultimately, the global model performance of the federated learning system is improved.

[0047] Embodiment 2: The clustering-based semi-asynchronous federated learning method includes the following steps: Obtain model training results and training processing time for local training of multiple client devices, and cluster them according to the training processing time to obtain multiple device clusters, each of which has training results of multiple client devices; select a coordinator in each device cluster to coordinate other client devices in the cluster to maintain synchronous communication and synchronous aggregation, and to communicate asynchronously with servers outside the cluster to maintain asynchronous updates; The server receives the training results uploaded by each device cluster, performs asynchronous updates between clusters based on the delay compensation mechanism, generates a new global model, and completes a round of federated learning training by sending the new global model back to the coordinator; through continuous iterations, the global model converges.

[0048] Combining the advantages of synchronous and asynchronous FL, it significantly reduces the total idle time on the device and alleviates the problem of falling behind in the system. Compared with the traditional federated learning synchronous solution, it can improve the training efficiency of the system, save training completion time, and solve the problem of device heterogeneity; compared with the traditional asynchronous federated learning solution, it can reduce the global model deviation and improve the overall system performance. Specifically, this embodiment adopts a clustering method and a semi-asynchronous method to cluster clients with similar training completion times first, and obtain a cluster model with higher accuracy through synchronous aggregation. Then, asynchronous updates are performed between clusters, and the impact of delays is alleviated through a delay compensation mechanism, and global updates are performed to improve the efficiency of federated learning while maintaining a high model accuracy. Ultimately, the global model performance of the federated learning system is improved.

[0049] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. Clustering-based semi-asynchronous federated learning system, characterized by: include: The clustering module is configured to: obtain model training results and training processing time of local training of multiple client devices, cluster them according to the training processing time to obtain multiple device clusters, select a coordinator in each device cluster to coordinate other client devices in the cluster to maintain synchronous communication and synchronous aggregation, and perform asynchronous communication with servers outside the cluster to maintain asynchronous updates; The model aggregation module is configured to: use the server to receive the training results uploaded by each device cluster, perform asynchronous updates between clusters based on the delay compensation mechanism, generate a new global model, and send the new global model back to the coordinator to complete a round of federated learning training; through continuous iterations, until the global model converges.

2. The clustering-based semi-asynchronous federated learning system according to claim 1, characterized in that: The coordinator in each device cluster is selected, specifically: the client device with the shortest processing time in each device cluster is selected as the coordinator with the strongest processing capability in the cluster.

3. The clustering-based semi-asynchronous federated learning system according to claim 2, characterized in that: The processing time includes the sum of the local training computation time and the communication time for local model updates.

4. The clustering-based semi-asynchronous federated learning system according to claim 1, characterized in that: Processing time is + ， , ; in, Compute time for local training, Communication time for local model updates, is the CPU frequency of the client device, The CPU cycles for training one data sample, is the dataset size, is the size of the uploaded data and the transmission rate , is the transmission bandwidth, is the channel gain, is the transmission power, is the background noise power.

5. The clustering-based semi-asynchronous federated learning system according to claim 1, characterized in that: The goal of clustering is as follows: minimize ; ， > 0； in, Represents the weight index, as the variance increases with the increase of .

6. The clustering-based semi-asynchronous federated learning system according to claim 5, characterized in that: cluster The time mean and variance of a round of intra-cluster aggregation are shown in the following formula: ； ； in, is the mean, is the variance, For clusters, For processing time, Represents a client device Is it a cluster? .

7. The clustering-based semi-asynchronous federated learning system according to claim 6, characterized in that: Represents a client device Is it a cluster? , as shown below: ； in, For clusters, is the weight, For processing time.

8. The clustering-based semi-asynchronous federated learning system according to claim 5, characterized in that: Weight The calculation and update method of is as shown below: ； ， 0< < 1； in, is the variance, For clusters, is the weight; p is a parameter, ranging from 0 to 1; is a constant that controls the influence of the previous weight index on the current update.

9. The clustering-based semi-asynchronous federated learning system according to claim 1, characterized in that: Delay compensation mechanism, specifically: ； ； in, is the global round number on the server, Cluster The round number of the uploaded cluster model. Every time the server receives an update, the global round count increases by 1. At the same time, the updated round count is sent back to the coordinator. When the uploaded cluster model is not stale, , if the cluster model sent to the server is stale, then The value is between 0 and 1.

10. A clustering-based semi-asynchronous federated learning method, implemented using the system according to any one of claims 1 to 9, characterized in that: The following steps are involved: Obtain model training results and training processing time for local training of multiple client devices, and cluster them according to the training processing time to obtain multiple device clusters, each of which has training results of multiple client devices; select a coordinator in each device cluster to coordinate other client devices in the cluster to maintain synchronous communication and synchronous aggregation, and to communicate asynchronously with servers outside the cluster to maintain asynchronous updates; The server receives the training results uploaded by each device cluster, performs asynchronous updates between clusters based on the delay compensation mechanism, generates a new global model, and completes a round of federated learning training by sending the new global model back to the coordinator; through continuous iterations, the global model converges.

Images