Semi-asynchronous federal learning training method based on intelligent client selection and adaptive weighting

Abstract

The invention provides a semi-asynchronous federal learning training method based on intelligent client selection and adaptive weighting, and the method comprises the steps: enabling a client to carry out the model initialization according to the broadcast of a central server, and enabling the central server to calculate the local data distribution according to a training set, clustering the clients according to a clustering algorithm on the basis of local data distribution of the clients; the central server randomly selects clients from the clusters of each cluster in a fixed period and broadcasts the latest global model weight; all the selected clients upload local model weights and version numbers immediately, and the unselected clients continue to carry out local training; the central server performs synchronous self-adaptive weighted aggregation on each selected client according to the uploaded local model weight and the local model version to update the global model weight; and S4, repeating the steps S2 to S4 to realize global training until the global model reaches the accuracy of the target model, and ending the training. The convergence speed and accuracy of the model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of federated learning in distributed machine learning, and particularly relates to a semi-asynchronous federated learning method. BACKGROUND

[0002] With the rapid development of artificial intelligence and Internet of Things, a large amount of valuable data is generated by user terminals such as sensors, unmanned aerial vehicles and intelligent vehicles. In traditional machine learning and deep learning, these data must be uploaded by users to a central server for centralized model training. This approach not only requires a large amount of communication resources, but also seriously threatens the privacy and security of the clients. As one of the new types of distributed machine learning paradigms, federated learning (FL) is a very promising solution to the above problems. In federated learning, a central server coordinates multiple clients to train a shared model. The client only uploads the local model trained by the local data to the central server and keeps the original data locally, so as to achieve distributed machine learning without leaking personal data. The most famous case of federated learning is Gboard, which is widely used for next word prediction.

[0003] However, traditional synchronous federated learning still faces many challenges, and the most urgent problem to be solved at present is the client heterogeneity problem, including system heterogeneity and data (statistical) heterogeneity. On the one hand, the system heterogeneity problem is caused by the performance difference between each client, such as local computing power, transmission power and battery capacity, etc. This greatly affects the local training and model uploading time of the client. However, in each round of training of federated learning, the server always needs to wait for all participating clients (including the slowest client) to upload the local model, which directly leads to slower training process and increased energy consumption, which is usually referred to as "straggler effect". On the other hand, the data heterogeneity problem is caused by the difference in data distribution of each client, such as dental hospitals and tumor hospitals in the same region, which have similar clients but completely different data volume and data type. This data heterogeneity leads to the problem of non-independent and identically distributed, which makes each user update the global shared model towards its own direction, reduces the performance and convergence speed of the global model, and seriously may cause the model to be difficult to converge.

[0004] In solving data heterogeneity, selecting representative clients to participate in training is an effective solution. The invention patent with publication number CN114819069A discloses a client selection federated learning method based on DBSCAN clustering, which has the following shortcomings: clustering operation needs to be performed on the model parameters uploaded by the client in each round of global model training, which increases the calculation of federated learning, and the application scenario is limited to scenarios with few global training rounds. The invention patent with publication number CN118965046A discloses a federated learning method and system based on client clustering selection under Non-IID data, which clusters individual clients according to loss values in each training round to achieve convergence of the global model under Non-IID data, but still has the following shortcomings: although the method of clustering clients according to loss values can reduce the calculation amount, it is still not suitable for scenarios with many communication rounds between the server and the client.

[0005] The invention patent with publication number CN119135689B discloses a federated learning model training method, which constructs a semi-asynchronous federated learning aggregation scheme with a fixed time period to ensure that data in all local clients can participate in model training; but still has the following shortcomings: during training, the client still needs to complete local training before the global model can be aggregated, resulting in slow clients having a small weight and participating in fewer rounds in the global model aggregation, still existing "biased selection" phenomenon. SUMMARY

[0006] To solve the technical problems of large calculation amount and inability to effectively overcome biased selection in existing methods when solving data heterogeneity, the present application proposes a semi-asynchronous federated learning training method based on intelligent client selection and adaptive weighting. Under the semi-asynchronous federated learning framework, the local model is trained asynchronously, and the server is aggregated synchronously to efficiently schedule each client. The clustering algorithm for the local data distribution of the client is only performed once after the start of federated learning, which greatly reduces the calculation amount under the premise of solving data heterogeneity, allows the client to upload the model without completing local training, enables each client to participate in global training fairly, prevents the "biased selection" phenomenon, and improves the convergence speed and accuracy of the model through adaptive weighting based on the data size and model obsolescence of each selected client, solving the system heterogeneity problem of the user.

[0007] To achieve the above purpose, the technical solution of the present application is as follows:

[0008] A semi-asynchronous federated learning training method based on intelligent client selection and adaptive weighting, the steps are as follows:

[0009] S1: The client initializes the model according to the center server broadcast, the center server calculates the local data distribution according to the training set, and clusters the clients based on the local data distribution of the clients according to the clustering algorithm;

[0010] S2: The center server selects a random client from each cluster in the cluster at a fixed period and broadcasts the latest global model weight;

[0011] S3: All selected clients immediately upload the local model weight and version number, and the unselected clients continue local training;

[0012] S4: The center server synchronously updates the global model weight according to the uploaded local model weight and local model version for each selected client;

[0013] S5: Repeat steps S2 to S4 to achieve global training, the center server tests the global model based on the updated global model weight using the test set until the global model reaches the target model accuracy, and ends the training.

[0014] Further, the implementation method of step S1 is:

[0015] S1.1 When the center server receives the federated learning task creation, broadcast the global initial model weight ω 0 , the global initial learning rate η and the local maximum training round L to each client;

[0016] S1.2 The local client updates the local model according to the received global initial model weight ω 0 and the global initial learning rate η to obtain the local model initial weight and the local model learning rate η i = η;

[0017] S1.3 The center server calculates the local data distribution P i (y) according to the local training set D i ;

[0018] S1.4 The center server clusters the clients based on the client local data distribution P i (y) according to the DBSCAN clustering algorithm.

[0019] Further, the implementation method of step S3 is:

[0020] S3.1 In the rth round of global training, any selected and locally trained client immediately uploads the locally trained model weight and local model version, and any selected and locally untrained client immediately uploads the local current model weight and model version;

[0021] S3.2 In the rth round of global training, the selected and locally trained client and the selected and locally untrained client update the local model version respectively and respectively send the global model weight ω r to start a new local model training as the local model weight;

[0022] S3.3 Any unselected and trained client enters an idle state waiting to be selected by the center server, and any unselected and locally untrained client continues the unfinished local training.

[0023] Further, the implementation method of step S4 is:

[0024] S4.1 Calculate the obsolescence degree of each client based on the model version number of the selected client;

[0025] S4.2 Calculate the adaptive weight factor c of each client based on the obsolescence degree of the client and the size of the data set i ;

[0026] S4.3 The center server performs weighted aggregation on all selected clients to obtain a new global model weight.

[0027] Further, the method for calculating the local data distribution P i (y) in step S1.3 is: count the sum of the sample numbers of all categories of the i-th client |D i | = |D i,1 | + |D i,2 | +... + |D i,m |, |D i,m | is the number of training samples of category m of the i-th client, and for each category m of the i-th client, calculate the proportion of the sample number to the total sample number Arrange the proportions of the m categories in order to form a local data distribution:

[0028]

[0029] The method for clustering clients according to the DBSCAN clustering algorithm described in step S1.4 is only performed before the global training starts.

[0030] Further, the method for updating the local model version is:

[0031]

[0032] Where S is the selected client set, is the model version number of the i-th client at the rth round of global training;

[0033] The method for performing new local model training is: updating the global model weight ω r The initial weight of the new local model training process Training is performed using an SGD optimizer:

[0034]

[0035] Wherein, The local model weight of the client i with the version number After the lth round of local training,

[0036] Indicates the SGD optimizer.

[0037] Further, the method for calculating the degree of staleness is:

[0038]

[0039] Wherein, staleness i Indicates the degree of staleness of the client i.

[0040] Further, the method for calculating the adaptive weight factor c i Is as follows:

[0041]

[0042] Wherein, lambda is an adjustable parameter for adjusting the size of the influence of the degree of staleness on the weight factor, and j represents any client selected from the selected clients.

[0043] Further, the value of the adjustable parameter lambda can be divided into three cases:

[0044] When lambda is in (0, 0.3], the central server has high tolerance to outdated clients and retains more information.

[0045] When lambda is in (0.3, 0.7], the model effect is moderate.

[0046] When lambda is in (0.7, 1], the central server has low tolerance to outdated clients and retains less information.

[0047] Further, the method for the central server to perform weighted aggregation on all selected clients to obtain new global model weight is:

[0048]

[0049] Wherein, alpha is an adjustable parameter, K is the total number of clusters.

[0050] The beneficial effects of the present application are:

[0051] The application can effectively solve the problem of data and system heterogeneity between clients in actual implementation, and improve the convergence speed and accuracy of model training. Specifically, in the implementation process, the center server first classifies each client by clustering algorithm based on local data distribution, and the clients with similar data distribution are regarded as a class. The center server periodically solves the problem of non-independent and identically distributed data caused by data heterogeneity from each cluster randomly selected representative client, and improves the convergence speed of the model. The selected client immediately uploads the local progress and performs new training, and the unselected client can continue to complete the local training. This local asynchronous training method improves the training efficiency of each client and solves the straggler effect caused by system heterogeneity. Subsequently, the center server synchronously receives the model parameters of the selected client, and performs weighting based on data volume and staleness. The application efficiently schedules all local clients in federated learning to participate in training by the way of "selecting representative client, local asynchronous training, and server synchronization aggregation", and improves the efficiency and convergence speed of model training. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 The training flowchart of the present application based on intelligent client selection and adaptive weighting of semi-asynchronous federated learning.

[0054] Figure 2 The training framework diagram of the present application based on intelligent client selection and adaptive weighting of semi-asynchronous federated learning.

[0055] Figure 3 The trend graph of loss function over time for comparison of the present application with other methods. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] A semi-asynchronous federated learning training method based on intelligent client selection and adaptive weighting, as shown in Figure 1 Based on the semi-asynchronous federated learning system, the semi-asynchronous federated learning system includes one central server and multiple local clients participating in training, as shown in Figure 2 The steps are as follows:

[0058] S1: The client initializes the model according to the center server broadcast, the center server calculates the local data distribution according to the training set, and clusters the clients based on the client local data distribution according to the clustering algorithm.

[0059] As an example, Figure 2 The training framework diagram of the method in the application process in the embodiment is given. In this embodiment, the CIFAR-10 dataset (the full name of CIFAR-10 is: Canadian Institute For Advanced Research-10 classes, and the CIFAR-10 dataset is a widely used color image dataset in the field of computer vision) is used. The CIFAR-10 dataset is composed of 60000 32x32 RGB color images, which are divided into 10 categories (including airplane, car, bird, cat, deer, dog, frog, horse, ship and truck), wherein the training set contains 50000 images and the test set contains 10000 images. Each category in the dataset has the same number of images, i.e. 5000 images per category in the training set and 1000 images per category in the test set.

[0060] In this experiment, each client trains a global model (ResNet-20) under the coordination of the server, and the optimizer of each client uses stochastic gradient descent (SGD) to verify the classification performance of the model on CIFAR-10.

[0061] Data heterogeneity: Based on Dirichlet distribution, the training dataset is divided into 100 clients, so as to construct the non-independent and identically distributed data between the clients. Specifically, assuming that there are N clients, the data distribution p N is determined by Dirichlet distribution p N ~ Dir(β), wherein the control parameter β ∈ (0, +∞). When β→0, the data heterogeneity is the strongest, and there is a big difference between the data amount and the category of each client. When β→+∞, there is no data heterogeneity, and the categories owned by each client and each category are the same. In this embodiment, β=0.5 is set, and under this setting, the data distribution between the clients is more consistent with the real situation.

[0062] System heterogeneity: To simulate the different local computing capabilities among clients, the time required for each client to perform a single local step is modeled as an exponential distribution random variable T ~ Exp(a), with probability density function f(t; a) = ae -at (t ≥ 0), the expected value of the exponential distribution to reflect the difference in computing speed of different clients. In implementation, 75 fast clients and 25 slow clients are set, and the values of a for fast and slow clients are and respectively. Then the average local training time of fast and slow clients is 2 and 8 respectively.

[0063] First, when the center server receives the federal learning task creation, it broadcasts the global initial model weight ω 0 , the global initial learning rate η and the local maximum training round L to each client;

[0064] Further, the client initializes the model according to the broadcast of the center server.

[0065] Model initialization: the local client updates the local model according to the received global initial model weight ω 0 and the global initial learning rate η to obtain the local model initial weight ω and the local model learning rate η i = η.

[0066] Further, the center server calculates the local data distribution P i (y):

[0067] Specifically, the center server counts the number of clients participating in training, obtains the local training set D i of each client, which is used to represent the original training data of each client:

[0068] D i = {x ij , y ij};

[0069] Where x ij is the jth training sample of the ith client, y ij is the sample label corresponding to the jth training sample, i ∈ [1, N], N is the total number of clients, j ∈ [1, |D i |], where |D i | is the size of the local data of the ith client.

[0070] Specifically, the local data distribution P i is calculated according to the local data set D i(y), representing the proportion of samples of each category:

[0071]

[0072] where |D i,m is the number of training samples of the m-th category of the i-th client, and this formula converts the original data into a distribution vector to measure data heterogeneity, for example, some client data is biased towards a particular category.

[0073] Further, the center server clusters the clients based on the local data distribution P i (y) according to the clustering algorithm: the center server takes the local data distribution P i (y) as the basis, before the start of global training, uses the DBSCAN-based clustering algorithm to put clients with similar data distribution into a cluster, ensuring that the data distribution of clients in the same cluster is similar, and the total number of clusters is K.

[0074]

[0075] where C a is the cluster where any client is located, and a ∈ [1, K].

[0076] In federated learning, client data is often not independent and identically distributed (such as some clients having only specific category data), resulting in bias in the global model. Clustering groups similar clients, ensuring that representatives are selected from each cluster in subsequent rounds, covering diverse data distributions. Clustering is only run once at initialization, avoiding repeated calculations and reducing computational overhead.

[0077] S2: The center server randomly selects a client from each cluster in the cluster of each cluster at a fixed period and broadcasts the latest global model weight.

[0078] The center server randomly selects a client from each cluster in the cluster of each cluster at a fixed period T (this embodiment includes a server waiting time of 5 seconds and a server processing information time of 1 second. Total 6 seconds):

[0079]

[0080] where c b is the set of selected clients participating in the next round of global training, and b ∈ [1, K].

[0081] The center server broadcasts the latest global model weight ω r to all selected clients c b , r ∈ [1, R], and R is the total number of global training rounds.

[0082] By randomly selecting clients, the model avoids bias towards specific clusters, ensuring that the global model is not biased towards any particular cluster.

[0083] S3: All selected clients immediately upload their local model weights and version numbers, while unselected clients continue local training.

[0084] Specifically, in the r-th round of global training, any selected client that has completed local training immediately uploads the completed local training data and the local model version, and any selected client that has not completed local training immediately uploads the model weights and model version obtained in the current local training round.

[0085] Furthermore, in the r-th round of global training, the selected clients that have completed local training and the selected clients that have not completed local training update their local model versions and respectively adjust the global model weights ω. r As the weights of the local model, a new local model is trained. In the r-th round of global training, any client asynchronously trains the local model.

[0086] The method to update the local model version is as follows:

[0087]

[0088] in, Let i be the version number of client i during the r-th round of global training.

[0089] The global model weights ω r As local model weights:

[0090]

[0091] in, The initial weights for the new training process of the local model.

[0092] The process of training a local model can be represented as follows:

[0093]

[0094] in, The version number is The local model weights of client i after the lth round of local training, η i Let i be the learning rate of client i. Let be the stochastic gradient descent (SGD) optimizer for client i.

[0095] Furthermore, any client that has not been selected but has completed training enters an idle state and waits to be selected by the central server. Any client that has not been selected and has not completed local training (the number of local training rounds has not reached the maximum number of local training rounds) continues to perform incomplete local training.

[0096] S4: The central server synchronously and adaptively aggregates and updates the global model weight for each selected client according to the uploaded local model weight and the local model version.

[0097] The method for synchronously and adaptively aggregating and updating the global model weight for each selected client is:

[0098] First, the obsolescence degree of each client is calculated based on the model version number of the selected client:

[0099]

[0100] Further, the adaptive weight factor c of each client is calculated based on the obsolescence degree and the data set size of the client i , the formula is:

[0101]

[0102] where λ is an adjustable parameter, λ∈(0,1], and the adjustable parameter λ can adjust the size of the influence of the obsolescence degree on the weight. The value of the adjustable parameter λ can be divided into three cases:

[0103] When λ∈(0,0.3], the central server has high tolerance for obsolete clients and retains more information.

[0104] When λ∈(0.3,0.7], the model effect is moderate.

[0105] When λ∈(0.7,1], the central server has low tolerance for obsolete clients and retains less information.

[0106] Further, the central server aggregates and obtains a new global model weight for all selected clients, and the formula is:

[0107]

[0108] where α is an adjustable parameter, K is the total number of clusters. The adjustable parameter α can adjust how much of the rth round global model weight is retained.

[0109] S5: Repeat steps S2 to S4 to realize global training, and the central server tests the global model based on the updated global model weight using the test set until the global model reaches the target model accuracy (accuracy higher than 90%), and ends the training.

[0110] For example, Figure 3As shown, the algorithm of the present application compares the classic synchronous federated learning framework FedAvg and the asynchronous federated learning framework FedBUFF, wherein the center server in FedAvg first randomly selects part of the clients and issues the global model, and then waits for all the selected clients to train and upload the end local model weight before model aggregation. The server in the FedBUFF framework does not need to wait for the client to upload, but sets a receiving buffer to receive the trained model of the client; the client trains locally asynchronously, and uploads the local model weight immediately after training and receives the latest global model weight. In this implementation, in order to simulate the characteristics of data and system heterogeneity in reality, 100 clients and 1 center server are set, of which 75 are fast clients and 25 are slow clients, and the local data of these clients p N ~ Dir(0.5).

[0111] The experimental results show that: although the FedBUFF framework is slightly better than the FedAvg framework in the final accuracy (corresponding to lower model loss), its training process presents significant fluctuations and poor convergence stability. In contrast, although the FedAvg framework can maintain a relatively stable training curve, the peak accuracy is the lowest among the three algorithms. It is worth noting that the algorithm proposed in the present application is significantly better than the comparative methods in terms of convergence speed and final accuracy, and exhibits excellent adaptability to data and system heterogeneity.

[0112] It should be understood that the above are only preferred embodiments of the present application and the application of technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the scope of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the specific embodiments described herein, and can include more other effective embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A semi-asynchronous federated learning training method based on intelligent client selection and adaptive weighting, characterized in that, The steps are: S1: the client initializes the model according to the center server broadcast, the center server calculates the local data distribution according to the training set, and clusters the clients based on the local data distribution of the clients according to the clustering algorithm; S2: the center server selects a random client from each cluster in the cluster at a fixed period and broadcasts the latest global model weight; S3: all selected clients immediately upload the local model weight and version number, and the unselected clients continue local training; S4: the center server synchronously updates the global model weight according to the uploaded local model weight and local model version for each selected client; S5: repeat steps S2 to S4 to realize global training, and the center server tests the global model based on the updated global model weight using the test set until the global model reaches the target model accuracy, and the training ends.

2. The method of claim 1, wherein the method is based on a smart client selection and adaptive weighting semi-asynchronous federated learning training method. The implementation method of step S1 is: S1.1 The central server broadcasts the global initial model weights ω, the global initial learning rate η, and the local maximum training round L to each client when receiving the federated learning task creation 0 ; S1.2 The local client updates the local model according to the received global initial model weight ω 0 and the global initial learning rate η to obtain the local model initial weight ω and the local model learning rate η i = η; S1.3 The central server computes the local training set D i Compute the local data distribution P i (y); S1.4 The central server distributes P based on the client local data i (y) clustering the clients according to the DBSCAN clustering algorithm.

3. The method of claim 2, wherein, The implementation method of step S3 is: S3.1 In the rth round of global training, any selected and locally trained client immediately uploads the locally trained model weight and local model version, and any selected and locally untrained client immediately uploads the local current model weight and model version; S3.2 In the rth round of global training, the selected and locally trained client and the selected and locally untrained client update the local model version respectively and respectively send the global model weight ω r start a new local model training as the local model weight S3.3 any unselected and trained client enters an idle state and waits to be selected by the center server, and any unselected and locally untrained client continues the unfinished local training.

4. The semi-asynchronous federated learning training method based on smart client selection and adaptive weighting according to any one of claims 1 to 3, characterized in that, The implementation method of step S4 is: S4.1 calculate the obsolescence degree of each client based on the model version number of the selected client; S4.2 An adaptive weight factor c is calculated for each client based on the client's staleness degree and the dataset size i ; S4.3 the center server aggregates all selected clients to obtain a new global model weight.

5. The semi-asynchronous federated learning training method based on intelligent client selection and adaptive weighting according to claim 2 or 3, characterized in that, Step S1.3 calculates the local data distribution P i The method of (y) is: count the sum of the sample numbers of all categories of the i-th client |D i | = |D i,1 | + |D i,2 | +... + |D i,m |, |D i,m | is the number of training samples of category m of the i-th client, and for each category m of the i-th client, the proportion of the sample number to the total sample number is calculated Arrange the proportions of the m categories in order to form a local data distribution: The method of clustering clients according to the DBSCAN clustering algorithm in step S1.4 is only performed before the start of global training.

6. The method of claim 3, wherein the method is based on a smart client selection and adaptive weighting based semi-asynchronous federated learning training method. The method of updating the local model version is: Wherein, S is the selected client set, is the model version number of the client i in the rth round of global training. The method for performing new local model training is to update the global model weight ω r initial weight of the local model new training process training with the SGD optimizer: wherein, represents the local model weight of the client i with version number the local model after the l-th round of local training, represents the SGD optimizer.

7. The method of claim 6, wherein, The method of calculating the obsolescence degree is: where staleness i represents the staleness of client i.

8. The method of claim 7, wherein, The adaptive weight factor c i The calculation method is: Where λ is an adjustable parameter for adjusting the size of the influence of the obsolescence degree on the weight factor, and j represents any client in the selected clients.

9. The method of claim 8, wherein, The value of the adjustable parameter λ can be divided into three cases: When λ∈(0,0.3], the center server has high tolerance to obsolete clients and retains more information. When λ∈(0.3,0.7], the model effect is moderate. When λ∈(0.7,1], the center server has low tolerance to obsolete clients and retains less information.

10. The method of claim 9, wherein, The method of the center server aggregating all selected clients to obtain a new global model weight is: wherein a is an adjustable parameter, K is the total number of clusters of the clustering.

Images