Federal learning method and system based on client clustering selection under non-iid data

Abstract

This invention discloses a federated learning method based on client clustering selection for Non-IID data. The method includes the following steps: S1. Constructing a federated learning system model for Non-IID data; S2. Clients download the global model, calculate the loss value and accuracy, and send the calculated loss value and accuracy to a central server; S3. The central server clusters the clients based on the loss value and selects clients to participate in federated learning training within each cluster based on the accuracy; S4. The selected clients update their local model parameters and send them to the central server. The central server performs global model aggregation, iterates, and determines whether the target accuracy or the maximum number of federated learning training rounds has been reached. If not, it returns to step S2; if yes, it stops the federated learning training rounds. This invention can relax the requirements for network communication bandwidth and achieve convergence of the global model for Non-IID data.

Description

Technical Field

[0001] This invention belongs to the field of federated learning and client selection technology, and particularly relates to a federated learning method and system based on client clustering selection for Non-IID (non-independent and identically distributed) data. Background Technology

[0002] In recent years, with the development of big data, the processing power, GPU memory, and memory bandwidth of a single machine are often insufficient to meet the needs of massive models, prompting the rise of distributed machine learning. However, the resulting data privacy leakage issues have also become increasingly prominent. To address this problem, federated learning has emerged. In traditional federated learning, the client performs model training locally and sends the updated local model parameters to a central server. The central server then aggregates the models based on the received parameter updates and updates the global model. This has provided opportunities for fields such as healthcare, finance, and the Internet of Things. However, the non-identical distribution (Non-IID) nature of data in practical applications presents numerous challenges to federated learning.

[0003] In Non-IID data, clients have different data distributions, and locally trained models may differ significantly. When the central server aggregates these scattered models, the global model convergence speed is slow, and the training accuracy is low. Furthermore, network bandwidth is limited, making it impractical for the central server to perform parallel model updates and aggregation on all clients. Therefore, this invention proposes a federated learning method and system based on client clustering selection for Non-IID data. By selecting suitable clients to participate in global model training, it achieves global model convergence under low bandwidth requirements. Summary of the Invention

[0004] The purpose of this invention is to address the aforementioned problems of federated learning under Non-IID data by providing a federated learning method and system based on client-side clustering selection for Non-IID data. This method improves training accuracy and reduces communication overhead through client-side clustering selection technology.

[0005] In order to achieve the objectives of this invention, the following technical solution is adopted:

[0006] A federated learning method based on client-side clustering selection for Non-IID data includes the following steps:

[0007] S1. Construct a federated learning system model for Non-IID data;

[0008] S2. The client downloads the global model, calculates the loss value and accuracy, and sends the calculated loss value and accuracy to the central server;

[0009] S3. The central server clusters the clients based on the loss value and selects clients to participate in federated learning training in each cluster based on the accuracy.

[0010] S4. The selected client updates its local model parameters and sends them to the central server. The central server completes global model aggregation and determines whether the target accuracy or the maximum number of federated learning training rounds has been reached. That is, if it has not been reached, it returns to step S2; if it has been reached, it stops federated learning training.

[0011] Furthermore, in step S1, a federated learning system model for Non-IID data is constructed. This model includes a central server and multiple clients, where M = {1, 2, ..., n} represents the set of all clients, S represents the set of selected clients, and the constant m represents the number of clients selected by the central server in each round, ω... t This represents the global model parameters in the t-th round of federated learning training. For client i, the local dataset distributed by the central server is defined as D. i The number of times a model is selected to participate in global model training is defined as C. i And define the loss value of client i in the t-th round of federated learning training as Model parameters updated to

[0012] Furthermore, step S2 specifically includes:

[0013] S21. The client downloads the global model from the central server and uses cross-entropy to calculate the loss value between the global model and the local dataset in round t. After being amplified, the data is sent to the central server. The amplification method is specifically represented as follows:

[0014]

[0015] Where i represents the client, and i∈M. This represents the cross-entropy loss value of client i in the t-th round of federated learning training, where λ is a constant representing the scaling factor. After scaling, They are sent to the central server to participate in clustering tasks;

[0016] S22. The client calculates the accuracy (acc) of the global small test set on the local model. i And send it to the central server. The specific calculation method is as follows:

[0017]

[0018] Where, n correct The number of samples correctly classified using the local model on the test set, n totalThis represents the total number of samples in the test set.

[0019] Furthermore, in step S3, the central server calculates the loss value of each client i after the expansion in round t. K-Center-Greedy clustering strategy is adopted. After clustering, the clients with the highest accuracy are selected from the m clusters to form a set S. The clients in set S participate in the global model update.

[0020] Furthermore, step S4 is detailed as follows:

[0021] S41. Clients i∈S calculate local model parameter updates and send them to the central server. Set S represents all selected clients. The calculation of local model parameter updates is as follows:

[0022]

[0023] in, Let ω be the local model parameter of client i in round t. t-1 These are the global model parameters at round t-1;

[0024] S42. The central server determines the number of times client i is selected, C. i and local dataset size |D i The score is calculated as follows:

[0025]

[0026] Among them, |D i | represents the size of the local dataset of client i, C i This represents the number of times client i is selected, and β is a floating-point number with a value between [0,1].

[0027] S43. The central server calculates the weights of client i in the global model training based on the score, and completes the global model aggregation, as follows:

[0028]

[0029] Among them, Score i The score obtained by client i through step S42;

[0030] S44. The central server determines the test set in the global model ω. t If the prediction accuracy reaches the target accuracy or if t reaches the maximum number of federated learning training rounds, then return to step S2; otherwise, stop federated learning training.

[0031] This invention also provides a federated learning system based on client-side clustering selection for Non-IID data, which, based on the above method, includes the following modules:

[0032] The model building module is used to build federated learning system models for Non-IID data;

[0033] The calculation module is used by the client to download the global model, calculate the loss value and accuracy, and send the calculated loss value and accuracy to the central server.

[0034] The selection module is used by the central server to cluster clients based on the loss value and select clients to participate in federated learning training in each cluster based on the accuracy.

[0035] The aggregation and iteration module is used to update the local model parameters of the selected client and send them to the central server. The central server completes global model aggregation and iterates continuously until the target accuracy or the maximum number of iterations is reached.

[0036] Compared with existing client selection techniques, this invention uses client loss value clustering. In each round of federated learning, the client sends its loss value and accuracy to the central server. The central server clusters the data based on the loss values ​​and selects the client with the highest accuracy in each cluster. This achieves convergence of the global model under Non-IID data and also reduces communication overhead. Attached Figure Description

[0037] Figure 1 This is a flowchart of a preferred embodiment of the federated learning method based on client-side clustering selection for Non-IID data according to the present invention.

[0038] Figure 2 This is a comparison chart of the accuracy of different client selection strategies when the Non-IID level is 0.5.

[0039] Figure 3 This is a comparison chart of the accuracy of different client selection strategies when the Non-IID level is 0.8.

[0040] Figure 4 This is a block diagram of a federated learning system based on client-side clustering selection for Non-IID data, according to a preferred embodiment of the present invention. Detailed Implementation

[0041] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0042] This embodiment presents a federated learning method based on client-side clustering selection for Non-IID data, such as... Figure 1 As shown, the specific steps include:

[0043] S1. Construct a federated learning system model for Non-IID data;

[0044] S2. The client downloads the global model and sends the calculated loss value and accuracy to the central server;

[0045] S3. The central server clusters the clients based on the loss value and selects clients to participate in federated learning training in each cluster based on the accuracy.

[0046] S4. The selected client updates its local model parameters and sends them to the central server. The central server completes global model aggregation and determines whether the target accuracy or the maximum number of federated learning rounds has been reached. If not, it returns to step S2; if so, it stops federated learning training.

[0047] This embodiment presents a federated learning method based on client clustering selection, aiming to address the challenges of federated learning with Non-IID data by selecting clients. The strategy is validated through a comparison of time, performance, and communication overhead between a client-based loss value clustering and accuracy selection scheme and other client selection strategies.

[0048] Specifically, in step S1, a federated learning system model for Non-IID data is constructed. This model includes a central server and multiple clients, where M = {1, 2, ..., n} represents the set of all clients, S represents the set of selected clients, and the constant m represents the number of clients selected by the central server in each round, ω... t Let represent the global model parameters in the t-th round of federated learning training, and define the Non-IID degree σ. The closer σ is to 1, the greater the Non-IID degree; the closer σ is to 0, the smaller the Non-IID degree. For client i, the local dataset distributed by the central server is defined as D. i The number of times a model is selected to participate in global model training is defined as C. i And define the loss value of client i in the t-th round of federated learning training as Model parameters updated to In this embodiment, a central server and 20 clients were included, and a comparative experiment was conducted for σ=0.5 and σ=0.8.

[0049] Step S2 in this embodiment specifically includes:

[0050] S21. The client downloads the global model from the central server and uses cross-entropy to calculate the loss value between the global model and the local dataset in round t. After being amplified, the data is sent to the central server. The amplification method is specifically represented as follows:

[0051]

[0052] Where i represents the client, and i∈M. This represents the cross-entropy loss value of client i in the t-th round of federated learning training, where λ is a constant representing the scaling factor. After scaling, It is sent to the central server to participate in the clustering task. In this embodiment, λ = 64 is set.

[0053] S22. The client calculates the accuracy (acc) of the global small test set on the local model. i And send it to the central server. The specific calculation method is as follows:

[0054]

[0055] Where, n correct The number of samples correctly classified using the local model on the test set, n total This represents the total number of samples in the test set.

[0056] In step S3, the central server calculates the loss value of each client i after the t-th round of federated learning training. K-Center-Greedy clustering is employed. After clustering, the clients with the highest accuracy from the m clusters are selected to form a set S. Clients in set S participate in the global model update. In this embodiment, m = 4, meaning that in each round of federated learning, the 20 clients are divided into 4 clusters based on their loss values, and one client is selected from each cluster to participate in the global model training.

[0057] In step S4, the selected client updates and calculates the local model parameters and sends them to the central server. The central server completes global model aggregation, iterating continuously until the target accuracy or the maximum number of iterations is reached. In this embodiment, the maximum number of iterations is set to 40, the target accuracy is set to 60% when σ = 0.5, and the target accuracy is set to 55% when σ = 0.8. The specific steps are as follows:

[0058] S41. Clients i∈S calculate local model parameter updates and send them to the central server. Set S represents all selected clients. The calculation of local model parameter updates is as follows:

[0059]

[0060] in, Let ω be the local model parameter of client i in round t. t-1 These are the global model parameters at round t-1;

[0061] S42. The central server determines the number of times client i is selected, C. i and local dataset size |Di The score is calculated as follows:

[0062]

[0063] Among them, |D i | represents the size of the local dataset of client i, C i This represents the number of times client i is selected, and β is a floating-point number ranging from [0,1]. In this embodiment, β is set to 0.2.

[0064] S43. The central server calculates the weights of client i in the global model training based on the score, and completes the global model aggregation, as follows:

[0065]

[0066] Among them, Score i The score obtained by client i through step S42;

[0067] S44. The central server determines the test set in the global model ω. t If the prediction accuracy reaches the target accuracy or if t reaches the maximum number of federated learning training rounds, then return to step S2; otherwise, stop federated learning training.

[0068] To evaluate the technical effectiveness of the proposed preferred embodiment, comparative experiments are conducted below to verify the accuracy and time consumption of the present invention, and to analyze the communication overhead of the present invention.

[0069] The experimental results are evaluated as follows:

[0070] 1. Accuracy

[0071] Figure 2 , 3 The figure shows the accuracy comparison experiment results between the client clustering selection scheme proposed in this invention and other existing client selection schemes. This embodiment selected two different Non-IID levels, 0.5 and 0.8, for comparative experiments. As can be observed from the figure, the client clustering selection scheme Loss-KCenter proposed in this invention performs similarly to K-Center in terms of accuracy. However, when the Non-IID level is higher, i.e., σ = 0.8, it is clearly shown that the client clustering selection scheme proposed in this invention is more stable.

[0072] 2. Time consumed

[0073] Table 1 compares the time consumption of the client-side loss value clustering selection scheme provided by this invention with other client-side selection schemes. Compared with other client-side selection schemes, the client-side loss value clustering selection scheme provided by this invention has a moderate time consumption, which is better than Pow-D based on direct loss value selection and Favor based on reinforcement learning.

[0074] Table 1 Comparison of Time Consumption for Each Client's Choice of Solution

[0075]

[0076] 3. Communication overhead

[0077] The Random and Favor strategies require minimal communication overhead as they only transmit updated model parameters to the central server. In contrast, the K-Center and K-medoids strategies require all clients to transmit their previously updated local model parameters to the central server, maximizing communication overhead. Furthermore, Pow-D and the Loss-KCenter strategy proposed in this invention require all clients to transmit only a one-dimensional array, much smaller than the model parameters, to the central server. Only selected clients need to transmit updated model parameters; this information is relatively small, resulting in lower communication overhead.

[0078] like Figure 4 As shown, this embodiment discloses a federated learning system based on client-side clustering selection for Non-IID data. Based on the above embodiment of the federated learning method based on client-side clustering selection for Non-IID data, it includes the following modules:

[0079] The model building module is used to build federated learning system models for Non-IID data;

[0080] The calculation module is used by the client to download the global model, calculate the loss value and accuracy, and send the calculated loss value and accuracy to the central server.

[0081] The selection module is used by the central server to cluster clients based on the loss value and select clients to participate in federated learning training in each cluster based on the accuracy.

[0082] The aggregation and iteration module is used to update the local model parameters of the selected client and send them to the central server. The central server performs global model aggregation and iterates continuously until the target accuracy or the maximum number of iterations is reached.

[0083] Other aspects of this embodiment can be found in the above method embodiments.

[0084] In summary, this invention discloses a federated learning method and system based on client clustering selection for Non-IID data. It proposes a client selection strategy based on loss value clustering and accuracy selection, called Loss-KCenter. The central server performs clustering based on the loss values ​​sent by each client and selects clients to participate in federated learning training within each cluster according to their accuracy. The selected clients calculate their local model parameters based on their local datasets and then send these updated parameters to the central server. The central server updates the global model using an aggregation algorithm based on data volume and the number of times a client is selected, until the target accuracy or the maximum number of iterations is reached. This invention relaxes the requirements for network communication bandwidth and achieves convergence of the global model under Non-IID data.

[0085] The above description is merely an illustration of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A federated learning method based on client-side clustering selection for Non-IID data, characterized in that, Includes the following steps: S1. Construct a federated learning system model for Non-IID data; S2. The client downloads the global model, calculates the loss value and accuracy, and sends the calculated loss value and accuracy to the central server; S3. The central server clusters the clients based on the loss value and selects clients to participate in federated learning training in each cluster based on the accuracy. S4. The selected client updates the local model parameters and sends them to the central server. The central server performs global model aggregation and determines whether the target accuracy or the maximum number of federated learning training rounds has been reached. If not, it returns to step S2; if so, it stops federated learning training. Step S2 is as follows: S21. The client downloads the global model from the central server and calculates the first step using cross-entropy. Loss values ​​of the global model and the local dataset , After being amplified, the data is sent to the central server. The amplification method is specifically represented as follows: in, Indicates the client, and , Represents the set of all clients. Indicates client In the Cross-entropy loss value for each training round of federated learning. is a constant, representing the scaling factor. After scaling, They are sent to the central server to participate in clustering tasks; S22. The client calculates the accuracy of the global small test set on the local model. And send it to the central server. The specific calculation method is as follows: in, The number of samples correctly classified using the local model on the test set. This represents the total number of samples in the test set; Step S4 is as follows: S41. Client Calculate local model parameter updates and send them to the central server, collection For all selected clients, the calculation of local model parameter updates is as follows: in, For the client In the Local model parameters during rounds, For the first Global model parameters during rounds; S42. The central server, based on the client... Number of times selected and local dataset size The score is calculated as follows: in, Indicates client The size of the local dataset. Indicates client The number of times it was selected. It is a floating-point number whose value ranges from [0, 1]. S43. The central server calculates client scores based on the client's performance. The weights participating in global model training are determined, and global model aggregation is completed as follows: in, For the client The score is calculated through step S42; S44. The central server determines whether the test set is in the global model. Has the prediction accuracy reached the target accuracy? Has the maximum number of federated learning training rounds been reached? If not, return to step S2; if yes, stop federated learning training.

2. The federated learning method based on client-side clustering selection for Non-IID data as described in claim 1, characterized in that, In step S1, the federated learning system model for Non-IID data includes a central server and multiple clients. Represents the set of all clients. Represents the set of selected clients, a constant. This indicates the number of clients selected by the central server in each round. Indicates the first Global model parameters for each round of federated learning training; for the client In this regard, the local dataset distributed by the central server is defined as The number of times a model is selected to participate in global model training is defined as follows: and define the client In the The loss value for each round of federated learning training is The model parameters are updated to .

3. The federated learning method based on client-side clustering selection for Non-IID data as described in claim 1, characterized in that, Step S3 is as follows: The central server, based on each client... In the Loss value after wheel enlargement Clustering was performed using the K-Center-Greedy strategy. After clustering, from Select the clients with the highest accuracy from each cluster to form a set. ,gather Clients in the system participate in the global model update.

4. A federated learning system for Non-IID data based on client-side clustering selection, based on the federated learning method for Non-IID data based on client-side clustering selection as described in any one of claims 1-3, characterized in that, Includes the following modules: The model building module is used to build federated learning system models for Non-IID data; The calculation module is used by the client to download the global model, calculate the loss value and accuracy, and send the calculated loss value and accuracy to the central server. The selection module is used by the central server to cluster clients based on the loss value and select clients to participate in federated learning training in each cluster based on the accuracy. The aggregation and iteration module is used to update the local model parameters of the selected client and send them to the central server. The central server completes global model aggregation and iterates continuously until the target accuracy or the maximum number of iterations is reached.

Images