Efficient Federated Learning Methods

Abstract

This invention discloses a communication-efficient federated learning method, comprising: constructing a proxy objective function and using the proxy objective function to replace the global objective function; using the Alternating Direction Multiplier Method (ADMM) to solve the parameter update formula in the proxy objective function; and designing a communication-efficient federated learning algorithm, including designing a central server algorithm and a client algorithm. This invention can reduce the computational load on the client in federated learning and reduce the communication load between the client and the central server, thereby reducing the algorithm convergence time and improving the efficiency of federated learning.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, and in particular to a federated learning method with high communication efficiency. Background Technology

[0002] Traditional federated learning frameworks typically involve a large number of clients participating in federated training. These clients need to communicate frequently with the central server, each communication transmitting large amounts of model parameters or gradient information. This not only leads to high communication costs but also makes algorithm convergence time lengthy, thus limiting the efficiency of federated learning. Since each round of communication requires the transmission of large amounts of data, this places enormous demands on network bandwidth and puts considerable pressure on communication latency and server load. In this situation, the efficiency of federated learning is significantly affected, and especially in resource-constrained environments such as mobile devices, communication costs become a major bottleneck restricting its application. Summary of the Invention

[0003] To address the problems existing in the prior art, the purpose of this invention is to provide a communication-efficient federated learning method. This invention can reduce the computational load on the client side in federated learning and reduce the communication load between the client and the central server, thereby reducing the algorithm convergence time and improving the efficiency of federated learning.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a communication-efficient federated learning method, comprising the following steps:

[0005] Step 1: Construct a proxy objective function and use the proxy objective function to replace the global objective function;

[0006] Step 2: Use the Alternating Direction Multiplier Method (ADMM) to solve for the parameter update formula in the surrogate objective function;

[0007] Step 3: Design a federated learning algorithm with high communication efficiency, including designing the central server algorithm and the client algorithm.

[0008] As a further improvement to the present invention, step 1 is specifically as follows:

[0009] Suppose a federated learning framework has a central server S and N clients {C1,,...,C...} N Client C i The local data possessed is D i Then the dataset set of all clients participating in federated learning is D = {D1, D2, ..., D...} N Assume F D(θ) represents the objective function based on the local data volume of all clients, also known as the global objective function; the global objective function is written as the sum of the local objective functions of each client, i.e.:

[0010]

[0011] in, This indicates the weight of each client, and F i (θ) represents the local objective function of the i-th client, as follows:

[0012]

[0013] Where, l(x) j ,y j ;θ) is the input x of the model with parameter θ for each instance j in the dataset. j and output y j The resulting loss function;

[0014] Next, we construct a proxy target function. To replace the global objective function F D (θ), for F D Taylor expansion of (θ) yields an infinite series:

[0015]

[0016] in, Let θ be any initial estimate. It is a computable constant, and <·,·> denote the inner product.

[0017] As a further improvement of the present invention, the global objective function F D When solving for (θ), a small sample dataset is first randomly selected from the local datasets of each client to construct a sample objective function. The higher-order derivatives of the sample objective function are then used to replace the higher-order derivatives of the global objective function. The specific steps are as follows:

[0018] Step 1.1: Set a sampling ratio α, and each client randomly samples a dataset of size d from its local dataset. i =αD i A subset of the samples constitutes the sampling sample set d = {d1, d2, ..., dn}. N Assume d is much smaller than D, that is Furthermore, as D→∞, d→∞, and the objective function based on the sampled data is expressed as:

[0019]

[0020] in, This indicates the weight of each client, and This indicates that the i-th client is based on the sampled sample d. i The local objective function, for the sample objective function F d Taylor expansion of (θ) yields an infinite series:

[0021]

[0022] in, Let θ be any initial estimate. It is a computable constant, and <·,·> denote the inner product;

[0023] Step 1.2: Replace the higher-order derivative of the global objective function with the higher-order derivative of the sample objective function, that is, let:

[0024]

[0025] Substituting equation (6) into equation (3), we obtain the global objective function F. D Approximation of (θ):

[0026]

[0027] Ignoring the constant term in equation (7) and simplifying, we obtain the proxy objective function:

[0028]

[0029] Equation (8) is called the objective function of the agent for efficient communication. It can be rewritten as the sum of the objective functions of the local samples of each client:

[0030]

[0031] The goal of federated learning is to obtain the optimal parameters θ' at the central server, which is represented by minimizing the agent's objective function, i.e.:

[0032]

[0033] As a further improvement of the present invention, step 2 is specifically as follows:

[0034] Decompose the minimization of the proxy objective function into minimization of the local objective functions of each client, let Equation (10) can be rewritten in the following form:

[0035]

[0036] in, This indicates the weight of each client, and This indicates that the i-th client is based on the sampled sample d. i The local objective function, Equation (11), is further expressed as the following globally consistent optimization problem:

[0037]

[0038] Where, θ i This represents the local model parameters of the i-th client, under constraint θ. i Under the influence of θ,i∈[1,2,···,N], equation (12) is equivalent to equation (11). Equation (12) is then transformed into the form required by the ADMM algorithm:

[0039]

[0040] Where B represents N identity matrices, Represents all θ i The set, i.e. The specific definition is:

[0041]

[0042] The augmented Lagrangian function corresponding to equation (13) is as follows:

[0043]

[0044] Where ρ>0 is the penalty parameter, and the dual variable u∈[u1,u2,···,u N ];

[0045] The update rules for the ADMM algorithm are given below:

[0046]

[0047]

[0048]

[0049] For equation (16), only client C i Relevant information, rewritten in the following form:

[0050]

[0051] right exist Using a second-order Taylor expansion, we can obtain Extensions:

[0052]

[0053] Where η is the coefficient of the penalty term; then The update formula is written as:

[0054]

[0055] Equation (21) is an approximation of equation (19). By solving equation (21), an exact solution to equation (19) can be obtained:

[0056]

[0057] For equation (17), which is a quadratic function of θ, the minimum point can be obtained by taking its partial derivative and setting it to zero.

[0058]

[0059] As a further improvement to the present invention, step 3 is specifically as follows:

[0060] make Before each round of federated training begins, the agent's objective function is first updated. Then update the parameters according to the formula;

[0061] The central server algorithm is as follows:

[0062] The central server first initializes the global model. In any iteration t∈T, the central server randomly selects client C to participate in the federated training. t and to client C t Send global model parameters θ t-1 The clients participating in this round of federated training receive the global model parameters θ. t-1 Then, sample data d are randomly sampled according to the sampling ratio α. i =αD i And calculate Each client then updates its local parameters according to its own algorithm and uploads them to the central server. After receiving the parameters from each client, the central server aggregates them and derives the iterative formula for the global model parameters of the central server based on equation (23):

[0063]

[0064] The client-side algorithm is as follows:

[0065] Clients participating in federated training randomly sample according to a sampling ratio α, and receive θ t-1 Post-calculation The parameter θ is obtained from equation (22) and the learning rate γ. iThe update formula is shown in equation (25). The dual variable u is updated according to equation (26). i ;

[0066]

[0067] u i ←u i +ρ(θ i -θ t-1 (26)

[0068] The client obtains the parameters θ after performing E rounds of local training. i and u i Then θ i u i and R i (θ t-1 Send to the central server.

[0069] The beneficial effects of this invention are:

[0070] This invention addresses the problems of long communication time and high communication cost between clients and central servers in federated learning scenarios. It proposes a highly efficient communication-based federated learning method (FedCSOF algorithm). The core idea of ​​this method is to use a surrogate objective function to replace the global objective function. This surrogate objective function can approximate the global objective function in a communication-efficient manner, reducing the amount of communication between the client and the central server without sacrificing model accuracy, thereby reducing the communication time between the client and the central server. The ADMM algorithm is used to solve the parameter update formula in the surrogate objective function to reduce computational load and further accelerate the convergence speed of the algorithm. Attached Figure Description

[0071] Figure 1 This is a preview image of the MNIST dataset in an embodiment of the present invention;

[0072] Figure 2 This is a preview image of the CIFAR-10 dataset in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram illustrating the accuracy of the model on the MNIST dataset under the IID environment in an embodiment of the present invention.

[0074] Figure 4 This is a schematic diagram illustrating the accuracy of the model on the CIFAR-10 dataset under the IID environment in an embodiment of the present invention.

[0075] Figure 5 This is a schematic diagram illustrating the number of communication rounds required for the MNIST dataset to achieve the target accuracy under the IID environment in this embodiment of the invention.

[0076] Figure 6 This is a schematic diagram illustrating the number of communication rounds required for the CIFAR-10 dataset to achieve the target accuracy under the IID environment in this embodiment of the invention.

[0077] Figure 7 This is a schematic diagram illustrating the accuracy of the model on the MNIST dataset in a Non-IID environment according to an embodiment of the present invention;

[0078] Figure 8 This is a schematic diagram illustrating the accuracy of the model on the CIFAR-10 dataset in a Non-IID environment according to an embodiment of the present invention.

[0079] Figure 9 This is a schematic diagram illustrating the number of communication rounds required for the MNIST dataset to achieve the target accuracy in a Non-IID environment according to an embodiment of the present invention.

[0080] Figure 10 This is a schematic diagram illustrating the number of communication rounds required for the CIFAR-10 dataset to achieve the target accuracy in a Non-IID environment, as described in this embodiment of the invention.

[0081] Figure 11 This is a schematic diagram of the final model's MSE under different sampling ratios in the MNIST dataset in an embodiment of the present invention;

[0082] Figure 12 This is a schematic diagram of the final model's MSE under different sampling ratios in the CIFAR-10 dataset in an embodiment of the present invention;

[0083] Figure 13 This is a schematic diagram illustrating the accuracy of the final model under different sampling ratios in the MNIST dataset in an embodiment of the present invention.

[0084] Figure 14 This is a schematic diagram illustrating the accuracy of the final model under different sampling ratios in the CIFAR-10 dataset in an embodiment of the present invention.

[0085] Figure 15 This is a schematic diagram illustrating the algorithm training time under different sampling ratios in the MNIST dataset in an embodiment of the present invention.

[0086] Figure 16 This is a schematic diagram illustrating the algorithm training time under different sampling ratios in the CIFAR-10 dataset in an embodiment of the present invention. Detailed Implementation

[0087] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0088] Example

[0089] A communication-efficient federated learning method includes the following steps:

[0090] 1. Construction of the proxy objective function:

[0091] To protect user privacy, federated learning requires all participants to train on the dataset locally before transmitting the model parameters to the server for aggregation. Assume a federated learning framework has a central server S and N clients {C1,,...,C...} N Client C i The local data possessed is D i Then the dataset set of all clients participating in federated learning is D = {D1, D2, ..., D...} N Assume F D (θ) represents the objective function based on the local data volume of all clients, also known as the global objective function. The global objective function can be written as the sum of the local objective functions of each client, i.e.:

[0092]

[0093] in, This indicates the weight of each client, and F i (θ) represents the local objective function of the i-th client, as follows:

[0094]

[0095] Where, l(x) j ,y j ;θ) is the input x of the model with parameter θ for each instance j in the dataset. j and output y j The resulting loss function.

[0096] Next, we construct a proxy target function. To replace the global objective function F D (θ). For F D Taylor expansion of (θ) yields an infinite series, as shown in equation (3):

[0097]

[0098] in, Let θ be any initial estimate. It is a computable constant, and <·,·> represents the inner product. Directly solving the higher-order derivative part in equation (3) is computationally difficult. Therefore, we can first randomly select a small sample dataset from each client's local dataset to construct a sample objective function, and use the higher-order derivative part of the sample objective function to replace the higher-order derivative part of the global objective function. The specific steps are as follows:

[0099] Step 1: Set a sampling ratio α, and each client randomly samples a dataset of size d from its local dataset. i =αD i A subset of the samples constitutes the sampling sample set d = {d1, d2, ..., dn}. N Assume d is much smaller than D, i.e. And as D→∞, d→∞. The objective function based on the sampled data can be expressed as:

[0100]

[0101] in, This indicates the weight of each client, and This indicates that the i-th client is based on the sampled sample d. i The local objective function. For the sample objective function F d (θ) is expanded using Taylor series to obtain an infinite series, as shown in equation (5):

[0102]

[0103] in, Let θ be any initial estimate. It is a computable constant, and <·,·> denote the inner product.

[0104] Step 2: Replace the higher-order derivative of the global objective function with the higher-order derivative of the sample objective function, that is, let:

[0105]

[0106] Substituting equation (6) into equation (3), we can obtain the global objective function F. D Approximation of (θ):

[0107]

[0108] Ignoring the constant term in equation (7) and simplifying, we obtain the proxy objective function:

[0109]

[0110] Equation (8) is called the Communication-efficient Surrogate Objective Function (CSOF). It can be rewritten as the sum of the objective functions of each client's local samples, as shown in Equation (9):

[0111]

[0112] The goal of federated learning is to obtain the optimal parameters θ' at the central server, which is represented by minimizing the agent's objective function, i.e.:

[0113]

[0114] 2. Alternating Direction Multiplier Method:

[0115] For problem (10), this embodiment uses the Alternating Direction Multiplier Method (ADMM) for solution. The advantage of the ADMM algorithm is that it can decompose a large-scale optimization problem into multiple small-scale subproblems and solve these subproblems in a distributed manner. Therefore, minimizing the proxy objective function can be decomposed into minimizing the local objective functions of each client. Let Rewrite question (10) in the following form:

[0116]

[0117] in, This indicates the weight of each client, and This indicates that the i-th client is based on the sampled sample d. i The local objective function. Problem (11) can be further expressed as the following globally consistent optimization problem:

[0118]

[0119] Where, θ i This represents the local model parameters for the i-th client. Under constraint θ... i Under the influence of θ, i∈[1,2,···,N], problem (12) is equivalent to problem (11). Problem (12) is transformed into the form required by the ADMM algorithm:

[0120]

[0121] Where B represents N identity matrices, Represents all θ i The set, i.e. The specific definition is:

[0122]

[0123] The augmented Lagrangian function corresponding to problem (13) is as follows:

[0124]

[0125] Where ρ>0 is the penalty parameter, and the dual variable u∈[u1,u2,···,u N The update rules for the ADMM algorithm are given below:

[0126]

[0127]

[0128]

[0129] Regarding question (16), this question only relates to client C. i This can be rewritten in the following form:

[0130]

[0131] Problem (19) typically does not allow closed-form solutions, as this increases computational complexity and cost. To accelerate local convergence on the client side, a linearized ADMM algorithm is used to simplify the solution. exist Using a second-order Taylor expansion, we can obtain Extensions:

[0132]

[0133] Where η is the coefficient of the penalty term. Then... The update formula can be written as:

[0134]

[0135] Equation (21) is an approximation of equation (19). By solving equation (21), an exact solution to equation (19) can be obtained:

[0136]

[0137] For problem (17), this is a quadratic function of θ. Taking its partial derivative and setting it to zero will give us the minimum point, i.e.:

[0138]

[0139] 3. Design of the FedCSOF algorithm:

[0140] Based on the iterative formula derived above, after simplification, we can obtain the highly efficient federated learning algorithm FedCSOF. Let... Before each round of federated training begins, the agent's objective function is first updated. Then the parameters are updated according to the formula.

[0141] (1) Central server algorithm design:

[0142] The training process of the central server for the FedCSOF algorithm can be mainly divided into the following steps:

[0143] Step 1: Initialize global model parameters.

[0144] Step 2: Randomly select clients to participate in this round of federated training.

[0145] Step 3: Send global model parameters to each client.

[0146] Step 4: Selected clients randomly sample data according to the sampling ratio.

[0147] Step 5: Each client performs local training based on Algorithm 2, and uploads the parameters after training is complete.

[0148] Step 6: Receive parameters uploaded by the client.

[0149] Step 7: Aggregate client parameters and update global model parameters according to the formula.

[0150] Step 8: Repeat steps 2 through 7 until the global communication rounds reach the specified value.

[0151] The pseudocode for the central server's algorithm is shown in Algorithm 1. The central server first initializes the global model. In any iteration t∈T, the central server randomly selects client C to participate in the federated training. t and to client C t Send global model parameters θ t-1 The clients participating in this round of federated training receive the global model parameters θ. t-1 Then, sample data d are randomly sampled according to the sampling ratio α. i =αD i And calculate Each client then updates its local parameters according to Algorithm 2 and uploads them to the central server. The central server receives the parameters from each client and aggregates them. Based on equation (23), the iterative formula for the global model parameters of the central server can be derived, as shown in equation (24).

[0152]

[0153] Table 1. Pseudocode of FedCSOF Client Algorithm

[0154]

[0155]

[0156] (2) Client-side algorithm design:

[0157] The client-side training process of the FedCSOF algorithm can be mainly divided into the following steps:

[0158] Step 1: Load the dataset and determine the number of local updates E and the sampling ratio α.

[0159] Step 2: The client randomly samples a dataset of size d from its local dataset. i =αD i The sample set.

[0160] Step 3: Based on the global model parameters θ t-1 Calculate R i (θ t-1 ).

[0161] Step 4: Update the local model parameters according to the learning rate γ until the current training round reaches E.

[0162] Step 5: Set parameter θ i u i and R i (θ t-1 Send to the central server.

[0163] Table 2 Pseudocode of FedCSOF Client Algorithm

[0164]

[0165] The client-side algorithm pseudocode is shown in Algorithm 2. The following is a detailed description of the client-side process for the FedCSOF algorithm. The client participating in federated training randomly selects samples according to a sampling ratio α, and receives θ... t-1 Post-calculation The parameter θ can be obtained from equation (22) and the learning rate γ. i The update formula is shown in equation (25). The dual variable u is updated according to equation (26). i The client obtains the parameters θ after performing E rounds of local training. i and u i Then θ i u i and R i (θ t-1 Send to the central server.

[0166]

[0167] u i ←u i +ρ(θ i -θ t-1 )#(26)

[0168] The following experiment will further illustrate this embodiment:

[0169] 1. Experimental environment:

[0170] The experiment simulated a federated learning environment with 100 clients and a central server. In each round, 20 clients were randomly selected to participate in federated training. All experiments were run on the same computer. The hardware and software environment for system operation and testing is shown in Table 3.

[0171] Table 3 Hardware and Software Environment of the Physical Machine System

[0172]

[0173] 2. Experimental data:

[0174] The datasets used were the MNIST dataset and the CIFAR-10 dataset, both of which are commonly used datasets in academia. Basic information about the MNIST and CIFAR-10 datasets is shown in Table 4.

[0175] Table 4 Experimental Dataset

[0176]

[0177] The MNIST (Modified National Institute of Standards and Technology) dataset is a classic handwritten digit recognition dataset widely used in machine learning and computer vision. This dataset consists of 70,000 28×28 pixel grayscale images, covering handwritten digits from 0 to 9, with 60,000 images used for training and 10,000 for testing. The handwritten images in the MNIST dataset were obtained from US high school students and Census Bureau staff. Figure 1 It consists of 10 classes from the MNIST dataset, and 10 random images for each class.

[0178] The CIFAR-10 (Canadian Institute for Advanced Research) dataset was designed and released by the Canadian Institute for Computer Science. The dataset consists of 60,000 32×32 pixel color images in 10 categories, with 50,000 images serving as training samples and the remaining 10,000 as test samples. These images are manually labeled and are suitable for training and testing various classification algorithms. Figure 2 It consists of 10 classes from the CIFAR-10 dataset, and 10 random images for each class.

[0179] The simulation experiment divides the data into two forms: Independent and Identically Distributed (IID) and Non-Identically Distributed (Non-IID), to simulate two scenarios of client data distribution. For the IID scenario, each client's training data is randomly sampled from the overall dataset. To simulate the IID environment, the original dataset is evenly distributed across all clients. For the Non-IID scenario, the dataset is divided into multiple disjoint subsets, and each client randomly receives several subsets. To simulate the Non-IID environment, the original dataset is first divided according to the classification labels, and then each client is randomly assigned data from any three or four of the ten classes.

[0180] 3. Experimental model:

[0181] To enhance the persuasiveness of the experiments, the simulation used a Multilayer Perceptron (MLP) and a Convolutional Neural Network (CNN) model as training models. Both MLP and CNN models are commonly used neural network models in computer image classification. The MLP model consists of an input layer, one hidden layer, and an output layer. The hidden layer has 512 neurons with the ReLU activation function. The output layer is a 10-class classification (digits 0-9), with 10 neurons and softmax as the activation function, mapping the input image to a 0-9 class; the class with the highest probability is the prediction result. The CNN model consists of two convolutional layers, two pooling layers, one fully connected layer, and one output layer. The convolutional kernels in the convolutional layers are all 5×5 with a stride of 1, and the ReLU activation function is used. The pooling layers all use MaxPool pooling with a 2×2 pooling window and a stride of 1. The fully connected layer has 512 neurons with the ReLU activation function. The output layer is for 10 categories, with 10 neurons, and softmax is chosen as the activation function. The expressions for ReLU and softmax activation functions are as follows:

[0182] relu(x)=max(0,x)#(27)

[0183]

[0184] The cross-entropy loss function is selected as the loss function for the simulation experiment. The calculation formula of the cross-entropy loss function is shown in equation (29):

[0185] F(θ,x)=-∑ylog(y')#(29)

[0186] Where y and y' represent the true value and predicted value of the input data x, respectively.

[0187] 4. Evaluation indicators:

[0188] The simulation experiment used mean squared error, model accuracy, and algorithm training time as metrics to evaluate the algorithm's performance.

[0189] (1) Mean Square Error (MSE)

[0190] Mean squared error is used as an evaluation metric for model performance, denoted by MSE, and its calculation formula is as follows:

[0191]

[0192] in To predict the output, y i Here, m represents the actual data labels, and m represents the number of data samples used in the calculation. A smaller MSE value indicates better model performance.

[0193] (2) Accuracy

[0194] For multi-class classification problems, accuracy is used as the evaluation metric for model performance. Accuracy refers to the percentage of correctly predicted results out of the total sample, and its calculation formula is as follows:

[0195]

[0196] Where T represents True, F represents False, P represents Positive, and N represents Negative. TP indicates a correct prediction (predicted label 1, actual label 1); FP indicates a wrong prediction (predicted label 1, actual label 0); TN indicates a correct prediction (predicted label 0, actual label 0); and FN indicates a wrong prediction (predicted label 0, actual label 1). All accuracy rates in the experimental results are for the test set.

[0197] (3) Training Time

[0198] To demonstrate the algorithm's improvement in communication, training time is used as an indicator to evaluate model performance and efficiency. Shorter training time indicates better algorithm performance.

[0199] 5. Parameter settings:

[0200] The simulation experiments set different hyperparameters for the MNIST and CIFAR-10 datasets, and the specific parameter settings are shown in Table 5 below.

[0201] Table 5. Hyperparameter settings for simulation experiments

[0202]

[0203] 4. Experimental Results and Analysis:

[0204] Simulation experiments compared the efficient federated learning algorithm FedCSOF proposed in this embodiment with the traditional federated learning algorithm FedAvg. To avoid random errors, all experiments were repeated 5 times, and the average of the 5 experiments was taken as the final experimental result.

[0205] (1) Performance analysis of FedCSOF algorithm in independent and identically distributed environments

[0206] The sampling ratio α was set to 0.005. For the MNIST dataset, 20 clients were randomly selected from 100 clients to participate in each round of federated training. The global communication rounds T were set to 30, the number of client-side model local updates E was set to 50, the learning rate γ was set to 0.001, the training batch size was set to 128, and the penalty parameter ρ was set to 1. MLP and CNN models were used as client-side local models for training, and the experimental results are as follows. Figure 3 As shown in the figure, the horizontal axis represents the number of communication rounds, and the vertical axis represents the model accuracy.

[0207] The experimental results show that, under the independent and identically distributed (IOD) environment, using the MLP model, the FedCSOF algorithm converges after 14 rounds of global communication, while the FedAvg algorithm converges after 16 rounds. Using the CNN model, the FedCSOF algorithm converges after 17 rounds of global communication, while the FedAvg algorithm converges after 21 rounds. Regardless of whether it's the MLP or CNN model, the FedCSOF algorithm converges faster than FedAvg, achieving a slightly higher accuracy.

[0208] For the CIFAR-10 dataset, in each round of federated training, 20 clients were randomly selected from 100 clients to participate. The global communication rounds T were set to 100, the number of client-side model local updates E was set to 20, the learning rate γ was set to 0.005, the training batch size was 128, and the penalty parameter ρ was set to 1. MLP and CNN models were used as client-side local models for training, and the experimental results are as follows. Figure 4 As shown in the figure, the horizontal axis represents the number of communication rounds, and the vertical axis represents the model accuracy.

[0209] The experimental results show that, under the independent and identically distributed (IOD) environment, using the MLP model, the FedCSOF algorithm converges after 45 rounds of global communication, while the FedAvg algorithm converges after 60 rounds. Using the CNN model, the FedCSOF algorithm converges after 41 rounds of global communication, while the FedAvg algorithm converges after 53 rounds. Regardless of whether it's an MLP or CNN model, the FedCSOF algorithm converges faster than FedAvg, reaching convergence more quickly, and its final accuracy is also slightly higher.

[0210] To further analyze the performance of the FedCSOF algorithm, the test set accuracy when FedAvg is close to convergence is used as the target accuracy. Based on... Figure 3 and Figure 4 The experimental results show that, under the independent and identically distributed (IOD) condition, the convergence accuracy of the MNIST dataset under the MLP and CNN models is approximately 91% and 90%, respectively. The convergence accuracy of the CIFAR-10 dataset under the MLP and CNN models is approximately 50% and 48%, respectively. To compare the convergence speed of different algorithms, the convergence accuracy of the MNIST dataset under the MLP and CNN models was set to 91% and 90%, respectively, and the convergence accuracy of the CIFAR-10 dataset under the MLP and CNN models was set to 50% and 48%, respectively. The number of global communication rounds (T) and training time (in hours) required to achieve the target accuracy for various training modes are shown in Table 6. Figure 5 and Figure 6 As shown.

[0211] Table 6. Number of communication rounds and training time required to achieve the target accuracy for various training modes in the IID environment.

[0212]

[0213] The experimental results show that, under the independent and identically distributed (IOD) environment, the FedCSOF algorithm achieves the target accuracy for each dataset with fewer communication rounds, and the training time is also significantly reduced. Specifically, for the MNIST dataset, the MLP model FedAvg requires 15 communication rounds to reach the target accuracy, with a training time of 0.506 hours, while FedCSOF only requires 10 communication rounds, with a training time of 0.385 hours, representing a 33.33% reduction in the number of communication rounds and a 23.91% reduction in training time. Using the CNN model FedAvg, 21 communication rounds are required to reach the target accuracy, with a training time of 0.697 hours, while FedCSOF only requires 15 communication rounds, with a training time of 0.531 hours, representing a 28.57% reduction in the number of communication rounds and a 23.82% reduction in training time. For the CIFAR-10 dataset, the MLP model FedAvg requires 58 communications to achieve the target accuracy, with a training time of 1.802 hours. FedCSOF requires 38 communications, with a training time of 1.302 hours, representing a 34.48% reduction in the number of communications and a 27.74% reduction in training time. Using the CNN model FedAvg, 53 communications are needed to achieve the target accuracy, with a training time of 1.683 hours. FedCSOF requires 35 communications, with a training time of 1.185 hours, representing a 33.96% reduction in the number of communications and a 29.59% reduction in training time. Overall, under the independent and identically distributed (IMDG) scenario, the FedCSOF algorithm reduces the number of communications by an average of 33.33% and the training time by an average of 27.41% compared to FedAvg. In federated learning, more communication rounds mean higher communication costs. For some clients with limited budgets, multiple rounds of communication may not be possible, negatively impacting the global model. The FedCSOF algorithm not only reduces the number of communication rounds between the client and the central server, thus lowering training time, but its final model accuracy is also slightly higher than that of the model trained by the FedAvg algorithm. This demonstrates that the FedCSOF algorithm can reduce communication costs and achieve efficient communication under independent and identically distributed conditions.

[0214] (2) Performance analysis of FedCSOF algorithm in non-independent and identically distributed environments

[0215] The sampling ratio α was set to 0.005. For the MNIST dataset, 20 clients were randomly selected from 100 clients to participate in each round of federated training. The global communication rounds T were set to 30, the number of client-side model local updates E was set to 50, the learning rate γ was set to 0.001, the training batch size was set to 128, and the penalty parameter ρ was set to 1. MLP and CNN models were used as client-side local models for training, and the experimental results are as follows. Figure 7 As shown in the figure, the horizontal axis represents the number of communication rounds, and the vertical axis represents the model accuracy.

[0216] The experimental results show that, in a non-independent and identically distributed environment, using the MLP model, the FedCSOF algorithm converges after 15 rounds of global communication, while the FedAvg algorithm converges after 18 rounds. Using the CNN model, the FedCSOF algorithm converges after 19 rounds of global communication, while the FedAvg algorithm converges after 23 rounds. Regardless of whether it's an MLP or CNN model, the FedCSOF algorithm converges faster than FedAvg, achieving a slightly higher accuracy.

[0217] For the CIFAR-10 dataset, in each round of federated training, 20 clients were randomly selected from 100 clients to participate. The global communication rounds T were set to 100, the number of client-side model local updates E was set to 20, the learning rate γ was set to 0.005, the training batch size was 128, and the penalty parameter ρ was set to 1. MLP and CNN models were used as client-side local models for training, and the experimental results are as follows. Figure 8 As shown in the figure. The horizontal axis represents the number of communication rounds, and the vertical axis represents the model accuracy.

[0218] The experimental results show that, in a non-independent and identically distributed environment, using the MLP model, the FedCSOF algorithm converges after 52 rounds of global communication, while the FedAvg algorithm converges after 60 rounds. Using the CNN model, the FedCSOF algorithm converges after 60 rounds of global communication, while the FedAvg algorithm converges after 70 rounds. Regardless of whether it's the MLP or CNN model, the FedCSOF algorithm converges faster than FedAvg, reaching convergence more quickly, and its final accuracy is also slightly higher.

[0219] Table 7. Number of communication rounds and training time required to achieve the target accuracy for various training modes in a Non-IID environment.

[0220]

[0221] To further analyze the performance of the FedCSOF algorithm, the test set accuracy when FedAvg is close to convergence is used as the target accuracy. Based on... Figure 7 and Figure 8The experimental results show that, under the non-independent and identically distributed condition, the convergence accuracy of the MNIST dataset under the MLP and CNN models is approximately 85% and 84%, respectively. The convergence accuracy of the CIFAR-10 dataset under the MLP and CNN models is approximately 45% and 43%, respectively. To compare the convergence speed of different algorithms, the convergence accuracy of the MNIST dataset under the MLP and CNN models was set to 85% and 84%, respectively, and the convergence accuracy of the CIFAR-10 dataset under the MLP and CNN models was set to 45% and 43%, respectively. The number of global communication rounds (T) and training time (in hours) required to achieve the target accuracy for various training modes are shown in Table 7. Figure 9 and Figure 10 As shown.

[0222] The table above shows that in non-independent and identically distributed environments, the FedCSOF algorithm achieves the target accuracy for each dataset with fewer communication rounds, and the training time is also significantly reduced. Specifically, for the MNIST dataset, the MLP model FedAvg requires 18 communication rounds and a training time of 0.612 hours, while FedCSOF only requires 13 communication rounds and a training time of 0.470 hours, representing a reduction of 27.78% in the number of communication rounds and a 23.20% reduction in training time. Using the CNN model FedAvg requires 23 communication rounds and a training time of 0.781 hours, while FedCSOF only requires 17 communication rounds and a training time of 0.606 hours, representing a reduction of 26.09% in the number of communication rounds and a 22.41% reduction in training time. For the CIFAR-10 dataset, the MLP model FedAvg requires 62 communications and 1.898 hours of training time, while FedCSOF requires 42 communications and 1.442 hours of training time, representing a 32.26% reduction in the number of communications and a 24.03% reduction in training time. Using the CNN model, FedAvg requires 66 communications and 2.099 hours of training time, while FedCSOF requires 48 communications and 1.576 hours of training time, representing a 27.27% reduction in the number of communications and a 24.92% reduction in training time. Overall, under the non-independent and identically distributed (IID) case, the FedCSOF algorithm reduces the number of communications by an average of 28.99% and the training time by an average of 24.04% compared to FedAvg. In federated learning, more communication rounds mean higher communication costs. For some clients with limited budgets, multiple rounds of communication may not be possible, negatively impacting the global model. The FedCSOF algorithm not only reduces the number of communication rounds between the client and the central server, thus lowering training time, but its final model accuracy is also slightly higher than that of the model trained by the FedAvg algorithm. This demonstrates that the FedCSOF algorithm can reduce communication costs and achieve efficient communication in the case of non-independent and identically distributed systems.

[0223] (3) Analysis of the impact of different sampling ratios α on the performance of the FedCSOF algorithm

[0224] Since different sampling ratios α can affect the performance of the FedCSOF algorithm, this section conducts an experimental analysis of the impact of the sampling ratio α. Several different sampling ratios were set for experimental analysis, i.e., α∈[0.001, 0.003, 0.005, 0.01, 0.05]. Experiments were conducted on the MNIST and CIFAR-10 datasets using a CNN model under an independent and identically distributed environment.

[0225] For the MNIST dataset, 20 clients are randomly selected from 100 clients for each round of federated training. The global communication rounds T are set to 30, the number of client-side model local updates E is set to 50, the learning rate γ is set to 0.001, the training batch size is 128, and the penalty parameter ρ is set to 1. For the CIFAR-10 dataset, 20 clients are randomly selected from 100 clients for each round of federated training. The global communication rounds T are set to 100, the number of client-side model local updates E is set to 20, the learning rate γ is set to 0.005, the training batch size is 128, and the penalty parameter ρ is set to 1. All clients use a CNN model as their local model for training. The mean squared error of the global model, the model accuracy, and the algorithm training time are recorded after reaching the specified number of iterations for various training modes.

[0226] The mean squared error histogram of the global model after various training modes have reached a specified number of iterations is shown below. Figure 11 and Figure 12 As shown in the figure, the horizontal axis represents various training modes, and the vertical axis represents the mean square error of the global model.

[0227] The experimental results show that, in the MNIST and CIFAR-10 datasets, the global model trained using the FedCSOF algorithm has a significantly lower mean squared error than the model trained using the FedAvg algorithm. Furthermore, as the sampling ratio α increases, the mean squared error of the model trained using the FedCSOF algorithm gradually decreases, but the rate of decrease gradually slows down.

[0228] The bar chart shows the global model accuracy after various training modes have reached a specified number of iterations. Figure 13 and Figure 3-14 As shown in the figure, the horizontal axis represents various training modes, and the vertical axis represents the accuracy of the global model.

[0229] The experimental results show that, on the MNIST and CIFAR-10 datasets, the global model trained using the FedCSOF algorithm has significantly higher accuracy than the model trained using the FedAvg algorithm. Furthermore, as the sampling ratio α increases, the accuracy of the model trained using the FedCSOF algorithm gradually increases, but the rate of increase gradually slows down.

[0230] The following is a bar chart showing the algorithm training time after reaching a specified number of iterations for various training modes. Figure 15 and Figure 16 As shown in the figure, the horizontal axis represents various training modes, and the vertical axis represents the algorithm training time.

[0231] The experimental results show that, on the MNIST and CIFAR-10 datasets, the training time using the FedCSOF algorithm is significantly shorter than that using the FedAvg algorithm. Furthermore, as the sampling ratio α increases, the training time using the FedCSOF algorithm gradually increases, and the rate of increase gradually intensifies.

[0232] The experimental results above demonstrate the feasibility and effectiveness of the FedCSOF algorithm. The parameter sampling ratio α has a certain impact on the performance of the algorithm. When α is larger, that is, the number of samples randomly drawn by the client is larger, the final model obtained by training will be better, but it will increase the amount of computation, thereby increasing the training time of the algorithm.

[0233] This embodiment addresses the problems of long communication time and high communication cost between clients and the central server in federated learning scenarios. It proposes a high-efficiency federated learning algorithm, FedCSOF. The core idea of ​​this method is to use a surrogate objective function to replace the global objective function. This surrogate objective function can approximate the global objective function in a communication-efficient manner, reducing the amount of communication between the client and the central server without sacrificing model accuracy, thereby reducing the communication time between the client and the central server. The ADMM algorithm is used to solve the parameter update formula in the surrogate objective function to reduce computational load and further accelerate the convergence speed of the algorithm. To verify the feasibility and effectiveness of the algorithm, this embodiment conducts a series of simulation experiments using the MNIST and CIFAR-10 datasets under independent and identically distributed (ISD) and non-independent ISD environments, respectively. Experimental results show that, under the ISD environment, the FedCSOF algorithm has a faster convergence speed than FedAvg on both the MNIST and CIFAR-10 datasets, and the final global model accuracy is slightly improved. The FedCSOF algorithm still performs well even under non-independent and identically distributed (i.I.D.) conditions. On the MNIST and CIFAR-10 datasets, FedCSOF's convergence speed is also superior to FedAvg, without affecting model accuracy. Using the test set accuracy of FedAvg near its convergence state as the target accuracy, we observed the number of iterations and training time required to reach the target accuracy under various training modes. The results show that under the i.I.D. condition, FedCSOF reduces the number of communication iterations by an average of 33.33% and the training time by an average of 27.41% compared to FedAvg. Under the non-independent and identically distributed (i.I.D.) condition, FedCSOF reduces the number of communication iterations by an average of 28.99% and the training time by an average of 24.04% compared to FedAvg. Less training time means lower communication costs. Finally, the parameters of the algorithm were experimentally analyzed. The results showed that the sampling ratio has a certain impact on the performance of the algorithm. When the sampling ratio is larger, that is, the number of samples randomly selected by the client is larger, the final model obtained by training will be better, but it will increase the amount of computation, thereby increasing the training time of the algorithm. Different sampling ratios can be selected according to different actual situations.

[0234] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A communication-efficient federated learning method, characterized in that, Includes the following steps: Step 1: Construct a proxy objective function and use the proxy objective function to replace the global objective function; Step 1 is described in detail as follows: Suppose a federated learning framework has a central server S and N clients {C1,,...,C...} N Client C i The local data possessed is D i Then the dataset set of all clients participating in federated learning is D = {D1, D2, ..., D...} N Assume F D (θ) represents the objective function based on the local data volume of all clients, also known as the global objective function; the global objective function is written as the sum of the local objective functions of each client, i.e.: in, This indicates the weight of each client, and F i (θ) represents the local objective function of the i-th client, as follows: Where, l(x) j ,y j ;θ) is the input x of the model with parameter θ for each instance j in the dataset. j and output y j The resulting loss function; Next, we construct a proxy target function. To replace the global objective function F D (θ), for F D Taylor expansion of (θ) yields an infinite series: in, Let θ be any initial estimate. It is a computable constant, and <·,·> denote the inner product; Step 2: Use the Alternating Direction Multiplier Method (ADMM) to solve for the parameter update formula in the surrogate objective function; Step 2 is described in detail below: Decompose the minimization of the proxy objective function into minimization of the local objective functions of each client, let Equation (10) can be rewritten in the following form: in, This indicates the weight of each client, and This indicates that the i-th client is based on the sampled sample d. i The local objective function, Equation (11), is further expressed as the following globally consistent optimization problem: Where, θ i This represents the local model parameters of the i-th client, under constraint θ. i Under the influence of θ, i∈[1,2,···,N], equation (12) is equivalent to equation (11). Equation (12) is then transformed into the form required by the ADMM algorithm: Where B represents N identity matrices, Represents all θ i The set, i.e. The specific definition is: The augmented Lagrangian function corresponding to equation (13) is as follows: Where ρ>0 is the penalty parameter, and the dual variable u∈[u1,u2,···,u N ]; The update rules for the ADMM algorithm are given below: For equation (16), only client C i Relevant information, rewritten in the following form: right exist Using a second-order Taylor expansion, we can obtain Extensions: Where η is the coefficient of the penalty term; then The update formula is written as: Equation (21) is an approximation of equation (19). By solving equation (21), an exact solution to equation (19) can be obtained: For equation (17), which is a quadratic function of θ, the minimum point can be obtained by taking its partial derivative and setting it to zero. Step 3: Design a federated learning algorithm with high communication efficiency, including designing the central server algorithm and the client algorithm; Step 3 is described in detail below: make Before each round of federated training begins, the agent's objective function is first updated. Then update the parameters according to the formula; The central server algorithm is as follows: The central server first initializes the global model. In any iteration t∈T, the central server randomly selects client C to participate in the federated training. t and to client C t Send global model parameters θ t-1 The clients participating in this round of federated training receive the global model parameters θ. t-1 Then, sample data d are randomly sampled according to the sampling ratio α. i =αD i And calculate Each client then updates its local parameters according to its own algorithm and uploads them to the central server. After receiving the parameters from each client, the central server aggregates them and derives the iterative formula for the global model parameters of the central server based on equation (23): The client-side algorithm is as follows: Clients participating in federated training randomly sample according to a sampling ratio α, and receive θ t-1 Post-calculation The parameter θ is obtained from equation (22) and the learning rate γ. i The update formula is shown in equation (25). The dual variable u is updated according to equation (26). i ; you i ←u i +ρ(θ i -θ t-1 (26) The client obtains the parameters θ after performing E rounds of local training. i and u i Then θ i u i and R i (θ t-1 Send to the central server.

2. The efficient federated learning method according to claim 1, characterized in that, For the global objective function F D When solving for (θ), a small sample dataset is first randomly selected from the local datasets of each client to construct a sample objective function. The higher-order derivatives of the sample objective function are then used to replace the higher-order derivatives of the global objective function. The specific steps are as follows: Step 1.1: Set a sampling ratio α, and each client randomly samples a dataset of size d from its local dataset. i =αD i A subset of the samples constitutes the sampling sample set d = {d1, d2, ..., dn}. N Assume d is much smaller than D, that is Furthermore, as D→∞, d→∞, and the objective function based on the sampled data is expressed as: in, This indicates the weight of each client, and This indicates that the i-th client is based on the sampled sample d. i The local objective function, for the sample objective function F d Taylor expansion of (θ) yields an infinite series: in, Let θ be any initial estimate. It is a computable constant, and <·,·> denote the inner product; Step 1.2: Replace the higher-order derivative of the global objective function with the higher-order derivative of the sample objective function, that is, let: Substituting equation (6) into equation (3), we obtain the global objective function F. D Approximation of (θ): Ignoring the constant term in equation (7) and simplifying, we obtain the proxy objective function: Equation (8) is called the objective function of the agent for efficient communication. It can be rewritten as the sum of the objective functions of the local samples of each client: The goal of federated learning is to obtain the optimal parameters θ on a central server. ′ It is expressed as minimizing the agent objective function, i.e.:

Images