A federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network

Abstract

This invention discloses a federated averaging algorithm based on residual networks, characterized by high accuracy and efficient communication: Step 1: The server partitions the MNIST dataset into IID and Non-IID categories and distributes them to various clients; Step 2: Each client locally constructs a residual network model; Step 3: The number of communication rounds R between the server and clients is set. In each round, the server randomly selects K clients to participate in the communication and sends the parameters of the global model; Step 4: The selected clients download the parameters of the global model and perform local machine learning training to obtain the trained model parameters; Step 5: The K clients upload the trained model parameters to the server; Step 6: The server updates the global model parameters according to a weighted average aggregation strategy and repeats steps 3-6 until the final global model parameters are obtained in the Rth round of communication. This invention significantly improves accuracy and communication efficiency compared to existing technologies.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a federated averaging algorithm based on residual networks with high accuracy and efficient communication. Technical Background

[0002] In the era of the data economy, the ability to fully extract useful information from massive amounts of data is crucial to the survival of many enterprises. From traditional machine learning to today's artificial intelligence, data remains central. The reality is that, apart from a few giant companies possessing vast amounts of data, most enterprises have data that is both small in quantity and poor in quality, thus insufficient to support the realization of artificial intelligence. Traditionally, data from various stakeholders is integrated onto a server for training and model generation. However, due to the enormous potential value of data and the increasing number of policies aimed at strengthening data protection, companies, and even departments within companies, often do not provide their data for aggregation. Therefore, data silos arise due to user privacy, trade secrets, and policy constraints. Thus, the urgent need arises to find ways to unite different organizations and institutions to integrate their raw data and jointly train a high-performance, powerful model.

[0003] Against this backdrop, federated learning emerged as a novel machine learning paradigm. Participating clients can jointly model data using data from other clients, while the data remains locally for training. Clients and servers only exchange network model parameters, thus building a shared machine learning model without exposing private data. Federated learning typically iterates through the following four steps: 1. The client requests model parameters from the server, and the server provides the latest model parameters to the client; 2. The client updates its model locally using local data and the latest model parameters; 3. The client sends the updated model parameters to the server; 4. The server aggregates the model parameters from each client and uses them to update the global model. However, federated learning encountered several bottlenecks, including frequent communication between clients and servers, resulting in a high communication cost, far exceeding the computational cost. Therefore, Google subsequently proposed the Federated Averaging algorithm, which improves upon this by iterating through multiple rounds during local client updates to obtain accumulated model parameters, which are then uploaded to the server for aggregation. This algorithm reduces communication costs at the expense of computation, achieving convergence with fewer communication rounds and resolving this bottleneck. Summary of the Invention

[0004] The FedAvg algorithm employs two models: MLP and LeNet5. These models limit test set accuracy, and their accuracy exhibits significant variance after stabilization, reflected in large fluctuations in the accuracy curve. In a few communication rounds, accuracy drops below 95%, and in some rounds, it even falls below 90%. This invention replaces the original network with an adjusted deep residual network for local updates. It simplifies the network structure by modifying relevant parameters of ResNet18 and deleting certain layers, forming ResNet18-E, achieving a good balance between accuracy and communication efficiency. A series of tests were conducted on the MNIST dataset using independent identically distributed (IID) and independent non-identical distributed (Non-IID) scenarios to evaluate the invention. Results show that the proposed scheme outperforms the original algorithm in both accuracy and stability. Although the more complex network structure increases the communication time per round, it reduces the number of communication rounds required to achieve the target accuracy, resulting in a decrease in the overall total communication time (the product of the number of communication rounds required to achieve the target accuracy and the communication time per round).

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A federated averaging algorithm based on residual networks, characterized by high accuracy and efficient communication, comprises the following steps:

[0007] Step 1: The server divides the MNIST dataset into IID and Non-IID partitions and distributes them to each client.

[0008] Step 2: Build the residual network model locally on each client, which includes the following sub-steps:

[0009] Step 21: Select ResNet18 as the backbone network of the residual network model;

[0010] Step 22: Based on the backbone network, construct the specific network structure of ResNet18-E; the ResNet18-E includes a 3×3 convolutional layer, convolutional groups Stage1, Stage2, Stage3, and a linear layer connected in sequence, wherein:

[0011] The convolutional groups Stage1, Stage2, and Stage3 each contain two residual blocks. Each residual block includes two 3×3 convolutional layers with the same number of channels. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The input jump of the residual block is added between the batch normalization layer and the ReLU activation function after its second convolutional layer. In addition, a downsampling structure is added to the first residual block of Stage2 and the first residual block of Stage3.

[0012] The linear layer consists of two sequentially connected linear layers;

[0013] Step 3: Set the number of communication rounds R between the server and the client. In each round of communication, the server randomly selects K clients from all clients C to participate in the round of communication and sends the parameters of the global model stored in the selected clients.

[0014] Step 4: The selected client downloads the parameters of the global model and performs local machine learning training within the preset number of local iterations. During the training process, the network loss is calculated using the cross-entropy loss function, the gradient is calculated using backpropagation, and finally the local model accuracy is continuously updated using the stochastic gradient descent algorithm to obtain the corresponding trained model parameters.

[0015] Step 5: After these K clients have finished updating, they will each upload the locally trained model parameters to the server.

[0016] Step 6: The server node receives the locally updated trained model parameters from K clients, updates the global model parameters according to the weighted average aggregation strategy, and saves them as the aggregation result. The updated global model parameters are used as the current global model parameters. That is, the global model parameter update of step 3-6 is re-executed to start a new round of communication until the global model parameters are obtained in the Rth round of communication, which are used as the final global model parameters.

[0017] Furthermore, step 1 specifically includes the following sub-steps:

[0018] Step 11: Divide the MNIST dataset into IIDs and distribute them to each client.

[0019] Step 12: Divide the dataset into Non-IID partitions and distribute them to each client.

[0020] Furthermore, there are a total of 100 clients.

[0021] Furthermore, step 11 specifically involves: first shuffling the MNIST dataset, and then randomly assigning 600 samples to each client.

[0022] Furthermore, step 12 specifically involves: sorting the MNIST dataset according to the data labels in order from 0 to 9, then dividing it into 200 data slices of size 300, and finally randomly assigning two slices to each user, meaning each client has 600 samples.

[0023] Furthermore, in step 22, the kernel size of the 3×3 convolutional layer is 3×3, the stride is 1, the padding is 1, and the output channels are 16.

[0024] Furthermore, in step 22, the downsampling structure consists of a 1×1 convolutional layer and a BatchNormalization layer.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] Extensive experiments were conducted on the MNIST dataset with both IID and independent Non-IID settings, demonstrating that the Federated Avg algorithm using the ResNet18-E model achieves higher accuracy and communication efficiency. Particularly in the Non-IID data partitioning within a federated learning context, the algorithm replacing ResNet18-E achieves approximately 1.2% and 0.31% higher upper bound accuracy and approximately 1.64% and 1.46% higher average accuracy compared to FedAvg using MLP and LeNet5, respectively, with speedups of 4.85× and 1.40×, respectively. Attached Figure Description

[0027] Figure 1 This is a flowchart of the algorithm of the present invention.

[0028] Figure 2 This is a structural diagram of the ResNet18-E network.

[0029] Figure 3 This is a diagram of the BasicBlock structure.

[0030] Figure 4 The figures show the accuracy curves of three network models on the test set under two partitioning methods, IID and Non-IID, in the MNIST dataset. Among them, (a) is the test set accuracy curve under IID; (b) is the test set accuracy curve under Non-IID; (c) is the maximum accuracy that can be achieved under Non-IID; and (d) is the average accuracy after 600 rounds under Non-IID.

[0031] Figure 5 The communication time required for three network models to achieve the target accuracy is given. Among them, (a) represents the network using the MLP model and the ResNet18-E network of the present invention; and (b) represents the network using the LeNet5 model and the ResNet18-E network of the present invention.

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments. Detailed Implementation

[0033] The federated averaging algorithm based on residual networks, which offers high accuracy and efficient communication, includes the following steps: Figure 1 As shown:

[0034] Step 1: The server partitions the MNIST dataset into Independent and Identically Distributed (IID) and Independent and Non-Identically Distributed (Non-IID) subsets and distributes them to various clients. This step is used to simulate the performance of the algorithm of this invention under different scenarios. Specifically, it includes the following sub-steps:

[0035] Step 11: Divide the MNIST dataset into IIDs and distribute them to each client.

[0036] The MNIST dataset is a classic dataset in the field of machine learning. It consists of 60,000 training samples and 10,000 test samples. Each sample is a 28×28 pixel grayscale image of a handwritten digit from 0 to 9, and the number of images for these 10 digits is the same.

[0037] Specifically, this invention sets up C=100 clients for the federated averaging algorithm, distributing images evenly across each client. For the IID partitioning of the MINIST dataset, the MNIST dataset is first shuffled, and then 600 samples are randomly assigned to each client, ensuring that each sample has an equal probability of appearing on each client, i.e., an IID distribution.

[0038] Step 12: Divide the dataset into Non-IID partitions and distribute them to each client.

[0039] Specifically, the MNIST dataset is sorted according to the data labels in order from 0 to 9, then divided into 200 data slices of size 300. Finally, each user is randomly assigned two slices, resulting in 600 samples per client, but each client only has one or two of the numbers from 0 to 9. This partitioning ensures that the data are independent, but the data on each client does not conform to the same distribution, i.e., a non-identical distribution. This simulated partitioning method is more consistent with the context of federated learning, where training occurs only on the local data of each client, and the data from each client is not aggregated, thus demonstrating the independent and non-identical distribution of the data. However, the experiment still assumes a specific IID distribution for the data, indicating that the sample points used for training have good overall representativeness, which can significantly reduce the possibility of isolated cases in the training samples.

[0040] Step 2: Build the residual network model locally on each client.

[0041] This invention selects the Residual Network (ResNet) model, which performs exceptionally well in image classification tasks. The authors provide ResNet structures of varying depths to suit different datasets. This is because ResNet is simple to implement and allows for flexible modification of network parameters or adjustment of the number of layers to adapt to diverse datasets. Furthermore, the residual structure in ResNet not only reduces the amount of knowledge that needs to be learned but also achieves higher classification accuracy and faster convergence speed while increasing the number of network layers.

[0042] The original FedAvg local update uses MLP and ResNet5 architectures. These two network models offer significant potential for improvement in both accuracy and communication efficiency in federated learning. This invention replaces the local update network model with the superior ResNet architecture to simultaneously improve both accuracy and communication efficiency. Specifically, it includes the following sub-steps:

[0043] Step 21: Select ResNet18 as the backbone network of the residual network model.

[0044] ResNet includes different depths such as ResNet18, ResNet34, and ResNet50. Due to the small and simple nature of the selected baseline dataset, ResNet18 is suitable; using a deeper structure can easily lead to overfitting and degrade classification performance. Testing showed that directly replacing the ResNet18 structure did not yield good results; the number of network layers increased exponentially after replacement, leading to a significant increase in training time. To find a good balance between accuracy and training efficiency, ResNet18 needs to be adjusted; the adjusted network is called ResNet18-E.

[0045] Step 22: Construct the specific network structure of ResNet18-E. It consists of 15 weighted layers, including 13 convolutional layers and 2 fully connected layers, as follows: Figure 2 As shown, the kernel size, padding, stride, etc. of each convolutional layer are given, as well as the changes in the number of image channels and size after passing through different network layers.

[0046] ResNet18-E consists of a series of interconnected 3×3 convolutional layers, convolutional group Stage 1, convolutional group Stage 2, convolutional group Stage 3, and linear layers. Among them:

[0047] ① 3×3 Convolutional Layer. This convolutional layer has a kernel size of 3×3, a stride of 1, padding of 1, and 16 output channels. The reason for this adjustment is as follows: Considering the small image size in the MNIST dataset, the 7×7 downsampling convolution and max pooling operations of the ResNet18 network are prone to losing some information. Therefore, the kernel size of the convolutional layer was changed to 3×3. Reducing the kernel size also reduces computation, thus decreasing the local update time per round. Furthermore, reducing the stride and padding size of this convolutional layer, and removing the max pooling layer, preserves as much information as possible from the original image, resulting in richer features. The data size output by the 3×3 convolutional layer is 16×28×28.

[0048] ② Convolutional groups Stage 1, Stage 2, and Stage 3. Each Stage 1, Stage 2, and Stage 3 contains two residual blocks. ResNet18-E uses a two-layer structure with residual blocks (BasicBlock), as shown below. Figure 3 It adopts the complete 3×3 convolutional layer design of VGG. Each residual block includes two 3×3 convolutional layers with the same number of channels, followed by a Batch Normalization layer and a ReLU activation function. The BasicBlock has two paths, F(x) and x. For a stacked layer structure, the feature learned when the input is x is denoted as F(x), and this path is called the residual path. Now, another branch is added, skipping the input of this residual block between the Batch Normalization layer and the ReLU activation function after the second convolutional layer. Figure 3 The jump connection on the right from x to the plus sign is called an identity mapping, or a short-circuit connection. The output of the short-circuit connection is still x. Therefore, the final output of BasicBlock is H(x) = F(x) + x.

[0049] In this design, due to the existence of this jump connection, the two convolutional layers can only be added if their input and output dimensions are the same. However, when the input and output dimensions or the number of channels of the residual block are inconsistent, a downsampling operation needs to be added to the residual block. The downsampling structure consists of a 1×1 convolutional layer and a Batch Normalization layer.

[0050] Specifically, in Stage 1, all four convolutional layers have a kernel size of 3×3, a stride of 1, padding of 1, and 16 output channels. Stage 1 does not change the input data size or the number of channels, therefore no downsampling structure is needed, and the output data size of Stage 1 is 16×28×28. In Stage 2, all four convolutional layers have 32 channels; only the stride of the first convolutional layer is changed to 2, while other settings remain unchanged. The stride of 2 halves the data size, requiring the addition of a downsampling structure to the first residual block of Stage 2 (see...). Figure 2 The first dashed arc on the left in the middle), the output data size of Stage 2 is 32×14×14. Stage 3 only changes the number of channels in the convolutional layer to 64, and the other settings are the same as Stage 2. It also needs to go through a downsampling structure (that is, a downsampling structure is added to the first residual block of Stage 2, see...). Figure 2 The second dashed arc on the left in the middle), the output data size of Stage 3 is 64×7×7. The 7×7 feature map space has very little information, and even using a 3×3 convolution kernel will not extract any effective features, so Stage 4 of ResNet18 is removed, which also reduces network complexity and computation. Figure 2 The solid arc in the text is represented by... Figure 3 The BasicBlock structure shown is a short-circuit connection. Figure 2 The residual blocks corresponding to the dashed arcs in the diagram also all adopted... Figure 3 The BasicBlock structure shown in the diagram, which has a short-circuit connection, only adds one downsampling operation compared to the solid arc.

[0051] ③ Linear Layers. Finally, two sequentially connected linear layers (FC) are used to complete the 10-class classification task.

[0052] Step 3: Set the number of communication rounds between the server and the client to R = 1000. In each round of communication, the server randomly selects K clients from all clients C to participate in the round of communication (K≤100, preferably 10) and sends the parameters of the global model stored in the selected clients.

[0053] Step 4: The selected client downloads the parameters of the global model and performs local machine learning training within a preset number of local iterations. During the training process, the network loss is calculated using the cross-entropy loss function, the gradient is calculated using backpropagation, and finally the local model accuracy is continuously updated using the stochastic gradient descent algorithm to obtain the corresponding trained model parameters.

[0054] Specifically, for machine learning problems, the loss for the i-th sample can be expressed as f i (w)=l(x i ,y i ;w), interpreted as using model parameters w on (xi ,y i The loss is predicted using x. i Data collected for end users, y i For x i The tags. K clients also correspond to K partitions of data. P k It is the index set of data points on client k, n k =|P k |,n k Let be the number of samples on client k, and n be the total number of samples from the K selected clients in this round. Therefore, the weighted average loss function for the selected clients under the current parameters is:

[0055]

[0056] Among them, F k (w) is the average loss of all data within a client, expressed as:

[0057]

[0058] With the learning rate fixed at η, the current model w is computed on each of the selected K clients. t Average gradient on local client data: g k =ΔF k (w t K clients perform stochastic gradient descent with local iterations, i.e.:

[0059]

[0060] An equivalent update is:

[0061]

[0062] Step 5: After these K clients have finished updating, they will each upload the locally trained model parameters to the server.

[0063] Step 6: The server node receives the locally updated trained model parameters from K clients. Update the global model parameters according to the weighted average aggregation strategy:

[0064]

[0065] The updated global model parameters are then saved as the aggregation result. Steps 3-6 are then re-executed to begin a new round of global updates, continuing until the 1000th round of communication yields the final global model parameters.

[0066] To demonstrate the feasibility and effectiveness of the method of the present invention, the following experiments are provided:

[0067] A. Test set accuracy test

[0068] With a fixed proportion of clients selected (C = 0.1) per communication round, and local iteration count (E = 5) and local batch size (B = 10), the test set accuracy curves for the MNIST dataset with IID and Non-IID partitioning methods are plotted over 1000 communication rounds, as shown below. Figure 4 As shown in (a) and (b), under the IID data partitioning, ResNet18-E significantly improves accuracy compared to MLP networks. Although it doesn't appear to offer an improvement over the maximum accuracy achievable by LetNet5, the replacement achieves higher accuracy in fewer communication rounds. Under the Non-IID data partitioning, the accuracy curve of the FedAvg algorithm using LeNet5 fluctuates greatly on the test set. In some communication rounds, the accuracy drops below 95%, and in others, it even falls below 90%. Replacing it with ResNet18-E not only increases the upper limit of accuracy but also ensures that the accuracy remains stable at a high level in subsequent communication rounds.

[0069] To illustrate the point more clearly, we conducted five repeated experiments. Since the accuracy increased very slowly and remained almost constant after 600 rounds, we recorded the maximum accuracy achievable by the algorithm under the three neural network structures and the average accuracy after 600 rounds, as follows: Figure 4 As shown in (c) and (d), the accuracy variances after 600 rounds were calculated to be approximately 0.2739 (MLP), 3.5593 (LeNet5), and 0.0152 (ResNet18-E), respectively. Therefore, this algorithm not only achieves an extremely high accuracy of 99.2% with Non-IID data partitioning, but also maintains a stable high accuracy of approximately 98.97% with very small variance in subsequent communication rounds.

[0070] B. Communication efficiency test

[0071] Maintaining the same C, E, and B settings as the previous experiment, communication efficiency is evaluated using the total communication time. This discussion focuses only on the Non-IID data partitioning method in the context of federated learning, and subsequent experiments will follow the same approach. Two sets of comparative experiments are set up. The first set uses the maximum accuracy achievable by the MLP as the target accuracy 1, comparing the total time required for the MLP and ResNet18-E networks to reach target accuracy 1 to evaluate the improvement of the algorithm. Similarly, the other set uses the maximum accuracy achievable by LeNet5 as the target accuracy 2, comparing the total time required for the LeNet5 and ResNet18-E networks to reach target accuracy 2. Each set is repeated five times, and the results are as follows: Figure 5 As shown in (a), it takes approximately 4.27 hours to reach the target accuracy using the MLP model, while it takes only 0.88 hours using the ResNet18-E network; and in (b), it takes approximately 5.73 hours to reach the target accuracy using the LeNet5 model, while it takes only 4.08 hours using the ResNet18-E network.

[0072] Theoretically, replacing ResNet18-E increases the communication time per round, but due to the significant reduction in the number of communication rounds required to achieve the target accuracy, the total communication time is significantly reduced in the end. Specifically, compared to the MLP model, the algorithm of this invention achieves a speedup of (4.85×) when using ResNet18-E, and a speedup of (1.40×) compared to the LeNet5 model. Therefore, the algorithm of this invention has a significant advantage in improving the communication efficiency of local updates.

Claims

1. A federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network, characterized in that, Specifically comprising the following steps: Step 1, the server divides the MNIST dataset into IID and Non-IID, and distributes to each client; Step 2, each client locally constructs a residual network model, specifically comprising the following sub-steps: Step 21, select ResNet18 as the backbone network of the residual network model; Step 22, based on the backbone network, construct the specific network structure of ResNet18-E; the ResNet18-E comprises a 3x3 convolution layer, a convolution group Stage1, Stage2, Stage3 and a linear layer connected in turn, wherein: The convolution group Stage1, Stage2, Stage3 each contains two residual blocks, each residual block includes two 3x3 convolution layers with the same number of channels, and each convolution layer is followed by a Batch Normalization layer and a ReLU activation function; and the input of the residual block is added to the Batch Normalization layer and the ReLU activation function after the second convolution layer; in addition, a down-sampling structure is added to the first residual block of Stage2 and the first residual block of Stage3 respectively; The linear layer is two linear layers connected in turn; Step 3, set the communication rounds R between the server and the client, in each round of communication, the server randomly selects K clients from all clients C to participate in this round of communication, and sends the parameters of the global model stored to the selected clients; Step 4, the selected clients download the parameters of the global model and perform local machine learning training within a predetermined number of local iterations; During training, the network loss is calculated using the cross-entropy loss function, the gradient is calculated by back propagation, and finally the local model accuracy is updated continuously using the stochastic gradient descent algorithm to obtain the corresponding trained model parameters; Step 5, when the K clients update is completed, they upload the locally trained model parameters to the server respectively; Step 6, the server node receives the local updated trained model parameters from the K clients, updates the global model parameters according to the weighted average aggregation strategy, and saves the aggregation result as the updated global model parameters, and takes the updated global model parameters as the current global model parameters, that is, re-executes steps 3-6 to start a new round of communication of global model parameter update, until the global model parameters are obtained in the Rth round of communication, as the final global model parameters.

2. The federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network according to claim 1, wherein, Step 1 specifically comprises the following sub-steps: Step 11, divide the MNIST dataset into IID and distribute to each client; Step 12, divide the dataset into Non-IID and distribute to each client.

3. The federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network according to claim 2, wherein, There are 100 clients in total.

4. The federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network of claim 3, wherein, Step 11 is: first, shuffle the MNIST dataset, then randomly allocate 600 samples to each client.

5. The federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network according to claim 3, wherein, Step 11 is: sort the MNIST dataset according to the data label in the order of 0-9, then divide it into 200 data slices with a size of 300, and finally randomly allocate two slices to each user, i.e. each client has 600 samples.

6. The federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network of claim 3, wherein, The convolution kernel size of the 3x3 convolution layer in step 22 is 3x3, the step is 1, the padding is 1, and the output channel is 16.

7. The federated averaging algorithm for high-accuracy and high-efficiency communication based on a residual network according to claim 1, wherein, In step 22, the down-sampling structure is composed of a 1x1 convolution layer and a Batch Normalization layer.

Images