Communication-efficient federated learning method based on compressed sensing

Abstract

The present invention provides a communication-efficient federated learning method based on compressed sensing, which can be summarized as follows: first, dictionary learning is performed using a quasi-validation dataset held by the server to learn a sparse representation of the model parameters; then, an adaptive compression ratio selection algorithm is used to determine the appropriate compression ratio based on the model training loss; finally, the linearity of compression is exploited on the server side to reduce the computational cost of global model recovery from n times to one execution of the reconstruction algorithm; in addition, the computational cost of compression is reduced through layered compression. With the present invention, non-sparse model parameters can be effectively compressed and accurately reconstructed, supporting not only uplink and downlink compression but also reducing overall communication costs without compromising test accuracy. Experiments on three image classification tasks demonstrate that our method consistently outperforms existing methods, achieving high reconstruction accuracy even when using a small quasi-validation dataset to learn the sparse dictionary.

Description

Technical Field

[0001] The present invention relates to a communication efficient federated learning method, and in particular to a communication efficient federated learning method based on compressed sensing Background Art

[0002] Federated learning, a distributed deep learning framework, not only enables multiple parties to collaborate on training a target model, but also eliminates the traditional deep learning concept of centralized server-side data sets, inherently protecting user privacy. Each distributed client (distributed node) only needs to store its own private dataset locally, without having to upload it to the server. After training the model locally, the client uploads the data to the server. The central server receives the uploaded data and aggregates it. The central server then sends the aggregated results to the client. Therefore, federated learning effectively protects client privacy compared to traditional centralized training methods.

[0003] However, federated learning training also brings a series of problems, one of the main ones being the communication cost and training time delay caused by the transmission of model updates. Specifically, for modern architectures with millions of parameters (which can be gigabytes in size), the total communication volume per client can easily exceed PB during hundreds of thousands of training iterations on large datasets. In addition, unreliable network conditions on the client side can cause severe delays. In other words, under limited bandwidth and unstable network transmission conditions, large-scale communication can significantly degrade the performance of federated learning. Therefore, it is crucial to find a federated learning method with high communication efficiency.

[0004] Existing methods for reducing communication costs in federated learning fall into two main categories: The first involves reducing the number of communication rounds, where participating clients perform a large number of local updates and periodically synchronize them through a parameter server. This approach can reduce both upstream and downstream communication, but performs poorly when the data is non-IID. The second approach involves reducing the size of each communication round, where clients transmit compressed data to a central server through sparsification, quantization, or a combination of both, to reduce the size of updates transmitted during each communication round. However, this approach fails to simultaneously meet the following three conditions: reducing the total number of communications, reducing the communication cost of each round, and supporting downlink compression. Summary of the Invention

[0005] The purpose of this invention is to further reduce the communication cost of each round based on FedAvg, so as to ultimately improve the performance of federated learning training.

[0006] In order to achieve the above object, the technical solution of the present invention is to provide a communication efficient federated learning method based on compressed sensing, which is characterized by comprising the following steps:

[0007] Step 1: The server uses the quasi-validation set X to learn the sparse dictionary Ψ locally, including the following steps:

[0008] Step 101: Select the quasi-validation set X as training data to train the global model G, and save the model parameters during the training of the global model;

[0009] Step 102 : Setting parameter values, including the length and number of atoms of the initial dictionary, the maximum number of linear combination atoms used in the sparse representation of the signal to be decomposed, the number of iterations of the K-SVD algorithm, and the number of atoms in the sample signal set;

[0010] Step 103: Based on the initial dictionary atom length N, select K atoms from the saved model parameters to form an initial dictionary Ψ, and select N atoms to form a sparse representation sample set S. Then, a decompression algorithm is used to obtain a sparse coefficient matrix Z under the initial dictionary of model parameters, i.e., S = Ψ × Z.

[0011] Step 104: continuously update the initial dictionary atoms according to the training iteration error of the K-SVD algorithm. If the number of iterations is reached, stop updating and construct a sparse dictionary Ψ that matches the features of the model parameters.

[0012] Step 2: The server first initializes the compression ratio δ 0 and the global model G 0 , whose parameters are determined by the n-dimensional Represented as n×n Gaussian random matrix as the measurement matrix Φ, then δ 0 、 and Φ are sent to each client; in the subsequent training phase, the server first calculates the compression ratio δ at the current t rounds based on the loss value of the global model t , and then use the sparse dictionary Ψ to obtain the global model parameters of the current t rounds The sparse representation and δ t Send to each client;

[0013] Step 3: Each client p i In obtaining and compression ratio δ t Afterwards, using its local dataset D i Iteratively run mini-batch gradient descent until the local model parameters are obtained And the local model parameters Flattened into an n×1 column vector, it is regarded as the signal to be compressed; then according to the current round compression ratio δ tThe size of the measurement matrix Φ sent by the server is extracted to form the m rows for the current round of compression The measurement matrix Φ δ , using the measurement matrix Φ δ Local model parameters To compress, Finally Send to the server;

[0014] Step 4: The server receives Afterwards, the decompression algorithm is run based on the sparse dictionary Ψ to decompress and reconstruct Then apply the weighted average aggregation protocol to obtain The server first performs local model aggregation and then uses the aggregated local model to reconstruct

[0015] Preferably, in step 2, during t rounds of training, the server calculates the average loss ι of the last θ rounds t , and calculate γ=ι t / ι 0 , where t>θ, ι 0 is the global model G after one round of training 0 The loss value is set to δ, and the new compression ratio is set to δ t =γ·δ 0 , if the model loss of the last θ round is unchanged, then further δ t Reduce to ∈·δ t ,∈ is the scaling factor.

[0016] Preferably, in step 3, the client uses the measurement matrix Φ δ Local model parameters When compressing, layered compression is used.

[0017] The present invention proposes a method based on compressed sensing, which further reduces the communication cost of federated learning on the basis of FedAvg. The present invention uses a quasi-validation set to learn a sparse representation dictionary, thereby achieving effective compression and reconstruction of non-sparse machine learning model parameters. The adaptive compression ratio adjustment technology adopted by the present invention allows the recovered model to gradually converge and achieve the approximate test accuracy of FedAvg, with a maximum compression ratio of 10 times the total compression ratio. In addition, on the server side, the present invention adopts joint model reconstruction, and on the client side, the present invention adopts hierarchical compression, which ensures low computational cost of compression. Experimental results show that the method provided by the present invention outperforms the other three methods in all three image classification tasks. The present invention can effectively compress and accurately reconstruct non-sparse model parameters, not only supports two-way communication compression, but also reduces the overall communication cost without reducing the test accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Flowchart of a communication-efficient joint learning method based on compressed sensing;

[0019] Figure 2 Flowchart for learning a sparse dictionary for the server using a quasi-validation set;

[0020] Figure 3(a) shows the convergence trend and communication cost of the Lenet5 onMNIST model on IID data;

[0021] Figure 3(b) shows the convergence trend and communication cost of the Lenet5 onF-MNIST model on IID data;

[0022] Figure 3(c) shows the convergence trend and communication cost of the CNN on CIFAR-10 model on IID data;

[0023] Figure 4(a) shows the convergence trend and communication cost of the Lenet5 onMNIST model on Non_IID (50% class) data;

[0024] Figure 4(b) shows the convergence trend and communication cost of the Lenet5 onF-MNIST model on Non_IID (50% class) data;

[0025] Figure 4(c) shows the convergence trend and communication cost of the CNN onCIFAR-10 model on Non_IID (50% class) data;

[0026] Figure 5(a) shows the convergence trend and communication cost of the Lenet5 onMNIST model on Non_IID (20% class) data;

[0027] Figure 5(b) shows the convergence trend and communication cost of the Lenet5 onF-MNIST model on Non_IID (20% classes) data;

[0028] Figure 5(c) shows the convergence trend and communication cost of the CNN on CIFAR-10 model on Non_IID (20% class) data;

[0029] Figure 6(a) shows the convergence trend of different participation rates λ on the Lenet5 onMNIST model IID data;

[0030] Figure 6(b) shows the convergence trend of different balance degrees μ on the IID data of the Lenet5 onMNIST model;

[0031] Figure 6(c) shows the convergence trend of different batch sizes b on the Lenet5 onMNIST model IID data;

[0032] Figure 7(a) Convergence curves of dictionary learning with different inputs when the quasi-validation set is 100;

[0033] Figure 7(b) Convergence curves of dictionary learning for different inputs when the quasi-validation set is 1000. DETAILED DESCRIPTION

[0034] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0035] like Figure 1 As shown, this embodiment discloses a communication-efficient federated learning method based on compressed sensing, including performing dictionary learning using a quasi-validation dataset held by a server to learn a sparse representation of model parameters; utilizing the linearity of compression on the server side to reduce the computational cost of global model recovery from n (the number of workers) to 1 execution of the reconstruction algorithm; reducing the computational cost of compression through hierarchical compression; and proposing an adaptive compression ratio selection algorithm that determines a suitable compression ratio based on the model training loss, thereby maximizing the communication compression ratio and minimizing the model loss.

[0036] Specifically, the implementation of the present invention includes the following steps:

[0037] Step 1: The server uses the quasi-validation set X to learn the N×K (K>N) sparse dictionary Ψ locally. The sparse dictionary Ψ will be used for the sparse representation of the model parameters transmitted between the server and the client, because the original W G0 It is not sparse.

[0038] Figure 2 The present invention provides a server for learning a sparse dictionary Ψ by using a quasi-validation set X. This method helps project non-sparse model parameters into a sparse domain, thereby making them sparse. The method mainly includes the following steps:

[0039] Step 101: Select the quasi-validation set X as training data to train the global model G, and save the model parameters during the training of the global model.

[0040] Step 102: Setting parameter values, including the length and number of atoms of the initial dictionary, the maximum number of linear combination atoms used in the sparse representation of the signal to be decomposed, the number of iterations of the K-SVD algorithm, and the number of atoms in the sample signal set.

[0041] Step 103: Based on the initial dictionary atom length N, select K atoms from the saved model parameters to form an initial dictionary Ψ (the size of the initial dictionary Ψ is N×K), and select N atoms to form a sparse representation sample set S (the size of the sample set S is N×N), and then obtain the sparse coefficient matrix Z under the initial dictionary of model parameters through the OMP algorithm (the size of the sparse coefficient matrix Z is K×N), that is, S=Ψ×Z.

[0042] Step 104: Continuously update the initial dictionary atoms according to the training iteration error of the K-SVD algorithm. If the number of iterations is reached, the update is stopped to construct a sparse dictionary Ψ that matches the characteristics of the model parameters.

[0043] Step 2: Initialization phase:

[0044] The server first initializes the compression ratio δ 0 and the global model G 0 , whose parameters are determined by the n-dimensional Represented as n×n Gaussian random matrix as the measurement matrix Φ, then δ 0 、 and Φ are sent to each client. In the subsequent training phase, the server first calculates the compression ratio δ at the current t rounds based on the loss value of the global model t , which means that the global model parameters of the current t rounds are The compressed version of is multidimensional (m = n / δ t ), then and δ t Distributed to each client.

[0045] Step 2 specifically includes the following steps:

[0046] Step 201: Determine relevant parameters, including the maximum training round θ with a constant compression ratio, the initial compression ratio δ used before the training round θ 0 , and scaling factor ∈(∈<1);

[0047] Step 202: Record the global model G after one round of training 0 The loss value ι 0 ;

[0048] Step 203: In t rounds of training (t>θ), the server calculates the average loss ι of the last θ rounds t , and calculate γ=ι t / ι 0 , where ι t Decreases with t and is non-negative, which means 0≤γ≤1. The server sets the new compression ratio to δ t =γ·δ 0 ;

[0049] If the server observes that the model loss for the last θ rounds is unchanged, which means that the most recent model is converged or has a large reconstruction error, then the server will δ t Reduce to ∈·δ t , attempting to further improve the model accuracy.

[0050] Step 3: Each client p i In obtaining and compression ratio δ t Afterwards, using its local dataset D i Iteratively run mini-batch gradient descent (MBGD) until a local model is obtained and will Flattened into an n×1 column vector, it is regarded as the signal to be compressed. Then according to the current round compression ratio δ t The size of the measurement matrix Φ sent by the server is extracted (according to the public selection algorithm, for example, the first m rows are selected) to form the matrix for the current round of compression. The measurement matrix Φ δ ,Right now Finally Sent to the server.

[0051] Step 4: The server receives After that, run the decompression algorithm (such as OMP) to decompress and reconstruct Then apply the weighted average aggregation protocol to obtain However, reconstructing the local models one by one on the server side would result in too much computational cost. Therefore, this paper uses a joint reconstruction method, in which the server only needs to execute the reconstruction algorithm once in each iteration. The theory is as follows:

[0052] For any m×n measurement matrix Φ, two n×1 signals W i and W j and the constant a i and a j have:

[0053]

[0054] This equation suggests that the server can perform weighted averaging before restoring the global model.

[0055] According to the equation Among them, D t refers to the total number of training samples in the tth round, that is,

[0056] Equation Established, where w i =|D i | / t , so it can be concluded that

[0057] Therefore, the server can first perform local model aggregation and then use the aggregated (compressed) local model to rebuild

[0058] That is, because reconstruction algorithms are typically computationally expensive, for example, OMP has a computational complexity of O(mns). Therefore, in a training round with l clients, the computational cost of recovering all local models is l·O(mns). In large-scale federated systems, it is desirable to reduce the computational complexity of reconstruction, and we can reduce it from l·O(mns) to O(mns). In other words, for each update of the global model, regardless of the number of local models, the servers jointly rebuild the model parameters once.

[0059] Step 5: Similarly, the above compression theory can be applied to downlink compression (according to the above compression method, downlink compression can be applied), where the server uses Φ δ compression Client (staff) uses Φ δ Refactoring This can be obtained from the server during the initialization step and further reduced in compression cost based on layered compression.

[0060] For example, consider a k-layer model with n parameters. Without loss of generality, we assume that each layer is of equal size and is compressed to δ = m / n, which means that the measurement matrix Φ is m × n. Then the computational complexity of compressing the model as a whole is O(n·m). In contrast, if we compress layer by layer, the size of Φ becomes Then the compression cost can be reduced to Therefore, we conclude that layered compression reduces the computation cost to 1 / k of its original cost.

[0061] The verification process of the present invention is as follows:

[0062] In this embodiment, a real learning scenario in federated learning is simulated on the distributed library of PyTorch. The experiment uses MNIST and Fashion MNIST to train the Lenet5 model, and uses CIFAR10 to train the CNN model. The total number of customers is 100, and the participation rate of customers in each round is 0.1. The present invention is compared with the FedAvg, FedPAQ and T-FedAvg methods in terms of model accuracy and communication cost, and experiments are conducted to see how different parameters in the present invention affect the performance of the model. Table I summarizes the hyperparameters and optimizers used in each learning task, and Table II summarizes other relevant parameters of the method in this article. The method of the present invention does not compress the first convolutional layers of the Lenet5 and CNN models because they are relatively small.

[0063]

[0064] TABLE I Optimizers and Hyperparameters

[0065]

[0066] Table II Other relevant parameters of this method

[0067] In this implementation, the proposed method was compared with FedAvg, FedPAQ, and T-FedAvg in terms of model accuracy and communication cost. Using IID (each customer holds all classes in the training data) and Non-IID (each customer holds 50% and 20% of the classes in the training data) data, the proposed method experimentally compared the communication cost and performance (i.e., test accuracy) of the four methods.

[0068] Figure 3 (IID) plots the convergence curves for the three learning tasks with IID data distribution. Figures 4 (Non_IID, 50% of classes) and 5 (Non_IID, 20% of classes) plot the convergence curves for the three learning tasks with Non_IID data distribution, respectively. These curves depict the average number of bits a client needs to upload to achieve a certain test accuracy. These results show that our method outperforms the other three methods in all cases. In the Lenet5 on MNIST task, regardless of data distribution, our method's final communication cost is only 1 / 10, 1 / 3, and 1 / 4 of that of FedAvg, FedPAQ, and T-FedAvg, respectively. In the Lenet5 on F-MNIST task, we find that on Non_IID data, our method and FedAvg maintain the same final test accuracy as on IID data, but FedPAQ and T-FedAvg suffer varying degrees of degradation. The greater the difference in the client's data distribution, the greater the decline in accuracy. In the CNN on CIFAR-10 task, on the Non_IID data, although the accuracy of all methods decreases, our method always shows a similar final accuracy to FedAvg, while the final accuracy of the other two methods is significantly lower than FedAvg. For example, on the IID data, the final accuracy of FedAvg is about 55%, while the final accuracy of T-FedAvg is 40%. In addition, our method can save 35%-50% of the communication cost.

[0069] The experiments also studied the effects of the following parameters on the accuracy of the proposed model. The following experiments were conducted on the Lenet5 on MNIST task using IID data.

[0070] 1) The impact of participation rate λ

[0071] In our experiments, we studied the impact of participation rate λ on performance. Given λ, we randomly selected λ·100 customers in each training round. Figure 6(a) depicts the test accuracy achieved by our method during training with different participation rates λ (0.1, 0.3, 0.5, 0.7). We found that our method is relatively robust to changes in λ. Although the learning rate of our method is slightly lower when λ=0.1, the final test accuracy is the same under different λ. This means that fewer participants can be involved in the federated learning task, which can save a lot of communication costs in the federated learning task. In addition, as λ increases, the performance fluctuation decreases slightly.

[0072] 2) The impact of data size imbalance μ

[0073] All the experiments we conducted above used a balanced split of the dataset, with all customers having the same number of samples. In this experiment, we studied the impact of data size imbalance on model performance in our approach. The degree of imbalance is defined as follows:

[0074]

[0075] Among them S N ={|D1|, .., |D N |}, μ = 0.1 indicates that most samples are stored by a small number of customers, while μ = 1 indicates that all customers hold an equal number of samples. We set μ to 0.1, 0.3, 0.5, 0.7, and 1. Figure 6(b) depicts the test accuracy achieved by our method during training for different degrees of data size imbalance μ (0.1, 0.3, 0.5, 0.7, 1). It can be seen that the imbalance does not significantly affect model convergence. We speculate that this is because the local model is able to correctly learn IID data even if the data is unevenly distributed across customers.

[0076] 3) Impact of batch size b

[0077] In our experiments, we studied how the batch size b affects the performance of our method. Setting λ = 0.1 and μ = 1, Figure 6(c) depicts the test accuracy achieved by our method during training with different batch sizes (32, 64, 100, 128). In general, appropriately increasing the batch size will make the model converge faster and smoother. We observed that increasing the batch size will cause the model to converge slower and show larger fluctuations. We examined the changes in the compression ratio during training and found that the compression ratio decreases more slowly as the batch size increases. We speculate that this leads to relatively low reconstruction accuracy in the early stages of federated training, and therefore relatively slow model convergence and large fluctuations.

[0078] 4) The impact of generated input on dictionary learning

[0079] In our experiments, we investigate how the input to dictionary learning affects the performance of our method. We generate quasi-validation sets of different sizes (100 and 1000 samples) and set ζ to 100 and 200. In addition, we employ three strategies for selecting the intermediate ζ model, namely, selecting the top ζ model, the last ζ model, and the average selection of the ζ models from the 1000 models. Figure 7 illustrates the test accuracy achieved during training of our method under different settings. We find that for a large quasi-validation set (1000 samples), our method is more robust, such that ζ and the selection strategy do not significantly affect its convergence. For a quasi-validation set with 100 samples, it can be seen that a larger ζ leads to faster convergence, and selecting the last ζ model is the best selection strategy. Importantly, the experimental results show that even with a small quasi-validation set, our method exhibits satisfactory performance in terms of test accuracy and convergence speed.

Claims

1. A communication-efficient federated learning method based on compressed sensing for image classification tasks, characterized by: The following steps are involved: Step 1: The server uses the quasi-validation set X to learn the sparse dictionary Ψ locally, including the following steps: Step 101: Select the quasi-validation set X as training data to train the global model G, and save the model parameters during the training of the global model; Step 102 : Setting parameter values, including the length and number of atoms of the initial dictionary, the maximum number of linear combination atoms used in the sparse representation of the signal to be decomposed, the number of iterations of the K-SVD algorithm, and the number of atoms in the sample signal set; Step 103: Based on the initial dictionary atom length N, select K atoms from the saved model parameters to form an initial dictionary Ψ, and select N atoms to form a sparse representation sample set S. Then, a decompression algorithm is used to obtain a sparse coefficient matrix Z under the initial dictionary of model parameters, i.e., S = Ψ × Z. Step 104: continuously update the initial dictionary atoms according to the training iteration error of the K-SVD algorithm. If the number of iterations is reached, stop updating and construct a sparse dictionary Ψ that matches the features of the model parameters. Step 2: The server first initializes the compression ratio δ 0 and the global model G 0 , whose parameters are determined by the n-dimensional Represented as n×n Gaussian random matrix as the measurement matrix Φ, then δ 0 、 and Φ are sent to each client; in the subsequent training phase, the server first calculates the compression ratio δ at the current t rounds based on the loss value of the global model t , and then use the sparse dictionary Ψ to obtain the global model parameters of the current t rounds The sparse representation and δ t Send to each client; Step 3: Each client p i In obtaining and compression ratio δ t Afterwards, using its local dataset D i Iteratively run mini-batch gradient descent until the local model parameters are obtained And the local model parameters Flattened into an n×1 column vector, it is regarded as the signal to be compressed; then according to the current round compression ratio δ t The size of the server is then extracted from the m rows of the measurement matrix Φ sent by the server to form the current round of compression The measurement matrix Φ δ , using the measurement matrix Φ δ Local model parameters To compress, Finally Send to the server; Step 4: The server receives Afterwards, the decompression algorithm is run based on the sparse dictionary Ψ to decompress and reconstruct Then apply the weighted average aggregation protocol to obtain The server first performs local model aggregation and then uses the aggregated local model to reconstruct .

2. The communication-efficient federated learning method based on compressed sensing according to claim 1, characterized in that: In step 2, during t rounds of training, the server calculates the average loss ι for the last θ rounds t , and calculate γ=ι t / ι 0 , where t>θ, ι 0 is the global model G after one round of training 0 The loss value is set to δ, and the new compression ratio is set to δ t =γ·δ 0 , if the model loss of the last θ round is unchanged, then further δ t Reduce to ∈·δ t ,∈ is the scaling factor.

3. The communication-efficient federated learning method based on compressed sensing according to claim 1, characterized in that: In step 3, the client uses the measurement matrix Φ δ Local model parameters When compressing, layered compression is used.

Images