A sparse communication method and system based on federated deep learning
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GENERAL HOSPITAL OF PLA
- Filing Date
- 2023-08-16
- Publication Date
- 2026-07-21
Smart Images
Figure CN117061617B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of federated learning technology, specifically relating to a sparse communication method and system based on federated deep learning. Background Technology
[0002] Federated learning plays a crucial role in enabling privacy-preserving deep learning. The process involves communicating and updating a large number of model parameters. However, in practical applications, the communication bandwidth of federated learning is limited, making it difficult to support the transmission of sufficient model parameter data. Therefore, communication has become one of the key bottlenecks in federated learning, and researching how to reduce the communication overhead is a critical issue that urgently needs to be addressed in this field.
[0003] Federated learning requires multiple communications between the client and server. During the communication process, dealing with the massive parameters of complex neural networks is an important measure to improve the communication efficiency of federated learning. Currently, communication compression methods are mainly divided into two categories: quantization and sparsification.
[0004] Typical sparse communication algorithms, through certain transformations, compress all gradients into a few gradients with update value for transmission, achieving communication compression of hundreds or even thousands of times. Due to their excellent communication compression effect, these algorithms are currently receiving close attention in the federated learning field. However, because deep neural networks are relatively large and complex, small differences between neural network layers can be amplified by the cascaded network, especially after sparse communication, which may cause the final result of federated deep learning to diverge, making model training impossible. Summary of the Invention
[0005] This invention provides a sparse communication method and system based on federated deep learning to solve the problem that sparse communication may cause the final result of federated deep learning to diverge.
[0006] In a first aspect, the present invention provides a sparse communication method based on federated deep learning, the method comprising the following steps:
[0007] Obtain the aggregated gradients transmitted by the servers participating in federated learning, and count the transmission rounds. The aggregated gradients are the convolutional layer aggregated gradients.
[0008] Determine whether the transmission round has reached the preset fusion round;
[0009] If the transmission round reaches the fusion round, then a preset client training model is trained based on the aggregated gradient and using a preset client dataset to obtain the global gradient of the client training model;
[0010] The global gradient is uploaded to the server;
[0011] If the transmission round has not reached the fusion round, then the client training model is trained based on the aggregated gradient and using the client dataset to obtain the local gradient of the client training model;
[0012] The local gradient is sparsified using a sparse algorithm to obtain the locally sparsified gradient;
[0013] The localized sparse gradient is uploaded to the server.
[0014] Optionally, before obtaining the aggregated gradients transmitted by the servers participating in federated learning, the following steps are also included:
[0015] Obtain the preset initial model parameters from the servers participating in federated learning;
[0016] Based on the initial model parameters, a preset client training model is trained using a preset client dataset to obtain the initial client gradient;
[0017] The initial client gradient is sparsified using a sparse algorithm to obtain the initial sparsified gradient.
[0018] The initial sparsified gradient is uploaded to the server.
[0019] Optionally, the global gradient of the client-trained model includes the gradients of the convolutional layers and the gradients of the normalized layers of the client-trained model.
[0020] Optionally, the step of training the client-side training model based on the aggregated gradient and using the client-side dataset to obtain the local gradient of the client-side training model includes the following steps:
[0021] Update the convolutional layer parameters in the client-trained model based on the aggregated gradient;
[0022] Based on the updated convolutional layer parameters and using the client dataset, the client training model is trained to obtain the local gradient of the client training model, which is the gradient of the convolutional layer.
[0023] Optionally, the sparse algorithm is a sparse ternary compression algorithm.
[0024] Secondly, the present invention also provides a sparse communication system based on federated deep learning, the system comprising:
[0025] The gradient acquisition module is used to acquire the aggregated gradients transmitted by the servers participating in federated learning and to count the transmission rounds. The aggregated gradients are the aggregated gradients of the convolutional layers.
[0026] The round determination module is used to determine whether the transmission round has reached the preset fusion round.
[0027] The asynchronous fusion module, when the round determination module determines that the transmission round has reached the fusion round, is used to train a preset client training model based on the aggregated gradient and using a preset client dataset to obtain the global gradient of the client training model.
[0028] The first gradient upload module is used to upload the global gradient to the server;
[0029] The local update module, when the round determination module determines that the transmission round has not reached the fusion round, is used to train the client training model based on the aggregated gradient and using the client dataset to obtain the local gradient of the client training model.
[0030] The sparsification module is used to sparsify the local gradient using a sparsity algorithm to obtain a locally sparsified gradient.
[0031] The second gradient upload module is used to upload the local sparse gradient to the server.
[0032] Optionally, the system further includes:
[0033] The parameter acquisition module is used to obtain the preset initial model parameters from the servers participating in federated learning.
[0034] The initial training module is used to train a preset client training model based on the initial model parameters and using a preset client dataset to obtain the initial client gradient.
[0035] An initial sparsification module is used to sparsify the initial client gradient using a sparsity algorithm to obtain an initial sparsification gradient.
[0036] An initial gradient upload module is used to upload the initial sparse gradient to the server.
[0037] Optionally, the global gradient of the client-trained model includes the gradients of the convolutional layers and the gradients of the normalized layers of the client-trained model.
[0038] Optionally, the local update module includes:
[0039] A parameter update unit is used to update the convolutional layer parameters in the client-trained model based on the aggregated gradient.
[0040] A local update unit is used to train the client training model based on the updated convolutional layer parameters and the client dataset to obtain the local gradient of the client training model, wherein the local gradient is the gradient of the convolutional layer.
[0041] Optionally, the sparse algorithm is a sparse ternary compression algorithm.
[0042] The beneficial effects of this invention are:
[0043] The sparsity communication method based on federated deep learning adopted in this invention includes the following steps: obtaining the aggregated gradients transmitted by the servers participating in federated learning and counting the transmission rounds; determining whether the transmission rounds have reached a preset fusion round; if the transmission rounds have reached the fusion round, training a preset client training model based on the aggregated gradients and using a preset client dataset to obtain the global gradient of the client training model; uploading the global gradient to the server; if the transmission rounds have not reached the fusion round, training a client training model based on the aggregated gradients and using the client dataset to obtain the local gradient of the client training model; sparsifying the local gradients using a sparsity algorithm to obtain the local sparse gradient; and uploading the local sparse gradient to the server.
[0044] Through the above steps, local parameter updates can be completed during the update process before the fusion round, ensuring that the model can learn global knowledge through shared parameters trained by other clients. During the update process before the fusion round, asynchronous fusion methods can be used to ensure stable update direction and convergence for the client during model training. Compared to the fully gradient-sparse communication method used in federated learning, this approach minimizes the possibility of divergence in the final result of the federated learning model. Attached Figure Description
[0045] Figure 1 This is a flowchart illustrating the sparse communication method based on federated deep learning in this invention.
[0046] Figure 2 This is a schematic diagram illustrating the communication methods between the client and server at different stages in this invention. Detailed Implementation
[0047] This invention discloses a sparse communication method based on federated deep learning.
[0048] In the sparse communication method based on federated deep learning disclosed in this invention, the executing entity is any one or more clients participating in federated learning.
[0049] Reference Figure 1 The sparse communication method based on federated deep learning specifically includes the following steps:
[0050] S101. Obtain the aggregated gradients transmitted by the servers participating in federated learning, and count the transmission rounds.
[0051] In federated learning, both the server and client first need to initialize. The server initializes the model parameters, while the client initializes its local data. The server sends the initialized model parameters to the client. Each client trains using its local data and the received model parameters. The client calculates gradients using its local data and sends the gradients to the server. After receiving gradients from multiple clients, the server performs gradient aggregation. Common aggregation methods include average aggregation and weighted aggregation. The aggregated gradient is obtained after gradient aggregation. In this embodiment, the gradients uploaded by the clients are all convolutional layer gradients obtained by updating the convolutional layers of the client's local model. Therefore, the aggregated gradient obtained by the server is the convolutional layer aggregated gradient. At this point, the client can obtain the aggregated gradient through step S101. After updating the parameters and gradients based on the aggregated gradient, the client can upload the gradient to the server again.
[0052] Furthermore, since federated learning models require continuous iterative training, the above steps need to be repeated until the model converges or reaches the predetermined number of iterations. At this point, the number of aggregated gradient transfer rounds can be counted, with one transfer round representing one training round of the model.
[0053] S102. Determine whether the transmission round has reached the preset fusion round. If the transmission round has reached the fusion round, proceed to step S103; if the transmission round has not reached the fusion round, proceed to step S105.
[0054] S103. Based on the aggregated gradient and using the preset client dataset, train the preset client training model to obtain the global gradient of the client training model.
[0055] Here, it is assumed that the preset client training model is in and The selection matrix is represented by d = d1 + d2, where d is the dimension of the model parameter vector, and t represents the transmission round or update iteration round. Upon receiving the aggregated gradient... Then use the client dataset To train and update the client-side training model for this round, the specific formula is as follows: In the formula η t This represents the learning rate, which is usually set to 0.001.
[0056] The updated client-side training model is obtained by training using stochastic gradient descent. And calculate the client parameters for the next round and the gradients to be aggregated. The calculation formula is Since the transmission rounds have reached the fusion rounds, the global gradients of the client-trained model obtained in this round include the gradients of the convolutional layers and the normalized layers of the client-trained model.
[0057] S104. Upload the global gradient to the server.
[0058] S105. Based on the aggregated gradients and using the client dataset, train the client-side training model to obtain the local gradients of the client-side training model.
[0059] Here, it is assumed that the preset client training model is in and The selection matrix is represented by d = d1 + d2, where d is the dimension of the model parameter vector, and t represents the transmission round or update iteration round. Upon receiving the aggregated gradient... Then use the client dataset To train and update the client-side training model for this round, the specific formula is as follows: In the formula η t This represents the learning rate, which is usually set to 0.001.
[0060] The updated client-side training model is obtained by training using stochastic gradient descent. And calculate the client parameters for the next round and the gradients to be aggregated. The calculation formula is Since the transmission round has reached the fusion round, the local gradients of the client-trained model in this round of update only include the gradients of the convolutional layers. The normalized layers do not participate in the aggregation of server gradients, and the client-local normalized layer gradients are used for updating.
[0061] S106. Use a sparse algorithm to sparsify the local gradient to obtain the locally sparsified gradient.
[0062] Among them, the sparse algorithm is the sparse ternary compression algorithm, which uses the sparse algorithm to reduce the local gradient. Sparsification yields the local sparsification gradient. in This is the sparse matrix obtained by sparse algorithm.
[0063] S107. Upload the localized sparse gradient to the server.
[0064] The local sparse gradient is sent to the server so that the server can perform gradient aggregation to obtain the aggregated sparse gradient, which is: in In the formula, F represents the aggregate function, and the function f(w) = (MC) h(t-τ) g represents the local sparsified gradient sent by the client to the server, and L represents the number of clients.
[0065] The implementation principle of this method is as follows:
[0066] Reference Figure 2 Taking the DenseNet network as an example, its convolutional layers are followed by normalization layers. During the local update phase, a client-side training model is trained using the client dataset based on aggregated gradients, yielding the local gradients of the client-side training model, i.e., the convolutional layer gradients. These gradients are then sparsified using a sparse algorithm such as STC and communicated with the server. The server aggregates the convolutional layer gradients using an aggregation function. The aggregated convolutional layer gradients are returned to the client to update its original convolutional layer parameters. The normalization layers do not participate in the server's gradient aggregation; instead, they are updated using the client's local normalized layer gradients.
[0067] During the asynchronous fusion phase, i.e. the preset τth communication, a preset client training model is trained based on the aggregated gradient and using the preset client dataset to obtain the global gradient of the client training model, i.e. the gradient of the convolutional layer and the gradient of the normalized layer. The global gradient is then sent to the server, aggregated by the aggregation function, and the aggregated gradient is returned to the client to update its original parameters.
[0068] Through the above steps, local parameter updates can be completed during the update process before the fusion round, ensuring that the model can learn global knowledge through shared parameters trained by other clients. During the update process before the fusion round, asynchronous fusion methods can be used to ensure stable update direction and convergence for the client during model training. Compared to the fully gradient-sparse communication method used in federated learning, this approach minimizes the possibility of divergence in the final result of the federated learning model.
[0069] In one implementation, before step S101, which involves obtaining the aggregated gradients transmitted by the servers participating in federated learning and counting the transmission rounds, the following specific steps are included:
[0070] Obtain the preset initial model parameters from the servers participating in federated learning;
[0071] The initial client gradient is obtained by training the preset client training model based on the initial model parameters and using the preset client dataset.
[0072] The initial client gradient is sparsified using a sparse algorithm to obtain the initial sparsified gradient.
[0073] The initial sparsified gradient is uploaded to the server.
[0074] In this embodiment, the server sends the initialized model parameters to the clients. Each client trains using its local data and the received model parameters. The client calculates gradients using its local data and sends the gradients to the server. After receiving gradients from multiple clients, the server performs gradient aggregation. Common aggregation methods include average aggregation and weighted aggregation. The aggregated gradient is obtained after gradient aggregation. In this embodiment, the gradients uploaded by the clients are all convolutional layer gradients obtained by updating the convolutional layers of the client's local model; therefore, the aggregated gradient obtained by the server is the convolutional layer aggregated gradient. The sparse algorithm used is the sparse ternary compression algorithm.
[0075] In one implementation, step S105, which involves training the client-side training model based on the aggregated gradients and using the client dataset to obtain the local gradients of the client-side training model, specifically includes the following steps:
[0076] Update the convolutional layer parameters in the client-trained model based on aggregated gradients;
[0077] Based on the updated convolutional layer parameters and using the client dataset to train the client-side training model, the local gradients of the client-side training model are obtained, and the local gradients are the convolutional layer gradients.
[0078] In one embodiment, the sparsity communication method based on federated deep learning in this invention is compared with the full gradient sparsity communication method, and the comparison results are shown in Tables 1 and 2.
[0079]
[0080] Table 1. Communication results of the full gradient sparse communication method in multiple rounds.
[0081]
[0082] Table 2. Communication results of multiple rounds using the sparse communication method based on federated deep learning.
[0083] In Table 1, because the gradient was not locally updated and asynchronously fused, but directly subjected to full gradient sparsity, it can be seen that the function loss diverged in the 4th round of communication, and the function loss could not be calculated by the neural network in the 9th round of communication. In contrast, Table 2 shows that the sparsity communication method based on federated deep learning in this invention steadily reduces the function loss in 9 rounds of communication, and the communication size in any round is smaller than that in Table 1.
[0084] This invention also discloses a sparse communication system based on federated deep learning, the system comprising:
[0085] The gradient acquisition module is used to acquire the aggregated gradients transmitted by the servers participating in federated learning and to count the transmission rounds. The aggregated gradients are the aggregated gradients of the convolutional layers.
[0086] The round determination module is used to determine whether the transmission round has reached the preset fusion round.
[0087] The asynchronous fusion module, when the round determination module determines that the transmission round has reached the fusion round, is used to train a preset client training model based on the aggregated gradient and using a preset client dataset to obtain the global gradient of the client training model.
[0088] The first gradient upload module is used to upload the global gradient to the server;
[0089] The local update module is used to obtain the local gradient of the client-trained model by training the client-trained model based on the aggregated gradient and using the client dataset when the round judgment module determines that the transmission round has not reached the fusion round.
[0090] The sparsification module is used to sparsify the local gradient using a sparsity algorithm to obtain a locally sparsified gradient.
[0091] The second gradient upload module is used to upload the local sparse gradient to the server.
[0092] In this embodiment, the global gradient of the client-trained model includes the gradients of the convolutional layers and the normalized layers, and the sparse algorithm is a sparse ternary compression algorithm. The gradient acquisition module obtains the aggregated gradients transmitted by the servers participating in federated learning and counts the transmission rounds. The round determination module determines whether the transmission rounds have reached the preset fusion rounds. If the transmission rounds have reached the fusion rounds, the asynchronous fusion module trains a preset client-trained model based on the aggregated gradients and using a preset client dataset to obtain the global gradient of the client-trained model, and then uploads the global gradient to the server through the first gradient upload module.
[0093] If the transmission rounds have not reached the fusion rounds, the local update module trains the client-side training model based on the aggregated gradient and using the client dataset to obtain the local gradient of the client-side training model. The local gradient is then sparsified using a sparsity algorithm through a sparsification module to obtain the locally sparse gradient. Finally, the locally sparse gradient is uploaded to the server through the second gradient upload module.
[0094] Through the steps performed by all the modules described above, local parameter updates can be completed during the update process before the fusion round, ensuring that the model can learn global knowledge through shared parameters trained by other clients. During the update process before the fusion round, asynchronous fusion methods can be used to ensure stable update direction and convergence during model training by the client. Compared to the fully gradient-sparse communication method used in federated learning, this approach minimizes the possibility of divergence in the final result of the federated learning model.
[0095] In one embodiment, the system further includes:
[0096] The parameter acquisition module is used to obtain the preset initial model parameters from the servers participating in federated learning.
[0097] The initial training module is used to train a preset client training model based on the initial model parameters and using a preset client dataset to obtain the initial client gradient.
[0098] The initial sparsification module is used to sparsify the initial client gradient using a sparsity algorithm to obtain the initial sparsified gradient.
[0099] The initial gradient upload module is used to upload the initial sparse gradient to the server.
[0100] In this embodiment, the sparse algorithm is the sparse ternary compression algorithm.
[0101] In one implementation, the partial update module includes:
[0102] The parameter update unit is used to update the convolutional layer parameters in the client-trained model based on the aggregated gradient.
[0103] The local update unit is used to train the client-trained model based on the updated convolutional layer parameters and the client dataset, and obtain the local gradient of the client-trained model. The local gradient is the convolutional layer gradient.
[0104] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.
[0105] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.
Claims
1. A sparse communication method based on federated deep learning, characterized in that, Includes the following steps: Obtain the aggregated gradients transmitted by the servers participating in federated learning, and count the transmission rounds. The aggregated gradients are the convolutional layer aggregated gradients. Determine whether the transmission round has reached the preset fusion round; If the transmission round reaches the fusion round, then a preset client training model is trained based on the aggregated gradient and using a preset client dataset to obtain the global gradient of the client training model; The global gradient is uploaded to the server; If the transmission round has not reached the fusion round, then the client training model is trained based on the aggregated gradient and using the client dataset to obtain the local gradient of the client training model; The local gradient is sparsified using a sparse algorithm to obtain the locally sparsified gradient; The localized sparse gradient is uploaded to the server.
2. The sparse communication method based on federated deep learning according to claim 1, characterized in that, Before obtaining the aggregated gradients transmitted by the servers participating in federated learning, the following steps are also included: Obtain the preset initial model parameters from the servers participating in federated learning; Based on the initial model parameters, a preset client training model is trained using a preset client dataset to obtain the initial client gradient; The initial client gradient is sparsified using a sparse algorithm to obtain the initial sparsified gradient. The initial sparsified gradient is uploaded to the server.
3. The sparse communication method based on federated deep learning according to claim 1, characterized in that, The global gradient of the client-trained model includes the gradients of the convolutional layers and the gradients of the normalized layers.
4. The sparse communication method based on federated deep learning according to claim 3, characterized in that, The process of training the client-side training model based on the aggregated gradient and using the client-side dataset to obtain the local gradient of the client-side training model includes the following steps: Update the convolutional layer parameters in the client-trained model based on the aggregated gradient; Based on the updated convolutional layer parameters and using the client dataset, the client training model is trained to obtain the local gradient of the client training model, which is the gradient of the convolutional layer.
5. The sparse communication method based on federated deep learning according to claim 1 or 2, characterized in that, The sparse algorithm is a sparse ternary compression algorithm.
6. A sparse communication system based on federated deep learning, characterized in that, The system includes: The gradient acquisition module is used to acquire the aggregated gradients transmitted by the servers participating in federated learning and to count the transmission rounds. The aggregated gradients are the aggregated gradients of the convolutional layers. The round determination module is used to determine whether the transmission round has reached the preset fusion round. The asynchronous fusion module, when the round determination module determines that the transmission round has reached the fusion round, is used to train a preset client training model based on the aggregated gradient and using a preset client dataset to obtain the global gradient of the client training model. The first gradient upload module is used to upload the global gradient to the server; The local update module, when the round determination module determines that the transmission round has not reached the fusion round, is used to train the client training model based on the aggregated gradient and using the client dataset to obtain the local gradient of the client training model. The sparsification module is used to sparsify the local gradient using a sparsity algorithm to obtain a locally sparsified gradient. The second gradient upload module is used to upload the local sparse gradient to the server.
7. The sparse communication system based on federated deep learning according to claim 6, characterized in that, The system also includes: The parameter acquisition module is used to obtain the preset initial model parameters from the servers participating in federated learning. The initial training module is used to train a preset client training model based on the initial model parameters and using a preset client dataset to obtain the initial client gradient. An initial sparsification module is used to sparsify the initial client gradient using a sparsity algorithm to obtain an initial sparsification gradient. An initial gradient upload module is used to upload the initial sparse gradient to the server.
8. The sparse communication system based on federated deep learning according to claim 6, characterized in that, The global gradient of the client-trained model includes the gradients of the convolutional layers and the gradients of the normalized layers.
9. The sparse communication system based on federated deep learning according to claim 8, characterized in that, The partial update module includes: A parameter update unit is used to update the convolutional layer parameters in the client-trained model based on the aggregated gradient. A local update unit is used to train the client training model based on the updated convolutional layer parameters and the client dataset to obtain the local gradient of the client training model, wherein the local gradient is the gradient of the convolutional layer.
10. The sparse communication system based on federated deep learning according to claim 6 or 7, characterized in that, The sparse algorithm is a sparse ternary compression algorithm.