A Federated Learning Acceleration Method Based on Parameter Sharing

Abstract

This invention relates to the field of federated learning technology, specifically to a method for accelerating federated learning based on parameter sharing, comprising the following steps: establishing a communication network among the participants in the federated learning; performing the traditional federated learning process, distributing the training task to each client, and the client starting training; after a client finishes training, encrypting the training parameters of the client using the public keys of the other clients; broadcasting the parameters through the established communication network; the beneficial effect is that the model parameters of the client that has completed training first in the horizontal federated learning task are broadcast to other clients through the encrypted communication network. After receiving the parameters, the other clients verify their effectiveness, compare them with their current effectiveness to determine whether to replace the parameters, and continue training for the remaining rounds.

Description

Technical Field

[0001] This invention relates to the field of federated learning technology, specifically to a method for accelerating federated learning based on parameter sharing. Background Technology

[0002] In recent years, with the increasing informatization and automation levels across numerous industries and the introduction of the Digital China policy, the amount of data generated by society has exploded. However, some data, due to its involvement in personal privacy or industry secrets, can only be stored and used within certain domains, resulting in numerous data silos. These silos cannot communicate with each other, hindering the full mining and utilization of data. How to fully realize the value of data while complying with regulations, without compromising user data privacy and security, and enabling data to truly play its significant role, has become a research topic in recent years. To address this issue, privacy computing has emerged.

[0003] In existing technologies, privacy computing is an interdisciplinary technology that encompasses many disciplines. Federated learning, as a mainstream sub-technology, has received significant attention from traditional security vendors and artificial intelligence vendors both domestically and internationally. Compared to traditional machine learning that uses a centralized approach, Federated Learning (FL) enables multi-party collaborative learning and training without leaving the local original data database. This is achieved through the circulation and processing of intermediate encrypted data, truly realizing the goal of "the model moves while the data remains stationary."

[0004] However, current federated learning frameworks require most clients to complete the current training round and upload the result parameters before subsequent parameter calculations such as FedAvg can be performed. However, due to the uneven performance of key indicators affecting model training speed, such as the hardware facilities, training optimization level, and data scale of each participant, the "barrel effect" is extremely obvious, resulting in low overall federated learning efficiency. Under these circumstances, how to improve the training speed of the "weakest" participants and improve the overall efficiency of federated learning has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a federated learning acceleration method based on parameter sharing to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a federated learning acceleration method based on parameter sharing, the federated learning acceleration method comprising the following steps:

[0007] Step 101: Establish a communication network among the participants in federated learning;

[0008] Step 102: Perform the traditional federated learning process, distribute the training tasks to each client, and the client begins training;

[0009] Step 103: After a client that has completed training appears, encrypt the training parameters of the client after training using the public keys of the other clients.

[0010] Step 104: Broadcast the parameters through the communication network established in Step 101;

[0011] Step 105: After receiving the encrypted training parameters, the client whose training has not yet ended decrypts them using its private key;

[0012] Step 106: Clients that have not yet finished training create a new training process, load the received parameters into the model, and use the local validation set to verify the effect.

[0013] Step 107: If the model using the parameters performs better than the model currently being trained, proceed to step 108; otherwise, proceed to step 109.

[0014] Step 108: Clients whose training has not yet finished will replace the parameters in the model that is being trained;

[0015] Step 109: Clients whose training has not yet ended continue to complete the remaining training rounds;

[0016] Step 110: After all clients have completed training, continue with the parameter aggregation task and continue using steps 103 to 109 in subsequent loops until the final aggregated model achieves the desired effect.

[0017] Preferably, to prevent parameter leakage, the communication network uses asymmetric encryption methods for encryption and decryption.

[0018] Preferably, after a client completes training, the training parameters f_param are encrypted using the public keys puk_n of the other clients and transmitted through the configured communication network.

[0019] Preferably, after receiving the encrypted training parameters, the client that has not yet finished training decrypts them using the private key pk_i, then creates a new training process, loads the received parameters param_f into the model, and uses the local validation set to verify the effect and obtain the accuracy acc_f. At the same time, the same validation set is used to verify the local model and obtain the accuracy acc_s.

[0020] Preferably, if the model using the parameters performs better than the currently trained model (i.e., acc_f > acc_s), then the parameters are replaced in the currently trained model, and the remaining training rounds are continued. If the performance is worse than the current model, then it is ignored.

[0021] Preferably, the communication network reuses the original federated learning communication network, uploads the parameters to the parameter server and then distributes them to each client, or builds a dedicated parameter broadcast communication network between clients, specifically using RabbitMQ for communication.

[0022] Preferably, the asymmetric encryption method uses public-key private-key encryption. Each participant stores the public key information of other participants. The sender uses the public keys of other participants to encrypt the data before sending it. The data receiver decrypts the information using its own private key after receiving it.

[0023] Preferably, traditional federated learning is based on the FedAvg horizontal federated learning framework. After each participant completes its training, it uploads its parameters to the parameter server. The parameter server performs a weighted average of the parameters uploaded by each participant to obtain the fused parameters, and then distributes the fused parameters to each participant to continue training. The effectiveness is verified by using a validation set to evaluate the model and obtain the model accuracy.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] This invention proposes a parameter-sharing-based accelerated federated learning method. In a horizontal federated learning task, the model parameters of the client that completes training first are broadcast to other clients via an encrypted communication network. Upon receiving these parameters, other clients verify their effectiveness, compare them with their own current performance to determine whether to replace the parameters, and continue training for the remaining rounds. This invention improves the federated learning process by increasing the training efficiency of slower clients through parameter sharing, thus alleviating the low efficiency and poor performance of federated learning caused by the "weakest link" effect. It can also improve the model training performance of clients with limited training data, thereby enhancing the overall performance of the federated learning model. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the structure of the present invention;

[0027] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1

[0030] Please see Figures 1 to 2 This invention provides a technical solution: a federated learning acceleration method based on parameter sharing, the federated learning acceleration method comprising the following steps:

[0031] Step 101: Establish a communication network among the participants in the federated learning process. This network uses asymmetric encryption to prevent parameter leakage.

[0032] Step 102: Perform the traditional federated learning process, distribute the training tasks to each client, and the client begins training;

[0033] Step 103: After a client finishes training, its training parameters are encrypted using the public keys of the other clients.

[0034] Step 104: Broadcast the parameters through the communication network established in Step 101;

[0035] Step 105: The client, whose training has not yet ended, receives the encrypted training parameters and decrypts them using its private key.

[0036] Step 106: The client that has not yet finished training creates a new training process, loads the received parameters into the model, and uses the local validation set to verify the effect.

[0037] Step 107: If the model using this parameter performs better than the model currently being trained, proceed to step 108; otherwise, proceed to step 109.

[0038] Step 108: The client whose training has not yet ended replaces the parameter in the model that is being trained;

[0039] Step 109: Clients whose training has not yet ended continue to complete the remaining training rounds;

[0040] Step 110: After all clients have completed training, continue with tasks such as parameter aggregation, and continue using steps 103 to 109 in subsequent loops until the final aggregated model achieves the desired effect.

[0041] Example 2

[0042] Building upon Example 1, a parameter-sharing-based method for accelerating federated learning first establishes a communication network among the participants. This network uses asymmetric encryption to prevent parameter leakage. Then, the traditional federated learning process is performed, distributing training tasks to each client, which begins training. Once a client finishes training, it encrypts its training parameters f_param using the public keys puk_n of the other clients and transmits them through the previously established communication network. Clients whose training is not yet complete receive the encrypted training parameters, decrypt them using their private keys pk_i, create a new training process, load the received parameters param_f into the model, and validate the model using a local validation set to obtain accuracy acc_f. Simultaneously, the same validation set is used to validate the local model to obtain accuracy acc_s. If the model using these parameters outperforms the currently trained model (acc_f > acc_s), the parameter is replaced in the currently trained model, and the remaining training rounds continue. If the performance is worse than the current model, the parameter is ignored. After all clients have completed training, continue with tasks such as parameter aggregation, and continue using the above methods in subsequent loops until the final aggregated model achieves the desired effect.

[0043] The communication network can reuse the original federated learning network, uploading parameters to the parameter server and then distributing them to each client. Alternatively, a dedicated parameter broadcast communication network can be built between clients, specifically using RabbitMQ for communication. The asymmetric encryption method employs public-key encryption, where each participant stores the public key information of other participants. The sender uses the public keys of other participants to encrypt the data before sending it, and the data receiver decrypts the information using its own private key. The traditional federated learning is a horizontal federated learning framework based on FedAvg. After each participant completes its training, it uploads its parameters to the parameter server. The parameter server performs a weighted average of the parameters uploaded by each participant to obtain fused parameters, which are then distributed to each participant for continued training. The performance verification involves using a validation set to evaluate the model and obtain its accuracy.

[0044] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A federated learning acceleration method based on parameter sharing, characterized in that: The federated learning acceleration method includes the following steps: Step 101: Establish a communication network among the participants in federated learning; Step 102: Perform the traditional federated learning process, distribute the training tasks to each client, and the client begins training; Step 103: After a client that has completed training appears, encrypt the training parameters of the client after training using the public keys of the other clients. Step 104: Broadcast the training parameters through the communication network established in Step 101; Step 105: After receiving the encrypted training parameters, the client whose training has not yet ended decrypts them using its private key; Step 106: Clients that have not yet finished training create a new training process, load the received training parameters into the model, and use the local validation set to verify the effect. Step 107: If the model using the parameters performs better than the model currently being trained, proceed to step 108; otherwise, proceed to step 109. Step 108: Clients whose training has not yet finished will replace the training parameters in the model that is being trained; Step 109: Clients whose training has not yet ended continue to complete the remaining training rounds; Step 110: After all clients have completed training, continue with the parameter aggregation task and continue using steps 103 to 109 in subsequent loops until the final aggregated model achieves the desired effect.

2. The federated learning acceleration method based on parameter sharing according to claim 1, characterized in that: To prevent parameter leakage, communication networks use asymmetric encryption methods for encryption and decryption.

3. The federated learning acceleration method based on parameter sharing according to claim 1, characterized in that: Once a client completes training, the training parameters are encrypted using the public keys puk_n of the other clients and transmitted through the configured communication network.

4. The federated learning acceleration method based on parameter sharing according to claim 3, characterized in that: After receiving the encrypted training parameters, the client that has not yet finished training decrypts them using the private key pk_i, then creates a new training process, loads the received parameters into the model, and uses the local validation set to verify the effect and obtain the accuracy acc_f. At the same time, the same validation set is used to verify the local model and obtain the accuracy acc_s.

5. The federated learning acceleration method based on parameter sharing according to claim 4, characterized in that: If the model using the training parameters performs better than the currently trained model (i.e., acc_f > acc_s), then the parameters are replaced in the currently trained model, and the remaining training rounds are completed. If the model using the training parameters performs worse than the currently trained model, then it is ignored.

6. The federated learning acceleration method based on parameter sharing according to claim 1, characterized in that: The communication network reuses the original federated learning communication network. After the parameters are uploaded to the parameter server, they are uniformly distributed to each client, or a dedicated parameter broadcast communication network is built between clients. Specifically, RabbitMQ is used for communication.

7. The federated learning acceleration method based on parameter sharing according to claim 1, characterized in that: Asymmetric encryption methods employ public-key and private-key encryption. Each participant stores the public key information of the other participants. The sender uses the public keys of the other participants to encrypt the data before sending it. The data receiver decrypts the information using its own private key after receiving it.

8. The federated learning acceleration method based on parameter sharing according to claim 1, characterized in that: Traditional federated learning is based on the FedAvg horizontal federated learning framework. After each participant completes its training, it uploads its parameters to the parameter server. The parameter server performs a weighted average of the parameters uploaded by each participant to obtain the fused parameters, and then distributes the fused parameters to each participant to continue training. The effectiveness is verified by using a validation set to evaluate the model and obtain the model accuracy.

Images