A federated learning method and a federated learning system
By uploading some model parameters in advance during the training process on edge devices, the problems of resource waste and bandwidth competition in federated learning are solved, and the model can achieve fast convergence and efficient training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU UNIV
- Filing Date
- 2023-04-14
- Publication Date
- 2026-06-12
AI Technical Summary
In existing federated learning protocols, there is a serious waste of computing and communication resources, the model convergence time is prolonged, and bandwidth contention is severe, resulting in underutilization of resources.
During the training process of edge devices, some model parameters are uploaded in advance using idle channels. Stable or highly variable parameters are selected through pre-upload rules to reduce bandwidth contention and improve model convergence efficiency.
This reduces the number of model parameters uploaded to edge devices, lowers upload time, and ensures model training accuracy and efficiency.
Smart Images
Figure CN116489142B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of distributed machine learning, specifically to a federated learning method and a federated learning system. Background Technology
[0002] The continuous development of science and technology has brought us increasingly powerful devices, such as smartphones, tablets, and smartwatches. While providing us with numerous conveniences, these devices also collect vast amounts of data. On the other hand, artificial intelligence technology is also developing rapidly. Deep learning, as one of the most important technologies, requires a massive amount of data as its foundation, and the data from these smart devices is undoubtedly very attractive. However, traditional machine learning involves collecting data and then training it centrally, posing a significant threat to the privacy of this information. Therefore, federated learning was born. Federated learning is a distributed machine learning technology. Unlike previous distributed machine learning technologies, federated learning does not collect user data. Instead, it keeps this data locally, allowing user devices to train machine learning models in-place and then upload the trained models to a server. In this way, the data does not leave the local device, ensuring the user's data privacy and security. At the same time, only the model parameters need to be transmitted, greatly reducing the communication burden.
[0003] Current research on federated learning mainly falls into two categories: privacy and security, and model learning efficiency. The efficiency aspect focuses on two main areas: resource scheduling and model learning efficiency. One focuses on resource scheduling, as mobile devices typically have limited resources such as battery, network, and computing power. Therefore, meticulous task allocation is crucial for federated learning tasks. The other focuses on addressing heterogeneity issues, including data heterogeneity, system heterogeneity, and model heterogeneity. Different devices, due to their different environments, often have access to different data. Training directly on this non-independent and identically distributed data can impair model performance. Specifically, in existing federated learning protocols, for each selected device, a training round involves three discontinuous processes: downloading, training, and uploading. Each stage waits for the previous stage to complete. This wastes the computing and communication resources of the selected devices, preventing their full utilization and prolonging model convergence time. Furthermore, since all selected devices submit their uploads after training in a round, the limited uplink bandwidth leads to severe resource contention due to a large number of simultaneous communications. Summary of the Invention
[0004] To address the aforementioned problems, the purpose of this disclosure is to provide a federated learning method and system that uses idle channels to upload some model parameters in advance during local training, thereby reducing bandwidth contention and accelerating model convergence.
[0005] To achieve the above objectives, this disclosure provides a federated learning method, which is applied to a federated learning system to perform machine learning on data. The method includes the following steps:
[0006] S1: Initialization: The server randomly initializes the global model parameters;
[0007] S2: Distribution: The server downloads all or part of the global model parameters and distributes them to multiple edge devices;
[0008] S3: Parameter Training: The edge device combines the received model parameters with the local model and performs training;
[0009] S4: Update and Upload: During training, the server downloads the remaining global model parameters and distributes them to the corresponding edge devices for parameter training. The edge devices upload some model parameter updates in advance according to the pre-upload rules. When training is completed, the final model parameter updates are uploaded.
[0010] S5: Aggregation: After receiving all model parameters, the server performs a weighted average of all model parameters, aggregates them into a model, and uses this model as the new global model.
[0011] S6: Repeat steps S2, S3, S4 and S5 sequentially for the new global model generated in step S5 until the global model reaches the performance standard.
[0012] Furthermore, in step S2, during the initial loop, all global model parameters are downloaded. In subsequent loops, if the selected device has been trained, the model generated by the device after the last training is compared with the global model of the current loop, and the model parameters with the greatest changes are downloaded. The number of parameters downloaded in each loop accounts for 10% to 60% of the total number of global model parameters. If the selected device has not been trained, all global model parameters are downloaded.
[0013] Furthermore, in step S4, in the initial loop, the server does not need to download the global model parameters again during the training process. In subsequent loops, the server downloads the remaining global model parameters corresponding to each edge device in the current loop.
[0014] Furthermore, the pre-upload rule includes pre-upload parameter selection and pre-upload timing selection. The pre-upload timing selection is based on the midpoint of the current training progress, and data pre-upload begins when the midpoint of the training progress is reached.
[0015] Furthermore, the pre-upload parameters are selected from the top 60% of model parameters whose current parameter stability is closer to 0 or the top 60% of model parameters with large parameter changes.
[0016] Furthermore, when the pre-upload parameters are selected as parameters whose stability is closer to 0, the final model parameter update is selected as all model parameters that have not yet been uploaded; when the pre-upload parameters are selected as parameters with large changes, the final model parameter update is selected as the top 60% of model parameters with the largest changes.
[0017] This embodiment also discloses a federated learning system for implementing the above-described federated learning method, the system comprising:
[0018] At least one server is used to initialize global model parameters and aggregate and distribute model parameters.
[0019] Multiple edge devices are used to receive model parameters sent by the server, train the model parameters, and upload the model parameters to the server during and after training.
[0020] Furthermore, the edge device has a dataset. in,
[0021] Represents the dataset The amount of data,
[0022] x k,j This represents the j-th input data from the k-th edge device.
[0023] y k,j x represents k,j The tag,
[0024] Represents the entire dataset.
[0025] This represents the total sample size.
[0026] Furthermore, the edge device minimizes the loss function on the dataset by training the model parameters w, as expressed as:
[0027]
[0028] in,
[0029] Where the loss function is f(w,x) k,j ,y k,j )Measure the model parameter w in the data pair {x k,j ,y k,j Error on}
[0030] Furthermore, the edge device calculates the parameter stability of the current model parameters and determines whether the model parameters are stable based on the parameter stability, where parameter stability can be defined as:
[0031]
[0032]
[0033] Among them, P k This is the parameter stability value after the Kth check.
[0034] E k It is a moving average of parameter updates.
[0035] It is a moving average of the absolute values of the parameter updates.
[0036] α is a smoothing factor, and its value is close to 1.
[0037] △ K This indicates the cumulative updates to the parameters since the last check.
[0038] The beneficial effects of this invention are:
[0039] 1. By proposing a federated learning method, some trained model parameters can be uploaded during the training phase of edge devices, which can reduce the amount of data uploaded by edge devices after the overall model parameters have been trained, thereby reducing the time for edge devices to upload the final model parameters.
[0040] 2. The proposed federated learning system can monitor the accuracy of the trained model and calculate the stability of the model parameters, providing a stability assessment of the model parameters for the federated learning method. This enables the uploading of more suitable model parameters, ensuring both training efficiency and model accuracy. Attached Figure Description
[0041] The accompanying drawings illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the principles of the invention. These drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification.
[0042] Figure 1 This is a flowchart of the federated learning method in this embodiment;
[0043] Figure 2 This is a schematic diagram of the federated learning system structure in this embodiment. Detailed Implementation
[0044] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings.
[0045] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present disclosure will now be described in detail with reference to the accompanying drawings and embodiments.
[0046] Example:
[0047] refer to Figure 1 A federated learning method, applied to federated learning systems to perform machine learning on data, specifically includes the following steps:
[0048] S1: Initialization: The server randomly initializes the global model parameters;
[0049] By using S1, the model can be effectively prevented from being locked into an incorrect local optimum, making it easier to obtain the global optimum in subsequent training.
[0050] S2: Distribution: The server downloads some or all of the global model parameters and distributes them to randomly selected edge devices;
[0051] In the initial loop, all global model parameters are downloaded. Without pre-training, it's impossible to separate parameters from the global model parameters that have a significant impact on the model's training image. Therefore, training all model parameters effectively avoids the mistaken elimination of parameters with large training images. Specifically, the server randomly selects multiple edge devices and distributes all global model parameters to each. In subsequent loops, for the selected edge devices, if the device has already been trained, the model generated by that device after the previous training is compared with the global model in the current loop, and the most changed model parameters are downloaded. The number of parameters downloaded in each loop accounts for 10% to 60% of the total global model parameters. These parameters have undergone fine-tuning in previous training and have a more significant impact on model training. Selecting parameters with greater changes allows the model trained on the device to better integrate with the global model. If the selected device has not been trained, all global model parameters for this loop are downloaded.
[0052] S3: Parameter Training: The edge device combines some of the received global model parameters with the local model and performs pre-training.
[0053] S4: Update and Upload: During training, the server selectively downloads the remaining global model parameters and distributes them to the corresponding edge devices for parameter training. The edge devices upload part of the model parameter updates in advance according to the pre-upload rules. When training is completed, the final model parameter updates are uploaded.
[0054] In the initial loop, since all global model parameters have been downloaded before training, the server does not need to download the remaining global model parameters during the training phase. Simultaneously, each of the selected edge devices retains a locally trained model upon completion of training. In subsequent loops, the server downloads the remaining global model parameters from the global model for each edge device in the current loop, after a partial download of model parameters in step S2. At this point, the model on the device side better represents the global model, thus improving the accuracy of the global model. Building upon this, the timing of parameter download during training is equally important. In this embodiment, the parameters are downloaded when the training task has fully adapted to the local model and converged.
[0055] Furthermore, the pre-upload rules include the selection of pre-upload parameters and the timing of pre-upload. The timing of pre-upload is determined by using the midpoint of the current training progress as a baseline; data pre-upload begins when the midpoint of the training progress is reached. In this embodiment, the pre-upload parameters are selected from the top 60% of model parameters whose current parameter stability is closer to 0. Parameters that are close to stable will not change significantly during training after pre-upload. Therefore, when training is finally completed, there is no need to upload these stable parameters, thus saving time during the final upload. In one or more embodiments, the pre-upload parameters can also be selected from the top 60% of model parameters with the largest parameter changes. These parameters have been significantly updated compared to the initial parameters, indicating that the model has learned a lot of valuable information from the local data. Prioritizing the upload of these parameters can greatly help the training process.
[0056] Furthermore, at the end of training, it's necessary to upload the final model parameter updates. Although some parameters were uploaded in advance, these only play a supporting role in the overall training results. Therefore, uploading the final model parameter updates is crucial to ensure training effectiveness. In the above discussion, pre-uploading can choose to upload more stable or more variable parameters. When more stable parameters are selected for pre-upload, the final model parameters are updated with parameters that haven't yet been uploaded during the overall training process, supplementing the pre-uploaded parameters and providing a complete training result. When more variable parameters are selected for pre-upload, the final model parameters are updated with the top 60% of the most variable model parameters from the current training. This method allows for flexible selection of the number of parameters to save, preventing the loss of important parameter updates.
[0057] S5: Aggregation: After receiving the final model parameters, the server performs a weighted average of all uploaded model parameters and then aggregates them into a new global model;
[0058] S6: Repeat steps S2, S3, S4 and S5 sequentially for the new global model generated in step S5 until the global model reaches the performance standard.
[0059] refer to Figure 2 This embodiment also discloses a learning system for implementing the federated learning method of this embodiment. Specifically, it is a novel pipelined federated learning framework, FedPipeline. FedPipeline reduces the number of model parameters that need to be transmitted before and after training by downloading some parameters to the device before training and uploading some training parameters to the server after training, thereby accelerating the training efficiency of federated learning while ensuring the accuracy of the model. The learning system includes at least one server and multiple edge devices. The server is used to initialize the global model parameters and aggregate and distribute the model parameters in this method. The edge devices have local data stored in their memory for model training, receive the model parameters distributed by the server, train the model parameters using the local data, and upload the updated model parameters during and after training.
[0060] Since the federated learning method in this embodiment involves calculating the stability of model parameters and the training effect of the model, the computation process of the edge devices in this federated learning system is equally important. Specifically, the local data stored on the edge devices is denoted as the dataset. in Represents the dataset Data volume, x k,j Let y represent the j-th input data of the k-th edge device. k,j x represents k,j The labels, the entire dataset can be represented as The total sample size is The goal of training is to derive model parameters w from training on edge devices to minimize the loss function on the dataset, expressed as:
[0061]
[0062] in, Where the loss function is f(w,x) k,j ,y k,j )Measure the model parameter w in the data pair {x k,j ,y k,j The error on the model is used to judge the learning effect of the current model. That is, the smaller the error, the higher the training quality of the model. When the error meets the threshold set by the current model, the model is judged to meet the performance requirements.
[0063] In step S4 of the federated learning method disclosed in this embodiment, the selection of pre-upload parameters needs to be judged by parameter stability. The edge device calculates the parameter stability of the current model parameters and judges whether the model parameters are stable based on the parameter stability. Parameter stability can be defined as:
[0064]
[0065]
[0066] Among them, P k E is the parameter stability value after the Kth check. k It is a moving average of parameter updates. It is a moving average of the absolute values of parameter updates, with a smoothing factor α that is close to 1, Δ K This represents the cumulative update of the parameters since the last check. If the model parameters are updated in the same direction each time, P is closer to 1. If the model parameters oscillate, P is closer to 0. In other words, the closer P is to 0, the more stable the parameters are during the model update process. By calculating the parameter stability, we can determine which model parameters are stable or have large changes during the model training process, which is convenient for the judgment of pre-uploaded parameters.
[0067] Those skilled in the art should understand that the above embodiments are merely for illustrating the present invention and are not intended to limit the scope of the invention. Those skilled in the art can make other changes or modifications based on the above invention, and these changes or modifications are still within the scope of this disclosure.
Claims
1. A federated learning method, said method for performing machine learning on data, characterized in that, The method includes the following steps: S1: Initialization: The server randomly initializes the global model parameters; S2: Distribution: The server downloads all or part of the global model parameters and distributes them to multiple edge devices; S3: Parameter Training: The edge device combines the received model parameters with the local model and performs training; S4: Update and Upload: During training, the server downloads the remaining global model parameters and distributes them to the corresponding edge devices for parameter training. The edge devices upload some model parameter updates in advance according to the pre-upload rules. When training is completed, the final model parameter updates are uploaded. S5: Aggregation: After receiving all model parameters, the server performs a weighted average of all model parameters, aggregates them into a model, and uses this model as the new global model. S6: Repeat steps S2, S3, S4 and S5 sequentially for the new global model generated in step S5 until the global model reaches the performance standard. In step S2, during the initial loop, all global model parameters are downloaded. In subsequent loops, if the selected device has been trained, the model generated by the device after the last training is compared with the global model of the current loop, and the model parameters with the greatest changes are downloaded. The number of parameters downloaded in each loop accounts for 10% to 60% of the total number of global model parameters. If the selected device has not been trained, all global model parameters are downloaded. In step S4, in the initial loop, the server does not need to download the global model parameters again during the training process. In subsequent loops, the server downloads the remaining global model parameters corresponding to each edge device in the current loop. The pre-upload rules include pre-upload parameter selection and pre-upload timing selection. The pre-upload timing selection is to start pre-uploading data when the midpoint of the current training progress is reached, based on the midpoint of the current training progress. The pre-upload parameters are selected from the top 60% of model parameters whose current parameter stability is closer to 0, or the top 60% of model parameters with large parameter changes.
2. The federated learning method according to claim 1, characterized in that, When the pre-upload parameters are selected as parameters whose stability is closer to 0, the final model parameter update is selected as all model parameters that have not yet been uploaded. When the pre-upload parameters are selected as parameters with large changes, the final model parameter update is selected as the top 60% of model parameters with the largest changes.
3. A federated learning system for implementing the federated learning method as described in any one of claims 1-2, characterized in that, The system includes: At least one server is used to initialize global model parameters and aggregate and distribute model parameters. Multiple edge devices are used to receive model parameters sent by the server, train the model parameters, and upload the model parameters to the server during and after training.
4. The federated learning system according to claim 3, characterized in that, The edge device has a dataset. in, ， Indicates the first k The first edge device j One input data, express The tag, Represents the entire dataset. This represents the total sample size.
5. The federated learning system according to claim 4, characterized in that, The edge device derives model parameters through training. w To minimize the loss function on the dataset, it is expressed as: ， Where the loss function Measuring model parameters w In data pairs Errors on.
6. The federated learning system according to claim 5, characterized in that, The edge device calculates the parameter stability of the current model parameters and determines whether the model parameters are stable based on the parameter stability. Parameter stability can be defined as: ， in, This is the parameter stability value after the Kth check. It is a moving average of parameter updates. It is a moving average of the absolute values of the parameter updates. It is a smoothing factor, and its value is close to 1. This indicates the cumulative updates to the parameters since the last check.