Federal learning method based on model deviation and variance distribution model
By calculating the variance and offset parameters of the client model on the server side, extracting personalized information, and providing the client with a personalized initial model, it solves the problem of high computational overhead on the client side by the existing personalized federated learning method, and improves prediction accuracy and privacy protection.
Patent Information
- Application Number
- CN202411614148.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-03-07
Smart Images
Figure CN119577826A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning, and in particular to a federated learning method based on a model bias and variance allocation model. Background Art
[0002] In the past, data often appeared in a smaller scale, and people only processed data by gathering them together and fitting and learning them with various models to obtain the desired models and parameters, and then applied them to various prediction tasks. However, as the amount of data grew and more and more people were involved, traditional model training methods gradually became incapable of keeping up.
[0003] First of all, it is difficult for people to gather data in one place for processing. In this era where everyone has a lot of data, it is a big challenge to gather the data of thousands or even hundreds of millions of people in one place, whether it is the storage capacity, transmission capacity or computing capacity of the device. If everyone's data can be processed locally and the required parameter information can be transmitted to the server, it will undoubtedly greatly reduce the burden on the server. Secondly, with the emergence of privacy issues, few people may be willing to make their data public for server training models. For example, this is like the rapid development of AI, which has caused painters to remove their works from media accounts and programmers to no longer want to open source their code. For example, your shopping records and personal photos, you probably won't be willing to share them with others. However, the training of various models requires data. Without the support of data, manufacturers will find it difficult to provide better services, which conflicts with the privacy and security of the public. If you can get the information needed for model training without making the data public, then you can solve the above problems. This leads to the distributed machine learning paradigm - federated learning.
[0004] In subsequent studies, heterogeneous data posed new challenges to federated learning. In order to make the model trained under the federated learning framework better fit the data of each client, the researchers proposed the concept of personalized federated learning. By training differentiated models for each client, they aim to achieve better performance on the data of each client.
[0005] However, existing personalized federated learning methods either require additional computing and space overhead at the client level to learn and store more information in order to learn personalized models; or transmit additional information to the server to extract personalized features of each client, which also means increased pressure on data transmission and an increased risk of privacy leakage.
[0006] Therefore, technicians in this field are committed to developing a federated learning method (FedDBV) based on a model bias and variance allocation model. The present invention proposes a personalized federated learning method on the server side, which extracts personalized information of each client by calculating the variance of the model uploaded by the client in each round on the server and learning the offset parameters of each client model relative to the global model, aiming to provide a personalized initial model for each round of local learning of the client. This method not only uses personalized model learning to make the learned model have better performance on local data, but also solves the problem that the existing personalized federated learning relies on additional overhead on the client. At the same time, there is no need to transmit additional data between the client and the server, which better protects the privacy of the client data. In addition, the present invention further improves the performance of the personalized model and improves the prediction accuracy of the data by hierarchical calculation of the model, while reducing the calculation and storage overhead in the learning process. Summary of the invention
[0007] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to solve the additional overhead problem of the existing personalized federated learning client, protect the privacy of client data, and improve the prediction accuracy.
[0008] To achieve the above objectives, the present invention provides a federated learning method based on a model bias and variance distribution model, which calculates the variance of each round of client models on the server, learns the offset parameters of each client model relative to the global model, extracts the personalized information of each client, and provides a personalized initial model for each round of local learning of the client.
[0009] Furthermore, the global data is regarded as a distribution, and the local data is a sample in the distribution. The process of calculating the variance and the global model fits this distribution, and the offset parameter represents the offset of the local data relative to the data center.
[0010] Furthermore, the client model includes personalized information of the client's local data.
[0011] Furthermore, the client model is used on the server to learn the offset parameters of each client, thereby learning the personalized information of the client on the server side.
[0012] Furthermore, only the client model is transmitted.
[0013] Furthermore, the model is subdivided into layers.
[0014] Furthermore, during the model training process, the bottom-level model learns the feature extraction method, and the top-level model classifies the extracted features.
[0015] Furthermore, by comparing the training results of different layers, the model layer containing the richest personalized information is found.
[0016] Further, the following steps are included:
[0017] Step 1, initialization parameters;
[0018] Step 2: Start the model training round and select the clients to participate in this round of training;
[0019] Step 3, first round: directly distribute the global model to each client; other rounds: for each client, first copy the global model parameters, then add the product of the variance and offset value corresponding to each client model, generate a personalized model, and distribute it to each client;
[0020] Step 4: After the client completes local training, the server collects the models of each client and aggregates them to form a new global model;
[0021] Step 5: Calculate the offset parameters of each client model;
[0022] Step 6: Repeat step 2 to step 5 until convergence or reaching a specified training round.
[0023] Furthermore, the step 5 comprises the following steps:
[0024] Step 5.1, calculate the variance of the client model parameters and the global model parameters;
[0025] Step 5.2, calculate the updated value of the offset;
[0026] Step 5.3: Calculate the offset parameters.
[0027] Existing personalized federated learning methods require additional computing and space overhead on the client. The present invention calculates the variance of the model uploaded by the client in each round on the server, learns the offset parameters of each client model relative to the global model, extracts the personalized information of each client, and provides a personalized initial model for each round of local learning of the client. If the global data is regarded as a distribution, the local data is equivalent to a sample in the distribution. The process of calculating the variance and the global model is equivalent to fitting the distribution, and the offset parameter represents the offset of the local data relative to the data center. Because the learned local model already contains the personalized information of the local data, the uploaded model is used on the server to approximate the position of the client data, and the offset parameters of each client are learned, so that the personalized information of the client can be learned only on the server. While the accuracy of the present invention on multiple data sets is higher than that of many SOTA methods, it only uses additional computing and storage overhead on the server side, and does not use additional resources on the client.
[0028] Existing personalized federated learning methods for learning personalized information on the server rely on additional information transmission between the client and the server, which increases the pressure on data transmission and the risk of privacy leakage. The present invention only transmits the most basic local model and does not need to transmit additional local information. During the learning process of this method, the training data required by the server is only the model uploaded by the client in the new round, and no more additional information is required. While the present invention learns more personalized information on the server side, it does not require more information to be uploaded, which not only saves the additional overhead of the transmission process, but also is beneficial to the privacy protection of the client.
[0029] The existing model hierarchical methods only consider the model as the front and back parts of the feature extractor and classifier. The present invention subdivides the model into layers and refines the impact of each layer on the training process. During the training process of the model, the information learned by each layer of the model is different. The bottom model learns the method of feature extraction, and the top model classifies the extracted features. This is also the basis of many previous hierarchical methods. However, for different heterogeneous data, the difference in the model may be more reflected in a specific layer, rather than the general upper and lower layers. By comparing the training results of different layers, the model layer with the richest personalized information can be found. The present invention only personalizes the model layer containing a large amount of personalized information. While learning sufficient personalized information, it retains the global information of the global layer, while improving the model performance, it reduces the redundant overhead of calculation and storage.
[0030] Compared with the prior art, the present invention has the following obvious substantial features and significant advantages:
[0031] 1. Technical advantages of the present invention:
[0032] The model learned by the present invention has better performance on local data and has higher prediction accuracy compared with existing SOTA methods on multiple data sets;
[0033] The present invention solves the problem that the existing personalized federated learning relies on the extra overhead on the client side, and only requires a certain increase in the amount of computing and data storage on the server side;
[0034] The present invention does not need to transmit additional data between the client and the server, thus better protecting the privacy of the client data;
[0035] The present invention reduces the computational and storage overhead in the learning process through hierarchical calculation of the model, further improves the performance of the personalized model and improves the prediction accuracy of the data.
[0036] 2. Performance indicators of the present invention: The accuracy of the present invention on several commonly used graphic classification data sets reaches or even exceeds that of existing advanced algorithms, especially on the Cifar100 data set, the accuracy is improved by 1-2 percentage points compared with other advanced algorithms; in terms of convergence, the convergence speed of the present invention is at the same level as other advanced algorithms; under the condition of specific data distribution, it shows better scalability, and the accuracy can be further improved when the number of participating clients is increased; in terms of time cost, it has no obvious disadvantage compared with other algorithms, the required computing resources are also small, and the utilization efficiency in terms of time and resources is good.
[0037] 3. The present invention is implemented in production:
[0038] Easy to deploy on a large scale: The present invention does not require additional hardware on the client side, and is easy to deploy and apply directly on the existing client terminal, reducing the transformation cost and equipment investment required in the industrialization process. At the same time, since the data is not interoperable, the technology is very attractive in areas where data privacy is sensitive, and can be quickly promoted and applied in the market.
[0039] Low operating cost: Since the present invention reduces the computing and storage requirements for the client, it not only prolongs the service life of the client device, but also significantly reduces the operation and maintenance costs of the system, making it suitable for large-scale, low-cost industrial applications.
[0040] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a training flow chart of the FedDBV method of a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0042] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.
[0043] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.
[0044] like Figure 1 As shown in the figure, the experimental running framework is: PFLlib personalized federated learning framework.
[0045] Code module: The DBV module plays a major role in the FedDBV algorithm, which contains three important function modules:
[0046] 1)generate_sigma: calculates the standard deviation between the global model parameters and the client model parameters;
[0047] 2) update_bias: Use standard deviation information to calculate the parameter offset value of each client model;
[0048] 3) distribute_model: Based on the parameters of the global model, for each client, the product of the offset and the variance is added to the global model parameters to generate a personalized model and send it to the client.
[0049] Parameter description: In the DBV algorithm, in addition to parameters such as the number of clients, data set, number of training rounds, and neural network type, the following hyperparameters for DBV are set:
[0050] 1)η: learning rate for updating the offset parameter;
[0051] 2) p: the power of the variance when calculating the offset parameter;
[0052] 3) λ: When updating the offset value, some parameter update values may be unreasonable (such as NaN or extreme values), so this hyperparameter is set to avoid extreme offset update values;
[0053] 4)Ω sel : Select a specific classification layer in the network for parameter update.
[0054] The following are the parameters generated during the training process:
[0055] 1) Offset parameter for client i in round t of training.
[0056] 2) σ t : The standard deviation of the client parameters in the yth round of training.
[0057] 3)Θ y : Global model parameters in the tth round of training.
[0058] 4) Local model parameters of client i in round t of training.
[0059] Run steps:
[0060] 1) Initialize model parameters and DBV parameters;
[0061] 2) Start the model training round and select the clients to participate in this round of training;
[0062] 3) First round: directly distribute the global model to each client; other rounds: for each client, first copy the global model parameters, then add the product of the variance and offset value corresponding to each client model to generate a personalized model, which is distributed to each client;
[0063] 4) After the client completes local training, the server collects the models of each client and aggregates them to form a new global model;
[0064] 5) For each client model:
[0065] a) Calculate the variance and standard deviation σ of the client model parameters and the global model parameters t ;
[0066] b) Calculate the updated value of the offset. The specific algorithm is to divide the difference between the client model parameters, the global model parameters and the original offset by the specified multiple of the variance (p) for the selected model layer to obtain the updated value of the offset. The calculation formula is: Here, we need to use λ to limit the update value range to filter out abnormal values and limit the update value to [-λ,λ]. or , it is corrected to λ or -λ.
[0067] c) Calculate the offset parameter. The calculation formula is:
[0068] This step is the core step of this method. If the parameters of the global model can be regarded as a fit to the data distribution of all clients, then the global model plus the variance and offset information of the client data distribution can also form a fit to the data distribution reflected by the specified client model. In this way, the global model can be personalized by only obtaining the weight parameter information of the client model, without the need for the client to provide additional information, and the generation of the personalized model does not make additional requirements on the client's resources, and the required calculations can be completed on the server side.
[0069] In addition, in this step, only the specified number of layers in the neural network can be calculated and updated, which further improves the training accuracy while saving computing resources.
[0070] 6) Repeat steps 2-5 until convergence or reaching the specified training round.
[0071] Results: The accuracy of the algorithm reaches SOTA level on datasets such as MNIST, Cifar10, and Cifar100, and it shows good scalability when the number of clients is adjusted.
[0072] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A federated learning method based on a model bias and variance distribution model, characterized in that: The variance of the client model in each round is calculated on the server, the offset parameters of each client model relative to the global model are learned, the personalized information of each client is extracted, and a personalized initial model is provided for the local learning of each client.
2. The federated learning method based on the model bias and variance allocation model according to claim 1, characterized in that: The global data is regarded as a distribution, and the local data is a sample in the distribution. The process of calculating the variance and the global model fits this distribution. The offset parameter represents the offset of the local data relative to the data center.
3. The federated learning method based on the model bias and variance allocation model as claimed in claim 2, characterized in that: The client model contains personalized information about the client's local data.
4. The federated learning method based on the model bias and variance allocation model as claimed in claim 3, characterized in that: The client model is used on the server to learn the offset parameters of each client, thereby learning the client's personalized information on the server.
5. The federated learning method based on the model bias and variance allocation model according to claim 1, characterized in that: Only the client model is transferred.
6. The federated learning method based on the model bias and variance allocation model according to claim 1, characterized in that: Subdivide the model into layers.
7. The federated learning method based on the model bias and variance allocation model according to claim 1, characterized in that: During the model training process, the bottom-level model learns the feature extraction method, and the top-level model classifies the extracted features.
8. The federated learning method based on the model bias and variance allocation model according to claim 7, characterized in that: By comparing the training results of different layers, we can find the model layer that contains the most personalized information.
9. The federated learning method based on the model bias and variance allocation model according to claim 1, characterized in that: The following steps are involved: Step 1, initialization parameters; Step 2: Start the model training round and select the clients to participate in this round of training; Step 3, first round: directly distribute the global model to each client; other rounds: for each client, first copy the global model parameters, then add the product of the variance and offset value corresponding to each client model, generate a personalized model, and distribute it to each client; Step 4: After the client completes local training, the server collects the models of each client and aggregates them to form a new global model; Step 5: Calculate the offset parameters of each client model; Step 6: Repeat step 2 to step 5 until convergence or reaching a specified training round.
10. The federated learning method based on the model bias and variance allocation model according to claim 9, characterized in that: The step 5 comprises the following steps: Step 5.1, calculate the variance of the client model parameters and the global model parameters; Step 5.2, calculate the updated value of the offset; Step 5.3: Calculate the offset parameters.