Efficient federal learning method

Abstract

The invention provides an efficient federal learning method, which comprises the following steps of: compressing global parameters after parameter aggregation at a server side, and compressing a local model after generating new local model parameters by combining a global model and a local private data set at a client side, so that a generated local sparse model is complementary with a global sparse model; an error feedback mechanism is introduced after compression of a client to return errors before and after compression of a local model to a model training process, so that parameters are adaptively adjusted to update and compensate the errors, and the training accuracy is improved. Compared with the traditional federated learning, the method provided by the invention has extremely high convergence and robustness, and can give consideration to the communication efficiency consisting of time complexity and communication overhead.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed machine learning, and in particular, to an efficient federated learning method. Background Art

[0002] Federated Learning (FL), as a distributed machine learning of privacy protection technology, breaks the "data silo" phenomenon that has long troubled the data science field. Federated learning was initially proposed by Google in 2017, officially starting the exploration of privacy protection collaborative training. In this paradigm, instead of concentrating scattered data on a central node for model training, local training is performed on each local device and the local model parameters are uploaded to the central node to improve the model effect. Federated learning demonstrates significant advantages in many aspects: under the federated learning framework, the status of each participant is equal, enabling fair cooperation; only the model parameters are uploaded to the server, and the original data does not need to be shared, effectively protecting user privacy; it can handle the heterogeneity of participating devices, including differences in hardware and software; it supports multi-party computing and can achieve secure computing in an untrusted environment. Due to these advantages, federated learning has now been widely applied in many fields such as medical image recognition, financial risk assessment, traffic flow prediction, and smart city construction. It can not only help multiple participants build a shared high-performance model but also meet the requirements of user privacy and data confidentiality.

[0003] However, in recent years, with the wide application of federated learning, some problems have also emerged: in traditional federated learning models, uploading all the parameters of the model increases the communication overhead of the system; at the same time, federated learning also faces challenges in terms of communication efficiency in practical applications. A core requirement of federated learning is to widely deploy the trained model on user nodes for local training of distributed data. However, model transmission is affected by factors such as limited communication bandwidth, insufficient power resources of mobile devices, and a large user scale, increasing the risk of client stragglers. In addition, federated learning needs to upload the complete training gradient in each round of iteration. As the scale of the model and parameters increases, the data volume of the gradient also increases sharply, which not only occupies a large amount of communication resources but also causes a sharp increase in the communication cost of gradient uploading and downloading, and the transmission efficiency is also greatly reduced, becoming one of the key factors restricting system performance. Summary of the Invention

[0004] According to the technical problems proposed above, an efficient federated learning method is provided. After local training is completed, before the participating parties upload the updated model and after the server-side parameter aggregation, the present invention compresses the local partial model and the global model parameters respectively, so that the compressed global model and local model parameters can complement each other into a complete dense model, so as to achieve the purpose of bidirectionally reducing the communication overhead and improving the communication efficiency during the federated learning upload and download processes.

[0005] The technical means adopted by the present invention are as follows:

[0006] An efficient federated learning method is applied to a number of local devices and a central server, including:

[0007] S1. The central server initializes the global model parameters;

[0008] S2. The server-side prunes the dense global model parameters;

[0009] S3. The client downloads the pruned global model and mask, and combines with the local private dataset to train to generate a local sparse model complementary to the global model and the corresponding inversion mask;

[0010] S4. Use the error feedback mechanism to retain the errors generated before and after sparsification locally for participating in the training of the next round of local models;

[0011] S5. Upload the sparsified local model and its corresponding inversion mask to the server-side, aggregate them into a dense model, and determine whether the dense model meets the preset requirements. If so, end the learning; if not, repeat steps S2 to S4 until convergence.

[0012] Further, step S1 specifically includes:

[0013] S11. The server-side establishes a communication channel with the client, and initializes the global model w0 of the server-side, the local models of each client, and the sample weights;

[0014] S12. The client trains from random weights and uploads the generated dense model w0 k back to the server-side;

[0015] S13. The server-side aggregates each local model using the averaging algorithm to generate the first-round dense global model w t+1 , as follows:

[0016]

[0017] where n k represents the number of samples of client k, n represents the total number of samples of the selected clients; K represents the total number of clients, Represents the k-th local parameter.

[0018] Further, step S2 specifically includes:

[0019] S21. The server prunes a certain proportion of the low-magnitude model weights, and the pruning function is as follows:

[0020] (w′ t+1 , mask) = prune(w t+1 )

[0021] where w t+1 represents the unpruned global model; w t ' +1 represents the pruned global model; the mask mask is a binary tensor, 0 indicates that the weight is removed, and 1 indicates that the weight is retained;

[0022] S22. Mark the pruned global model w t ' +1 with the pruning mask and send it to the client together.

[0023] Further, step S3 specifically includes:

[0024] S31. Each participating client n downloads the pruned global sparse model w t ' and trains it using the local dataset x n to generate a new local model θ t+1,n , as follows:

[0025] θ t+1,n = w′ t - ηg n

[0026] where η represents the learning rate, and g n represents the gradient of client n;

[0027] S32. The client calculates the inverse mask according to the pruning mask of the global model The mask weight of the server is 0, and the corresponding inverse mask is 1. The mask weight of 1 corresponds to an inverse mask of 0;

[0028] S33. Obtain the new local sparse parameter by multiplying the local parameter by the inverse mask ⊙, as follows:

[0029]

[0030] where θ t ' +1,n represents the new local sparse parameter; θ t+1,n represents the local parameter; represents the inverse mask.

[0031] Further, step S4 specifically includes:

[0032] S41. The gradient after sparsification of the local model jointly trained by the nth client combining the local dataset and the sparsified parameters left through the error feedback mechanism in the previous round is calculated as follows:

[0033]

[0034] Among them, represents the gradient after sparsification of the local model; error t-1 represents the error left through the error feedback mechanism in the (t - 1)th round;

[0035] S42. The error information is returned to the model training process through the error feedback mechanism, enabling the model to adaptively adjust the parameter update to compensate for the error, thereby improving the training accuracy. The calculation formulas for the error before and after sparsification are as follows:

[0036]

[0037] Among them, error t represents the error before and after sparsification.

[0038] Further, in step S5, a new dense model is aggregated as follows:

[0039]

[0040] Among them, w t+2 represents the new global dense model; N represents the number of clients participating in the training, represents the local model complementary to the global sparse model after sparsification of the nth client.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] 1. An efficient federated learning method provided by the present invention can achieve low communication overhead and high communication efficiency both at the client side and the server side, reduce the two-way communication overhead between the server and the client, minimize the computational overhead of the system, and achieve good model performance.

[0043] 2. An efficient federated learning method provided by the present invention, in which the inverted mask maintains the weights of the client models that were initially zero in the global sparse model. Therefore, the global sparse model pruned by the server side in the previous round is complementary to the local sparse models aggregated by each client, and the aggregation generates a completely dense model.

[0044] 3. An efficient federated learning method provided by the present invention returns error information to the model training process through an error feedback mechanism, enabling the model to adaptively adjust parameter updates to compensate for errors, thereby improving the accuracy of training.

[0045] For the above reasons, the present invention can be widely promoted in the fields of distributed machine learning and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0047] Figure 1 It is the overall flowchart of the efficient federated learning method of the present invention.

[0048] Figure 2 It is the framework diagram of the efficient federated learning method of the present invention.

[0049] Figure 3 It is the structural diagram of the complementary sparsification algorithm provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0052] In view of the foregoing problems and scenarios, that is, on the premise that data does not leave the local client, multiple parties collaborate to reduce the two-way communication overhead between the server and the client while ensuring privacy and security during the upload and download processes, improving communication efficiency without sacrificing model accuracy.

[0053] As Figure 1 , 2 shown, the present invention provides an efficient federated learning method, which is applied to a system including a number of local devices and a central server, and includes:

[0054] S1. The central server initializes the global model parameters;

[0055] Specifically, as a preferred embodiment of the present invention, step S1 specifically includes:

[0056] S11. The server establishes a communication channel with the client, and initializes the global model w0 of the server, the local models of each client, and the sample weights;

[0057] S12. The client trains from random weights and uploads the generated dense model back to the server;

[0058] S13. The server aggregates the local models using the averaging algorithm to generate the first-round dense global model w t+1 , as follows:

[0059]

[0060] where n k represents the number of samples of client k, n represents the total number of samples of the selected clients; K represents the total number of clients, represents the k-th local parameter.

[0061] S2. The server prunes the dense global model parameters;

[0062] Specifically, as a preferred embodiment of the present invention, step S2 specifically includes:

[0063] S21. The server prunes a certain proportion of the low-magnitude model weights, and the pruning function is as follows:

[0064] (w′ t+1 , mask) = prune(w t+1 )

[0065] where w t+1 represents the unpruned global model; w t ' +1Denote the pruned global model; the mask is a binary tensor, where 0 indicates that the weight is removed and 1 indicates that the weight is retained;

[0066] S22. Mark the pruned global model w with the pruning mask t ' +1 and send them to the client together.

[0067] S3. The client downloads the pruned global model and the mask, and combines with the local private dataset to train and generate a local sparse model complementary to the global model and the corresponding inverse mask;

[0068] Specifically, as a preferred embodiment of the present invention, step S3 specifically includes:

[0069] S31. Each participating client n downloads the pruned global sparse model w t ' and uses the local dataset x n to train and generate a new local model θ t+1,n as follows:

[0070] θ t+1,n = w′ t - ηg n

[0071] where η represents the learning rate, and g n represents the gradient of client n;

[0072] S32. The client calculates the inverse mask according to the pruning mask of the global model The mask weight of the server is 0, and the corresponding inverse mask is 1. The mask weight of 1 corresponds to the inverse mask of 0;

[0073] S33. Obtain the new local sparse parameters through the product ⊙ of the local parameters and the inverse mask, as follows:

[0074]

[0075] where θ t ' +1,n represents the new local sparse parameters; θ t+1,n represents the local parameters; represents the inverse mask.

[0076] S4. Use the error feedback mechanism to retain the errors generated before and after sparsity locally for participating in the training of the next round of local models;

[0077] Specifically, as a preferred embodiment of the present invention, step S4 specifically includes:

[0078] S41. The gradient after sparsification of the local model jointly trained by the nth client using the local dataset and the parameters sparsified in the previous round through the error feedback mechanism is calculated as follows:

[0079]

[0080] Wherein, represents the gradient after sparsification of the local model; error t-1 represents the error left through the error feedback mechanism in the (t - 1)th round;

[0081] S42. Return the error information to the model training process through the error feedback mechanism, enabling the model to adaptively adjust parameter updates to compensate for errors, thereby improving the training accuracy. The calculation formulas for the errors before and after sparsification are as follows:

[0082]

[0083] Wherein, error t represents the error before and after sparsification.

[0084] S5. Upload the sparsified local model and its corresponding inversion mask to the server side, aggregate them into a dense model, and determine whether the dense model meets the preset requirements. If so, end the learning; if not, repeat steps S2 to S4 until convergence.

[0085] Specifically, as a preferred implementation manner of the present invention, in step S5, a new dense model is aggregated as follows:

[0086]

[0087] Wherein, w t+2 represents the new global dense model; N represents the number of clients participating in the training, represents the local model complementary to the global sparsified model after sparsification of the nth client. As Figure 3 shown, it is the structural diagram of the complementary sparsification algorithm.

[0088] In summary, the efficient federated learning method provided by the present invention includes:

[0089] The client and the server establish a secure channel. In the first round, each client uploads the trained model to the server side for aggregation to generate a dense model. After the server side aggregates, according to the magnitude of the model weights, the model parameters with low weights are trimmed and deleted to generate a sparsified global model.

[0090] The client downloads the sparse global model and the mask for pruning weights corresponding thereto, where 0 represents that the weight is pruned and 1 represents that it is not pruned. The client combines with the local private dataset for training to generate a local model, prunes the local parameters according to the inverted mask of the global sparse model to generate a local sparse model complementary to the global sparse model, and uses the error feedback mechanism to retain the locally sparsified model parameters locally to adjust the parameter update compensation error and participate in the next round of training.

[0091] Each client uploads the sparsified local model parameters and their corresponding index information. The server aggregates the local gradients through FedAvg averaging to generate a dense model, and then sparsifies it again, and so on until the model meets the standard.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient federated learning method, characterized in that, Applied to several local devices and a central server, including: S1. The central server initializes the global model parameters; S2. The server side prunes the dense global model parameters; S3. The client downloads the pruned global model and mask, and combines with the local private dataset to train and generate a local sparse model complementary to the global model and the corresponding inversion mask; S4. Use the error feedback mechanism to retain the errors generated before and after sparsification locally for participating in the training of the next round of local models; S5. Upload the sparsified local model and its corresponding inversion mask to the server side, aggregate them into a dense model, and determine whether the dense model meets the preset requirements. If so, end the learning; if not, repeat steps S2 to S4 until convergence.

2. An efficient federated learning method according to claim 1, characterized in that, Step S1 specifically includes: S11. The server side establishes a communication channel with the client, initializes the global model w0 of the server side, the local models of each client, and the sample weights; S12. The client trains from random weights and uploads the generated dense model w0 k back to the server side; S13. The server aggregates each local model using the averaging algorithm to generate the first-round dense global model w t+1 , as follows: where n k represents the number of samples of client k, and n represents the total number of selected samples of all clients; K represents the total number of clients, represents the k-th local parameter.

3. An efficient federated learning method according to claim 1, characterized in that, Step S2 specifically includes: S21. The server prunes a certain proportion of low-magnitude model weights, and the pruning function is as follows: (w' t+1 , mask) = prune(w t+1 ) Among them, w t+1 represents the unpruned global model; w' t+1 represents the pruned global model; the mask mask is a binary tensor, where 0 indicates that the weight is removed and 1 indicates that the weight is retained; S22. Mark the pruned global model w' with a pruning mask t+1 and send them to the client together.

4. An efficient federated learning method according to claim 1, characterized in that, Step S3 specifically includes: S31. Each participating client n downloads the trimmed global sparse model w' t and uses the local dataset x n to train and generate a new local model θ t+1,n as follows: θ t+1,n = w' t - ηg n where η represents the learning rate, and g n represents the gradient of client n; S32. The client calculates the inverse mask according to the pruning mask of the global model The mask weight of the server is 0, and the corresponding inverse mask is 1; the mask weight is 1, and the corresponding inverse mask is 0; S33. Obtain the new local sparse parameters through the product ⊙ of the local parameters and the inversion mask, as follows: Among them, θ' t+1,n represents the new local sparsity parameter; θ t+1,n represents the local parameter; represents the inversion mask.

5. An efficient federated learning method according to claim 1, characterized in that, Step S4 specifically includes: S41. The gradient after sparsification of the local model generated by the nth client by jointly training the local dataset and the parameters sparsified and left through the error feedback mechanism in the previous round, and its calculation formula is as follows: Among them, represents the gradient after sparsifying the local model; error t-1 represents the error left through the error feedback mechanism in the (t - 1)-th round; S42. Use the error feedback mechanism to return the error information to the model training process, so that the model can adaptively adjust the parameter update to compensate for the error, thereby improving the training accuracy. The calculation formulas for the errors before and after sparsification are as follows: Among them, error t represents the error before and after sparsification.

6. An efficient federated learning method according to claim 1, characterized in that, In step S5, aggregate to generate a new dense model, as follows: Among them, w t+2 represents a new round of global dense model; N represents the number of clients participating in the training, represents the local model complementary to the global sparse model after sparsification of the nth client.

Images