Federal learning method for resisting lagging behind based on summer secret sharing
By using the Chamil secret sharing scheme in federated learning to segment and distribute user data and model parameters, the problems of data leakage, high computing complexity and fallen behind in federated learning are solved, and a more efficient and safe training process is achieved.
Patent Information
- Application Number
- CN202411522395.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-03-07
AI Technical Summary
There are risks of data breaches, high computational complexity, and behind-the-scenes problems in federated learning, resulting in long training time and reduced model accuracy.
The federated learning method based on Chamil secret sharing is adopted, and the user data and model parameters are divided and distributed in the initialization stage using a secret sharing scheme, allowing users to train locally, reduce the complexity of data interaction computing, and ensure data security through a secret sharing scheme.
It effectively reduces the complexity of data interaction calculation in federated learning, ensures security during data transmission, solves the problem of fallen behind, and improves training efficiency and model accuracy.
Smart Images
Figure CN119578585A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of privacy protection technology, and in particular to a federated learning method based on Chamier secret sharing to resist stragglers. Background Art
[0002] With the rapid development of artificial intelligence, the demand for big data is increasing. However, due to the sensitivity of user data, data cannot be directly sent to the server for calculation, resulting in a large amount of data cannot be effectively used. Therefore, how to use data while protecting the privacy of user data has become the main problem at present. In this context, privacy computing has gradually become a research hotspot. Federated Learning (FL) is a mainstream direction in privacy computing. The introduction of federated learning solves the above problems. The federated server coordinates multiple participants to jointly train a global optimal model, and conducts collaborative learning while protecting data privacy.
[0003] Federated learning is divided into horizontal federated learning, vertical federated learning, and federated transfer learning. The frameworks of the three types of federated learning are as follows: they consist of a central server and multiple distributed clients, the training data is owned by different clients, and different features are privately stored on different clients.
[0004] Currently, there is a risk of data leakage in federated learning because users need to upload the intermediate parameters of the training model to the server; the current vertical federated learning based on homomorphic encryption has high computational complexity and long calculation time, and the training effect is poor in practical applications; the performance of vertical federated learning is degraded due to the dispersion between different clients, which is specifically manifested in the different computer performance and large resource differences between different users, resulting in the problem of stragglers, which is specifically manifested in excessive training time and reduced training model accuracy. Summary of the invention
[0005] The purpose of the present invention is to provide a federated learning method based on Charmilles secret sharing to resist stragglers, using a secret sharing scheme to split user data and user models in the initial stage, so that users can train data locally and reduce the complexity of data interaction calculation in federated learning.
[0006] To achieve the above object, the present invention provides a federated learning method based on Charmilles secret sharing to resist stragglers, comprising the following steps:
[0007] Step 1: Before training begins, each user processes the local data and divides the data into dimensions suitable for the polynomial network; then each user transfers his local data D iUse the Charmilles secret sharing scheme to split the data according to the number of users participating in the training, keep your own part, and distribute the split data to different participants;
[0008] Step 2: Before each training phase, each user receives model parameters from the server and first quantizes the model parameters. Then, each user uses the Charmilles secret sharing scheme to split its own model parameters, retain the split local parameters, and distribute the split parameters to different participants.
[0009] Step 3: All users use the segmented data distributed by different users and the received model parameters to train the model and upload the trained results to the server;
[0010] Step 4: The server uses the Charmier secret sharing scheme to reconstruct the data received from different users, quantizes the reconstructed data, updates the server model parameters, and sends the model parameters to the users;
[0011] Step 5: The user receives the parameters sent by the server, performs local gradient update calculations, updates local model parameters, and performs the next round of training until the model is fitted.
[0012] Optionally, the Charmilles secret sharing scheme is an optimized secret sharing scheme, generating a public parameter N=p in the initialization phase. 2 *q, and distribute N to each user participating in the training, and then restrict the coefficient generation of the secret polynomial in the secret polynomial generation phase;
[0013] The polynomial coefficients are restricted to a1, a2, …, a k-1 ∈p·q·Z p , the secret polynomial Where s is the secret value, i.e. the user’s data part and the model parameters received by the user from the server, and N is the public parameter generated in the initialization phase.
[0014] Optionally, during the execution of step 1, each user first quantizes the local data X i , Suppose the polynomial after the user uses the secret sharing scheme is F(x), and then the optimized secret sharing scheme is used to split the data. After the split, the data obtained by each participant n is
[0015] Optionally, the execution process of step 2 is specifically that the server distributes the model parameter W to each participant, and the model parameter held by each participant user n is W n, the total model parameter of the server is W0;
[0016] Quantify the model parameters of the user, the model parameters of user n Suppose the polynomial after the user uses the secret sharing scheme is g(x), use the optimized secret sharing scheme to split the data and distribute the data to different participants, then the model data obtained by each user n is
[0017] Optionally, during the execution of step 3, after receiving the user data and model parameters of different users, each user n now has the following data:
[0018] For each user, use the data to train the embedding. in Then the user will calculate the result Upload to the central server.
[0019] Optionally, during the execution of step 4, the server accepts calculation results uploaded by different users. The Lagrange interpolation algorithm is used to reconstruct the secret polynomial. The polynomial interpolation polynomial is: The server then uses the gradient descent method to update the model and distributes the total gradient to different users so that they can update the gradient.
[0020] Optionally, the execution process of step 5 is that the user receives the aggregated gradient from the server as Use the total gradient to update the local gradient W n , the local gradient after user update is The user then continues training with the updated parameters until the model is fit.
[0021] The present invention provides a federated learning method for resisting stragglers based on Shamier secret sharing. In view of the risk of data leakage in federated learning due to the need for users to upload intermediate parameters of training models to a server, a Shamier secret sharing scheme is adopted to distribute and encrypt data, thereby ensuring the security of data during transmission. In view of the high computational complexity of longitudinal federated learning based on homomorphic encryption, the present invention proposes to use a Shamier secret sharing scheme to replace the homomorphic encryption scheme, thereby replacing the complex ciphertext calculation of homomorphic encryption, and greatly reducing the complexity of data interaction calculation in federated learning. In view of the problem of stragglers in federated learning caused by different client resources and computing performance, the present invention adopts an improved Shamier secret sharing scheme, and distributes user data and user model parameters to different users using Shamier secret sharing in the initialization stage of training, so that the training stage is the data training stage for all users, and the embedded aggregation of all customers is losslessly reconstructed by decrypting the computational sharing of non-colluding customers. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 It is a schematic flow chart of the steps of a federated learning method for resisting stragglers based on Charmilles secret sharing of the present invention.
[0024] Figure 2 It is a schematic diagram comparing the performance of the stragglers problem in several situations in a specific embodiment of the present invention.
[0025] Figure 3 It is a schematic diagram of the difference between the process of the present invention and the traditional solution. DETAILED DESCRIPTION
[0026] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0027] See also Figure 1 The present invention provides a federated learning method for resisting stragglers based on Charmilles secret sharing, comprising the following steps:
[0028] S1: Before training begins, each user processes the local data and divides the data into dimensions suitable for the polynomial network; then each user transfers his local data D i Use the Charmilles secret sharing scheme to split the data according to the number of users participating in the training, keep your own part, and distribute the split data to different participants;
[0029] S2: Before each training phase, each user receives model parameters from the server and first quantizes the model parameters. Then, each user uses the Charmilles secret sharing scheme to split its own model parameters, retain the split local parameters, and distribute the split parameters to different participants.
[0030] S3: All users use the segmented data distributed by different users and the received model parameters to train the model and upload the trained results to the server;
[0031] S4: The server uses the Charmier secret sharing scheme to reconstruct the data received from different users, quantizes the reconstructed data, updates the server model parameters, and sends the model parameters to the users;
[0032] S5: The user receives the parameters sent by the server, performs local gradient update calculations, updates local model parameters, and performs the next round of training until the model is fitted.
[0033] The Charmilles secret sharing scheme is an optimized secret sharing scheme. In the initialization phase, a public parameter N=p is generated. 2 *q, and distribute N to each user participating in the training, and then restrict the coefficient generation of the secret polynomial in the secret polynomial generation phase;
[0034] Unlike traditional secret sharing schemes, the polynomial coefficients are not randomly selected, but are restricted to a1, a2, …, a k-1 ∈p·q·Z p , the secret polynomial Where s is the secret value, i.e. the user’s data part and the model parameters received by the user from the server, and N is the public parameter generated in the initialization phase.
[0035] During the execution of step S1, each user first quantizes the local data X i , Suppose the polynomial after the user uses the secret sharing scheme is F(x), and then the optimized secret sharing scheme is used to split the data. After the split, the data obtained by each participant n is Any user has a share of the data of every participant, but no user can obtain valid information about the user's real data through the share.
[0036] The execution process of step S2 is as follows: the server distributes the model parameter W to each participant. The model parameter held by each participant user n is W n , the total model parameter of the server is W0;
[0037] Quantify the model parameters of the user, the model parameters of user n Suppose the polynomial after the user uses the secret sharing scheme is g(x), use the optimized secret sharing scheme to split the data and distribute the data to different participants, then the model data obtained by each user n is
[0038] During the execution of step S3, after receiving the user data and model parameters of different users, each user n now has the following data:
[0039] For each user, use the data to train the embedding. in Then the user will calculate the result Upload to the central server.
[0040] During the execution of step S4, the server accepts calculation results uploaded by different users. The Lagrange interpolation algorithm is used to reconstruct the secret polynomial, and in this process, the training results of all users can be perfectly aggregated without any loss.
[0041] The polynomial interpolation polynomial is: The server then uses the gradient descent method to update the model and distributes the total gradient to different users so that they can update the gradient.
[0042] The execution process of step S5 is as follows: the user receives the aggregated gradient from the server: Use the total gradient to update the local gradient W n , the local gradient after user update is The user then continues training with the updated parameters until the model is fit.
[0043] Specifically, the method of the present invention can be mainly divided into the following parts:
[0044] (1) Construct a secret sharing multiplication sub-protocol that does not require communication. The design starts from the parameter initialization stage. By limiting the relationship between the coefficients of the secret polynomial and the modulus of the polynomial, the degree of the polynomial after multiplication is reduced. For the Charmilles threshold secret sharing scheme with a threshold of k, only k users are needed to restore the secret. It also supports polynomial calculations of arbitrary depth, which meets the deep calculation problem of neural networks. At the same time, security is guaranteed by using the difficulty of large prime number decomposition problem.
[0045] (2) Using polynomial networks instead of traditional neural networks can better calculate the user data segmented by the secret sharing scheme and the secret share data after the user model, and using the Lagrange interpolation method to speed up the calculation.
[0046] The polynomial network involved in the federated learning method for resisting stragglers based on Charmilles secret sharing described in the present invention is: This system uses such a simplified architecture, that is, for a PN polynomial network with D layers, the output embedding h∈R h is the input x∈R d Among them where x i is the i-th power of the input data, W i ∈R h*d ,b i is the weight matrix and bias vector of the i-th layer. D is the number of layers of the polynomial network. It is worth noting that the quadratic function x 2 as an activation function for polynomial networks.
[0047] After the server receives the data uploaded by the user, it uses the Lagrange interpolation method to reconstruct the gradient and restore the gradient value. The present invention uses the average method to represent the overall gradient of the server. After the server calculates the total gradient uploaded by the user, it performs an average process, that is, the total gradient is divided by the number of users, which represents the average gradient. The advantage of this is that it can maximize the training results representing the user. Where N represents the number of users, H represents the average gradient value of all users calculated by the server, and H n Represents the gradient value of local training uploaded by each user to the server.
[0048] Furthermore, the present invention is further explained by comparing specific embodiments:
[0049] like Figure 2The figure shows the impact of three server aggregation methods on the model when facing the stragglers problem. The three aggregation methods are connection aggregation, average aggregation, and average maximum aggregation. Each method performs four processing at the same time, namely, no stragglers problem, ignoring stragglers problem, waiting for stragglers problem, and asynchronous processing. Specifically, first for Figure 2 .a, the test is about the impact of the four processing methods on the server training model in terms of time and training accuracy under the connection aggregation method. It can be seen that among the four processing methods, the one without the problem of stragglers is the most efficient, followed by the methods of ignoring stragglers and using asynchronous aggregation. Figure 2 .b and Figure 3 .b are average aggregation and maximum aggregation respectively. It can be seen that in the four processing situations, the two aggregation methods have the fastest model training speed and the highest model accuracy when ignoring the problem of stragglers and processing without stragglers. Figure 2 By comparing several methods, it can be seen that when there are no stragglers in the model, the model training time is the shortest and the trained model accuracy is the highest. Compared with the best performing aggregation method, the connection method, the element average method performs equally well when there are no stragglers; 2) For the element average method, the server does not necessarily need to know the embedding information of a single client to calculate their sum, so it is possible to achieve a higher level of privacy protection. In order to achieve this embedding averaging, the present invention requires the embedding dimensions to be the same.
[0050] like Figure 3 It is shown that the solution of the present invention is different from the traditional solution. In the traditional vertical federated learning using homomorphic encryption, data between different users is not exchanged, that is, all users only use local data and local models for training. At the same time, encryption processing is required during the training process, which will result in high computational complexity and long calculation time in the training process. Figure 3 In the initialization phase of training, we use a secret sharing scheme to segment the user's data and the model parameters received from the server, and distribute the segmented data to each user participating in the training. This can avoid the complex encryption process. At the same time, since the data and model parameters have been segmented and distributed in the initialization phase, during the training process, only a threshold number of users need to upload trained data to the server, and the server can restore the data of all participants through Lagrange polynomial interpolation.
[0051] In summary, the present invention has the following beneficial effects:
[0052] 1. Compared with the existing vertical federated learning scheme based on homomorphic encryption, the present invention is a vertical federated learning based on a secret sharing scheme, which fundamentally reduces the high computational complexity and long computational time caused by the need for homomorphic encryption to perform intermediate values during the calculation process. At the same time, by using a secret sharing scheme in the initialization phase to split and distribute user data and user model parameters, there is no need to calculate intermediate values during the calculation process, and there is no need for communication between users.
[0053] 2. Compared with the existing vertical federated learning scheme, the present invention is a vertical federated learning based on a secret sharing scheme, which fundamentally eliminates the problem of data leakage caused by transmitting data to the server after the user training is completed. Due to the use of the secret sharing scheme, the user's data and user parameters have been divided, and the result of the user's local training is only a part of the secret share of all participants' training. Under the current scientific computing conditions, it is impossible to deduce user data and user model parameters from this. The security of the overall system is enhanced.
[0054] 3. Due to the differences in resources and computing performance between different clients, there will be a problem of stragglers during the training phase. The problem of stragglers is that some users with fast training speed can upload training data to the server, while some users take a long time to train, resulting in inconsistent server aggregated gradients, poor overall model training accuracy, and long training time. This solution uses a secret sharing scheme to split and distribute user data and user model parameters to different users.
[0055] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. A federated learning method based on Charmilles secret sharing to resist stragglers, characterized in that: The following steps are involved: Step 1: Before training begins, each user processes the local data and divides the data into dimensions suitable for the polynomial network; then each user transfers his local data D i Use the Charmilles secret sharing scheme to split the data according to the number of users participating in the training, keep your own part, and distribute the split data to different participants; Step 2: Before each training phase, each user receives model parameters from the server and first quantizes the model parameters. Then each user uses the Charmilles secret sharing scheme to split its own model parameters, retain the split local parameters, and distribute the split parameters to different participants. Step 3: All users use the segmented data distributed by different users and the received model parameters to train the model and upload the trained results to the server; Step 4: The server uses the Charmier secret sharing scheme to reconstruct the data received from different users, quantizes the reconstructed data, updates the server model parameters, and sends the model parameters to the users; Step 5: The user receives the parameters sent by the server, performs local gradient update calculations, updates local model parameters, and performs the next round of training until the model is fitted.
2. The method for federated learning based on Charmilles secret sharing to resist stragglers as claimed in claim 1, characterized in that: The Charmilles secret sharing scheme is an optimized secret sharing scheme. In the initialization phase, a public parameter N=p is generated. 2 *q, and distribute N to each user participating in the training, and then restrict the coefficient generation of the secret polynomial in the secret polynomial generation phase; The polynomial coefficients are restricted to a1, a2, …, a k-1 ∈p·q·Z p , the secret polynomial Where s is the secret value, i.e. the user’s data part and the model parameters received by the user from the server, and N is the public parameter generated in the initialization phase.
3. The method for federated learning based on Charmilles secret sharing to resist stragglers as claimed in claim 2, characterized in that: In the execution of step 1, each user first quantizes the local data X i , Suppose the polynomial after the user uses the secret sharing scheme is F(x), and then the optimized secret sharing scheme is used to split the data. After the split, the data obtained by each participant n is 4. The method for federated learning based on Charmilles secret sharing to resist stragglers as claimed in claim 3, characterized in that: The execution process of step 2 is as follows: the server distributes the model parameter W to each participant. The model parameter held by each participant user n is W n , the total model parameter of the server is W0; Quantify the model parameters of the user, the model parameters of user n Suppose the polynomial after the user uses the secret sharing scheme is g(x), use the optimized secret sharing scheme to split the data and distribute the data to different participants, then the model data obtained by each user n is 5. The method for federated learning based on Charmilles secret sharing to resist stragglers as claimed in claim 4, characterized in that: During the execution of step 3, after receiving the user data and model parameters of different users, each user n now has the following data: For each user, use the data to train the embedding. in Then the user will calculate the result Upload to the central server.
6. The method for federated learning based on Charmilles secret sharing to resist stragglers as claimed in claim 5, characterized in that: During the execution of step 4, the server accepts calculation results uploaded by different users. The Lagrange interpolation algorithm is used to reconstruct the secret polynomial. The polynomial interpolation polynomial is: The server then uses the gradient descent method to update the model and distributes the total gradient to different users so that they can update the gradient.
7. The method for federated learning based on Charmilles secret sharing to resist stragglers as claimed in claim 6, characterized in that: The execution process of step 5 is as follows: the user receives the aggregated gradient from the server: Use the total gradient to update the local gradient W n , the local gradient after user update is The user then continues training with the updated parameters until the model is fit.