Decentralized longitudinal federated learning privacy protection method
Through blockchain technology and Byzantine consensus, local model parameters are verified and aggregated in vertical federated learning, the single point failure caused by centralized architecture and low model performance are solved, decentralized vertical federated learning is realized, and the model is improved Robustness and accuracy, and ensure data privacy security.
Patent Information
- Application Number
- CN202410776614.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-17
- Publication Date
- 2024-11-15
AI Technical Summary
The vertical federated learning system based on centralized architecture is prone to single point failures, and the local model cannot be verified, resulting in low global model training accuracy.
Adopt the decentralized vertical federated learning privacy protection method, realize decentralized vertical federated learning through blockchain technology, use Byzantine consensus and weighted aggregation to form the aggregation intermediate results, verify the quality of local model parameters, and pass the blockchain verification. local intermediate results are aggregated to ensure the privacy and security of the model.
It significantly improves the robustness and accuracy of the model, reduces the negative impact of poor quality local samples on the global model, ensures the local data privacy and security of participants, solves the single point of failure problem, and improves model performance.
Smart Images

Figure CN118965416A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a decentralized vertical federated learning privacy protection method, and belongs to the technical field of machine learning. Background Art
[0002] The current vertical federated learning based on a centralized architecture faces a single point of failure problem because the entire system relies on a centralized server for data processing, model updates, and communication coordination. If this server has hardware failures, network problems, software vulnerabilities, or is attacked maliciously. Therefore, once the central server fails, the training of the entire vertical federated learning system will be interrupted. Blockchain provides a technical path for realizing decentralized vertical federated learning training. However, the local model of the existing vertical federated learning cannot be verified. When the participants have poor quality local data, it will have a negative impact on the global model training and reduce the accuracy of the global model.
[0003] Therefore, how to use blockchain technology to solve the single point failure and low model performance caused by the centralized architecture of vertical federated learning has become an urgent problem to be solved. Summary of the invention
[0004] The purpose of this invention is to solve the technical problems of single point failure and low model performance caused by the centralized architecture of vertical federated learning, and to propose a decentralized vertical federated learning privacy protection method.
[0005] The objective of the present invention is achieved through the following technical solutions:
[0006] The decentralized vertical federated learning privacy protection method of the present invention comprises the following steps:
[0007] Step 1: The member nodes of the participants in the decentralized vertical federated learning model include active parties, passive parties, and validators.
[0008] Step 2: After privacy intersection of the active party and the passive party of the participants described in step 1, sample-aligned participant data is formed, thereby forming active party data samples and passive party data samples.
[0009] Step 3: Initialize the decentralized vertical federated learning model, that is, construct the correspondence between the data samples and sample features of the participating member nodes described in step 1.
[0010] Step 4: Based on the passive party sample features formed by the corresponding relationship between the passive party data samples obtained in step 2 and step 3, the passive party performs forward training using the underlying model of the passive party sample features.
[0011] Step 4.1: After the passive party initializes the hyperparameters of the underlying model, it uses the underlying model of the passive party sample features for forward training to obtain the t-round local intermediate results after training, as shown in formula (1).
[0012]
[0013] Among them, i represents the i-th passive party, t represents the use of sample features Xi and the underlying model θ i Conduct the tth round of model training.
[0014] Step 4.2: The passive party sends the intermediate result to the verifier.
[0015] Step 5: The verifier receives the t-round local intermediate results from step 4 for verification, and after Byzantine consensus on the verified local intermediate results, forms an aggregated intermediate result in a weighted aggregation manner.
[0016] Step 5.1: The verifier calculates the verification distance between the local intermediate result of round t and the local intermediate result of round t-1 in the manner shown in formula (2), and compares the verification distance with a predetermined threshold to obtain a verification result as shown in formula (3).
[0017]
[0018] Among them, formula (2) represents the Euclidean distance between the local model parameters of the ith passive party in the tth round and the local model parameters in the previous round, i.e., the t-1th round, where represents the local model parameters of the i-th passive party in the t-th round, represents the local model parameters of the i-th passive party in the t+1th round.
[0019]
[0020] Among them, formula (3) is mainly used to express the verification result, where v represents whether the current local model of the passive party has passed the verification, and the verification is marked as 1, and the verification is not passed is marked as 0; represents the local model parameters of the i-th passive party in the t-th round.
[0021] Step 5.1.1: When verifying the distance > the predetermined threshold L, it indicates that the verification result fails; v is set to 0.
[0022] Step 5.1.2: When verifying the distance When < the predetermined threshold L, it means the verification result is passed; v is set to 1.
[0023] Step 5.2: Submit the local intermediate result that has passed the verification and conduct Byzantine consensus, thereby forming the on-chain intermediate result of the blockchain.
[0024] Step 5.3: Aggregate the intermediate results on the chain described in step 5.2 using an average aggregation method, as shown in formula (4), to form a global intermediate result.
[0025]
[0026] Where VT represents the total number of verified local intermediate results.
[0027] Step 6: The active party uses the global intermediate results obtained by aggregation described in step 5 to train the top-level model.
[0028] Step 6.1: The active party trains the top-level model on the aggregated intermediate results to obtain the prediction result shown in formula (5).
[0029] y p =θ top (h t ) (5)
[0030] Among them, θ top represents the top model owned by the active party, h t Represents the global intermediate result obtained by aggregation.
[0031] Step 6.2: Calculate the loss value of the prediction result obtained in step 6.1 by using the loss function and label assignment method, as shown in formula (6).
[0032] L t =F(Y,y p ) (6)
[0033] Among them, F represents the loss function, and the prediction result is y p , the label value is Y, and the loss value is L t .
[0034] Step 6.3: Using the gradient descent method and the loss value described in step 6.2, obtain the gradient value G of the active party top model, as shown in formula (7).
[0035]
[0036] Among them, formula (7) represents the process loss function L using the chain rule to calculate the intermediate gradient value t The gradient relative to the intermediate variable h. t represents the loss value of the tth round, h represents the intermediate result of the passive party, and y represents the output value of the network or the output value of an intermediate layer.
[0037] Step 7: The passive party uses the gradient value G of the active party's top model obtained in step 6 to reversely update the local model parameters.
[0038] Step 7.1: Use the gradient value G of the active top model to calculate the gradient value g of the local model i As shown in formula (8).
[0039]
[0040] Step 7.2: Use the gradient value g of the local model obtained in step 7.1 i And gradient descent method to update local model parameters.
[0041] Step 8: Repeat steps 3 to 7 until the training model reaches the expected convergence accuracy or reaches the pre-agreed maximum number of training rounds, at which point the training is stopped;
[0042] The vertical federated learning privacy protection method disclosed in the present invention aims at the problems of single point failure and low accuracy in existing vertical federated learning, and proposes a decentralized vertical federated learning privacy protection method to eliminate single failures and improve the performance of accuracy. The contribution value of the passive party is judged by the consensus group node in the blockchain, and the local value of the passive party after the aggregation node aggregates the consensus, so as to realize the decentralized vertical federated learning privacy protection method.
[0043] Beneficial effects:
[0044] 1. The present invention has good robustness. When the client has local data of poor quality, the present invention can significantly improve the performance of the model and reduce the negative impact of poor quality local samples on the accuracy of the global model.
[0045] 2. The present invention realizes decentralized vertical federated learning. The present invention uses blockchain technology to realize decentralized vertical federated learning, and uses consensus group nodes to verify the quality of the current client local training, solving the single failure problem faced by traditional federated learning.
[0046] 3. The present invention ensures the privacy and security of the local data of the participants. The present invention implements multi-party collaborative global model training through the embedded values of the local data of the interactive client, rather than sharing the original data, thus protecting the privacy and security of the original data. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a schematic diagram of the architecture of the present invention;
[0048] Figure 2 It is a schematic diagram of the process of the present invention;
[0049] Figure 3 This is the accuracy and loss result graph of the traditional longitudinal federated learning model;
[0050] Figure 4 This is a graph of the accuracy and loss value results of the decentralized longitudinal federated learning model. DETAILED DESCRIPTION
[0051] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.
[0052] This embodiment provides a decentralized vertical federated learning privacy protection method, using Figure 1 In the architecture shown, the system first initializes an image classification task and randomly selects K participants. The K participants represent clients that own local data and participate in vertical federated learning. , K is set to 5, indicating that there are currently 5 participants cooperating to train a global model for vertical federated learning. The local dataset is MNIST, images with known image classification labels. The image classification labels are 0 to 9. , Set the partitioned neural network as the training model for vertical federated learning, where one client owns the top model and the other clients own the bottom model. , The case uses PyTorch to implement vertical federated learning training and uses FISCO BCOS as the underlying blockchain development platform.
[0053] Example
[0054] like Figure 2 As shown, the decentralized vertical federated learning privacy protection method of the present invention is specifically implemented in the following steps:
[0055] Step 1: The member nodes of the participants in the decentralized vertical federated learning model include active parties, passive parties, and validators.
[0056] Step 2: After privacy intersection of the active party and the passive party of the participants described in step 1, sample-aligned participant data is formed, thereby forming active party data samples and passive party data samples.
[0057] Step 3: Initialize the decentralized vertical federated learning model, that is, construct the correspondence between the data samples and sample features of the participating member nodes described in step 1.
[0058] Step 4: Based on the passive party sample features formed by the corresponding relationship between the passive party data samples obtained in step 2 and step 3, the passive party performs forward training using the underlying model of the passive party sample features.
[0059] Step 4.1: After the passive party initializes the hyperparameters of the underlying model, it uses the underlying model of the passive party sample features for forward training to obtain the t-round local intermediate results after training, as shown in formula (1).
[0060]
[0061] Among them, i represents the i-th passive party, t represents the use of sample features Xi and the underlying model θ i Conduct the tth round of model training.
[0062] Step 4.2: The passive party sends the intermediate result to the verifier.
[0063] Step 5: The verifier receives the t-round local intermediate results from step 4 for verification, and after Byzantine consensus on the verified local intermediate results, forms an aggregated intermediate result in a weighted aggregation manner.
[0064] Step 5.1: The verifier calculates the verification distance between the local intermediate result of round t and the local intermediate result of round t-1 in the manner shown in formula (2), and compares the verification distance with a predetermined threshold to obtain a verification result as shown in formula (3).
[0065]
[0066] Among them, formula (2) represents the Euclidean distance between the local model parameters of the ith passive party in the tth round and the local model parameters in the previous round, i.e., the t-1th round, where represents the local model parameters of the i-th passive party in the t-th round, represents the local model parameters of the i-th passive party in the t+1th round.
[0067]
[0068] Among them, formula (3) is mainly used to express the verification result, where v represents whether the current local model of the passive party has passed the verification, and the verification is marked as 1, and the verification is not passed is marked as 0; represents the local model parameters of the i-th passive party in the t-th round.
[0069] Step 5.1.1: When verifying the distance > the predetermined threshold L, it indicates that the verification result fails; v is set to 0.
[0070] Step 5.1.2: When verifying the distance When < the predetermined threshold L, it means the verification result is passed; v is set to 1.
[0071] Step 5.2: Submit the local intermediate result that has passed the verification and conduct Byzantine consensus, thereby forming the on-chain intermediate result of the blockchain.
[0072] Step 5.3: Aggregate the intermediate results on the chain described in step 5.2 using an average aggregation method, as shown in formula (4), to form a global intermediate result.
[0073]
[0074] Where VT represents the total number of verified local intermediate results.
[0075] Step 6: The active party uses the global intermediate results obtained by aggregation described in step 5 to train the top-level model.
[0076] Step 6.1: The active party trains the top-level model on the aggregated intermediate results to obtain the prediction result shown in formula (5).
[0077] y p =θ top (h t ) (5)
[0078] Among them, θ top represents the top model owned by the active party, h t Represents the global intermediate result obtained by aggregation.
[0079] Step 6.2: Calculate the loss value of the prediction result obtained in step 6.1 by using the loss function and label assignment method, as shown in formula (6).
[0080] L t =F(Y,y p ) (6)
[0081] Among them, F represents the loss function, and the prediction result is y p , the label value is Y, and the loss value is L t .
[0082] Step 6.3: Using the gradient descent method and the loss value described in step 6.2, obtain the gradient value G of the active party top model, as shown in formula (7).
[0083]
[0084] Among them, formula (7) represents the process loss function L using the chain rule to calculate the intermediate gradient value t The gradient relative to the intermediate variable h. t represents the loss value of the tth round, h represents the intermediate result of the passive party, and y represents the output value of the network or the output value of an intermediate layer.
[0085] Step 7: The passive party uses the gradient value G of the active party's top model obtained in step 6 to reversely update the local model parameters.
[0086] Step 7.1: Use the gradient value G of the active top model to calculate the gradient value g of the local model i As shown in formula (8).
[0087]
[0088] Step 7.2: Use the gradient value g of the local model obtained in step 7.1 i And gradient descent method to update local model parameters.
[0089] Step 8: Repeat steps 3 to 7 until the training model reaches the expected convergence accuracy or reaches the pre-agreed maximum number of training rounds, at which point the training is stopped;
[0090] To further demonstrate the superiority of the present invention, the present invention conducts the following training experiment for illustration. In this experiment, the passive party submits incorrect results during the training process to observe the model performance of traditional vertical federated learning and decentralized vertical federated learning. The accuracy and loss value results of the traditional vertical federated learning model are shown in the experimental results. Figure 3 The accuracy and loss results of the decentralized longitudinal federated learning model are shown in Figure 4 shown.
[0091] The results show that when the passive party submits incorrect local training, the training accuracy of traditional vertical federated learning has obvious fluctuations in the 5th, 10th, 15th, 20th and 25th rounds of training. In contrast, under the same training conditions, the model accuracy of decentralized vertical federated learning is not affected by the number of rounds, realizing decentralized vertical federated learning with higher model performance.
Claims
1. A decentralized vertical federated learning privacy protection method, characterized by: The following steps are included: Step 1: The member nodes of the decentralized vertical federated learning model participants include active parties, passive parties, and validators; Step 2: After privacy intersection of the active party and the passive party of the participants described in step 1, sample-aligned participant data is formed, thereby forming active party data samples and passive party data samples; Step 3: Initialize the decentralized vertical federated learning model, that is, construct the corresponding relationship between the data samples and sample features of the participating member nodes described in step 1; Step 4: Based on the passive party sample features formed by the corresponding relationship between the passive party data samples obtained in step 2 and step 3, the passive party performs forward training using the underlying model of the passive party sample features; Step 5: The verifier receives the t-round local intermediate results from step 4 for verification, and after the Byzantine consensus is performed on the verified local intermediate results, an aggregated intermediate result is formed in a weighted aggregation manner; Step 6: The active party uses the global intermediate results obtained by aggregation in step 5 to train the top-level model; Step 7: The passive party uses the gradient value G of the active party's top model obtained in step 6 to reversely update the local model parameters; Step 8: Repeat steps 3 to 7 until the training model reaches the expected convergence accuracy or reaches the pre-agreed maximum number of training rounds, at which point the training is stopped.
2. The decentralized vertical federated learning privacy protection method according to claim 1, characterized in that: Step 4 is implemented as follows: Step 4.1: After the passive party initializes the hyperparameters of the underlying model, it uses the underlying model of the passive party sample features for forward training, thereby obtaining the t-round local intermediate results after training, as shown in formula (1); Among them, i represents the i-th passive party, t represents the use of sample features Xi and the underlying model θ i Conduct the tth round of model training; Step 4.2: The passive party sends the intermediate result to the verifier.
3. The decentralized vertical federated learning privacy protection method according to claim 1, characterized in that: Step 5 is implemented as follows: Step 5.1: The verifier calculates the verification distance between the local intermediate result of round t and the local intermediate result of round t-1 in the manner shown in formula (2), and compares the verification distance with a predetermined threshold to obtain a verification result as shown in formula (3); Among them, formula (2) represents the Euclidean distance between the local model parameters of the ith passive party in the tth round and the local model parameters in the previous round, i.e., the t-1th round, where represents the local model parameters of the i-th passive party in the t-th round, represents the local model parameters of the i-th passive party in the t+1th round; Among them, formula (3) is mainly used to express the verification result, where v represents whether the current local model of the passive party has passed the verification, and the verification is marked as 1, and the verification is not passed is marked as 0; represents the local model parameters of the i-th passive party in the t-th round; Step 5.2: Submit the local intermediate results that have passed the verification and conduct Byzantine consensus, thereby forming the on-chain intermediate results of the blockchain; Step 5.3: Aggregate the intermediate results on the chain described in step 5.2 using an average aggregation method; as shown in formula (4), a global intermediate result is formed; Where VT represents the total number of verified local intermediate results.
4. The decentralized vertical federated learning privacy protection method according to claim 3, characterized in that: Step 5.1 is implemented as follows: Step 5.1.1: When verifying the distance When , it means the verification result fails; set v to 0; Step 5.1.2: When verifying the distance , it means the verification result passed; set v to 1.
5. The decentralized vertical federated learning privacy protection method according to claim 1, characterized in that: Step 6 is implemented as follows: Step 6.1: The active party trains the top-level model on the aggregated intermediate results to obtain the prediction results shown in formula (5); y p =θ top (h t ) (5) Among them, θ top represents the top model owned by the active party, h t Represents the global intermediate result obtained by aggregation; Step 6.2: Calculate the loss value of the prediction result obtained in step 6.1 by using the loss function and label assignment method, as shown in formula (6); L t =F(Y,y p ) (6) Among them, F represents the loss function, and the prediction result is y p , the label value is Y, and the loss value is L t ; Step 6.3: Using the gradient descent method and the loss value described in step 6.2, obtain the gradient value G of the active party top model; as shown in formula (7); Among them, formula (7) represents the process loss function L using the chain rule to calculate the intermediate gradient value t The gradient relative to the intermediate variable h; where L t represents the loss value of the tth round, h represents the intermediate result of the passive party, and y represents the output value of the network or the output value of an intermediate layer.
6. The decentralized vertical federated learning privacy protection method according to claim 1, characterized in that: Step 7 is implemented as follows: Step 7.1: Use the gradient value G of the active top model to calculate the gradient value g of the local model i As shown in formula (8); Step 7.2: Use the gradient value g of the local model obtained in step 7.1 i And gradient descent method to update local model parameters.
