An efficient heterogeneous longitudinal federated learning method based on unsupervised learning
Patent Information
- Application Number
- CN202311593450.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-27
- Publication Date
- 2026-09-01
- Estimated Expiration
- 2043-11-27
AI Technical Summary
通信成本是限制纵向联邦学习广泛应用的另一个关键挑战
[0050] The method of the present invention has the following advantages compared with the prior art:
Smart Images
Figure CN117648994B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a heterogeneous vertical federated learning method, and more particularly to an efficient heterogeneous vertical federated learning method based on unsupervised learning, belonging to the field of distributed federated learning technology in artificial intelligence. Background Technology
[0002] Data silos formed between mobile and IoT devices severely hinder secure data flow. Federated learning, a new paradigm of distributed machine learning, makes secure data flow possible. Federated learning achieves this by obtaining the local models of each participant and aggregating them to obtain a global model.
[0003] In vertical federated learning, the participating clients are typically divided into one active party and multiple passive parties. The active party possesses features and labels, while the passive parties only have sample features. During training, the passive parties assist the active party in training the model. In practical applications, the computing power and resources of the participating parties are often heterogeneous. To accommodate these heterogeneous participants, it is necessary to train heterogeneous network models of different sizes with the same functionality for each participant.
[0004] However, in existing vertical federated learning techniques, the active party aggregates intermediate data or prediction results to gain local feature knowledge of the client. Generally, these methods assume that the participants have the same model structure. Furthermore, existing vertical federated learning techniques barely consider that the convergence of the global model is simultaneously affected by the local model structure and the samples. Therefore, existing vertical federated learning methods are not suitable for collaboratively training the global model among heterogeneous participants. How to achieve vertical federated learning in heterogeneous environments is one of the challenges.
[0005] Furthermore, communication cost is another key factor affecting the widespread application of vertical federated learning techniques. In principle, existing research on reducing communication in vertical federated learning includes data compression and local multi-round training. Generally, compressing local data involves passively compressing the local data before communication begins. Therefore, it can reduce the transmission of data dimensions and lower communication costs. However, in vertical federated learning, compressing local data is not an effective way to reduce communication costs. This is because the passive party needs to communicate with the active party in each round to calculate the loss value in each round of local training. Vertical federated learning involves a large number of training rounds, and reducing the amount of communication data also requires reducing the number of communication rounds to lower communication costs.
[0006] To reduce communication between the active and passive parties, existing research has proposed local multi-round training. That is, the client first performs multiple rounds of local training, followed by comprehensive training. However, existing research on local multi-round training almost always assumes that the passive party possesses a copy of the label, which is inconsistent with real-world applications. This means that in vertical federated learning, only one active party possesses the label, while other passive parties only possess the characteristics of the sample. The passive parties will assist the active party in implementing training. Communication costs are another key challenge limiting the widespread application of vertical federated learning.
[0007] Therefore, in scenarios where the local models of the participants are heterogeneous, how to effectively reduce communication between the participants and improve the performance of heterogeneous vertical federated learning is a key issue we are currently facing. Summary of the Invention
[0008] The purpose of this invention is to address the problems and shortcomings of existing technologies by creatively proposing an efficient heterogeneous vertical federated learning method based on unsupervised learning in order to realize a vertical federated learning method with heterogeneous participants and reduce the communication volume of global model training.
[0009] First, the concepts and contents involved in this invention will be explained.
[0010] Vertical federated learning: refers to the federated learning training process in which participants achieve sample alignment and feature joint processing while ensuring that the local data of each participant does not leave the domain.
[0011] Participants: refers to clients that possess local data and participate in federated learning and training.
[0012] Active party: The participant that possesses label values and local features is called the active party.
[0013] Passive party: A participant that only possesses local features but not labels is called a passive party.
[0014] Representation layer: This refers to the part of the local heterogeneous network before the activation function. The representation layer embeds all local features into the same space and outputs local embedding values.
[0015] Decision layer: refers to the part after the activation function of the local heterogeneous network. The prediction results of the local heterogeneous network are obtained through the decision layer.
[0016] Prototype: Refers to the local embedding value output by the representation layer of a heterogeneous network. Since all participants embed their local features into the same space, the local embedding value is called the prototype.
[0017] The present invention is achieved using the following technical solution.
[0018] An efficient heterogeneous longitudinal federated learning method based on unsupervised learning includes the following steps:
[0019] Step 1: Model Initialization. Each participant initializes its local model parameters and local optimization algorithm.
[0020] Step 2: Without exposing the participants' local data, each participant finds the intersection of their data using a privacy set. Participants include passive and active parties.
[0021] Step 3: The passive party trains a local pre-trained model using local features and unsupervised learning methods.
[0022] Specifically, step 3 includes the following steps:
[0023] Step 3.1: The passive side initializes the parameters required for unsupervised learning, including the random target value C and the learning rate η;
[0024] Step 3.2: The passive side calculates the optimal perturbation matrix P for the random target C in this round, which can be achieved using the Hungarian algorithm;
[0025] Step 3.3: The passive party calculates the loss value L of the local model in this round based on the optimal perturbation matrix P;
[0026] Step 3.4: The passive side calculates the local model gradient g based on the loss value L;
[0027] Step 3.5: The passive side uses gradient descent to update the local model parameters using the local model gradient g;
[0028] Step 3.6: Repeat steps 3.2 to 3.5 until the local model reaches the expected convergence accuracy or reaches the pre-negotiated maximum number of training rounds, at which point training is stopped;
[0029] Step 4: The passive party uses local features and the representation layer of the pre-trained model to compute local embedding values (prototypes).
[0030] Specifically, step 4 includes the following steps:
[0031] Step 4.1: The passive side uses local features as input to the representation layer of the pre-trained model. After the representation layer calculates the values, it outputs the local embedding values.
[0032] Step 4.2: The passive party uses the local embedding value as the input to the prediction layer of the pre-trained model. After the prediction layer calculates, it outputs the local prediction result.
[0033] Step 4.3: The passive party sends the local embedded values and local prediction results to the active party (the labeled participant).
[0034] Step 5: The active party calculates the aggregate weight value of each passive party.
[0035] Specifically, step 5 includes the following steps:
[0036] Step 5.1: The active party receives the local prediction results sent by all passive parties;
[0037] Step 5.2: The active party calculates the weight value of the passive party based on the local labels and the prediction results sent by the passive party;
[0038] Step 6: Train the global model.
[0039] Specifically, step 6 includes the following steps:
[0040] Step 6.1: The active party uses local features and the representation layer of the global model to calculate the local embedding value.
[0041] Step 6.2: The active party uses weights to aggregate the local embedding values of each passive party to obtain the global embedding value of the passive party;
[0042] Step 6.3: The active party adds the passive party's global embedding value to the local embedding value to obtain the global embedding value;
[0043] Step 6.4: The active party uses the global embedding value as input to the global model prediction layer to calculate the global prediction value;
[0044] Step 6.5: The active party calculates the global loss function based on the global predicted values and labels;
[0045] Step 6.6: The active party calculates the gradient value based on the global loss function;
[0046] Step 6.7: The active side uses gradient descent to update the global model;
[0047] Step 6.8: Repeat steps 6.1 to 6.7 until the trained model reaches the expected convergence accuracy or the pre-negotiated maximum number of training rounds, at which point training is stopped.
[0048] Through the aforementioned technical means, efficient heterogeneous vertical federated learning based on unsupervised learning has been achieved without data leakage or privacy infringement.
[0049] Beneficial effects
[0050] The method of the present invention has the following advantages compared with the prior art:
[0051] 1. This invention improves the accuracy of training models in vertical federated learning with heterogeneous participants. The invention employs a weighted aggregation prototype method to obtain the local knowledge values of all participants, reducing reliance on heterogeneous model information. This reduces the impact of heterogeneous model information from other participants on the accuracy of the training model, thereby improving its accuracy.
[0052] 2. In this invention, the passive party uses unsupervised learning to achieve multiple rounds of local training, thereby reducing the number of communication rounds between the passive party and the active party and improving the training efficiency of vertical federated learning.
[0053] 3. This invention offers strong privacy protection. It aggregates prototype data rather than raw data, thus reducing the risk of raw data leakage. Furthermore, by injecting random numbers into the local embedded values, the initiating party cannot obtain the participating party's local embedded values, further protecting the participating party's raw data. Attached Figure Description
[0054] Figure 1 This is a schematic diagram of the method of the present invention. Detailed Implementation
[0055] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments.
[0056] Example
[0057] First, the system initializes an image classification task and randomly selects K participants, which are divided into 1 active participant and K-1 passive participants. The K participants then use heterogeneous longitudinal federated learning to train the model.
[0058] Furthermore, in this embodiment, K is set to 4, with 1 active party and 3 passive parties.
[0059] Furthermore, the local dataset is MNIST, containing images with known image classification labels. The image classification labels range from 0 to 9.
[0060] Furthermore, three heterogeneous local models were set up, including fully connected neural network MLP, convolutional neural network CNN, and convolutional neural network LetNet.
[0061] like Figure 1 As shown. An efficient heterogeneous longitudinal federated learning method based on unsupervised learning includes the following steps:
[0062] Step 1: Model initialization.
[0063] Each participant selects one of the three heterogeneous local models as its local model and chooses stochastic gradient descent as its local optimization algorithm. The local model selected by the active party is then determined as the global model.
[0064] Step 2: Without exposing the participants' local data, the four participants use privacy set intersection technology to find the intersection of all participants' data.
[0065] Step 3: Each passive party trains a local pre-trained model in parallel using local features and unsupervised learning methods.
[0066] Specifically:
[0067] Step 3.1: The first k The passive party initializes the parameters required for unsupervised learning, including random target values. and learning rate
[0068] Step 3.2: The first k The passive party uses the Hungarian algorithm to calculate the random target in round t. Optimal perturbation matrix
[0069] Step 3.3: The passive side uses the optimal perturbation matrix. Calculate the loss value L of the local model in this round. (t) :
[0070]
[0071] in, Indicates the lth k A local model for the passive party; Indicates the lth k Features of the passive participant model training; This represents the prediction result obtained by the neural network; This represents the optimal perturbation matrix for round t. denoted as random target; n represents the total number of features used in training.
[0072] Step 3.4: The passive party determines the loss value. Calculate the gradient g of the local model (t) :
[0073]
[0074] Step 3.5: The passive side uses gradient descent and the local model gradient g. (t) Update local model parameters:
[0075]
[0076] in, Indicates the lth k The learning rate of each participant.
[0077] Step 3.6: Repeat steps 3.2 to 3.5 until the local model reaches the expected convergence accuracy or the pre-negotiated maximum number of training rounds, at which point training is stopped.
[0078] Step 4: Each passive party computes local embedding values (prototypes) using local features and the representation layer of the pre-trained model.
[0079] Specifically, it includes the following steps:
[0080] Step 4.1: The k-th passive party will perform local feature processing. Input to the representation layer of the pre-trained model After the presentation layer performs calculations, the local embedded value is output.
[0081] Step 4.2: The k-th passive party uses the local embedded value As the prediction layer of the pre-trained model The input is processed by the prediction layer, and the local prediction result is output.
[0082] Step 4.3: The k-th passive party embeds the local value. and local prediction results Send to the initiating party (the tagged participant).
[0083] Step 5: The active party calculates the aggregate weight value of each passive party.
[0084] Specifically, it includes the following steps:
[0085] Step 5.1: The active party receives the local prediction results sent by all passive parties.
[0086] Step 5.2: The active party calculates the passive party's weight value based on the local labels and the prediction results sent by the passive party.
[0087] Step 6: Global model training.
[0088] Specifically, it includes the following steps:
[0089] Step 6.1: The initiating party uses local features and global model Presentation layer Calculate the local embedding value
[0090] Step 6.2: The active party uses weights to aggregate the local embedding values of each passive party to obtain the global embedding value E of the passive party. p , Where K represents the total number of passive parties, and k represents the kth participant.
[0091] Step 6.3: The active party adds the passive party's global embedding value E to its local embedding value. p Obtain the global embedding value E.
[0092] Step 6.4: The active party uses the global embedding value E as the global model prediction layer. The input is used to calculate the global predicted value.
[0093] Step 6.5: The active party calculates the global loss function based on the global predicted value and the label.
[0094] Step 6.6: The active party calculates the gradient value g based on the global loss function.
[0095] Step 6.7: The active side updates the global model using gradient descent. This represents the learning rate of the active party l1.
[0096] Step 6.8: Repeat steps 6.1 to 6.7 until the trained model reaches the expected convergence accuracy or the pre-negotiated maximum number of training rounds, at which point training is stopped.
[0097] This enables efficient heterogeneous vertical federated learning based on unsupervised learning without leaking data or infringing on privacy.
[0098] The training results are summarized in Tables 1 and 2.
[0099] Table 1 Comparison of Algorithm Performance
[0100]
[0101] Table 2 Comparison of Weighted Aggregation Performance
[0102]
[0103] The above description is merely a preferred embodiment of the present invention, and the present invention should not be limited to the content disclosed in this embodiment and the accompanying drawings. Any equivalent or modified embodiments made without departing from the spirit of the present invention fall within the scope of protection of the present invention.
Claims
1. An efficient heterogeneous longitudinal federated learning method based on unsupervised learning, characterized in that, The system initializes an image classification task and randomly selects K participants, which are divided into 1 active participant and K-1 passive participants. The K participants call a heterogeneous longitudinal federated learning model for training. Three heterogeneous local models are set up, including a fully connected neural network MLP, a convolutional neural network CNN, and a convolutional neural network LetNet. Includes the following steps: Step 1: Each participant initializes its local model parameters and local optimization algorithm; Step 2: Without exposing the local data of the participants, each participant finds the intersection of their data by using privacy sets; Among them, the participating parties refer to the clients that possess local data and participate in federated learning training, including passive parties and active parties; An active party refers to a participant that possesses both label values and local characteristics; A passive party refers to a participant that possesses only local characteristics but not labels; Step 3: The passive party uses local features and unsupervised learning methods to train a local pre-trained model; Step 4: The passive party uses local features and the representation layer of the pre-trained model to compute local embedding values; Step 5: The active party calculates the aggregate weight value of each passive party; Step 5.1: The active party receives the local prediction results sent by all passive parties; Step 5.2: The active party calculates the weight value of the passive party based on the local labels and the prediction results sent by the passive party; Step 6: Train the global model; Step 6.1: The active party uses local features and the representation layer of the global model to calculate the local embedding value; Step 6.2: The active party uses weights to aggregate the local embedding values of each passive party to obtain the global embedding value of the passive party; Step 6.3: The active party adds the passive party's global embedding value to the local embedding value to obtain the global embedding value; Step 6.4: The active party uses the global embedding value as input to the global model prediction layer to calculate the global prediction value; Step 6.5: The active party calculates the global loss function based on the global predicted values and labels; Step 6.6: The active party calculates the gradient value based on the global loss function; Step 6.7: The active side uses gradient descent to update the global model; Step 6.8: Repeat steps 6.1 to 6.7 until the trained model reaches the expected convergence accuracy or the pre-negotiated maximum number of training rounds, at which point training is stopped.
2. The efficient heterogeneous vertical federated learning method based on unsupervised learning as described in claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: The passive side initializes the parameters required for unsupervised learning, including the random target value C and the learning rate. ; Step 3.2: The passive side calculates the optimal perturbation matrix P for the random target C in this round; Step 3.3: The passive party calculates the loss value L of the local model in this round based on the optimal perturbation matrix P; Step 3.4: The passive side calculates the local model gradient based on the loss value L. ; Step 3.5: The passive side uses gradient descent to utilize the local model gradient. To update the local model parameters; Step 3.6: Repeat steps 3.2 to 3.5 until the local model reaches the expected convergence accuracy or the pre-negotiated maximum number of training rounds, at which point training is stopped.
3. The efficient heterogeneous vertical federated learning method based on unsupervised learning as described in claim 1, characterized in that, Step 4 includes the following steps: Step 4.1: The passive side uses local features as input to the representation layer of the pre-trained model. After the representation layer calculates the values, it outputs the local embedding values. Step 4.2: The passive party uses the local embedding value as the input to the prediction layer of the pre-trained model. After the prediction layer calculates, it outputs the local prediction result. Step 4.3: The passive party sends the local embedded values and local prediction results to the active party.