Longitudinal federal continuous learning privacy protection method based on prototype
By constructing class prototypes and enhancing previous task knowledge, the problem of model forgetting in vertical federated learning is solved, and the learning performance and accuracy of the model is improved, and it is suitable for continuous learning in dynamic environments.
Patent Information
- Application Number
- CN202510514790.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-09-02
AI Technical Summary
In vertical federated learning, over time, the model's forgetting knowledge of previous tasks leads to a decrease in learning effect, making it difficult to maintain memory of the learned knowledge while constantly adding new tasks and new data.
The vertical federated continuous learning method based on prototype is adopted to achieve continuous learning of the model by building class prototypes and enhancing the knowledge of previous tasks, combining current knowledge, and fixing model parameters important to the old tasks on the client.
This reduces the catastrophic forgetting of previous task knowledge by the model, improves the training performance and accuracy of the model, and realizes stable learning in a dynamic environment.
Smart Images
Figure CN120579199A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a prototype-based vertical federated continuous learning privacy protection method, belonging to the technical field of federated learning. Background Art
[0002] In modern society, data privacy protection is becoming increasingly important, and awareness of data privacy protection is rapidly increasing. In specific sectors, such as healthcare, finance, and the internet, privacy protection requirements for sensitive data are even stricter. In these situations, data often exists in silos across organizations and institutions, significantly reducing its value. Traditional centralized data processing methods are no longer able to meet data privacy protection requirements. Therefore, how to share and utilize data without compromising user privacy has become a key topic in current research and application.
[0003] Federated learning, an emerging distributed machine learning technology, provides an effective solution for sharing and utilizing data without compromising user privacy. The core concept of federated learning is to train a model on each device, exchanging only model parameters rather than data directly, thereby protecting user privacy. Specifically, federated learning allows multiple participants, such as devices and institutions, to jointly train a global model, but each participant's data remains locally and is not uploaded or exchanged. This allows participants to improve the performance of the global model through joint model training while ensuring data privacy.
[0004] In the federated learning framework, participants only share locally trained model updates, such as gradients and weights, rather than the data itself. All training occurs locally, and data is retained locally, significantly enhancing privacy protection. This is particularly true in vertical federated learning, where a single active party typically initiates joint training, with multiple passive parties collaborating. The goal of vertical federated learning is to address data silos through collaborative training across diverse data sources while ensuring privacy. While vertical federated learning provides technical support for addressing data silos and protecting privacy, in practice, participants' sample data, such as features and categories, continues to grow over time, shifting the model's training tasks and data distribution. This can lead to a sometimes catastrophic loss of knowledge learned from previous tasks. Consequently, ensuring that the model retains its learned knowledge as new tasks and data are added becomes a pressing challenge.
[0005] Therefore, achieving vertical federated continuous learning, mitigating the loss of knowledge from previous tasks as new data and tasks are added over time, is a pressing technical challenge. Solving this challenge will enable broader applications of federated learning in dynamic environments, particularly in scenarios involving long-term learning and continuous updates, enabling a more efficient and stable learning process. Summary of the Invention
[0006] To address the problem in existing technologies that models tend to forget previous task knowledge over time, the present invention creatively proposes a prototype-based vertical federated continuous learning privacy protection method.
[0007] In order to explain the present invention clearly, some concepts involved in the present invention are first explained.
[0008] Vertical federated learning: This refers to the federated learning training process in which each participant achieves sample alignment and feature combination while ensuring that the participant's local data does not leave the domain.
[0009] Continuous learning: This means that as time goes by, the number of categories of training data owned by the participants continues to increase, and the number of categories recognized by the model also continues to increase;
[0010] Client: refers to the participant who only has some local features of the sample and participates in model training;
[0011] Server: refers to the party that owns the sample labels and participates in model training;
[0012] Local model: refers to the local heterogeneous network owned by the client, embeds local features into the same space, and outputs local embedding values;
[0013] Global model: refers to the global network owned by the server, and the aggregated local embedding values are input into the network to obtain the prediction results;
[0014] Class prototype: refers to the prototype of each class constructed according to the local embedded values of the participants, that is, one class corresponds to one prototype.
[0015] The present invention provides a prototype-based vertical federated continuous learning privacy protection method, comprising the following steps:
[0016] Step 1: Model initialization and task initialization. Each client initializes the local model parameters and local optimization algorithm.
[0017] Step 2: Without exposing the local data of the participants, the client and server use the privacy set intersection technique to obtain the intersection of all training data, thereby achieving sample alignment of the training data.
[0018] Step 3: The client uses the local sample features it has to perform forward training on the local model to obtain the local embedding value.
[0019] Step 4: The client sends its local embedded value to the server.
[0020] Step 5: The server generates a global embedding value and a class prototype by aggregating local embedding values. Step 5 further includes the following sub-steps:
[0021] Sub-step 5.1: The server aggregates all local embedding values using the average aggregation method to obtain the current global embedding value;
[0022] Sub-step 5.2: The server obtains the embedding value corresponding to each label based on the current global embedding value and the sample label;
[0023] Sub-step 5.3: The server averages the embedding values of each label to obtain the class prototype.
[0024] Step 6: The server enhances the previous task knowledge, said step 6 further comprising the following sub-steps:
[0025] Sub-step 6.1: The server determines whether the current task is the first task. If the current task is the first task, the server proceeds to the next step, i.e., step 7; otherwise, the server proceeds to the following sub-steps 6.2 to 6.4;
[0026] Sub-step 6.2: Get the previous prototype based on the global prototype list and the current prototype category;
[0027] Sub-step 6.3: Obtain an enhanced previous class prototype based on the previous prototype and the current prototype;
[0028] Sub-step 6.4: Construct a global embedding value of the prior knowledge based on the prior class prototype and random sampling algorithm;
[0029] Sub-step 6.5: Add the global embedding value of the previous knowledge to the current global embedding value.
[0030] Step 7: The server updates the global model according to the current global embedding value. Step 7 further includes the following sub-steps:
[0031] Sub-step 7.1: The server performs forward propagation of the global model based on the current global embedding value to obtain the current prediction value;
[0032] Sub-step 7.2: The server obtains the current loss value based on the current prediction value and the current sample label value;
[0033] Sub-step 7.3: The server updates the global model based on the current loss value and stochastic gradient descent.
[0034] Sub-step 7.4: The server sends the gradient value corresponding to each client to each client.
[0035] Step 8: The client updates the local model according to the corresponding gradient value, and step 8 further includes the following sub-steps: sub-step 8.1, sub-step 8.2 and sub-step 8.3;
[0036] Sub-step 8.1: The client determines whether the current task is the first task. If so, execute sub-step 8.3; otherwise, execute sub-step 8.2.
[0037] Sub-step 8.2: The client fixes some local model parameters that are important for the previous task;
[0038] Sub-step 8.3: The client updates the unpinned local model based on the gradient value.
[0039] Step 9: Repeat steps 3 to 8 until the training model reaches the expected convergence accuracy or reaches the pre-agreed maximum number of training rounds, at which point the training is stopped, thereby achieving a privacy-preserving method for class-enhanced longitudinal federated learning.
[0040] Beneficial effects:
[0041] 1. The present invention constructs a class prototype, uses prototype enhancement to estimate old task knowledge, and uses current knowledge and estimated previous task knowledge to update the global model, thereby enabling the global model to maintain continuous learning, reducing the catastrophic forgetting of previous task knowledge, and improving the training performance of the model.
[0042] 2. During the update process of the local model on the client, the present invention fixes some model parameters that are important for the old task, so that the local model has both current knowledge and previous knowledge, reduces the catastrophic forgetting of previous task knowledge, and improves the performance of the local model.
[0043] 3. The present invention realizes the transfer of previous task knowledge by adopting prototypes and fixing some local model parameters in the global model training process, thereby solving the problem of catastrophic forgetting of previous tasks and improving the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 It is a schematic flow diagram of the present invention. DETAILED DESCRIPTION
[0045] In order to better illustrate the purpose and advantages of the present invention, the following further describes the invention in conjunction with the accompanying drawings and examples. It should be noted that the embodiments and examples of the present invention are only used to further explain the present invention in detail, but do not constitute any limitation to the present invention.
[0046] This embodiment provides a prototype-based, longitudinal federated continuous learning privacy protection method for a CNN neural network. The system initializes an image classification task, namely handwriting recognition, and three participants: two clients and a server. Furthermore, the local dataset is MNIST, which contains images with known image classification labels. The image classification labels range from 0 to 9. Furthermore, the CNN neural network is set as the local model, and the fully connected MLP neural network is set as the global model.
[0047] Example
[0048] See the accompanying drawings in the specification. Figure 1 The present invention provides a privacy protection method for vertical federated continuous learning based on prototypes, and the specific implementation steps are as follows:
[0049] Step 1: Model initialization and initialization task t is 1; Client A and Client B initialize the local model to θ A and θ B , server C initializes the global model as θ C ; The local optimization algorithm is specifically the stochastic gradient descent method.
[0050] Step 2: Without exposing the local data of the participants, Client A, Client B, and Server C use the privacy set intersection technique to obtain the intersection of all training data, thereby achieving sample alignment of the training data.
[0051] Step 3: Client A and client B use their local sample features X A Perform forward training of the local model to obtain the local embedding value E A and E B , as shown in formula (1).
[0052] E Z =X z θ Z (1)
[0053] Among them, X Z represents the local data owned by participant Z, θ Z Represents the local model parameters owned by participant Z, participant Z∈{client A, client B}.
[0054] Step 4: Client A and Client B each embed the value E locally A and E B Send to server C.
[0055] Step 5: The server aggregates the local embedded value E A and E B Generate the global embedding value E and class prototype U, which includes the following sub-steps:
[0056] Sub-step 5.1: The server aggregates all local embedding values E using the average aggregation method A and E B Get the current global embedding value E, as shown in formula (2):
[0057]
[0058] Sub-step 5.2: The server obtains the embedding value E corresponding to each label based on the current global embedding value E and the sample label Y i , where i represents the i-th label value;
[0059] Sub-step 5.3: The server generates the embedding value E for each tag. i Take the average to get the class prototype u=∪u i , as shown in formula (3):
[0060]
[0061] Where i represents the i-th label value, |D i | represents the total number of samples corresponding to the i-th label;
[0062] Sub-step 5.4: The server saves the obtained class prototype u into the global prototype table as u t , where t represents the current task t.
[0063] Step 6: The server enhances the previous task knowledge, which includes the following sub-steps:
[0064] Sub-step 6.1: The server determines whether the current task t is the first task. If the current task t=1 is the first task, the server executes the next step, i.e., step 7; otherwise, the server executes the following sub-steps:
[0065] Sub-step 6.2: Based on the global prototype list u and the current prototype category u t-1 Get the previous prototype u';
[0066] Sub-step 6.3: Obtain the enhanced previous class prototype u based on the previous prototype u' and the current prototype u * ;
[0067] Sub-step 6.4: Based on the previous class prototype u * And random sampling algorithm to construct the global embedding value E of previous knowledge * ;
[0068] Sub-step 6.5: Global embedding value E of prior knowledge * Add to the current global embedding value E = E ∪ E * .
[0069] Step 7: The server updates the global model θ based on the current global embedding value E C , this step specifically includes the following sub-steps:
[0070] Sub-step 7.1: The server performs forward propagation of the global model based on the current global embedding value E to obtain the current predicted value Y', as shown in the following formula:
[0071] Y′=θ C *E
[0072] Sub-step 7.2: The server obtains the current loss value L(Y',Y) and gradient value based on the current predicted value Y' and the current sample label value Y
[0073] Sub-step 7.3: The server updates the global model θ based on the current loss value and stochastic gradient descent C ,θ C =θ C -η C g, where η C represents the learning rate of server C.
[0074] Sub-step 7.4: The server sends the gradient values g corresponding to clients A and B A and g B Sent to each client A and B;
[0075] Step 8: The client updates the local model based on the corresponding gradient value, which includes sub-steps 8.1, 8.2, and 8.3:
[0076] Sub-step 8.1: Client A and Client B determine whether the current task t is the first task. If the current task t is the first task, execute step 8.3; otherwise, execute step 8.2.
[0077] Sub-step 8.2: Client A and Client B fix some local model parameters θ' that are important for the previous task A and θ' B ;
[0078] Sub-step 8.3: Client A and Client B update the unfixed local model θ according to the gradient value A =(θ A -θ' A )-η A g A ,θ B =(θ B -θ' B )-η B g B , where η A and η BRepresent the learning rates of the client A and client B models respectively.
[0079] Step 9: Repeat steps 3 to 8 until the training model reaches the expected convergence accuracy or the pre-agreed maximum number of training rounds is reached, at which point training is stopped. This achieves verifiable longitudinal federated learning using Shapley values.
[0080] The present invention has been described in detail above through the examples. It should be noted that the above examples are only preferred embodiments of the present invention, not all embodiments of the present invention, and do not constitute any limitation to the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims of this application document. Under the premise of not departing from the spirit of the present invention, those skilled in the art who make equivalent substitutions or other various transformations of the technical features of the present invention and obtain technical solutions or embodiments fall within the scope of protection of the present invention.
Claims
1. A prototype-based privacy-preserving method for vertical federated continuous learning, characterized by: The method comprises the following steps: Step 1: Model initialization and task initialization, where each client initializes local model parameters and local optimization algorithms; Step 2: Without exposing the local data of the participants, the client and server use a private set intersection technique to obtain the intersection of all training data, thereby achieving sample alignment of the training data; Step 3: The client uses its local sample features to perform forward training on the underlying model to obtain local embedding values; Step 4: The client sends its respective local embedding value to the server; Step 5: The server aggregates the local embedding values to generate a global embedding value and a class prototype; Step 6: The server enhances the previous task knowledge; Step 7: The server updates the global model according to the current global embedding value; Step 8: The client updates the local model according to the corresponding gradient value; and Step 9: Repeat Step 3 to Step 8 until the training model reaches the expected convergence accuracy or reaches the pre-agreed maximum number of training rounds, and stop training.
2. The prototype-based vertical federated continuous learning privacy protection method according to claim 1, characterized in that: The step 5 includes the following sub-steps: Sub-step 5.1: the server aggregates all local embedding values using an average aggregation method to obtain a current global embedding value; Sub-step 5.2: the server obtains the embedding value corresponding to each label based on the current global embedding value and the sample label; And sub-step 5.3: The server averages the embedding values of each label to obtain the class prototype.
3. The prototype-based vertical federated continuous learning privacy protection method according to claim 2, characterized in that: The step 6 includes the following sub-steps: Sub-step 6.1: the server determines whether the current task is the first task. If the current task is the first task, the next step, i.e., the step 7, is executed; otherwise, the following sub-steps are executed: Sub-step 6.2: the previous prototype is obtained based on the global prototype list and the current prototype category; Sub-step 6.3: the enhanced previous class prototype is obtained based on the previous prototype and the current prototype; Sub-step 6.4: the global embedding value of the previous knowledge is constructed based on the previous class prototype and the random sampling algorithm; and Sub-step 6.5: the global embedding value of the previous knowledge is added to the current global embedding value.
4. The prototype-based vertical federated continuous learning privacy protection method according to claim 3, characterized in that: The step 7 includes the following sub-steps: sub-step 7.1: the server performs forward propagation of the global model based on the current global embedding value to obtain the current prediction value, sub-step 7.2: the server obtains the current loss value based on the current prediction value and the current sample label value, sub-step 7.3: the server updates the global model based on the current loss value and the stochastic gradient descent method, and sub-step 7.4: the server sends the gradient value corresponding to each client to each client.
5. The prototype-based vertical federated continuous learning privacy protection method according to claim 4, characterized in that: The step 8 includes sub-steps 8.1, 8.2 and 8.3: Sub-step 8.1: the client determines whether the current task is the first task. If the current task is the first task, execute sub-step 8.3; otherwise, execute sub-step 8.2: the client fixes some local model parameters that are important to the previous task. Sub-step 8.3: the client updates the unfixed local model according to the gradient value.
6. The prototype-based vertical federated continuous learning privacy protection method according to claim 5, characterized in that: In step 3, the client uses its local sample features to perform forward training on the underlying model to obtain the local embedding value. The calculation formula is as follows: E Z =X z i Z Among them, Z represents the client, X Z represents the local data owned by client Z, θ Z Represents the local model parameters owned by client Z.
7. The prototype-based vertical federated continuous learning privacy protection method according to claim 7, characterized in that: In the sub-step 5.3, the server embeds the value E of each tag i Take the average to get the class prototype u=∪u i , as shown in the following formula: Among them, i represents the i-th label value, D i Indicates the total number of samples corresponding to the i-th label.