Federal forgetting learning method based on knowledge distillation

Abstract

The invention discloses a federal forgetting learning method based on knowledge distillation, and the method simplifies the two steps of target data influence removal in the forgetting process and model performance maintenance after forgetting into one step through a teacher-student model structure. When the server receives a forgetting request, the server issues a forgetting instruction and a global model copy to each client; the client divides data needing to be forgotten and reserved data according to local data and generates three types of teacher models; carrying out local training on the student model under the guidance of a plurality of teacher models, learning random distribution on target forgotten data so as to eliminate the influence of trained knowledge, and meanwhile, keeping prediction performance consistent with an original model and a real label on reserved data; then, each client uploads the updated model to the server, and the server aggregates the update and judges whether forgetting is completed or not; by means of the method, efficient data forgetting can be achieved under the condition that the global model is not retrained, and meanwhile reduction of model performance on reserved data is avoided.

Description

Technical Field

[0001] This invention belongs to the field of information security technology and relates to a federated forgetting learning method based on knowledge distillation. Background Technology

[0002] With the rapid development of Artificial Intelligence (AI) technology, AI has been widely applied in key fields such as healthcare, finance, transportation, industrial manufacturing, and intelligent recommendation, becoming an important engine for promoting the informatization and intelligent transformation of society. The powerful performance of AI relies on high-quality, large-scale data support. Data, as the core "fuel" for AI model training, directly determines the upper limit of model performance through its quantity, quality, and diversity. Simultaneously, the widespread adoption of smart terminals and IoT devices has further propelled the rapid evolution of AI through the generation and circulation of massive amounts of multi-source data. However, the widespread application of data has also brought severe privacy and compliance challenges. Against this backdrop, the "right to be forgotten" has gradually become an important component of global data protection legislation. This right grants data subjects the legal right to request data controllers to delete their personal information under specific conditions, reflecting not only respect for individual data autonomy but also posing new requirements for traditional data processing and model training mechanisms. To realize the technological implementation of the "right to be forgotten," researchers have proposed the concept of "Machine Unlearning" (MU). Forgetting learning aims to effectively remove specified training data and its impact on model parameters and predictive behavior without retraining the entire model, making the model after forgetting processing resemble a state of "never having learned from the data." Forgetting learning provides AI systems with the ability to actively forget specific information and is an important technical path to satisfying the right to be forgotten.

[0003] Federated Learning (FL), a typical distributed machine learning paradigm, aims to achieve joint modeling across clients while protecting user privacy. While protecting user privacy, FL has shown great potential in privacy-sensitive fields such as healthcare and finance. However, with the implementation of laws and regulations related to the "right to be forgotten," FL also faces new challenges. When users request to forget some data, simply deleting the target data at the database level is insufficient, as the impact of the target data on model parameters is already implicit in the client and the global model. The most direct solution is to retrain the model, but this method has extremely high computational and communication overhead and is often infeasible in scenarios with strict privacy protection. Therefore, researchers have proposed Federated Unlearning (FU), which aims to effectively remove the influence of target data on the model without retraining the entire model, while maintaining the model's performance on the remaining data. Compared to MU in a centralized environment, FU faces more challenges: First, the data distribution in FL is usually non-iid, with differences and correlations between data from different clients, making it complex to completely remove the influence of specific data from the global model; second, FL's privacy protection principles dictate that data cannot leave the client, limiting FU's direct manipulation of the original data; in addition, FU needs to complete the forgetting operation within a limited number of communication rounds, which limits the amount of data that can be transmitted and the complex operations that can be executed.

[0004] Current Full Rendering (FU) methods typically consist of two phases: influence removal and performance recovery. Influence removal refers to removing the influence of the target data from the model through specific mechanisms after the target data has been identified, so that the forgotten model behaves as if it had never seen the data. Existing influence removal methods include gradient-based operations, historical information reconstruction, loss function approximation, and knowledge distillation. However, these methods often suffer from high computational resource costs, the need for additional storage of historical update information, and inaccurate data influence estimation. More importantly, influence removal often leads to a performance degradation of the model on non-target data, thus requiring an additional performance recovery phase. Performance recovery is generally achieved through post-training, fine-tuning, or regularization, which, while improving performance, also introduces additional computational overhead and training time. In summary, existing FU methods are complex, computationally and storage-intensive, and struggle to simultaneously achieve effective forgetting and performance preservation. Therefore, how to ensure the effectiveness of forgetting while avoiding model performance degradation and reducing computational and storage costs has become a pressing issue in the FU field. The objective of this invention is to propose a streamlined federated forgetting learning method that combines effect removal and performance preservation, thereby simplifying the FU process, improving forgetting efficiency, and effectively addressing the shortcomings of existing technologies. Summary of the Invention

[0005] Purpose of the invention: This invention provides a federated forgetting learning method based on knowledge distillation, which simplifies and integrates the two-stage model of "separation of influence removal and performance recovery" in existing federated forgetting learning, thereby avoiding additional performance recovery steps while ensuring the forgetting effect, so that the method can be more efficiently and widely applied in diverse federated learning scenarios.

[0006] The federated forgetting learning method based on knowledge distillation described in this invention includes the following steps: S1, the server is in the... After receiving a forget request, the round sets and publishes a global forget control parameter group to each client. ,in, The distillation temperature. To accommodate forgetting constraints, the value can be selected based on privacy requirements and data sensitivity. To maintain performance constraints, the server then sets the current global model parameters. Distribute to each client; S2, each client Its local dataset Divide into a subset of targets to be forgotten With reserved subset In order to obtain the samples needed for subsequent training and testing; S3, The client is based on the received global parameters. Constructing a multi-teacher set Specifically, it includes: Forgotten Teachers : Random initialization parameters that are isomorphic to the global model are used to generate predictions with an approximately uniform distribution, so as to guide the student model away from the original representation on the forgotten subset; Performance maintenance teacher :for A frozen copy is used to maintain the predictive power of the original model on preserved data; Labeling Teachers : One-hot vectors of the real labels Defined to provide standard monitoring signals; In step S3, the multi-teacher temperature-based output distribution is defined as follows:

[0007] in, Teacher Model The output log odds, The one-hot vector of the real label; S4, Client uses student model With the main body as the guide, local optimization is implemented under the guidance of multiple teachers: The output is approximated to a uniform distribution or the teacher's output is forgotten, thus eliminating the influence of the target sample; Simultaneously, alignment performance is maintained to ensure that the teacher's predicted distribution matches the true label distribution, thus guaranteeing model performance. After one round of optimization, updated student model parameters are obtained. .

[0008] In step S4, the client's local objective function It consists of three distillations and one labeling supervision:

[0009] in, The cross-entropy function, Indicates the student model under temperature Normalized predicted distribution This represents the log-odds ratio of the student model's output. for The class is uniformly distributed. The divergence is Kullback–Leibler.

[0010] S5. After completing local optimization, the client performs tests on its local forgotten subset and retained subset to obtain the forgetting accuracy. With retention accuracy And the above results, along with the updated model parameters Upload them to the server together.

[0011] In step S5, the global aggregation uses a federated average (FedAvg) or its equivalent with sample size weights:

[0012] Alternatively, write an equivalent update based on local gradients:

[0013] in, For server-side learning rate, For the client In the round The local gradient.

[0014] S6. The server uses a federated average method with sample size weights (FedAvg) to aggregate the parameters uploaded by the client, obtaining new global model parameters:

[0015] Based on the test results uploaded by all clients, the global forgetting accuracy and global retention accuracy were calculated respectively:

[0016] If the following conditions are met:

[0017] If the forgetting process is complete, then normal federal training should resume; otherwise, [then...]. Continue executing steps S4 to S6 until the judgment condition is met.

[0018] Beneficial Effects: This invention proposes an efficient forgetting mechanism based on knowledge distillation for the implementation of the "right to be forgotten" within the federated learning framework. By constructing forgetting teachers, performance preservation teachers, and label preservation teachers on the client side, and guiding student models to simultaneously align distributions on target and non-target data, it achieves unified optimization of forgetting and performance preservation, avoiding the additional computational overhead caused by the separation of "influence removal" and "performance recovery" in traditional methods. This method eliminates the need to store historical gradients or model snapshots, reducing storage burden and communication consumption. Under the forgetting determination mechanism, it ensures that the influence of target data is effectively eliminated while maintaining prediction accuracy on non-target data, guaranteeing both forgetting effectiveness and model performance. Compared to existing methods, this invention improves forgetting efficiency, enhances the robustness of the algorithm in non-independent and identically distributed data environments, and possesses stronger practicality and promotional value. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the client process in an example of the present invention; Figure 2 This is a schematic diagram of the server-side process in an example of the present invention. Detailed Implementation

[0020] The above-described solution will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the protection scope of the present invention.

[0021] This invention addresses the problems of existing federated learning (FL) methods, such as reliance on retraining when user-specified data is forgotten, excessive computational and communication overhead, and separation of forgetting and performance recovery steps. It proposes an improved solution. By designing a multi-teacher-student model structure based on knowledge distillation, this invention integrates the removal of target data influence and the maintenance of model performance during the forgetting phase into a single training process, effectively reducing communication rounds and system resource consumption.

[0022] In terms of application, the method described in this invention can be widely applied to privacy-sensitive scenarios such as intelligent healthcare, financial risk control, and personalized recommendations. In the field of intelligent healthcare, when a patient withdraws their authorization to use data, this method can achieve rapid forgetting without retraining the model, ensuring the compliant use of medical data. In financial risk control systems, when a user withdraws credit information or account data, the method described in this invention can effectively remove the influence of related data on model prediction results, preventing privacy abuse. In recommendation systems, when a user deletes historical behavior records, this method can update the model state in a timely manner, avoiding interference from old preferences on recommendation results. Through the above mechanisms, this invention maintains the accuracy and stability of the global model while ensuring the controllability and forgetting capability of user data, significantly improving the privacy security and engineering practicality of federated learning systems in practical deployments.

[0023] The method described in this invention first constructs forgetting teacher, performance-preserving teacher, and label-preserving teacher models on the client side. A joint distillation loss function guides the student model to "forget" on the target data while preserving performance on non-target data. Next, model optimization is performed locally and parameter differences are uploaded. Finally, the server side performs secure aggregation to generate a global model, thereby ensuring model performance while satisfying the right to be forgotten. Specifically, the implementation steps of the federated forgetting learning method based on knowledge distillation described in this invention are as follows: S1, the server is in the... After receiving a forget request, the round sets and publishes a global forget control parameter group to each client. .in, The distillation temperature. For forgetting constraints, the value can be selected based on privacy needs and data sensitivity, typically within... , To maintain performance constraints, the server then sets the current global model parameters. Distribute to each client.

[0024] S2, each client Its local dataset Divide into a subset of targets to be forgotten With reserved subset In order to obtain the samples needed for subsequent training and testing.

[0025] S3, The client is based on the received global parameters. Constructing a multi-teacher set Specifically, it includes: Forgotten Teachers : Random initialization parameters that are isomorphic to the global model are used to generate predictions with an approximately uniform distribution, so as to guide the student model away from the original representation on the forgotten subset; Performance maintenance teacher :for A frozen copy is used to maintain the predictive power of the original model on preserved data; Labeling Teachers : One-hot vectors of the real labels Defined to provide standard monitoring signals.

[0026] In step S3, the multi-teacher temperature-based output distribution is defined as follows:

[0027] in, Teacher Model The output log odds, This is the one-hot vector of the real label.

[0028] S4, Client uses student model With the main body as the guide, local optimization is implemented under the guidance of multiple teachers: The output is approximated to a uniform distribution or the teacher's output is forgotten, thus eliminating the influence of the target sample; Simultaneously, alignment performance is maintained to ensure that the teacher's predicted distribution matches the true label distribution, thus guaranteeing model performance. After one round of optimization, updated student model parameters are obtained. .

[0029] In step S4, the client's local objective function It consists of three distillations and one labeling supervision:

[0030] in, The cross-entropy function, Indicates the student model under temperature Normalized predicted distribution This represents the log-odds ratio of the student model's output. for The class is uniformly distributed. The divergence is Kullback–Leibler.

[0031] S5. After completing local optimization, the client performs tests on its local forgotten subset and retained subset to obtain the forgetting accuracy. With retention accuracy And the above results, along with the updated model parameters Upload them to the server together.

[0032] In step S5, the global aggregation uses a federated average (FedAvg) or its equivalent with sample size weights:

[0033] Alternatively, write an equivalent update based on local gradients:

[0034] in, For server-side learning rate, For the client In the round The local gradient.

[0035] S6. The server uses a federated average method with sample size weights (FedAvg) to aggregate the parameters uploaded by the client, obtaining new global model parameters:

[0036] Based on the test results uploaded by all clients, the global forgetting accuracy and global retention accuracy were calculated respectively:

[0037] If the following conditions are met:

[0038] If the forgetting process is complete, then normal federal training should resume; otherwise, [then...]. Continue executing steps S4 to S6 until the judgment condition is met.

[0039] Combination Figure 1 The specific client-side process steps in the knowledge distillation-based federated forgetting learning method are as follows: Step 1: The client receives the global model parameters and forget request from the server, and then processes the local dataset. The data is divided into a subset of targets to be forgotten, as specified by the user. With reserved subset If the target forgotten data is empty, then only the non-target data is retained for training.

[0040] Step 2: Construct a teacher model locally based on the partitioning results, including the forgotten teacher model. In target data The system generates near-uniformly distributed predictions to guide student models away from the target data representation; performance is maintained while preserving teacher performance. In non-target data The original prediction distribution is maintained to preserve model performance; the labels are maintained for teachers. A label distribution is generated based on real labels from non-target data to supervise the student model output.

[0041] Step 3: The client builds a student model, using the global model distributed by the server as initial parameters, and defines a joint distillation loss function, combining the forgetting distillation loss, performance preservation distillation loss, and label preservation loss according to weights. The client uses the student model... Local optimization was performed under the guidance of multiple teachers, and the loss function was... for:

[0042] exist Approximate a uniform distribution or forget the teacher's output to remove the target influence; Simultaneously aligning performance to maintain teacher and real labels to preserve performance, resulting in updated parameters.

[0043] Step 4: The client performs tests on its local forgotten subset and retained subset respectively to obtain the forgetting accuracy. With retention accuracy

[0044] Step 5: Upload from client , and This is sent to the server for global aggregated updates.

[0045] Combination Figure 2 The specific steps of the server-side process in the knowledge distillation-based federated forgetting learning method are as follows: Step 1: The server is at the... After the round receives the forgotten data, initialize the global model parameters. At the same time, determine the sufficiency constraint of forgetting. and performance constraints .

[0046] Step 2: Set the current global model and related control parameters Distribute to each client to initiate the forgetting process.

[0047] Step 3: After the local forgetting process is completed, the server collects all data. Parameter differences uploaded by each client And forgetting accuracy With retention accuracy The parameter difference records the updates to the student model after local joint distillation training.

[0048] Step 4: The server uses the FedAvg method to perform weighted aggregation of the parameter differences uploaded by the client. The calculation formula is as follows:

[0049] in, For the current global parameters, For the updated global parameters, Indicates the client The number of samples.

[0050] Step 5: The server validates the aggregated global model, evaluating the forgetting effect and performance retention on both the target and non-target datasets. Based on the test results uploaded by all clients, the global forgetting accuracy is calculated. Compared with global retention accuracy Determine if the following conditions are met:

[0051] Step 6: If the forgetting effect meets the preset threshold and the performance degradation does not exceed the allowable range, the server determines that the forgetting task is completed and outputs the final global model; otherwise, the server will send the new global model back to the client and enter the next round of iterative training until the dual constraints of forgetting and performance are met.

[0052] The experimental results of this invention are as follows. The datasets used in the experiments included CIFAR-10, CIFAR-100, and DBpedia. The CIFAR-10 dataset contains 60,000 color images with a resolution of 32×32 pixels, divided into 10 categories, with 6,000 images in each category. The CIFAR-100 dataset is similar to CIFAR-10 but contains 100 categories, with 600 images in each category. The DBpedia dataset contains 560,000 training samples and 70,000 test samples, divided into 14 categories. Regarding model selection, CIFAR-10 uses the ResNet18 network structure, CIFAR-100 uses ResNet44, and DBpedia uses a two-layer LSTM network structure. In the federated learning experimental environment, the number of clients was set to N=20. During the standard training phase, 25% of the clients were randomly selected to participate in training in each iteration; during the forgetting learning phase, all clients were required to participate to ensure that the target data was completely forgotten. The training data is divided using a Dirichlet distribution, resulting in a non-independent and identically distributed data distribution characteristic. The aggregation algorithm uses the FedAvg method.

[0053] The experiment evaluates the Federated Forgetting Learning (FU) method proposed in this invention from three dimensions: (1) Forgetting effectiveness: measures the degree of forgetting of the model on the target data, specifically by comparing the prediction accuracy of the model on the target data before and after forgetting. When forgetting is complete, the accuracy should decrease significantly. (2) Forgetting fidelity: measures the performance retention of the model on non-forgotten data, specifically by comparing the difference in prediction accuracy of the model on non-forgotten data before and after forgetting. When the difference is small, it indicates that the method can maintain the usability of the model while forgetting. (3) Forgetting efficiency: measures the time and communication overhead required to complete forgetting, specifically by statistically analyzing the number of communication rounds required for forgetting and the speedup ratio relative to the retraining method. In addition, retraining, as the most primitive and naive federated forgetting method, is widely used as an evaluation benchmark. This method has the best performance in terms of forgetting effectiveness and forgetting fidelity by retraining the model after deleting the target data, but it performs very poorly in terms of efficiency because it consumes a lot of computational resources and does not meet the privacy protection requirements of federated learning.

[0054] Table 1. Accuracy of the model before and after forgetting on different datasets.

[0055] Table 2. Number of communication rounds and speedup ratio required for forgetting

[0056] Table 1 shows that we compared the prediction accuracy of the model on forgotten data before and after forgetting learning. Theoretically, the more thoroughly the model learns, the closer this value is to the prediction accuracy of retraining, which is close to 0. We randomly selected classes of data from each dataset for forgetting learning and recorded the mean of the results. For the image datasets CIFAR-10 and CIFAR-100, the prediction accuracy of the proposed method after forgetting learning was almost the same as the retraining baseline, close to 0. On the additional text dataset Dbpedia, the proposed method also achieved a prediction accuracy similar to retraining, also close to 0. From the perspective of prediction accuracy, the proposed method demonstrates a forgetting learning effect comparable to retraining, effectively achieving comprehensive clearing of forgotten data.

[0057] Furthermore, Table 1 compares the model's prediction accuracy on retained data before and after forgetting learning. Ideally, an effective forgetting learning process should only affect the forgotten data while maintaining the model's performance on retained data. When forgetting learning is performed using retraining, the prediction accuracy on retained data is almost identical to that before forgetting learning. Similarly, after forgetting learning using the method of this invention, the prediction accuracy on retained data remains close to its original level.

[0058] As shown in Table 2, we use two metrics to measure the forgetting efficiency of these FU methods: the number of communication rounds and the time speedup compared to retraining. Regarding the number of communication rounds, results across all datasets show that the method of this invention requires significantly fewer communication rounds to complete forgetting than retraining, averaging only 1-2 rounds, significantly reducing communication overhead. In terms of the speedup compared to retraining, the method of this invention consistently exhibits a higher speedup across all datasets, averaging 80.02 times, highlighting its superior computational efficiency. This indicates that SFU not only effectively achieves forgetting but also significantly reduces computational costs, making it a highly efficient federated forgetting method.

[0059] This invention can protect users' original data from being obtained by attackers, resist attackers with arbitrary background knowledge, and prevent privacy attacks from untrusted third parties. At the same time, it solves the problem that existing local differential privacy key-value data acquisition methods do not consider two-dimensional data, achieve more granular protection, and reasonably protect sensitive data while ensuring the security of privacy data, thereby improving the utility of estimating privacy data.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A federated forgetting learning method based on knowledge distillation, characterized in that, Includes the following steps: S1, the server is in the... After receiving a forget request, the round sets and publishes a global forget control parameter group to each client. ,in, The distillation temperature. To accommodate the forgetting constraint, the value is selected based on privacy requirements and data sensitivity. To maintain performance constraints, the server then sets the current global model parameters. Distribute to each client; S2, each client Its local dataset Divide into a subset of targets to be forgotten With reserved subset In order to obtain the samples needed for subsequent training and testing; S3, The client is based on the received global parameters. Constructing a multi-teacher set Specifically, it includes: Forgotten Teachers : Random initialization parameters that are isomorphic to the global model are used to generate predictions with an approximately uniform distribution, so as to guide the student model away from the original representation on the forgotten subset; Performance maintenance teacher :for A frozen copy is used to maintain the predictive power of the original model on preserved data; Labeling Teachers : One-hot vectors of real labels Defined to provide standard monitoring signals; S4, Client uses student model With the main body as the guide, local optimization is implemented under the guidance of multiple teachers: The output is approximated to a uniform distribution or the teacher's output is forgotten, thus eliminating the influence of the target sample; Simultaneously, alignment performance is maintained to ensure that the teacher's predicted distribution matches the true label distribution, thus guaranteeing model performance. After a batch optimization, updated student model parameters are obtained. ; S5. After completing local optimization, the client performs tests on its local forgotten subset and retained subset to obtain the forgetting accuracy. With retention accuracy And the above results, along with the updated model parameters Upload them to the server together; S6. The server uses a federated average method with sample size weights to aggregate the parameters uploaded by the client to obtain new global model parameters: ， Based on the test results uploaded by all clients, the global forgetting accuracy and global retention accuracy were calculated respectively: ， If the following conditions are met: ， If the forgetting process is complete, then normal federal training should resume; otherwise, [then...]. Continue executing steps S4 to S6 until the judgment condition is met.

2. The federated forgetting learning method according to claim 1, characterized in that, In step S3, the temperature-based output distribution of multiple teachers is defined as follows: ， ， ， in, These are the log-odds output by the teacher model, respectively. This is the one-hot vector of the real label.

3. The federated forgetting learning method according to claim 1, characterized in that, The client's local objective function in step S4 It consists of three distillations and one labeling supervision: ， in, , This is the cross-entropy function.

4. The federated forgetting learning method according to claim 1, characterized in that, The global aggregation in step S5 uses a federated average with sample size weights or its equivalent: ， Alternatively, write an equivalent update based on local gradients: ， in, For server-side learning rate, For the client In the round The local gradient.

5. The federated forgetting learning method according to claim 1, characterized in that, When class imbalance or client data is not independently and identically distributed, class reweighting is applied to local samples: ， in, For the client The Middle Number of class samples It is a smoothing constant. This is the loss for a single sample.

6. The federated forgetting learning method according to claim 1, characterized in that, Forgotten Teachers The equivalent form of adopting a uniform objective, that is, in The forgotten item should be rewritten as: and with temperature amplification factor Gradient scaling is applied to maintain dimensionality consistent with the temperature loss from knowledge distillation.

Images