Federal forgetting learning method based on small sample knowledge distillation

Abstract

The invention discloses a federal forgetting learning method based on small sample knowledge distillation. The method comprises the following steps: initializing a global model by adopting a server and broadcasting the global model to each client; each client carries out local training on the received global model by using local data, and after the local training is completed, the global model is updated and sent back to the server; the server stores and aggregates the global models sent by all the clients, is used for updating the local global model of the server, and repeats the federal learning process from S1 to S3; a target client makes a forgetting request at a certain moment, federal learning is stopped, and at the moment, a server side uses a forgetting module to generate a forgetting model and takes the forgetting model as a student model; the server side finely adjusts and compares the language image pre-training model according to local small sample data, so that the language image pre-training model adapts to downstream data and serves as a teacher model; the server uses local small sample data to carry out recovery training through a knowledge distillation method, so that catastrophic forgetting caused by client data revocation is avoided, and the performance of a forgetting model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of data security and privacy protection, and relates to federated learning and federated forgetting learning technology, specifically a federated forgetting learning method based on small sample knowledge distillation. Background Art

[0002] With the rapid digitalization of various industries, the emergence of large amounts of data has led to increasing concerns about data privacy and security. Traditional deep learning methods usually rely on obtaining large amounts of data and training them through neural network models. However, due to data privacy and security restrictions, it is difficult to share data between different institutions, such as medical record data in hospitals. Federated learning, as an emerging machine learning method, optimizes the model by performing distributed training among multiple devices or computing nodes without sharing the original data, thereby effectively solving privacy and security issues. In addition, as public concern about the "right to be forgotten" has increased, recent legislation (such as the Cybersecurity Law) explicitly stipulates that users have the right to request the deletion of their contributions to trained models, further strengthening data control and privacy protection.

[0003] Federated forgetting learning aims to effectively remove the data contribution of specific clients in the global model and eliminate the negative impact of forgetting without infringing user privacy and without compromising model performance. Knowledge distillation, as an efficient model optimization technique, has been applied by researchers to the model recovery phase of federated forgetting to mitigate the negative impact of forgetting on the global model. The performance of the student model is improved by transferring the knowledge in the teacher model to a smaller or structurally different student model. However, in the federated learning scenario where the client needs to forget data or exit training, the existing federated forgetting learning method based on knowledge distillation relies on a large amount of high-quality image data on the server for training, ignoring the scenario where the availability of image data is limited. Summary of the invention

[0004] In response to the problem of limited data resources on the server side, the present invention discloses a federated forgetting learning method based on small sample knowledge distillation. The forgetting module is adopted to effectively erase the historical contribution of the target client, and a small sample knowledge distillation mechanism is introduced in the recovery stage. The fine-tuned CLIP model is used to transfer visual and semantic knowledge, enhance the performance of the student model, and finally eliminate the negative impact of forgetting and restore the accuracy of the forgetting model.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] Step 1: The server initializes the global model and broadcasts it to each client.

[0007] Step 2: Each client uses local data to locally train the received global model and sends its model updates back to the server after local training is completed.

[0008] Step 3: The server saves and aggregates the model updates from all clients to update the local global model on the server, and repeats the federated learning process from steps 1 to 3.

[0009] Step 4: At a certain moment, the target client makes a forget request and stops federated learning. At this time, the server uses the forget module to erase the historical contribution of the target client model saved in the last global model, generates a forget model, and uses it as the student model.

[0010] Step 5: The server fine-tunes the CLIP model based on the local small sample data. The fine-tuning process introduces the original CLIP model into the neural network module as an adapter to make it better adapt to downstream data. The model is trained until it converges and is used as the teacher model.

[0011] Step 6: The server uses local small sample data to perform recovery training through the knowledge distillation method, using the rich visual and semantic knowledge of the teacher model to enhance the student model's understanding of the image semantic context, ultimately improving the performance of the forget model.

[0012] Furthermore, the step 1 specifically includes:

[0013] Step 1.1: The total number of clients participating in the training during the federated forget learning process is N, which can be expressed as N = {n 1 ,n 2 ,n 3 ,…,n N}, the corresponding local data can be expressed as D = {D 1 ,D 2 ,D 3 ,…,D N}, their data is not shared between clients and is not independent and identically distributed.

[0014] Step 1.2: The server initializes the global model W and broadcasts it to each client n.

[0015] Furthermore, the step 2 specifically includes:

[0016] Step 2.1: Each client, as a participant, uses its own local data to train the model received from the server. Suppose a client n i In round t, the training process is expressed using formula (1):

[0017]

[0018] Where η is the learning rate used during training, L is the loss function, and D i Indicates client n i Local data, and (x ij ;y ij ) represents the jth sample and its label.

[0019] Step 2.2: After each client completes training based on local data, it sends the updated local model back to the server.

[0020] Furthermore, the step 3 specifically includes:

[0021] Step 3.1: The server saves the model parameters from each client into a list for subsequent forget operations.

[0022] Step 3.2: The server aggregates the model updates of all clients using average aggregation. This process is expressed using formula (2):

[0023]

[0024] where u i Indicates client n i Local data D i , and U represents the total amount of local data of all clients.

[0025] Step 3.3: Repeat the federated learning process from steps 1 to 3.

[0026] Furthermore, the step 4 specifically includes:

[0027] Step 4.1: At a certain moment, the K target client sends a forget request to the server. At this time, the server responds to this request and stops federated learning and enters forget learning.

[0028] Step 4.2: The server uses the forget module to erase the target client's historical model parameter information that was previously saved. The erasure process is expressed using formula (3):

[0029]

[0030] in is the global model after forgetting, W K is the global model of round K, Indicates the target client n i The incremental model update during the first K rounds.

[0031] Step 4.3: Use the above forgetting model as the student model for subsequent knowledge distillation.

[0032] Furthermore, the step 5 specifically includes:

[0033] Step 5.1: The server-side small sample dataset contains images I and their corresponding category names {class}. These natural language category names C = {C 1 ,...,C k} into the preset hard prompt template P, where k represents the total number of categories, and the final generated text description is “a photo of a{C i}", where C i Belongs to C.

[0034] Step 5.2: Keep the original CLIP backbone network unchanged during fine-tuning and use the original image encoder and text encoder Extract image features f(v) and text features f(t), and express the extraction process using formulas (4) and (5) respectively:

[0035]

[0036]

[0037] Step 5.3: Introduce two bottleneck linear layers and activation functions to form a feature adapter. During the fine-tuning process, only the feature adapter of the image branch is fine-tuned, but the feature adapter of the text branch is not fine-tuned. The fine-tuning process is expressed by formula (6):

[0038]

[0039] Among them A I (·) Feature adapter for the image branch, ReLU is the activation function, f(v) T is the converted image feature, and are the weights of the two bottleneck linear layers in the image branch.

[0040] Step 5.4: To avoid forgetting the original knowledge of the pre-trained CLIP encoder and prevent overfitting, residual connections are further used to dynamically fuse the fine-tuned knowledge with the original knowledge of the contrastive language image pre-training model (CLIP). The process is expressed using formula (7):

[0041] f * =αA I (f(v)) T +(1-α)f(v) (7)

[0042] Where α is the residual ratio, f * is the final extracted image feature.

[0043] Step 5.5: Further use the contrastive loss function of the original CLIP model to maximize the similarity between the features of each image sample and the text features of its correct category, while minimizing its similarity with the text features of all other categories.

[0044] Step 5.6: Update the initial parameters W of the CLIP model using gradient descent CLIP , until the contrast loss L θ Converge to a smaller value, and use the final model as the teacher model for subsequent knowledge distillation. The process is expressed by formula (8):

[0045]

[0046] Where W CLIP is the initial parameter of CLIP, are the CLIP model parameters after fine-tuning.

[0047] Furthermore, the step 6 specifically includes:

[0048] Step 6.1: The server uses the local small sample data to perform recovery training through knowledge distillation, and uses the forgotten learning model in step 4.3 as the student model W s , the Contrastive Language Image Pre-trained model (CLIP) fine-tuned in step 5.6 is used as the teacher model W t .

[0049] Step 6.2: In the knowledge distillation process, by defining a hard loss L h ard and soft loss L soft To guide students to model W s Learning, the hard loss consists of the cross entropy loss between the student model’s prediction and the true label, and the process is expressed using formula (9):

[0050]

[0051] Where M is the number of training samples, y i is the true label of the i-th sample, is the predicted label of the student model for the i-th sample.

[0052] Step 6.3: The soft loss is based on the output of the teacher model. By calculating the Kullback-Leibler divergence between the student model and the teacher model output, the similarity is measured to make the student model close to the output distribution of the teacher model. The temperature parameter T is used to smooth the probability distribution and enhance the information of the soft label. The process is expressed by formula (10):

[0053]

[0054] Where P t (y i |x i ) and P s (y i |x i ) is the teacher model W t and student model W s The softmax output of the i-th sample at temperature T is expressed using formulas (11) and (12):

[0055]

[0056] in They are respectively the teacher model W t and student model W S The logits vector output on the sample dataset.

[0057] Step 6.4: The total loss function combines the hard loss and the soft loss and can be expressed by formula (13):

[0058] L total (W s )=βL h ard (W s )+(1-β)L soft (W s ) (13)

[0059] The weight factor β can be adjusted to balance the contribution of hard loss and soft loss. A higher β value means more reliance on the true label (hard loss), while a lower β value means more reliance on the output of the teacher model (soft loss).

[0060] Step 6.5: Iterate and update the small sample knowledge distillation process above, using the rich visual and semantic knowledge of the teacher model to make up for the deficiencies caused by limited data resources, and ultimately improve the performance of the forgetting model.

[0061] Due to the adoption of the above technical scheme, the present invention provides a federated forgetting learning method based on small sample knowledge distillation, which solves the problem that the existing federated forgetting learning recovery method relies too much on a large amount of high-quality data on the server side for training, and fully considers the limited availability of server data. The present invention effectively erases the historical contribution of the target client by introducing a forgetting module, and then adopts a small sample knowledge distillation mechanism in the model recovery stage, and transfers visual and semantic knowledge with the help of the fine-tuned contrastive language image pre-training model (CLIP), significantly improves the performance of the student model, eliminates the negative impact of forgetting, and restores the accuracy of the forgetting model. In the fine-tuning stage, the present invention introduces an adapter to efficiently fine-tune the contrastive language image pre-training model (CLIP), only adjusts the bottleneck linear layer in the image branch, keeps the text encoder stable, avoids large-scale adjustments, and retains the pre-training knowledge of the contrastive language image pre-training model (CLIP) to the maximum extent. The new knowledge after fine-tuning is dynamically fused with the original knowledge through residual connections to avoid forgetting of pre-training knowledge and reduce the risk of model overfitting. In addition, the soft label distribution is adjusted by the temperature parameter to enhance the learning ability of the student model in distillation, and finally restore the performance of the forgetting model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 It is a flow chart of the federated forgetting model recovery method based on small sample knowledge distillation of the present invention;

[0064] Figure 2 It is a structural schematic diagram of the federated forgetting model recovery method based on small sample knowledge distillation of the present invention and the compared method;

[0065] Figure 3 Schematic diagram of the overall structure of the federated forgetting model recovery method based on small sample knowledge distillation of the present invention.

[0066] Figure 4 Schematic diagram of the feature adapter structure of the federated forgetting model recovery method based on small sample knowledge distillation of the present invention. DETAILED DESCRIPTION

[0067] In order to make the technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention:

[0068] like Figure 1 The present invention uses the public MNIST dataset, SVHN dataset, CIFAR-10 dataset and CIFAR-100 dataset as input data to conduct experiments. In this embodiment, the CIFAR-10 dataset is used as input data. It contains 60,000 color images of 10 common objects, which are divided into 50,000 training images and 10,000 test images. Figures 2 to 4 As shown, the training includes the following steps:

[0069] Step 1: Normalize the CIFAR-10 dataset and then split the data. Select 10,000 images from the training set of the dataset as the small sample data source for the server. Use the Dirichlet function to generate non-independent and identically distributed data for the remaining 40,000 training images and distribute them to 10 clients.

[0070] Step 2: The server initializes the global model and broadcasts it to 10 clients. Each client receives it and performs local training, using the stochastic gradient descent optimization algorithm with an initial learning rate of 0.001, a batch size of 64, a momentum of 0.9, and a weight decay of 0.0005 to optimize the learnable parameters of the network. After each client completes one epoch of training, the model update is sent back to the server;

[0071] Step 3: The server saves and aggregates the model update parameters of the participating clients. During aggregation, the total amount of data from each client is used as a proportional constraint for average aggregation.

[0072] Step 4: Repeat steps 2-3, and forget the two target clients after 30 epochs. The server uses the forget module to erase the historical contribution of the target clients according to the proportion constraint during aggregation to generate a forget model.

[0073] Step 5: The server randomly extracts 20% of the 10,000 images in step 1 as small sample data, and uses this data to fine-tune the CLIP model, adjusting only the feature adapter of the image branch without changing the text branch, where the weighted feature parameter α is 0.2. The network parameters are updated by backpropagation, with the initial learning rate set to 0.005 and the batch size to 8. Within 200 epochs, the stochastic gradient descent optimization algorithm with a cosine annealing learning rate adjustment strategy is used for learning, with a momentum of 0.9 and a weight decay of 0.001 to optimize the learnable parameters of the network;

[0074] Step 6: The server uses local small sample data to perform recovery training through the knowledge distillation method. The forgotten model obtained in step 4 is used as the student model, and the fine-tuned model obtained in step 5 is used as the teacher model. The parameters of the teacher model are frozen to prevent updates. In the process, the hard loss uses cross entropy loss, and the soft loss uses Kullback-Leibler divergence. With the help of the rich visual and semantic knowledge of the teacher model, the student model's understanding of the image semantic context is enhanced. The network parameters are updated by backpropagation, with the initial learning rate set to 0.005 and the batch size of 8. Within 200 epochs, the stochastic gradient descent optimization algorithm is used for learning, with a momentum of 0.9 and a weight decay of 0.001. The temperature T during the distillation process is 3, and the total loss weight factor β is 0.2 to restore the performance of the forgotten model.

[0075] The present invention uses the standard prediction accuracy index to represent the accuracy ACC, and the process is represented by formula (15):

[0076]

[0077] where y i is the true label of the i-th sample, is the predicted label of the i-th sample.

[0078] According to the above steps, the present invention is compared with the method of using the pre-forgetting model as the teacher model for knowledge distillation in a scenario with limited small sample data resources, such as Figure 2 As shown in Table 1, it can be seen that the evaluation indicators of the method proposed in the present invention on the four data sets are better than those of the compared methods, where the number of clients is 10 and the number of target customers is 2.

[0079] Table 1 Performance comparison with comparison methods on four datasets

[0080] Dataset Model Before forgetting After forgetting Comparison Method The present invention MNIST LeNet 89.41％ 9.58％ 87.71％ 93.48％ SVHN ResNet-18 86.75％ 19.59％ 74.42％ 76.06％ CIFAR-10 ResNet-18 57.42％ 10.00％ 47.71％ 55.27％ CIFAR-100 ResNet-50 21.90％ 1.00％ 8.64％ 12.89％

[0081] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A federated forgetting learning method based on small sample knowledge distillation, characterized by include: S1: Use the server to initialize the global model and broadcast it to each client at the same time; S2: Each client uses local data to perform local training on the received global model, and after the local training is completed, sends its global model update back to the server; S3: The server saves and aggregates the global model sent by all clients to update the local global model on the server and repeat the federated learning process from S1 to S3; S4: At a certain moment, the target client makes a forget request and stops federated learning. At this time, the server uses the forget module to generate a forget model and uses it as the student model; S5: The server fine-tunes the contrastive language image pre-training model based on the local small sample data to adapt it to the downstream data and uses it as the teacher model; S6: The server uses local small sample data to perform recovery training through the knowledge distillation method, thereby avoiding catastrophic forgetting caused by client data withdrawal, thereby improving the performance of the forgetting model.

2. The method for federated forgetting learning based on small sample knowledge distillation according to claim 1, characterized in that: When generating the forget model: At a certain moment, the K target clients send a forget request to the server to stop federated learning. The server erases the saved historical model parameter information of the target client through the forget module and generates a forget model as the student model for subsequent knowledge distillation. The erasure process is performed using formula (1): Where W K is the global model, Indicates the model update delta for the target client.

3. The method for federated forgetting learning based on small sample knowledge distillation according to claim 1, characterized in that: S51: The server-side small sample data includes image I and category, and a text description is generated according to the category. The original CLIP backbone network is kept unchanged during fine-tuning, and the image encoder and text encoder are used to extract image features f(v) and text features f(t); S52: Two bottleneck linear layers and activation functions are introduced to form a feature adapter. During the fine-tuning process, only the feature adapter of the image branch is fine-tuned. The process is expressed by formula (2): Where ReLU is the activation function, f(v) T is the converted image feature, and are the weights of the two bottleneck linear layers; S53: Use α as the residual ratio to dynamically fuse the fine-tuned knowledge with the original knowledge of the contrastive language image pre-training model. The process is expressed by formula (3): f * =αA I (f(v)) T +(1-α)f(v) (3) S54: Use the contrast loss function of the original CLIP model to perform similarity calculation, and use the gradient descent method to update the contrastive language image pre-training model, and use the final model as the teacher model for subsequent knowledge distillation.

4. The method for federated forgetting learning based on small sample knowledge distillation according to claim 1, characterized in that: S61: The server uses local small sample data for knowledge distillation and guides the student model learning by defining hard loss and soft loss. The hard loss is composed of the cross entropy loss between the student model's prediction and the true label. The process is expressed by formula (4): Where M is the number of training samples, y i is the true label, is the predicted label; S62: The soft loss is based on the output of the teacher model. It measures the similarity by calculating the Kullback-Leibler divergence between the student model and the teacher model output, and uses the temperature parameter T to smooth the probability distribution and enhance the soft label information. The process is expressed by formula (5): Where P t (·) and P s (·) is the softmax output of the teacher and student models at temperature T; S63: The total loss function combines hard loss and soft loss, and adjusts the contribution of both through the weight factor β, which is expressed by formula (6); iteratively update the knowledge distillation process to ultimately improve the performance of the forgetting model L total (W s )=βL hard (W s )+(1-β)L soft (W s ) (6)。

Images