Method for federated learning model knowledge forgetting under heterogeneous data

Abstract

The application discloses a federated learning model knowledge forgetting method under heterogeneous data, clusters and slices user data, trains a sub-federated model on each separate slice, and finally aggregates to generate a global model. In the reduction learning task, only the corresponding sub-federated model is retrained, while the data privacy and accurate data deletion are ensured, the training time is greatly reduced, and the private data and harmful data of the federated user are quickly and accurately cleaned up in the heterogeneous data environment.

Description

TECHNICAL FIELD

[0001] The present application relates to machine learning and federated learning technology, in particular to a federated learning model knowledge forgetting method under heterogeneous data. BACKGROUND

[0002] The learning reduction is a method of deleting specific data from the model, which is usually operated in a centralized framework, where all data is collected on the server for unified operation. However, given the growing concern for privacy, data owners are often reluctant to share their data. In this case, federated learning based on the distributed machine learning paradigm enables multiple participants to collaboratively train a shared model without the need to directly exchange raw data, which has attracted widespread attention from both industry and academia. However, when a participating user requests to delete its data, the original federated model will not be able to continue to be used. To solve this problem, the industry has proposed a federated learning reduction method.

[0003] The main goal of federated learning reduction is to achieve complete and effective deletion of participant or client contributions while preserving data privacy and security. The most direct method is retraining, which requires retraining the model from scratch to completely eliminate the impact of deleted data. Although this method guarantees the accuracy and consistency of the model, it introduces a large amount of computational overhead and time cost. Deleting a data from a deep neural network can require retraining millions of parameters, which can take several hours or even days. Therefore, in practical applications, the method of retraining is usually costly.

[0004] Another invention SISA method adopts a data partitioning strategy. This strategy divides the training data into pieces, allowing for targeted retraining of the corresponding data pieces. By training smaller models on these pieces and then aggregating them into a global model, the SISA method achieves more effective data deletion while balancing computational cost. However, this method mainly operates in a centralized framework, limiting their applicability to distributed federated learning settings.

[0005] The concept of federated learning reduction approximate forgetting was first proposed in the FedEraser method. This method uses approximation techniques to mitigate the impact of data that needs to be deleted on the model without the need for complete retraining from scratch. Although these methods have advantages in saving computational resources and time, they only provide statistical privacy protection for deleted data and do not effectively eliminate the impact of data on the model. Therefore, this is contrary to the original intention of privacy protection in federated learning.

[0006] A recent study, Rapid Retraining, extends the concept of FedEraser by updating all users' models. This approach requires all users to retrain a model for each forgetting task, ensuring that each user completes the forgetting task to guarantee that there is no stored data that needs to be deleted. However, in the case of multiple consecutive forgetting tasks, it can lead to a large number of users lacking the necessary device resources to support repeated training, and using this approach is obviously flawed.

[0007] In summary, although there are many technologies and inventions trying to solve the problem of users requesting to delete private data under federated learning, the existing methods have high training costs or are not suitable for distributed federated learning frameworks. The other part of the method mainly relies on approximation to induce model forgetting, and cannot ensure complete elimination of related data, still existing privacy invasion risk, and contradicting with existing legal framework, and destroying the original purpose of federated learning, i.e. privacy protection. SUMMARY

[0008] The purpose of the present application is to solve the problems existing in the prior art and provide a federated learning model knowledge forgetting method under heterogeneous data. The present application (FedCSA) adopts a divide-and-conquer strategy to cluster and slice user data, and then trains sub-federated models on each individual slice, and their aggregation produces a global model. In the restoration learning task, only the corresponding sub-federated model is retrained, which greatly reduces the training time while ensuring the comprehensive and accurate removal of requested data and data privacy, effectively solving the privacy protection problem commonly existing in current work.

[0009] TECHNICAL SOLUTIONThe federated learning model knowledge forgetting method under heterogeneous data of the present application comprises the following steps:

[0010] Step 1, balanced slicing data, i.e. slicing and dividing the user set U based on the model bias value clustering method on the center server (reducing the level of non-independent and identically distributed within each slice); the specific method is as follows:

[0011] Step 1.1, after the end of the first round of pre-training, according to the model bias value W uploaded by all users U in the user set U, the correlation D between each pair of users is calculated using cosine similarity; i

[0012]

[0013] In the above formula, the user set U contains all users participating in federated learning, w σ is the initial weight of the global model, and w σ is used as a reference point to measure the correlation between each user and w σ , w iFor all users after a round of training model weights, W is all user model weight set; D i Indicates w i Correlation with w σ , D is all user correlation set;

[0014] Step 1.2, the user is clustered based on the correlation set D, wherein the users with highly similar data types are grouped into the same cluster set C;

[0015]

[0016] In the above formula, D i , D j Indicate the correlation of user i and user j and the initial global model, and D i , D j Are elements in the correlation set D, C i Is an element in the cluster set C;

[0017] Step 1.3, the users in the cluster set C are evenly distributed to obtain the divided slice set S;

[0018]

[0019] The slice set S contains r slices S j , wherein each individual slice contains N / r users, and N is the total number of users, so that the non-independent identically distributed condition in each slice is alleviated, which is beneficial to model training;

[0020] Step 2, train the sub-federal model M j On each slice S j , all sub-federal models share the same architecture during training and are trained in parallel, thereby speeding up the training process;

[0021] Step 3, the trained sub-federal model is aggregated to form a global model, that is, when a restoration learning task needs to be performed, the corresponding sub-federal model is retrained after deleting the corresponding user or related data, and then all sub-federal models are aggregated into a new global model.

[0022] The present application performs the above steps 1 to 3 on the central server, and the performance of the central server is far superior to that of the client, so that the participation of the client is minimized, which helps to improve the overall speed.

[0023] Further, the step 2 randomly selects m users in the slice for single training in a single slice;

[0024]

[0025] Within the slice, the user participating in the current training first uploads the respective model update, denoted as w i Then, the model updates of each round of training session are summarized using the FedAvg algorithm, as follows:

[0026]

[0027] The above process is repeated until the model within the slice converges.

[0028] w t is the update of the tth round of federated model; minimize() means minimizing the loss to obtain the optimized model parameters; FedAvg() is a classic algorithm in federated learning, used to average the parameters of the models of all users participating in federated training, and the aggregated parameters are the global model parameters.

[0029] Further, the detailed method of step 3 is:

[0030] All slices upload their respective model updates, denoted as w j Then, the model updates of each round of training session are summarized using the FedAvg algorithm, as follows:

[0031] W={FedAvg(w j )|j=1,2,3...r}

[0032] When there is a user who proposes a forgetting request, only the sub-federated model of the slice area where the user is located needs to be retrained according to the above step 2, and then all slice models are aggregated to complete the task of clearing the user's private data; the performance of each sub-federated model in the above process is similar, thereby avoiding the case that a sub-federated model with poor performance may have a negative impact on the performance of the aggregated global model. The aggregation process ensures that the overall performance of the global model is not affected.

[0033] Advantages: Compared with the prior art, the present application has the following advantages:

[0034] (1) The present application proposes a user-based slice clustering method specially designed for federated learning, which clusters users according to model bias values, facilitates the division of data into slices, and ensures the best performance of the global model by reducing the degree of non-independent and identically distributed within each slice.

[0035] (2) The present application can accurately complete the forgetting task while improving the retraining efficiency. In addition, the present application is not limited to the type of model, and the federated learning model knowledge forgetting method under the whole heterogeneous data is irrelevant to the model, and can be seamlessly integrated into the federated learning framework without any additional burden to the user.

[0036] (3) All learning tasks in the application are coordinated by the central server, simplifying the process; realizing the rapid and accurate removal of private data and harmful data of the federal user in a heterogeneous data environment. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a schematic diagram of the overall framework of the application;

[0038] Figure 2 is a schematic diagram of the overall process of the application;

[0039] Figure 3 is a schematic diagram of the accuracy of the sub-models under the division of 5 slices in the embodiment of the application;

[0040] Figure 4 is a schematic diagram of the accuracy of the sub-models under the division of 10 slices in the embodiment of the application;

[0041] Figure 5 is a schematic diagram of the accuracy of the sub-models under the division of 20 slices in the embodiment of the application. DETAILED DESCRIPTION

[0042] The technical solutions of the application will be described in detail below, but the protection scope of the application is not limited to the described embodiments.

[0043] As shown in Figure 1 and Figure 2 , a federated learning model knowledge forgetting method under heterogeneous data of the application includes the following steps:

[0044] Step 1, balanced slicing data, that is, slicing division of the user set U on the central server based on the model bias value clustering method, the specific method is:

[0045] Step 1.1, after the first round of pre-training, according to the model bias value W uploaded by all users U in the user set U, the correlation D between each pair of users is calculated by using cosine similarity; i

[0046]

[0047] In the above formula, the user set U contains all users participating in federated learning, w σ is the initial weight of the global model, w σ is used as a reference point to measure the correlation between each user and w σ , w i is the model weight of all users after one round of training, W is the model weight set of all users; D i represents the correlation of w i and w σ , D is the correlation set of all users; ​

[0048] Step 1.2, the users are clustered based on the relevance set D, wherein the users with highly similar data types are grouped into the same cluster set C;

[0049]

[0050] In the above formula, D i , D j respectively represent the relevance of user i and user j and the initial global model, and D i , D j are elements in the relevance set D, C i is an element of the cluster set C;

[0051] Step 1.3, the users in the cluster set C are evenly distributed to obtain a divided slice set S;

[0052]

[0053] The slice set S contains r slices S j , wherein each individual slice contains N / r users, and N is the total number of users, and the non-independent identically distributed condition in each slice is alleviated, which is beneficial to model training;

[0054] Step 2, train a sub-federated model M j on each slice S j , and all sub-federated models share the same architecture and are trained in parallel during the training process;

[0055] Step 3, aggregate the trained sub-federated models to form a global model, that is, when a restoration learning task needs to be performed, retrain the corresponding sub-federated model after deleting the corresponding user or relevant data, and then aggregate all sub-federated models into a new global model.

[0056] Figure 1 M r in the formula refers to a slice model trained using data of pre-forgetting users; M r ′ in the formula refers to a slice model trained after removing the data of pre-forgetting users.

[0057] In step 2 of the embodiment, m users are randomly selected in the slice for single training in the single slice;

[0058]

[0059] In the slice, the users participating in the current training first upload their respective model updates, denoted as w i ; then the FedAvg algorithm is used to aggregate the model updates of each round of training session, and the formula is as follows:

[0060]

[0061] Repeat the above process until the model within the slice converges;

[0062] w t This is for updating the federated model in round t; minimize() means minimizing the loss to obtain the optimized model parameters; FedAvg() is a classic algorithm in federated learning, used to average the parameters of all user models participating in federated training, and the aggregated parameters are the global model parameters.

[0063] The detailed method for step 3 in this embodiment is as follows:

[0064] Each slice uploads its own model update, denoted as w. j Then, the FedAvg algorithm is used to summarize all slice models, as shown in the following formula:

[0065] W = {FedAvg(w j |j=1,2,3...r}

[0066] When a user requests to forget, simply retrain the sub-federation model for the user's area according to step 2 above, and then aggregate all slice models to complete the task of clearing the user's private data.

[0067] Example 1

[0068] This embodiment algorithmizes the relevant steps in the method of the present invention, as follows:

[0069] 1. Define and initialize parameters: Set the set of clients as U, the total number of communication rounds as T, the number of local model update rounds performed by each client as E, and the number of clients per communication round as m. For S i The model on the area is M i The t-th round of communication S i Global model w on the region t .

[0070] For the communication round at t=0, perform the following steps:

[0071] The server initializes the global model parameters M0 and sends them to all clients in the client set U; after the first round of training, each user uploads their respective model bias value W; the server uses cosine similarity to calculate the correlation D between each pair of users; the users are clustered based on the calculated correlation D, where users with highly similar data types are grouped into the same cluster C;

[0072] Distribute the users in set C evenly to obtain the slice set S;

[0073] 2. Slice Model Training: In each slice S i Training the sub-federated model M i Models trained on all slices share a unified model structure, which facilitates subsequent model aggregation. For training a sub-federated model within a single slice, m users are randomly selected within the slice to participate in the training.

[0074] For each communication round t from 1 to T-1, perform the following steps:

[0075] The server-side operations are as follows: The server retrieves data from slice U... s Select m clients to form a client set O. t The server will use the current global model parameters w t Distribute to set O t All clients in;

[0076] The client operations are as follows: For each selected client k, perform the following operations: Using its local data, process the downloaded w... t Perform gradient descent updates in round E to obtain new model parameters. Client k updates its model parameters Uploaded to the server;

[0077] 3. Global Model Aggregation: First, the models of each region are aggregated, and then the final global model is formed. The server uses the parameters for aggregating each region as described in this invention. Get each region S i Model M i Aggregate the model M of each region i The updated global model M is obtained. g .

[0078] 4. Fast Forgetting Algorithm:

[0079] A client x sends a request for forgotten data to the central server; the server responds quickly and determines the region S in which the client is located. x And delete the complete model M of that area. x ;

[0080] The server schedules client retraining within this region to update model M in that region. x ′ The central server re-aggregates the sub-federal models of each region {M}. x ′ +(M g -M x )}, thus obtaining the global model M after forgetting. u .

[0081] Example 2

[0082] 1) Experimental equipment and software environment

[0083] This example conducts experiments on three publicly available real datasets, which are widely used to evaluate the performance of federated learning models: MNIST, Fashion-MNIST, and CIFAR-10. All experiments are developed using Python 3.9 and PyTorch 1.12, and are executed on a standard computing platform equipped with Ubuntu 20.04, Platinum 8255C CPU and RTX 3080 GPU.

[0084] 2) Simulation of non-independent and identically distributed data

[0085] To simulate non-independent and identically distributed data, this example has each user randomly select two unconnected data classes, with an equal number of data samples for each class. In addition, all users have the same number of data. In MNIST and Fashion-MNIST, the number of users is set to 100. Each user has 600 images, with 300 images belonging to one class and the other 300 images belonging to another class. For the CIFAR-10 dataset, each user has 500 images.

[0086] 3) Comparison method

[0087] A comparative analysis of the most advanced machine restoration learning methods for two different models is conducted. These two models are commonly used in the field of federated learning. Details of the two models are as follows:

[0088] MLP: A 3-layer perceptron neural network model, including an input layer, an output layer, and multiple hidden layers, is commonly used for classification tasks, including MNIST and Fashion-MNIST classification tasks.

[0089] CNN: The widely adopted convolutional neural network (CNN) model is commonly used in image processing, consisting of convolutional layers, pooling layers, and fully connected layers. In this paper, we apply the CNN model to perform classification tasks on the CIFAR-10 dataset.

[0090] Here, three machine rejuvenation learning methods are compared: retraining, SISA, and the FedCSA of the present application; the retraining method includes deleting target data and training a new model from scratch, and the retraining method is taken as an experimental baseline; SISA includes randomly dividing data into multiple slices, independently training small models on each slice, and then aggregating these small models to form a final model. The FedCSA method of the present application uses a model bias value clustering method to divide users into different slices.

[0091] Table 1 Dataset Details:

[0092] Dataset Picture size Class Training set Test set MNIST 28*28 10 60000 10000 Fasion-MNIST 28*28 10 60000 10000 CIFAR-10 28*28 10 50000 10000

[0093] Experimental Results Details:

[0094] 1) Model Performance Comparison

[0095] This embodiment initially evaluates the performance of models generated by different machine rejuvenation learning methods. The number of users participating in federated learning is configured to 100, and for the rejuvenation learning method, the slice is set to 5, 10, and 20. The experimental results are shown in Table 2.

[0096] Table 1 Comparison of global model accuracy generated by different methods in federated rejuvenation learning

[0097]

[0098] From the above results, it can be seen that the existing retraining method shows low learning efficiency in the case of large original data sets, and requires a large amount of training time. In contrast, the FedCSA of the present application is more efficient than SISA. When the number of slices is 10 and 20, the efficiency of the FedCSA of the present application is improved by about 4% and 5% compared with the global model of SISA.

[0099] As Figures 3 to 5 shown, in the case of using 5 slices, the performance of the model trained on each slice by the FedCSA of the present application is always better than SISA. When 10 slices are selected, the FedCSA of the present application is better than SISA on most slices. In the case of using 20 slices, compared with the scenario of using 5 slices, the performance of all small models has declined.

[0100] There is a relatively uniform performance difference between the small models trained on the slices allocated by the FedCSA of the present application. On average, the small models generated by the slices allocated by the FedCSA of the present application exhibit better performance than the slices allocated by SISA. This observation indicates that the slices divided by the FedCSA of the present application have lower non-uniformity, thereby improving the generalization performance of each small model and ultimately improving the overall performance of the model.

[0101] The significant performance of the FedCSA of the present application benefits from the clustering algorithm based on model bias values proposed herein, which effectively reduces the non-identically distributed features of each slice and improves the performance of the small model trained on each slice. According to the performance of the global model on each dataset, we observe a correlation between the performance of the global model and the number of slices. The fewer the number of slices, the better the performance of the global model. However, this is at the expense of reducing the efficiency of the forgetting task.

[0102] 2) Comparison of forgetting efficiency

[0103] The present application simulates non-identically distributed data. Subsequently, the retraining time of the corresponding sub-federal model and the time required for the aggregated global model are recorded, and the average time of these instances is calculated. All experiments are performed on the same machine to ensure consistent results. Due to the limitations of simulation experiments in capturing the performance differences of client devices, the number of training rounds for each model is used as an indication of time consumption. The results obtained are shown in Table 3:

[0104] Table 3 Comparison of training time (unit: number of rounds to model convergence point)

[0105]

[0106]

[0107] Compared with existing retraining methods, the present application FedCSA improves the execution efficiency of the retraining forgetting task by nearly 4 times. FedCSA adopts a clustering strategy, classifies users before assigning slices, and then divides slices according to different categories. This method minimizes the non-identically distributed nature of the data within the slice, greatly reduces the fluctuation level of the loss value, stabilizes and accelerates the convergence of the loss value. Thus, the number of training rounds required by the model is reduced, and the purpose of fast training can be achieved.

[0108] Although both the present application FedCSA and SISA adopt a divide-and-conquer approach, the partitioning mechanism of the existing SISA method involves random partitioning, which can result in a high degree of non-identically distributed within the slice, causing early loss value fluctuations to be severe, thereby hindering convergence and thus prolonging model training time. In addition, the inherent ability of the present application FedCSA method can alleviate the non-identically distributed nature in each slice, thereby allowing more slices to be divided, thereby improving the efficiency of the learning process without affecting the performance of the global model.

Claims

1. A federated learning model knowledge forgetting method under heterogeneous data, characterized in that, Comprising the following steps: Step 1, equalizing slice data, that is, slicing the client set U based on the model bias value clustering method on the center server, the specific method is: Step 1.1, after the first round of pre-training, according to the uploaded model bias value W of all users U in the client set U i , the correlation D between each pair of users is calculated using cosine similarity; In the above formula, the client set U contains all users participating in federated learning, w σ is the initial weight of the global model, and w σ is used as a reference point to measure the correlation between each user and w σ , w i is the model weight of all users after a round of training, and W is the model weight set of all users; D i represents the correlation between w i and w σ , and D is the correlation set of all users; Step 1.2, the client is clustered based on the correlation set D, wherein the users with high similarity in data type are grouped into the same cluster set C; In the above formula, D i D j Let D represent the correlations between user i and user j and the initial global model, respectively, and D i D j All are elements in the correlation set D, C i It is an element of cluster set C; Step 1.3, evenly distribute the users in the cluster set C to obtain the divided slice set S; S = {S1, S2,..., Sr} is a set of r slices j where each individual slice contains N / r users, N being the total number of users, and where the non-identically distributed case is mitigated in each slice. Step 2, training sub-federated model M on each slice S j training sub-federated model M on each slice S j , all sub-federated models share the same architecture during the training process and are trained in parallel; the specific method is: in each slice S i training sub-federated model M on each slice S i， The models trained on all slices share a unified model structure, which facilitates subsequent model aggregation. For the training of a sub-federated model in a single slice, m users are randomly selected within the slice to participate in the training; For each communication round of t from 1 to T-1, the following steps are performed: The server-side operation is as follows: the server selects m clients from the slice U s to form a client set O t ; The server distributes the current global model parameters w t to all clients in the set O t . The client operates as follows: for each selected client k, perform the following: using its local data, compute the gradient of the loss function with respect to the model parameters t Perform an E-round of gradient descent updates to obtain new model parameters Client k uploads its updated model parameters to the server; Step 3, the trained sub-federal model is aggregated to form a global model, when it is necessary to perform the task of restoring learning, the corresponding sub-federal model is retrained after deleting the corresponding user or related data, and then all the sub-federal models are aggregated into a new global model; wherein the server aggregates the parameters of each tile obtains a model M i on each tile S i ; aggregates the model M i of each tile, obtaining an updated global model M g .

2. The federated learning model knowledge forgetting method under heterogeneous data according to claim 1, characterized in that, The step 2 is that when training the federal model in a single slice, m users in the slice are randomly selected to participate in the federal training; Within a slice, users participating in the current training first upload their respective model updates, denoted as w i Then, the model updates of each round of training session are aggregated using the FedAvg algorithm, as follows: Repeat the above process until the model in the slice converges; w t is the update of the federated model for the t-th round; minimize() means minimizing the loss to obtain the optimized model parameters; FedAvg() is a classic algorithm in federated learning, which is used to average the parameters of all user models participating in federated training, and the aggregated parameters are the global model parameters.

3. The federated learning model knowledge forgetting method under heterogeneous data according to claim 1, characterized in that, The detailed method of the step 3 is: All slice uploads respective model updates, denoted as w j ; then use FedAvg algorithm to aggregate all slice models, the formula is as follows: W = {FedAvg(w j )|j = 1,2,3...r} When there is a user who proposes a forgetting request, only the sub-federal model of the slice area where the user is located needs to be retrained according to the above step 2, and then all the slice models are aggregated to complete the task of clearing the user's private data.

Images