A hybrid privacy-preserving image classification method based on federated model distillation

Abstract

The present invention discloses a hybrid privacy protection image classification method based on federated model distillation, and the steps include: 1. The client pre-trains a local neural network; 2. The central server obtains a public data set and randomly samples it, and then sends it to the client; 3. The client predicts the sent data; 4. The client performs norm clipping on the prediction result and randomly splits it into prediction slices, and sends it to the corresponding client according to the number; 5. The client aggregates the prediction slices to obtain a confused prediction and uploads it to the central server; 6. The central server aggregates the confused prediction and adds noise to obtain a global prediction, and then sends it to the client; 7. The client trains the local neural network by distilling the global prediction, and then retrains the local neural network; 8. The client uses the local neural network to classify images. Based on the federated model distillation algorithm, the present invention uses the secret sharing idea and differential privacy technology to achieve privacy-protected image classification.

Description

Technical Field

[0001] The present invention relates to the field of anomaly detection, and in particular to a hybrid privacy-preserving image classification method based on federated model distillation. Background Art

[0002] Image classification is to assign a corresponding class label to each image from a given set of categories. For the human visual system, it is very easy to distinguish the category of an image. However, for computers, it is impossible to immediately obtain the semantic information of the image and complete the task of classifying the image like humans. At present, the implementation of image classification is mainly based on deep learning methods. A deep learning model trained with high-quality data can accurately classify images, but nowadays high-quality data is difficult to integrate due to privacy issues.

[0003] The emergence of Federated Learning (FL) has alleviated the difficulty of data integration caused by privacy issues to a certain extent. By uploading the parameters of the deep learning model trained locally on the client, Federated Learning enables the server to train a high-precision deep learning model without integrating the client's local data, which protects the user's data privacy to a certain extent. However, while Federated Learning protects user data privacy, it also brings about the problem of huge communication overhead, making it difficult to deploy in bandwidth-constrained environments. To solve this problem, Federated Model Distillation (FedMD) came into being. By uploading the predictions of the local deep learning model on the data, it effectively reduces the communication overhead.

[0004] However, the existing image classification methods based on federated model distillation still have the risk of local data privacy leakage. Therefore, it is very important to protect local data privacy in the image classification methods based on federated model distillation. Summary of the invention

[0005] In order to address the shortcomings of the above-mentioned prior art, the present invention proposes a hybrid privacy-preserving image classification method based on federated model distillation, in order to utilize differential privacy and secret sharing technology to protect the client from the threat of privacy leakage, thereby achieving high-precision image classification while protecting the client data privacy.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0007] The hybrid privacy-preserving image classification method based on federated model distillation of the present invention is characterized in that it is applied to a system consisting of a central server and N clients and is performed in the following steps:

[0008] Step 1. The i-th client uses local labeled private image data to pre-train the local neural network;

[0009] Step 1.1. The i-th client preprocesses the local labeled private image data to obtain the preprocessed private image dataset denoted as D i ={(x i,1 ,y i,1 ),(x i,2 ,y i,2 ),...,(x i,j ,y i,j ),...,(x i,J ,y i,J )},x i,j and i,j They represent the j-th private image sample and its corresponding category after preprocessing of the i-th client, i=1,2,...,N, N is the total number of clients, j=1,2,...,J, J is the total number of private image samples of the i-th client;

[0010] Step 1.2. The i-th client uses formula (1) to construct a local neural network M Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M Gi The weights of the pre-trained local neural network M′ are obtained when the supervised loss value tends to be stable Gi ;

[0011]

[0012] In formula (1), is a symbolic function representing the jth private image sample x of the i-th client i,j Whether it belongs to category c, if represents the jth private image sample x i,j belongs to category c, if represents the jth private image sample x i,j The category does not belong to category c; Represents the local neural network M of the i-th client Gi Predict the jth private image sample x i,j The probability of belonging to category c; c∈[1,m], m is the number of categories;

[0013] Step 2. The central server obtains the unlabeled public image dataset and randomly samples it, and then sends it to the client;

[0014] Step 2.1. Define the number of collaborative training rounds as t, and initialize t = 1;

[0015] Step 2.2. The central server randomly samples the unlabeled public image dataset and obtains the t-th round of public image data subset, which is recorded as D P,t ={x 1,t ,x 2,t ,...,x k,t ,...,x K,t} and sent to N clients respectively, x k,t represents the kth sampled public image sample in the tth round of collaborative training, k = 1, 2, ..., K, K is the total number of sampled public image samples in the tth round of collaborative training;

[0016] Step 3. N clients use the public image dataset D issued in round t P,t Make predictions:

[0017] The i-th client uses the pre-trained local neural network M′ Gi Calculate the public image dataset D issued in round t P,t The prediction results Y of each public image sample in t,i ={f i (x 1,t ),f i (x 2,t ),...,f i (x k,t ),...,f i (x K,t )}; where f i (x k,t ) represents the kth public image sample x issued in the tth round calculated by the i-th client k,t The predicted value, k = 1, 2, ..., K, K is the total number of public image samples issued in the tth round;

[0018] Step 4. N clients evaluate the prediction results. 1 - After norm pruning, the prediction fragments are randomly split into prediction fragments, and then sent to the corresponding clients according to the numbers;

[0019] Step 4.1. Prediction result Y of round t for the i-th client according to the clipping threshold C t,i Carry out 1 -norm clipping, get the prediction result Y after the tth round of clipping t,i / max(1,||Y t,i || 1 / C);

[0020] Step 4.2. The i-th client takes the prediction result Y after the t-th round of cropping t,i / max(1,||Y t,i || 1 / C) Randomly split and generate N-1 predicted results Y t,i Random vectors of the same size And as the first N-1 prediction slices, where Indicates that the prediction result of the tth round of cropping of the i-th client should be sent to the j-th client;

[0021] Step 4.3. Calculate the prediction result Y of the i-th client for the t-th round t,i With random vectors and The difference between The N-1 predicted shards before the tth round and the Nth prediction shard in round t Composing the prediction shard set for round t

[0022] Step 5. The i-th client receives the t-th round prediction fragments sent by N-1 other clients and aggregates them with one of its own prediction fragments to obtain the t-th round aggregated confusion prediction result. And sent to the central server, where It indicates the t-th round prediction fragment sent by the j-th client and received by the i-th client;

[0023] Step 6. The central server calculates all the t-th round confusion prediction results {Y′ t,1 ,Y′ t,2 ,...,Y′ t,i ,...,Y′ t,N After aggregation, add Laplace noise Lap(C / ε) to get the global prediction result of round t And sent to N clients respectively, where ε represents the privacy protection strength;

[0024] Step 7. The i-th client uses the t-th round global prediction result Distillation trains the local neural network M′ Gi , and then use the private dataset D i The local neural network M″ after distillation training Gi Conduct review training;

[0025] Step 7.1. The i-th client uses formula (2) to construct the local neural network M′ after the distillation training Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M′ Gi The weights of the local neural network M″ after distillation training are obtained when the supervised loss value tends to be stable Gi;

[0026]

[0027] In formula (2), is a symbolic function, representing the public data subset D issued by the i-th client in the t-th round P,t The kth public sample x in k,t Whether the prediction category of belongs to the global prediction category c, if represents the kth public sample x k,t The prediction category belongs to the global prediction category c. If represents the kth public sample x k,t The prediction category does not belong to the global prediction category c; represents the local neural network M′ after distillation training of the i-th client Gi Predict the kth public sample x k,t The probability of belonging to the global prediction category c;

[0028] Step 7.2. The i-th client uses formula (3) to construct the local neural network M″ after distillation training Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M″ Gi The weights of the local neural network M″′ after review training are obtained when the supervised loss value tends to be stable Gi ;

[0029]

[0030] In formula (3), is a symbolic function representing the jth private image sample x of the i-th client i,j Whether it belongs to category c, if represents the jth private image sample x i,j belongs to category c, if represents the jth private image sample x i,j The category does not belong to category c; Represents the local neural network M″ of the i-th client Gi Predict the jth private image sample x i,j The probability of belonging to category c;

[0031] Step 7.3. After assigning t+1 to t, return to step 2.2. Execute sequentially until the maximum number of rounds is reached, thereby obtaining the local neural network after the i-th client collaborative training

[0032] Step 8. The i-th client uses the local neural network trained by the collaborative training Classify images.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. This paper proposes a hybrid privacy-preserving federated model distillation framework HPFMD, which uses secret sharing and differential privacy to provide perfect privacy protection for client model prediction.

[0035] 2. To address the problem of untrustworthy central servers, the present invention proposes prediction obfuscation based on the idea of ​​secret sharing to protect model predictions, preventing semi-trusted central servers from launching reasoning attacks on normal clients and protecting the data privacy of clients when facing untrusted servers. The proposed method achieves strong privacy protection while ensuring image classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0037] In this embodiment, see Figure 1 , a hybrid privacy-preserving image classification method based on federated model distillation is applied in a network consisting of a central server and N clients, and proceeds as follows:

[0038] In this embodiment, two collaborative tasks are performed respectively. One is to use the MNIST dataset as public data and a partial subset of the FEMNIST dataset as client private data to predict Task 1 of the FEMNIST dataset. The other is to use the CIFAR-10 dataset as public data and a partial subset of the CIFAR-100 dataset as client private data to predict Task 2 of the CIFAR-100.

[0039] Step 1. The i-th client pre-trains a local neural network using a private labeled image dataset;

[0040] Step 1.1. The i-th client preprocesses the local labeled private data to obtain the preprocessed private image dataset denoted as D i ={(x i,1 ,y i,1 ),(x i,2 ,y i,2 ),...,(x i,j ,y i,j ),...,(x i,J ,y i,J )},x i,j and i,jThey represent the jth private image sample and its corresponding category after preprocessing by the ith client, i=1,2,...,N, N is the total number of clients, and in this embodiment, N is 10; j=1,2,...,J, J is the total number of private image samples of the ith client; when task 1 is performed and the collaborative tasks are independent and identically distributed (IID), that is, each client randomly obtains the same number of handwritten letter images as private data, in this embodiment, J is 18, and the ith client randomly obtains 18 training samples of letter [a, b, c, d, e, f] images as private image dataset D i When task 1 is performed and the collaborative tasks are not independent and identically distributed (Non Independently Identically Distributed, referred to as NonI.ID), that is, each client only obtains handwritten letter images from the same author as private image data, but needs to correctly classify handwritten letters from other authors. In this embodiment, J is 20, and the i-th client randomly obtains 20 training samples of letter images from one author as the local data set D i ; When task 2 is performed and the collaborative tasks are independent and identically distributed, that is, each client randomly obtains the same number of CIFAR-100 training samples as private image data. In this embodiment, J is 120, and the i-th client randomly obtains 120 training samples as the private image data set D i ; When task 2 is performed and the collaborative tasks are not independent and identically distributed, that is, each client only has one subclass data of each superclass (such as lions among large carnivores, but lacks bears, cheetahs, tigers and wolves) as private image data, in this embodiment, J is 120, but it is necessary to correctly classify the superclasses of all test data. The i-th client randomly obtains 120 training samples as the private image dataset D i ; In order to verify the collaborative training effect of heterogeneous models in the federated model distillation, each client in this embodiment uses a convolutional neural network with different numbers of layers or channels to participate in the training;

[0041] Step 1.2. The i-th client uses formula (1) to construct a local neural network M Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M Gi The weight of the learning rate lr in this embodiment i L Take 1e-3, and get the pre-trained local neural network M′ when the supervised loss value tends to be stable Gi ;

[0042]

[0043] In formula (1), is a symbolic function representing the jth private image sample x of the i-th client i,j Whether it belongs to category c, if represents the jth private image sample x i,j belongs to category c, if represents the jth private image sample x i,j The category does not belong to category c; Represents the local neural network M of the i-th client Gi Predict the jth private image sample x i,j The probability of belonging to category c; c∈[1,m], m is the number of categories, when performing task 1, m is 10, when performing task 2, m is 100;

[0044] Step 2. The central server obtains the unlabeled public image dataset and randomly samples it, and then sends it to the client;

[0045] Step 2.1. Define the number of collaborative training rounds as t, and initialize t = 1;

[0046] Step 2.2. The central server randomly samples the unlabeled public image dataset and obtains the t-th round of public image data subset, which is recorded as D P,t ={x 1,t ,x 2,t ,...,x k,t ,...,x K,t} and sent to N clients respectively, x k,t represents the kth sampled public image sample in the tth round of collaborative training, k=1,2,...,K, K is the total number of sampled public image samples in the tth round of collaborative training, and K is 5000 in this embodiment; when performing task 1, 5000 MNIST data sets are randomly selected as public samples in each round, and when performing task 2, 5000 CIFAR-10 data sets are randomly selected as public samples in each round;

[0047] Step 3. N clients use the public image dataset D issued in round t P,t Make predictions:

[0048] The i-th client uses the pre-trained local neural network M′ Gi Calculate the public image dataset D issued in round t P,t The prediction results Y of each public image sample in t,i ={f i (x 1,t ),fi (x 2,t ),...,f i (x k,t ),...,f i (x K,t )}; where f i (x k,t ) represents the kth public image sample x issued in the tth round calculated by the i-th client k,t The predicted value, k = 1, 2, ..., K, K is the total number of public image samples issued in the tth round;

[0049] Step 4. N clients evaluate the prediction results. 1 - After norm pruning, the prediction fragments are randomly split into prediction fragments, and then sent to the corresponding clients according to the numbers;

[0050] Step 4.1. According to the clipping threshold C, the prediction result Y of the i-th client for the t-th round t,i Carry out 1 -norm clipping, get the prediction result Y after the tth round of clipping t,i / max(1,||Y t,i || 1 / C);

[0051] Step 4.2. The i-th client takes the prediction result Y after the t-th round of cropping t,i / max(1,||Y t,i || 1 / C) Randomly split and generate N-1 predicted results Y t,i Random vectors of the same size And as the first N-1 prediction slices, where Indicates that the prediction result of the tth round of cropping of the i-th client should be sent to the j-th client;

[0052] Step 4.3. Calculate the prediction result Y of the i-th client for the t-th round t,i With random vectors and The difference between The N-1 predicted shards before the tth round and the Nth prediction shard in round t Composing the prediction shard set for round t

[0053] Step 5. The i-th client receives the t-th round prediction fragments sent by N-1 other clients and aggregates them with one of its own prediction fragments to obtain the t-th round aggregated confusion prediction result. And sent to the central server, where It indicates the t-th round prediction fragment sent by the j-th client and received by the i-th client;

[0054] Step 6. The central server calculates all the t-th round confusion prediction results {Y′ t,1 ,Y′ t,2 ,...,Y′ t,i ,...,Y′ t,N After aggregation, add Laplace noise Lap(C / ε) to get the global prediction result of round t And sent to N clients respectively, where ε represents the privacy protection strength;

[0055] Step 7. The i-th client uses the t-th round global prediction result Distillation training of local neural network M′ Gi , and then use the private dataset D i The local neural network M″ after distillation training Gi Conduct review training;

[0056] Step 7.1. The i-th client uses formula (2) to construct the local neural network M′ after distillation training Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M′ Gi The weight of the learning rate lr in this embodiment i L Take 1e-3 and get the local neural network M″ after distillation training when the supervised loss value tends to be stable Gi ;

[0057]

[0058] In formula (2), is a symbolic function, representing the public data subset D issued by the i-th client in the t-th round P,t The kth public sample x in k,t Whether the prediction category of belongs to the global prediction category c, if represents the kth public sample x k,t The prediction category belongs to the global prediction category c. If represents the kth public sample x k,t The prediction category does not belong to the global prediction category c; represents the local neural network M′ after distillation training of the i-th client Gi Predict the kth public sample x k,t The probability of belonging to the global prediction category c;

[0059] Step 7.2. The i-th client uses formula (3) to construct the local neural network M″ after distillation training Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M″ Gi The weight of the learning rate lr in this embodiment i L Take 1e-3, and get the local neural network M″′ after review training when the supervised loss value tends to be stable Gi ;

[0060]

[0061] In formula (3), is a symbolic function representing the jth private image sample x of the i-th client i,j Whether it belongs to category c, if represents the jth private image sample x i,j belongs to category c, if represents the jth private image sample x i,j The category does not belong to category c; Represents the local neural network M″ of the i-th client Gi Predict the jth private image sample x i,j The probability of belonging to category c;

[0062] Step 7.3. After assigning t+1 to t, return to step 2.2. Execute sequentially until the maximum number of rounds is reached, thereby obtaining the local neural network after collaborative training of the i-th client.

[0063] Step 8. The i-th client uses the local neural network trained by the co-training Perform image classification.

[0064] Example:

[0065] In order to verify the effectiveness of the method of the present invention, two collaborative tasks are performed in this embodiment, and the average test accuracy of the local model is used as a quantitative evaluation standard.

[0066] In this embodiment, three methods are selected to compare the effects with the HPFMD method of the present invention. The selected methods are the federated model framework FedMD-NP without privacy protection, the central differential privacy federated framework FedMD-CDP and the local differential privacy federated framework FedMD-LDP. According to the experimental results, the results are shown in Table 1:

[0067] Table 1 Experimental results of the proposed method and the three selected comparison methods on the FEMNIST and CIFAR-100 datasets at different privacy protection levels

[0068]

[0069] The experimental results show that at the same privacy protection level, the method of the present invention is better than the FedMD-LDP method, has similar effects to the FedMD-CDP method, and better protects privacy data than the FedMD-NP method, thereby proving the feasibility of the method proposed in the present invention.

Claims

1. A hybrid privacy-preserving image classification method based on federated model distillation, characterized by: Applied to a system consisting of a central server and N clients, and proceed as follows: Step 1. The i-th client uses local labeled private image data to pre-train the local neural network; Step 1.

1. The i-th client preprocesses the local labeled private image data to obtain the preprocessed private image dataset denoted as D i ={(x i,1 ,y i,1 ),(x i,2 ,y i,2 ),...,(x i,j ,y i,j ),...,(x i,J ,y i,J )},x i,j and i,j They represent the j-th private image sample and its corresponding category after preprocessing of the i-th client, i=1,2,...,N, N is the total number of clients, j=1,2,...,J, J is the total number of private image samples of the i-th client; Step 1.

2. The i-th client uses formula (1) to construct a local neural network M Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M Gi The weights of the pre-trained local neural network M′ are obtained when the supervised loss value tends to be stable Gi ; In formula (1), is a symbolic function representing the jth private image sample x of the i-th client i,j Whether it belongs to category c, if represents the jth private image sample x i,j belongs to category c, if represents the jth private image sample x i,j The category does not belong to category c; Represents the local neural network M of the i-th client Gi Predict the jth private image sample x i,j The probability of belonging to category c; c∈[1,m], m is the number of categories; Step 2. The central server obtains the unlabeled public image dataset and randomly samples it, and then sends it to the client; Step 2.

1. Define the number of collaborative training rounds as t, and initialize t = 1; Step 2.

2. The central server randomly samples the unlabeled public image dataset and obtains the t-th round of public image data subset, which is recorded as D P,t ={x 1,t ,x 2,t ,...,x k,t ,...,x K,t } and sent to N clients respectively, x k,t represents the kth sampled public image sample in the tth round of collaborative training, k = 1, 2, ..., K, K is the total number of sampled public image samples in the tth round of collaborative training; Step 3. N clients use the public image dataset D issued in round t P,t Make predictions: The i-th client uses the pre-trained local neural network M′ Gi Calculate the public image dataset D issued in round t P,t The prediction results Y of each public image sample in t,i ={f i (x 1,t ),f i (x 2,t ),...,f i (x k,t ),...,f i (x K,t )}; where f i (x k,t ) represents the kth public image sample x issued in the tth round calculated by the i-th client k,t The predicted value, k = 1, 2, ..., K, K is the total number of public image samples issued in the tth round; Step 4. N clients perform l1-norm cropping on the prediction results and randomly split them into prediction fragments, which are then sent to the corresponding clients according to their numbers; Step 4.

1. Prediction result Y of round t for the i-th client according to the clipping threshold C t,i Perform l1-norm clipping to obtain the prediction result Y after the tth round of clipping t,i / max(1,||Y t,i ||1 / C); Step 4.

2. The i-th client takes the prediction result Y after the t-th round of cropping t,i / max(1,||Y t,i ||1 / C) randomly split and generate N-1 predicted results Y t,i Random vectors of the same size And as the first N-1 prediction slices, where Indicates that the prediction result of the tth round of cropping of the i-th client should be sent to the j-th client; Step 4.

3. Calculate the prediction result Y of the i-th client for the t-th round t,i With random vectors and The difference between The N-1 predicted shards before the tth round and the Nth prediction shard in round t Composing the prediction shard set for round t Step 5. The i-th client receives the t-th round prediction fragments sent by N-1 other clients and aggregates them with one of its own prediction fragments to obtain the t-th round aggregated confusion prediction result. And sent to the central server, where It indicates the t-th round prediction fragment sent by the j-th client and received by the i-th client; Step 6. The central server calculates all the t-th round confusion prediction results {Y′ t,1 ,Y′ t,2 ,...,Y′ t,i ,...,Y′ t,N After aggregation, add Laplace noise Lap(C / ε) to get the global prediction result of round t And sent to N clients respectively, where ε represents the privacy protection strength; Step 7. The i-th client uses the t-th round global prediction result Distillation trains the local neural network M′ Gi , and then use the private dataset D i The local neural network M″ after distillation training Gi Conduct review training; Step 7.

1. The i-th client uses formula (2) to construct the local neural network M′ after the distillation training Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M′ Gi The weights of the local neural network M″ after distillation training are obtained when the supervised loss value tends to be stable Gi ; In formula (2), is a symbolic function, representing the public data subset D issued by the i-th client in the t-th round P,t The kth public sample x in k,t Whether the prediction category of belongs to the global prediction category c, if represents the kth public sample x k,t The prediction category belongs to the global prediction category c. If represents the kth public sample x k,t The prediction category does not belong to the global prediction category c; represents the local neural network M′ after distillation training of the i-th client Gi Predict the kth public sample x k,t The probability of belonging to the global prediction category c; Step 7.

2. The i-th client uses formula (3) to construct the local neural network M″ after distillation training Gi The supervision loss Then use the stochastic gradient descent optimization method with a learning rate lr i L To update the local neural network M″ Gi The weights of the local neural network M″′ after review training are obtained when the supervised loss value tends to be stable Gi ; In formula (3), is a symbolic function representing the jth private image sample x of the i-th client i,j Whether it belongs to category c, if represents the jth private image sample x i,j belongs to category c, if represents the jth private image sample x i,j The category does not belong to category c; Represents the local neural network M″ of the i-th client Gi Predict the jth private image sample x i,j The probability of belonging to category c; Step 7.

3. After assigning t+1 to t, return to step 2.

2. Execute sequentially until the maximum number of rounds is reached, thereby obtaining the local neural network after the i-th client collaborative training Step 8. The i-th client uses the local neural network trained by the collaborative training Classify images.

Images