A privacy-preserving method for federated meta-learning based on dataset compression

Abstract

The present invention belongs to the technical field of image data processing, and more specifically, to a method for privacy protection of federated meta-learning based on data set compression. The method includes each client initializing a small synthetic data set to replace the original data set of the client, randomly initializing each synthetic data set and the corresponding learning rate; inputting the initial local model parameters for training to obtain a new round of local model parameters; optimizing the synthetic data set and its corresponding learning rate; continuously iterating the optimization to obtain the final synthetic data set and the final learning rate; the federated meta-learning obtains the real gradient based on the final synthetic data set and the final learning rate, and then optimizes the virtual data set and the virtual gradient through the DLG attack embedded in the federated meta-learning; adding noise to the image data in the final synthetic data set. The present invention solves the problem that the DLG attack will have a higher success rate in federated meta-learning to guess the batch size and attack, thereby causing privacy leakage.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and more specifically, to a method for privacy protection of federated meta-learning based on data set compression. Background Art

[0002] Federated learning has been widely used in real-life tasks such as healthcare and image recognition. It can effectively solve the problem of data silos and ensure the privacy of user data. But at the same time, federated learning also faces the problem of inconsistent data distribution among clients. By combining federated meta-learning with meta-learning algorithms, personalized models are trained for each client in federated learning, reducing the differences in model performance on different clients. Meta-learning's ability to prevent overfitting and quickly adapt to new tasks is particularly suitable for solving the problem of inconsistent data distribution among different clients in federated learning.

[0003] Chinese invention patent CN117035009A discloses a personalized communication method based on data set compression, including: S1: performing data distillation on the graph data stored by the k-th client to obtain the k-th synthetic data set; S2: constructing an optimization target based on the synthetic data set and optimizing it; S3: determining the training learning rate of the global model on the synthetic data based on the optimization target; S4: the central server constructs a pre-trained model based on the learning rate, and sends the pre-trained model to each client for personalized training.

[0004] The amount of data in federated meta-learning is usually small, which provides a good attack environment for Deep Leakage From Gradients (DLG). DLG attacks can infer the original data before training through the gradients generated during model training. DLG attacks are very effective but require relatively strict prior conditions. DLG attackers need to know the batch size of training, and the batch size cannot be too large, otherwise the number of iterations required for the attack will be too large. The amount of data in the support set and query set in federated meta-learning is relatively small, and their batches are smaller than those in traditional machine learning. This naturally provides the necessary conditions for DLG attacks. Therefore, compared with other attack environments, DLG attacks will have a higher success rate in federated meta-learning to guess the batch size and attack, thereby causing privacy leakage. This problem has not been noticed in previous studies. Some current privacy protection methods, the idea of ​​differential privacy is to ensure the privacy security of user original data by adding noise perturbations to user information. The disadvantage is that noise will inevitably cause a certain degree of data loss, which may affect the utility of the model. Homomorphic encryption protects against privacy leaks from malicious central servers by encrypting sensitive data for transmission, and the honest party obtains the result by decrypting it. The disadvantage is that the use of homomorphic encryption algorithms leads to high complexity. Secure multi-party computation, multiple participants simulate a fully trusted third party through cryptographic collaboration, which allows distributed participants to collaboratively calculate an objective function without revealing their own data. The disadvantage is that the use of secure aggregation protocols also leads to high communication costs. Summary of the invention

[0005] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a method for privacy protection of federated meta-learning based on data set compression to solve the problem that the federated meta-learning solution is easily vulnerable to privacy attacks based on reasoning, and due to the characteristics of small samples of federated meta-learning, the DLG attack has a higher probability of success in the federated meta-learning environment. A data set compression method is introduced to process the original data set, so as to protect federated meta-learning from DLG attacks and reduce the risk of privacy leakage.

[0006] The present invention also provides a device for privacy protection of federated meta-learning based on data set compression.

[0007] The present invention also provides a computer-readable storage medium for implementing the above method to solve the problem of effective communication between software applications and simulation networks in a simulation environment.

[0008] The detailed technical scheme of the present invention is as follows:

[0009] A method for privacy protection of federated meta-learning based on data set compression, the method comprising:

[0010] S1. Each client independently initializes a synthetic dataset to replace the original dataset of the client in federated meta-learning, and then randomly initializes the synthetic dataset and learning rate through the generative adversarial network. This process is performed locally by the client.

[0011] S2. Input the synthetic data set and learning rate into the initial local model parameters for training to obtain a new round of local model parameters; then minimize the loss of the original data set on the new local model parameters, optimize the synthetic data set and learning rate; continuously iterate the optimization until the maximum mean difference converges to a preset value or reaches a preset number of iterations, and obtain the final synthetic data set and final learning rate;

[0012] Most existing dataset compressions involve solving expensive two-layer optimization: first optimizing the model, and then optimizing the synthetic data along the additional second-order derivative calculations, which undoubtedly increases the computational cost of the synthetic dataset, resulting in complex and inaccurate results. Therefore, the dataset concentration method based on distribution matching in the present invention effectively solves the two-layer optimization, uses the maximum mean difference (MMD) to minimize the distance between the original data and the synthetic data distribution, and embeds the training data into a lower dimensional space by using a family of parametric functions without directly evaluating the original data distribution.

[0013] S3. Federated meta-learning obtains the real gradient based on the final synthetic data set and the final learning rate, and then obtains the virtual data set and virtual gradient through the DLG attack embedded in the federated meta-learning; finally, the original gradient of the original data in the federated meta-learning is calculated, and noise is added to the image data in the final synthetic data set according to the loss maximization of the original gradient and the virtual gradient to obtain the final synthetic data set with noise. .

[0014] Furthermore, the S2 specifically includes:

[0015] S21. Synthetic Dataset and learning rate Initial local model parameters sent from the central server θ 0 Perform a round of training in and get a new round of local model parameters θ 1：

[0016] (1);

[0017] In formula (1), is θ 0 The loss on is the gradient, is the learning rate, θ1 is the new round of local model parameters;

[0018] S22. Calculate the parameters of the new local model for the original data set θ 1 The loss on the data set is used as the optimization target to minimize the loss to optimize the synthetic data set and its corresponding learning rate:

[0019] (2);

[0020] In formula (2), Initializing local model parameters θ 0 The loss function under , which measures the use of synthetic datasets and learning rate The new round of local model parameters obtained after a round of gradient descent θ 1 The average loss on the original dataset x;

[0021] S23. The optimization results of one round are as follows: and optimize the learning rate :

[0022] (3);

[0023] In formula (3), is a synthetic dataset for one round of optimization and a learning rate for one round of optimization;

[0024] S24, a round of optimized synthetic dataset and optimize the learning rate Local model parameters after one round of training θ 1 Retraining on θ 2 , and continuously iterate and optimize until the maximum mean difference converges to the preset value or reaches the preset number of iterations, and the final optimized synthetic data set is obtained and optimize the learning rate , the final optimization goal is as follows:

[0025] (4);

[0026] In formula (4), is the final optimized synthetic dataset and the corresponding learning rate;

[0027] The final synthetic dataset is used for subsequent federated learning model training. This synthetic dataset is small enough to reduce communication and storage requirements while retaining the key features of the original dataset. The original dataset is hidden and does not directly participate in the formal federated learning model training.

[0028] S3, federated meta-learning obtains the real gradient based on the final synthetic data set and the final learning rate, and then obtains the virtual data set and virtual gradient through the DLG attack embedded in the federated meta-learning; finally, the original gradient of the original data in the federated meta-learning is calculated, and noise is added to the image data in the final synthetic data set according to the loss maximization of the original gradient and the virtual gradient to obtain the noisy final synthetic data set and the noisy learning rate;

[0029] The DLG attack in federated meta-learning first randomly generates a virtual data set, that is, virtual input and labels; then performs model training on the generated virtual data set in the global model of the central server; after deriving the virtual gradient from the virtual data set, the virtual input and label are optimized to minimize the distance between the virtual gradient and the real gradient, and gradient matching is performed to make the virtual data set close to the original data; after continuous iterations, the original data will be fully displayed, exposing customer privacy data.

[0030] Based on the final synthetic data set, the present invention uses the DLG attack in federated meta-learning to make the virtual data set close to the final synthetic data set to avoid exposing customer privacy data. The S3 specifically includes:

[0031] S31. First, the n clients participating in the training input the final synthetic data set, and then train it in the local model of the client. Furthermore, the batch size of the data in the final synthetic data set is generally much smaller than the batch size in traditional machine learning.

[0032] S32, the final synthetic data set is input and trained on the local model to obtain the generated gradient, and the W 1 arrive W n There are n client gradients in total;

[0033] S33, upload the client gradient generated by local model training to the central server, and then the central server aggregates and averages to obtain the true gradient W;

[0034] S34, then the DLG attack uses the central server as an attacker and randomly initializes a virtual data set in Python using the numpy library. The initial virtual data set includes virtual input , virtual label The batch size of the initial virtual dataset is equal to the batch size of the final synthetic dataset. The initial virtual dataset is trained in the global model of the central server to obtain the virtual client gradient, which corresponds to n clients. arrive A total of n virtual client gradients;

[0035] S35: Aggregate and average the n generated client virtual gradients to obtain a virtual gradient , virtual gradient It is not like the real gradient W is used to update the virtual dataset instead of the global model.

[0036] (5);

[0037] In formula (5), , are virtual input and virtual label respectively, W is the parameter of the global model, L is the loss function, is the differential derivative, is a differentiable model, is the generated virtual gradient;

[0038] S36, the central server will be the real gradient W is sent to each client for updating and iterating the global model, continuously optimizing the virtual gradient to make it fit the real gradient, and at the same time, making the virtual data set fit the final synthetic data set;

[0039] The distance between the virtual gradient and the real gradient is used as the loss to iteratively update the virtual dataset (calculate the square of the difference between the virtual gradient and the real gradient, back propagate according to the square of the difference, and update the virtual dataset) to obtain the final virtual dataset:

[0040] (6);

[0041] In formula (6), , They are the virtual input and virtual label after the update is completed. W is the true gradient from the final synthetic dataset;

[0042] S37, the n normal clients participating in the training input the original data in the original data set, and then train in the local model of the client to obtain arrive There are n original client gradients in total, and the central server aggregates and averages them to get the original gradient , through federated meta-learning, iterative updates are continuously made to obtain the final original gradient ;

[0043] Reverse optimization is performed through gradient ascent, which maximizes the loss between the virtual data set and the original data set by maximizing the loss of the final virtual gradient and the final original gradient:

[0044] (7);

[0045] In formula (7), is the final original gradient; It is the final virtual gradient corresponding to the last round of iterative update of the final virtual gradient;

[0046] S38, adding noise to the image data in the final synthetic data set according to the maximum loss between the virtual data set and the original data;

[0047] Further, a random number generator is used to generate a Gaussian distribution random value with the same size as the image, and then the generated Gaussian noise is added to the image data in the final synthetic data set;

[0048] S39, repeat the step S2 again with the final synthetic data set after adding noise until convergence, and obtain the final synthetic data set with noise and the noisy learning rate ;

[0049] (8);

[0050] In formula (8), is the noised final synthetic dataset and the noised learning rate, is the local model parameter θ n-1 The loss function under The mean is 0 and the variance is Gaussian distribution of

[0051] Through the above feedback optimization, the virtual data set used in the DLG attack is closer to the synthetic data and further away from the original data, thereby further strengthening the protection of the original data set.

[0052] In another aspect of the present invention, a method for privacy protection of federated meta-learning based on data set compression is provided, the device comprising:

[0053] at least one processor; and

[0054] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to perform the method for privacy protection of federated meta-learning based on data set compression as described above.

[0055] In another aspect of the present invention, a machine-readable storage medium is provided, which stores executable instructions, and when the instructions are executed, the machine performs the method of privacy protection of federated meta-learning based on data set compression as described above.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] The present invention provides a method, device and computer-readable storage medium for privacy protection of federated meta-learning based on data set compression. The original data is processed by a data set compression method, compressed from a large training data set to a small data set, and a final synthetic data set is obtained. These data do not need to come from the correct data distribution. By replacing the original data set with the final synthetic data set as the training data of the model, the trained model will be similar to the model trained on the original data. The original data set is processed by this method and then put into model training. Finally, the DLG attack successfully takes effect and only obtains a synthetic data set with a model training effect close to the original data set. Noise is then added to the final synthetic data set by maximizing the loss of the original gradient and the virtual gradient. These data do not even have the original data distribution characteristics, thereby avoiding the privacy leakage of the original data, thereby achieving the purpose of protecting the privacy information in the original data set and defending against the DLG attack in federated meta-learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of a method for privacy protection of federated meta-learning based on data set compression described in the present invention.

[0059] Figure 2 This is a schematic diagram of data set compression in Example 1 of the present invention.

[0060] Figure 3 This is a schematic diagram of the federated meta-learning DLG attack in Example 1 of the present invention.

[0061] Figure 4 These are five groups of clear compressed images in Example 1 of the present invention.

[0062] Figure 5 It is a schematic diagram of the effect of DLG on the synthetic image in Example 1 of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0064] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0065] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0066] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0067] Example 1

[0068] Ginseng Figure 1 This embodiment provides a method for privacy protection of federated meta-learning based on data set compression, the method comprising:

[0069] S1. Initialize synthetic dataset and learning rate:

[0070] Each client independently initializes a small synthetic dataset to replace the original dataset in federated meta-learning, and then randomly initializes each synthetic dataset through generative adversarial networks (GANs) And the corresponding learning rate ,This process is performed locally by the client, and each client corresponds to a synthetic dataset and a learning rate;

[0071] Preferably, the original data set is a public data set such as MNIST, Cifar-10 or FashionMNIST, and the synthetic data set is a randomly sampled picture or randomly generated noise;

[0072] S2. Optimizing synthetic datasets through dataset compression: And the corresponding learning rate Input the initial local model parameters for training to obtain a new round of local model parameters; then minimize the loss of the original data set on the new local model parameters, optimize the synthetic data set and its corresponding learning rate; continue to iterate and optimize until the maximum mean difference converges to the preset value or reaches the preset number of iterations, and obtain the final synthetic data set and the final learning rate .

[0073] Most dataset compression involves solving an expensive two-level optimization: first optimizing the model, then optimizing the synthetic data along with additional second-order derivative calculations, which undoubtedly increases the computational cost of the synthetic dataset, thus obtaining complex and inaccurate results.

[0074] The dataset enrichment method based on distribution matching in the present invention effectively solves the two-level optimization, uses the maximum mean difference (MMD) to minimize the distance between the original data and the synthetic data distribution, and embeds the training data into a lower dimensional space by using a family of parameter functions without directly evaluating the original data distribution, such as Figure 2 As shown, specifically including:

[0075] S21, a round of training of the local model obtains a new round of local model parameters;

[0076] Synthetic Dataset And the corresponding learning rate Initial local model parameters sent from the central server θ 0 Perform a round of training in and get a new round of local model parameters θ 1 ;

[0077] (1);

[0078] In formula (1), is θ 0 The loss on is the gradient, is the learning rate, θ 1 is the new round of local model parameters;

[0079] S22. Calculate the parameters of the new local model for the original data set θ 1 The loss on the , take this loss as the optimization target, that is, matching loss, minimize the loss and optimize the synthetic data set and learning rate:

[0080] (2);

[0081] In formula (2), Initializing local model parameters θ 0 The loss function under , which measures the use of synthetic datasets and learning rate The new round of local model parameters obtained after a round of gradient descent θ 1 The average loss on the original data set x; preferably, the loss is cross entropy loss (ce loss);

[0082] S23. The optimization results of one round are as follows: and optimize the learning rate ,

[0083] (3);

[0084] In formula (3), is the synthetic dataset and corresponding learning rate after a round of optimization;

[0085] S24, the synthetic dataset after one round of optimization and optimize the learning rate Local model parameters after one round of training θ 1 Retraining on θ 2 , and continuously iterate and optimize until the MMD (maximum mean difference) converges to a preset value or reaches a preset number of iterations; each iteration aims to further reduce the distribution difference between the synthetic dataset and the original dataset. After optimization, the final synthetic dataset is obtained. and the final learning rate , the final optimization goal is as follows:

[0086] (4);

[0087] In formula (4), is the final synthetic dataset and final learning rate; are the local model parameters of the mth iteration.

[0088] The final synthetic dataset is used for subsequent federated learning model training. This synthetic dataset is small enough to reduce communication and storage requirements while retaining the key features of the original dataset. The original dataset is hidden and does not directly participate in the formal federated learning model training.

[0089] The key point of DLG attack and the factor that makes defense difficult is that the attacker of the central server knows the specific gradient aggregation method. In federated meta-learning, the gradient aggregation method must be transparent to the central server, so it is difficult to consider defense methods from the perspective of gradient aggregation. Since the gradient comes from the data, data aggregation can be considered for defense. Moreover, the aggregation of gradients in federated meta-learning occurs in the cloud central server, and the data processing is performed locally throughout the process, which will further provide better protection for data privacy. The compressed synthetic data set and synthetic learning rate are randomly initialized, and the final synthetic data set and synthetic learning rate are obtained by continuously calculating the loss gradient descent and optimizing. In order to obtain a better model with a small amount of gradient descent, a suitable learning rate is also obtained through learning.

[0090] S3. Federated meta-learning based on the final synthetic dataset and the final learning rate Get the real gradient, and then get the virtual data set and virtual gradient through the DLG attack embedded in federated meta-learning;

[0091] Finally, the original gradient of the original data in the federated meta-learning is calculated, and noise is added to the image data in the final synthetic dataset according to the loss maximization of the original gradient and the virtual gradient to obtain the final synthetic dataset with noise added. and.

[0092] Implementing the DLG attack in federated meta-learning: The DLG attack in federated meta-learning first randomly generates a virtual dataset, that is, virtual input and label, such as an image containing a cat, whose label is cat; then performs model training on the generated virtual dataset in the global model of the central server (CS); after deriving the virtual gradient from the virtual dataset, the virtual input and label are optimized to minimize the distance between the virtual gradient and the real gradient, and gradient matching is performed to make the virtual dataset close to the original data; after continuous iterations, the original data (including input and label) will be fully displayed, which will expose customer privacy data.

[0093] The optimization idea of ​​this embodiment is to make the virtual dataset close to the final synthetic dataset through the DLG attack in federated meta-learning based on the final synthetic dataset to avoid exposing customer privacy data, such as Figure 3 As shown, the specific steps are as follows:

[0094] S31, the n clients participating in the training obtain the final synthetic data set, and then train it in the local model of the client; further, the batch size of the data in the final synthetic data set is generally much smaller than the batch size in traditional machine learning;

[0095] S32, based on the final synthetic data set, the local model is trained to obtain the generated gradient, W 1 arrive W n There are n client gradients in total;

[0096] S33, upload the client gradient generated by local model training to the central server, and then the central server aggregates and averages to obtain a true gradient W; Under normal circumstances, the true gradient of this aggregation W is used for subsequent update iterations of the global model;

[0097] S34, but when the central server acts as an attacker, it can first use the numpy library in Python to randomly initialize a virtual data set, including virtual input , virtual label The batch size of these virtual data sets is equal to the batch size of the final synthetic data set. The virtual data sets are trained in the global model sent by the central server to obtain virtual client gradients, which correspond to n clients. arrive A total of n virtual client gradients;

[0098] S35: Aggregate and average the n generated client virtual gradients to obtain a virtual gradient , whether it is the real gradient ∇W or the virtual gradient , are aggregated and averaged by the central server, and the way of processing gradients by aggregate averaging is the same. In addition, unlike the real gradient mentioned above, the virtual gradient It is not like the real gradient W is not used to update the global model, but to update the virtual dataset.

[0099] (5);

[0100] In formula (5), , are virtual input and virtual label respectively, W is the parameter of the global model, L is the loss function, is the differential derivative, is a differentiable model, is the generated virtual gradient;

[0101] S36, continuously optimize the virtual gradient to make it fit the real gradient, and also make the virtual data set close to the final synthetic data set, use the distance between the virtual gradient and the real gradient as the loss for iterative optimization, calculate the square of the difference between the virtual gradient and the real gradient, and back propagate based on the square of the difference, and update the virtual data set so that its calculated gradient can match the real gradient. By updating the virtual data set to match the gradient, the virtual data set is close to the data of the final synthetic data set.

[0102] (6);

[0103] In formula (6), , They are the virtual input and virtual label after the update is completed. W is the true gradient from the final synthetic dataset;

[0104] S37. The DLG attack is extremely powerful and can almost completely restore the final synthetic data set. Although the synthetic data set is virtually generated from scratch and is small enough, the DLG attack is still possible to infer some information about the original data from some key features of the original data contained in the final synthetic data. The following is further optimized for the final synthetic data set:

[0105] The loss of the obtained virtual data set is calculated with the original data. The n normal clients participating in the training input the original data in the original data set, and then train in the local model of the client to obtain arrive There are n original client gradients in total, and the central server aggregates and averages them to get the original gradient , through federated meta-learning, iterative updates are continuously made to obtain the final original gradient ;

[0106] Reverse optimization is performed through gradient ascent, which maximizes the loss between the virtual data set and the original data set by maximizing the loss of the final virtual gradient and the final original gradient:

[0107] (7);

[0108] In formula (7), is the final original gradient; It is the final virtual gradient corresponding to the last round of iterative update of the final virtual gradient;

[0109] S38, adding noise to the image data in the final synthetic data set according to the maximum loss between the virtual data set and the original data, using a random number generator to generate a Gaussian distribution random value with the same size as the image, and then adding the generated Gaussian noise to the image data in the final synthetic data set;

[0110] S39, repeat S2 again with the final synthetic data set after adding noise until convergence, and obtain the final synthetic data set with noise and the noisy learning rate ;

[0111] (8);

[0112] In formula (8), is the noised final synthetic dataset and the noised learning rate, is the local model parameter θ n-1 The loss function under The mean is 0 and the variance is Gaussian distribution of

[0113] Through the above feedback optimization, the virtual data set used in the DLG attack is closer to the synthetic data and further away from the original data, thereby further strengthening the protection of the original data set.

[0114] Furthermore, the effectiveness of dataset compression is evaluated through comparative experiments. In this embodiment, the dataset of each client is compressed using the compression ratio of 5 images per category and 10 images per category on three types of datasets (MNIST, FaMNIST, and CIFAR10). α is the global learning rate of the comparative experiment, and ipc is the number of images of each category in the dataset. As shown in Table 1.

[0115] Table 1 Accuracy of compressed data on three datasets

[0116]

[0117] Then, the compressed dataset of each client is visualized. Five compressed synthetic images of clients are randomly selected, such as Figure 4 As shown, there are five groups of digital images: 3, 9, 8, 3, 9; five groups of clothing images: skirt, bag, boots, bag, short sleeves; five groups of object images: cat, boat, truck, ship, airplane. The synthesized MNIST images are clear and noise-free, while the number of images synthesized by previous methods contains obvious noise and some unnatural strokes. The synthetic images of FaMNIST clearly distinguish the categories of various products, which is consistent with the clarity of the MNIST dataset. Compared with other existing compression methods, the synthetic images of CIFAR10 are also very clear.

[0118] During the iterations, the mean squared error (MSE) between the real and virtual gradients is recorded. Starting from random Gaussian noise (first column), as the MSE between the real and virtual gradients decreases, DLG tries to match the gradients produced by the pseudo data and the real data. As the gradients decrease, the pseudo data becomes closer to the real data, but still cannot get a clear image of the real data. For MNIST and FaMNIST, the synthetic image is output every 60 iterations, while for CIFAR10, the synthetic image is output every 180 iterations. According to Figure 5 , based on CIFAR10, iterates 180, 360, and 540 times respectively. It can be seen that for the synthetic image, DLG is completely unable to restore the original image and even cannot show any features, which shows that the method of the present invention effectively fights against DLG.

[0119] Example 2

[0120] This embodiment provides a device for implementing a method for privacy protection of federated meta-learning based on data set compression, the device comprising:

[0121] at least one processor; and

[0122] A memory storing instructions, which, when executed by the at least one processor, enables the at least one processor to perform the method for privacy protection of federated meta-learning based on data set compression as described above.

[0123] In this embodiment, electronic devices include but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0124] Example 3

[0125] This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to perform the method of privacy protection of federated meta-learning based on data set compression as described above.

[0126] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0127] In this case, the program code itself read from the machine-readable medium can realize the function of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0128] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code may be downloaded from a server computer or a cloud via a communication network.

[0129] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for privacy protection of federated meta-learning based on dataset compression, characterized in that: The method comprises: S1. Each client independently initializes a synthetic dataset to replace the original dataset of the client in federated meta-learning, and then randomly initializes the synthetic dataset and learning rate through the generative adversarial network; S2, inputting the synthetic data set and learning rate into the initial local model parameters for training to obtain a new round of local model parameters; Then minimize the loss of the original dataset on the new local model parameters, and optimize the synthetic dataset and learning rate; Continuously iterate the optimization until the maximum mean difference converges to the preset value or reaches the preset number of iterations, and obtain the final synthetic data set and the final learning rate; S3, federated meta-learning obtains the real gradient based on the final synthetic dataset and the final learning rate, and then obtains the virtual dataset and virtual gradient through the DLG attack embedded in the federated meta-learning; Finally, the original gradient of the original data in the federated meta-learning is calculated, and noise is added to the image data in the final synthetic data set according to the loss maximization of the original gradient and the virtual gradient to obtain the noisy final synthetic data set and the noisy learning rate; The S2 specifically includes: S21. Synthetic Dataset And the corresponding learning rate Initial local model parameters sent from the central server θ 0 to perform a round of training and obtain a new round of local model parameters θ 1： (1)； In formula (1), is θ The loss on 0, is the gradient, is the learning rate, θ 1 is the new round of local model parameters; S22. Calculate the parameters of the new local model for the original data set θ 1, and use this loss as the optimization target to minimize the loss to optimize the synthetic dataset and learning rate: (2)； In formula (2), Initializing local model parameters θ Loss function under 0; S23. The optimization results of one round are as follows: and a round of optimized learning rate , (3)； In formula (3), is the synthetic dataset and learning rate after one round of optimization; S24, a round of optimized synthetic dataset and a round of optimized learning rate Local model parameters after one round of training θ 1 is trained again to get θ 2. Continuously iterate and optimize until the maximum mean difference converges to the preset value or reaches the preset number of iterations to obtain the final optimized synthetic data set and optimize the learning rate , the final optimization goal is as follows: (4)； In formula (4), is the final optimized synthetic dataset and the corresponding learning rate; The S3 specifically includes: S31, the n clients participating in the training input the final synthetic data set, and then train it in the local model of each client; S32, the final synthetic data set input is trained on the local model to obtain the generated gradient, and the W1 to W n There are n client gradients in total; S33, upload the client gradient generated by the local model training to the central server, and then the central server aggregates and averages the real gradient, and continuously iterates to obtain the real gradient W; S34, then the DLG attack uses the central server as an attacker and randomly initializes a virtual data set in Python using the numpy library. The initial virtual data set includes virtual input , virtual label ; The batch size of the initial virtual dataset is equal to the batch size of the final synthetic dataset; The initial virtual dataset is trained in the global model of the central server to obtain the virtual client gradient, which corresponds to n clients. arrive A total of n virtual client gradients; S35: Aggregate and average the n generated client virtual gradients to obtain a virtual gradient , used to update the virtual dataset; (5)； In formula (5), , are virtual input and virtual label respectively, W is the parameter of the global model, L is the loss function, is the differential derivative, is a differentiable model, is the final virtual gradient generated; S36, the central server will be the real gradient W is sent to each client for iterative update of the global model, continuously optimizing the virtual gradient to make it fit the real gradient, and at the same time, making the updated virtual data set fit the final synthetic data set; The distance between the virtual gradient and the real gradient is used as the loss to iteratively update the virtual dataset to obtain the final virtual dataset: (6)； In formula (6), , They are the virtual input and virtual label after the update is completed. W is the true gradient from the final synthetic dataset; S37, the n normal clients participating in the training input the original data in the original data set, and then train in the local model of the client to obtain arrive There are n original client gradients in total, and the central server aggregates and averages them to get the original gradient , through federated meta-learning, iterative updates are continuously made to obtain the final original gradient ; Reverse optimization is performed through gradient ascent, which maximizes the loss between the virtual data set and the original data set by maximizing the loss of the final virtual gradient and the final original gradient: (7)； In formula (7), is the final original gradient; It is the final virtual gradient corresponding to the last round of iterative update of the final virtual gradient; S38, adding noise to the image data in the final synthetic data set according to the maximum loss between the virtual data set and the original data; S39, repeat the step S2 for the final synthetic data set after adding noise until convergence, and obtain the final synthetic data set with noise and the noisy learning rate ; (8)； In formula (8), is the noised final synthetic dataset and the noised learning rate, is the local model parameter θ n-1 The loss function under The mean is 0 and the variance is Gaussian distribution of The adding noise to the image data in the final synthetic data set comprises: Use a random number generator to generate Gaussian distributed random values ​​of the same size as the image, and then add the generated Gaussian noise to the image data in the final synthetic dataset.

2. A device for privacy protection of federated meta-learning based on data set compression, characterized in that: The device comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the method for privacy protection of federated meta-learning based on data set compression as described in claim 1 are implemented.

3. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 are implemented.

Images