Differential privacy data set distillation method and system based on image generation data

Abstract

The invention discloses a differential privacy data set distillation method and system based on image generation data, and belongs to the technical field of data privacy protection and machine learning. The method comprises the following steps: firstly, synthesizing a synthetic data set meeting Gaussian differential privacy, training a feature extractor on the synthetic data set, and finely adjusting an expert model by using original data differential privacy; multiple rounds of iterative optimization are carried out on the distillation data set initialized according to the classes, wherein a feature extractor is randomly selected according to the classes in each round to align the features of the original data added with the noise and the distillation data, and an expert model is utilized to align the semantics of the distillation data hard labels and the semantics of the synthetic data soft labels; and finally outputting a distillation data set meeting the total privacy budget. According to the method, effective priori is provided by utilizing generated data, extra noise injection in the alignment process is reduced, convergence is accelerated, and better privacy protection and data availability balance compared with a previous method is realized under the same differential privacy budget.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data privacy protection and machine learning, and particularly relates to a differential privacy dataset distillation method and system based on image generated data. BACKGROUND

[0002] In recent years, deep learning has achieved great success in the field of computer vision, natural language processing, etc., which all rely on training on large-scale datasets. However, the huge amount of data poses a significant threat to the training efficiency of the model and the storage of data. The data set distillation technology is proposed to solve the problem of massive data. As a data compression technology, it aggregates the original data into a smaller distillation dataset with high knowledge density. The downstream model trained on the small distillation dataset can achieve comparable generalization performance to the model trained on the real data. Currently, the dataset distillation technology is mainly applied to image classification tasks, and the generated distillation data can be applied to continuous learning and privacy protection downstream applications.

[0003] The distillation dataset provides a solution for the privacy data publishing scenario. The distillation data has certain privacy protection properties by replacing the original data publishing through its compression characteristics. However, the standard dataset distillation lacks formal privacy guarantees, which allows attackers to infer sensitive information in the original data through reverse engineering or membership inference attacks. For example, the distillation set of medical images or face data may leak the identity or biometric features of patients, resulting in serious privacy risks. Therefore, for strict privacy protection scenarios, differential privacy guarantees need to be provided for the distillation data. Differential privacy can control the strength of privacy protection through a privacy budget, while theoretically providing an upper limit to privacy leakage, thereby providing reliable privacy guarantees to ensure that the output information is not affected by a single privacy sample.

[0004] Existing differential privacy dataset distillation methods directly inject noise during the distillation process, which provides certain privacy protection, but the noise interference can cause the generated data to be severely distorted, significantly reducing the visual authenticity and semantic consistency, and further affecting the performance of the downstream model. At the same time, existing methods usually use the Laplace differential privacy mechanism to control the amount of noise, and excessive noise can cause the distribution of the distillation data to deviate significantly from the private data, thereby damaging the quality of the distillation samples. SUMMARY

[0005] In view of the above problems, there is an urgent need for a distillation method that balances privacy protection and data utility to meet the application requirements of privacy protection data publishing scenarios. The present application provides a differential privacy dataset distillation method and system based on image generated data, which guides the distillation process through differential privacy generated images to improve the usability of the distillation data.

[0006] To achieve the above object, the present application adopts the following technical solutions.

[0007] In a first aspect, the present application discloses a differential privacy dataset distillation method based on image generated data, comprising the following steps:

[0008] (1) using the original private dataset to fine-tune the pre-trained generator with differential privacy, or using the random generated image of the pre-trained generator to perform differential privacy nearest neighbor voting, to generate a synthetic dataset satisfying Gaussian differential privacy;

[0009] (2) using the synthetic dataset to train a plurality of feature extractors;

[0010] (3) using the original private dataset to fine-tune the pre-trained expert model with differential privacy to obtain an expert model;

[0011] (4) generating an initial distillation dataset by randomly sampling or clustering from the synthetic dataset by category;

[0012] (5) for the same category of samples in the distillation dataset, the following operations are iteratively performed:

[0013] randomly selecting a feature extractor, respectively extracting features from a subset of the original private dataset and a subset of the distillation dataset under the same category, and optimizing the distillation dataset by adding differential privacy feature matching loss with Gaussian noise;

[0014] using the expert model to perform semantic alignment between the hard label of the output of the distillation dataset under the same category and the soft label of the synthetic dataset under the same category, to obtain an expert guidance loss to optimize the distillation dataset;

[0015] traversing all categories;

[0016] (6) iteratively performing step (5) until a preset number of times is reached, and outputting a final distillation dataset whose total privacy budget satisfies:

[0017]

[0018] wherein μ total is the privacy parameter of Gaussian differential privacy, ∈ and δ are the privacy parameters of (∈, δ)-differential privacy, wherein ∈ is the privacy budget and δ is the probability of privacy failure; μ E represents the privacy parameter of expert model guidance, μ G represents the privacy parameter of synthetic data, μ F represents the privacy parameter of feature matching, and Φ represents the cumulative distribution function of standard normal distribution.

[0019] Further, the differential privacy fine-tuning of step (1) refers to performing gradient clipping after calculating the gradient of the back propagation of the original private data set, injecting Gaussian noise updating the pre-trained generator parameters; wherein, refers to a normal distribution with mean 0 and covariance .

[0020] updating formula of the pre-trained generator parameters θ G is as follows:

[0021]

[0022] wherein η G is the learning rate, B is the batch size, g i is the gradient of the i-th original private data set sample, σ G is the noise scale of the synthetic data, and ξ is the unit noise, is a Gaussian distribution, I is an identity matrix, clipping(.) is a clipping operation, and C G is the clipping threshold of the synthetic data;

[0023] using the updated generation model to directly generate a synthetic data set satisfying μ G -Gaussian differential privacy.

[0024] Further, the differential privacy nearest neighbor voting of step (1) includes:

[0025] using each sample in the original private data set to vote for the randomly generated image of the pre-trained generator to obtain a voting vector;

[0026] injecting Gaussian noise into the voting vector to sample from updating the voting vector and normalizing; wherein, refers to a normal distribution with mean 0 and covariance , and σ G is the noise scale of the synthetic data, and I is an identity matrix;

[0027] sampling the generated samples with higher probability in the voting vector to input the pre-trained generator again, obtaining generated images and voting again, repeating the iteration until a preset number of rounds is reached, and the generated images of the last round constitute the synthetic data set.

[0028] Further, when voting, the similarity of each sample in the original private data set and the randomly generated image in the CLIP model feature extraction is calculated, and each sample votes for the randomly generated image with the highest similarity.

[0029] Further, the pre-trained expert model is obtained by randomly selecting a feature extractor, or is trained on the synthetic data set.

[0030] Further, the differential privacy fine-tuning of step (1) refers to sampling the original private data set, performing gradient clipping after calculating the gradient of the back propagation of the sampled data, and injecting Gaussian noise updating the expert model parameters; wherein, refers to a normal distribution with mean 0 and covariance ;

[0031] The updating formula of the expert model parameters is as follows:

[0032]

[0033] wherein, η E is the learning rate, B is the batch size, g i is the gradient of the i-th original private data set sample, σ E is the noise scale guided by the expert model, and ξ is the unit noise, is a Gaussian distribution, I is a unit matrix, clipping(.) is a clipping operation, C E is the clipping threshold guided by the expert model.

[0034] Further, the feature extractor and the expert model adopt a convolutional neural network and are trained based on an image classification loss.

[0035] Further, the differential privacy feature matching loss is:

[0036]

[0037] wherein, represents the differential privacy feature matching loss, represents feature clipping, x * represents an input private image or a distilled image x S , represents a small batch of original private data subsets, represents a small batch of distilled data subsets, φ F is a randomly selected feature extractor, represents a distribution, C F represents the clipping threshold of feature matching, σ F represents the noise scale of feature matching, and I represents a unit matrix.

[0038] The noise scale σ F required for feature matching is calculated by the privacy budget allocation formula:

[0039]

[0040] wherein, T represents the number of iterations, Indicates the batch subsampling rate. Indicates the size of the small subset of the original private data. Represents the original private dataset Size.

[0041] Furthermore, the aforementioned expert guidance losses include:

[0042]

[0043] in, Indicates the total loss. Represents cross-entropy loss, Let λ represent the KL divergence loss, and let λ represent the weighting weight. Represents the distillation dataset, y S Represents the distillation image and its label, φ E Representing expert models, Represents the reference sample set Chinese reference sample The predicted probability distribution Indicates label y S Distillation image The predicted probability distribution, D KL Denotes KL divergence, Represents a reference sample set of the same category from a synthetic dataset. The calculated average soft label.

[0044] Secondly, this invention proposes a differential privacy dataset distillation system based on image generation data, used to implement the aforementioned differential privacy dataset distillation method.

[0045] The beneficial effects of this invention are:

[0046] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the present invention not only obtains better prior knowledge by using generated data, which can accelerate the convergence of distillation data optimization, but also reduces additional noise injection and improves the usability of distillation data by replacing the original data during the alignment process. Compared with previous methods, the distillation data generated by the present invention can achieve the best trade-off between privacy protection and data usability under the same differential privacy budget. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A total framework diagram of the differential privacy dataset distillation method based on image generated data provided for the examples of the present application is shown in the figure;

[0049] Figure 2 An implementation block diagram of the expert model training algorithm provided for the examples of the present application is shown in the figure.

[0050] Figure 3 An algorithm implementation block diagram of the differential privacy dataset distillation method based on image generated data provided for the examples of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.

[0052] To solve the problems of existing differential privacy dataset distillation technology, such as reducing data availability while reducing the fidelity of distilled data by directly injecting Gaussian noise in the distillation process. The present embodiment provides a differential privacy dataset distillation method based on image generated data, the framework of which is shown in the figure Figure 1 The differential privacy dataset distillation method based on image generated data includes the following modules:

[0053] (1) Differential privacy synthetic data generation module

[0054] In order to make the generated model have better generation effect, first use the diffusion model Pre-train on public dataset (not involving private information, such as ImageNet), and then After the forward process of gradually adding noise, learn the denoising reverse process on the U-Net network architecture, predict the noise by inputting the noise picture and the corresponding time step, use the mean square error of the predicted noise and the real noise as the loss function, and update the parameters of the diffusion model using the back propagation algorithm. After that, use the private original data Differential privacy fine-tuning of the generated model, or differential privacy nearest neighbor voting iteration of the generated model output result to obtain synthetic data Differential privacy fine-tuning refers to fine-tuning the pre-trained diffusion model

[0055] using the original private dataset In each training batch, the differential privacy stochastic gradient descent algorithm is adopted, that is, after calculating the gradient g of back propagation, gradient clipping (clipping threshold C G ) is performed, and Gaussian noise is injected. The parameter update formula of the model generated in fine-tuning is:

[0056]

[0057] where η G is the learning rate, B is the batch size, D is the number of samples of the original private data set, g i is the gradient of the i-th sample, σ G is the noise scale of the synthetic data, and ξ is the unit noise, is a Gaussian distribution, and I is an identity matrix. Finally, the updated diffusion model can be used to generate synthetic data that satisfies μ G - Gaussian differential privacy synthetic data set μ G is the privacy parameter of the synthetic data, and clipping(.) is the clipping operation.

[0058] Differential privacy nearest neighbor voting refers to an evolutionary algorithm for generating samples. First, the pre-trained CLIP model is directly used to extract features from the original private data set , and then each sample in the original private data set is used to vote for N generated samples, and the sample with the nearest neighbor in the CLIP model feature extraction is selected to obtain the final 1xN voting vector. In order to ensure the differential privacy of the voting result, noise needs to be added to the voting vector. Since the voting vector has a sensitivity of 1, it is not necessary to control the sensitivity by clipping. Gaussian noise is added to the voting vector to update the voting vector and normalize it.

[0059] The generated sample with a larger probability in the voting vector is sampled and re-input into the pre-trained CLIP model to generate N samples again. The iteration is repeated until the preset number of rounds is reached, and the generated sample in the last round is the final synthetic data that satisfies μ G - Gaussian differential privacy synthetic data set

[0060] The privacy parameter μ G of the above two differential privacy synthetic data sets requires a noise scale σ G to be determined by binary search, so that the FID index of the image generation under the total privacy parameter μ total is 1.2 times.

[0061] (2) Differential privacy feature matching module

[0062] Feature matching is a common paradigm for data set distillation, which is used to align the distillation samples with the private original samples in the feature space to learn the knowledge of the private original samples . The present application designs differential privacy for feature matching to ensure that the private original samples The query is differentially private.

[0063] (2.1) Distillation data initialization

[0064] In order to make the distillation data have prior knowledge, the present application generates an initial distillation data set from the synthetic data according to the category, or performs feature k-means clustering , where each category contains IPC samples, IPC represents the number of images per category, controls the size of the distillation data set, and is usually taken as 1, 10, 50, etc., representing the compression degree of the distillation data set.

[0065] (2.2) Feature extractor training

[0066] For the feature extractor, the feature matching paradigm usually selects multiple models pre-trained on private original data to more accurately capture sample features and enhance the adaptability of the distillation data to different initialized model training. In order to control the privacy budget and not introduce additional noise, the present application uses synthetic data to replace private original data to train the feature extractor, thereby obtaining n feature extractors

[0067] Here, the feature extractor is implemented by a convolutional neural network, and a fully connected layer is spliced after the feature extractor in the training stage to realize image classification. The loss is calculated based on the image classification result to update the feature extractor parameters.

[0068] (2.2) Differentially private feature matching

[0069] Randomly select a feature extractor φ F , and perform feature extraction on private original data and distillation data for different categories. First, Poisson sampling is performed on private original data and distillation data to obtain small batch subsets and Feature extraction on the original private data batch involves private information, so noise processing is needed to ensure differential privacy. In order to control the same scale, the present application implements the same clipping (clipping threshold C F ) on the original private data small batch subset and the distillation data small batch subset , and injects Gaussian noise to calculate the feature matching loss and update x S , and the loss formula is:

[0070]

[0071] where, represents the feature clipping to control the two-norm sensitivity of the feature vector, x * represents the input private image or distilled image x S , φ F represents a randomly selected feature extractor, C F represents the clipping threshold of feature matching, σ F represents the noise scale of feature matching, I represents the identity matrix. The distilled data is updated by gradient descent, and the noise scale of feature matching σ F is determined by the privacy parameter allocation strategy, and the privacy parameter μ G determining the synthesized data can be calculated in priority E and the privacy parameter μ E of the expert model guidance G is calculated by combining the subsampling theorem:

[0072]

[0073] where T is the number of iterations, represents the batch subsampling rate, μ E represents the privacy parameter of the expert model guidance, μ G represents the privacy parameter of the synthesized data, μ total is the privacy parameter of the Gaussian differential privacy of the total process of dataset distillation.

[0074] (3) Differential privacy expert model guidance module

[0075] Due to the introduction of noise in differential privacy in the feature matching process of the distillation process, the distilled samples deviate from the original label distribution, resulting in poor availability of distilled data. The expert model guidance module is designed to guide the distilled samples to not deviate from the original label distribution in the feature matching process.

[0076] (3.1) Expert model training

[0077] The feature extractor of the differential privacy feature matching module can be used for feature extraction of the sample, but the capture of the output soft label of the data is not accurate, so an expert model is needed to provide accurate sample soft labels. The training of the expert model φ E is shown in Figure 2 , a pre-trained feature extractor φ F may be selected or the expert model is obtained after pre-training on the differential privacy synthesized data and then trained on the private original data by differential privacy gradient descent algorithm.

[0078] Here, the base model is implemented by a convolutional neural network, and a fully connected layer is spliced after the base model in the training stage to realize image classification. The loss is calculated based on the image classification result to update the base model parameters as a pre-trained expert model.

[0079] When the pre-trained expert model is fine-tuned with differential privacy, the private original data is subsampled, and gradient clipping (clipping threshold C E ) is performed after calculating the gradient g of the sampled data, and Gaussian noise is injected to update the expert model φ E .

[0080]

[0081] where η E is the learning rate, B is the batch size, g i is the gradient of the i-th original private data set sample, σ E is the noise scale guided by the expert model, ξ is the unit noise, is a Gaussian distribution, I is a unit matrix, clipping(.) is a clipping operation, and C E is the clipping threshold. The noise scale σ E guided by the expert model can be obtained by binary search to achieve 90% classification accuracy of the expert model under the total privacy parameter.

[0082] (3.2) Expert model guided output alignment

[0083] To ensure that the distilled data maintains semantic consistency during feature matching, the present application proposes a double-label alignment strategy based on differential privacy generated data, combining hard label supervision and soft label distribution matching. For each sample in the distilled data set , a reference sample set is randomly sampled from the same category of the differential privacy synthetic data to provide an intra-class feature distribution prior. The expert model φ E is used to predict the reference sample set , and the intra-class average soft label is calculated. Where is the predicted probability distribution of the reference sample . The present application constrains the consistency of the prediction result of the distilled sample with its label by cross-entropy loss , and aligns the soft label distribution of the distilled sample with the reference sample by combining KL divergence loss . This module combines the two losses with weights:

[0084]

[0085] where λ is a weight coefficient. Update the distilled data by the stochastic gradient descent method The optimization direction is to minimize the difference with the reference distribution while preserving the class label information.

[0086] Repeat the feature matching and expert guidance steps until a preset number of iterations is reached, and output the final distilled dataset

[0087] (4) Privacy budget control

[0088] The above three modules are used to ensure differential privacy, and each has a noise scale σ G , σ F , and σ E , and the independent privacy parameters are μ G , μ F , and μ E . The privacy parameter μ total of the entire data set distillation process of the present application can be calculated by the composition theorem of Gaussian differential privacy

[0089] The present application also follows the standard (ε, δ)-differential privacy, and the conversion theorem of Gaussian differential privacy and standard differential privacy can be used , where Φ represents the cumulative distribution function of the standard normal distribution. Where μ is the privacy parameter of Gaussian differential privacy, and ε and δ are the privacy parameters of (ε, δ)-differential privacy. A random algorithm is said to satisfy (ε, δ)-differential privacy if for any two adjacent data sets D1 and D2, and any possible output set of the algorithm satisfies the following inequality:

[0090]

[0091] Where ε is the privacy budget, which measures the degree of change in the probability distribution of the algorithm's output when a single individual's data is added or deleted from the data set; δ represents the probability of privacy failure, which represents the probability that the privacy protection commitment may be broken. Generally, ε and δ are predetermined, and then converted into Gaussian differential privacy parameter μ, so that the noise scale required by each module can be further confirmed.

[0092] As shown in Figure 3 , the differential privacy data set distillation method based on image generated data is realized by combining the above modules, which includes:

[0093] S1, convert the Gaussian differential privacy parameter to obtain the privacy parameter of each stage, satisfying:

[0094] The total privacy budget satisfies:

[0095]

[0096] wherein μ total is a privacy parameter of Gaussian differential privacy, ∈ and δ are privacy parameters of (e, δ)-differential privacy, wherein ∈ is a privacy budget and δ is a probability of privacy failure in (e, δ)-differential privacy; μ E represents a privacy parameter of expert model guidance, μ G represents a privacy parameter of synthetic data, μ F represents a privacy parameter of feature matching, and Φ represents a cumulative distribution function of a standard normal distribution.

[0097] S2, using the original private data set to perform differential privacy fine-tuning on the pre-trained generator, or performing differential privacy nearest neighbor voting on the randomly generated images of the pre-trained generator, to generate a synthetic data set satisfying Gaussian differential privacy;

[0098] S3, training a plurality of feature extractors using the synthetic data set, and performing differential privacy fine-tuning on a pre-trained expert model using the original private data set to obtain an expert model;

[0099] S4, generating an initial distillation data set by randomly sampling or clustering from the synthetic data set according to categories;

[0100] S5, for the same category of samples in the distillation data set, iteratively performing the following operations:

[0101] randomly selecting a feature extractor, respectively performing feature extraction on a subset of the original private data set and a subset of the distillation data set under the same category, and optimizing the distillation data set by adding a differential privacy feature matching loss of Gaussian noise;

[0102] using the expert model to perform semantic alignment on the output of the distillation data set under the same category using its hard label and the soft label of the synthetic data set under the same category, to obtain an expert guidance loss to optimize the distillation data set;

[0103] traversing all categories;

[0104] iteratively performing step S5 until a preset number of times is reached, and outputting a final distillation data set.

[0105] To verify the effect of the present application, the present application uses three data sets of CIFAR-10, CIFAR-100 and CelebA for experimental evaluation. CIFAR-10 contains 60,000 color images, each image with a resolution of 32x32 pixels, and is divided into 10 different categories: airplane, car, bird, cat, deer, dog, frog, horse, ship and truck. CIFAR-100 contains 60,000 color images, each image with a resolution of 32x32 pixels. It is divided into 100 categories. These categories are further grouped into 20 superclasses, each containing several related subclasses (e.g. the "animal" superclass includes the "beaver" subclass, etc.). The CelebA dataset contains over 200,000 color celebrity images, each image with a resolution of 128x128 pixels, and with annotations of 40 binary attributes, including "smile", "young", "glasses", "blond hair", etc. The classification experiment only classifies the gender attribute into two categories. The feature extractor, expert model, and test model architecture used are three identical convolutional blocks, each containing a 3x3 convolutional layer with 128 kernels.

[0106] In the baseline model selection, the overview differential privacy data related work can be divided into two categories: the first category is differential privacy data generation work. The primary purpose of this type of work is to generate images that protect privacy while approximating the original private data set in visual effect. The present embodiment selects DP-MERF based on feature matching, PE based on evolutionary algorithm, and PrivImage based on differential privacy fine-tuning; the second category is differential privacy distillation data generation work. This type of work combines differential privacy and data set distillation, so that the model performance can still be good when the data amount is small. The present embodiment selects PSG based on gradient matching and NDPDC based on feature matching.

[0107] In the experimental setup, the selection of each type of data is 1, 10, and 50, i.e. the distillation volume of 1, 10, and 50 of each type of data is used for model training, reflecting the model performance when improving training efficiency under the scale of distillation data. In the privacy parameter, for CIFAR-10 and CIFAR-100, δ=10 -5 ; for CelebA, δ=10 -6 , experiments with ∈=1 and ∈=10 are conducted. The evaluation index is the test set accuracy (%) of the trained model.

[0108] Table 1 Test accuracy evaluation of the trained model of the distillation data set under the same privacy budget

[0109]

[0110] Table 1 illustrates the optimal utility-privacy tradeoff of this invention compared to the baseline model at the same privacy budget, given the scale of the distilled data. For example, on the CIFAR-10 dataset with a privacy budget of ∈=10 and 50 graphs per class, this invention achieves a test set accuracy of 65.5%, surpassing existing differential privacy data generation methods and differential privacy dataset distillation methods. Furthermore, differential privacy data generation methods directly generate data the size of the distilled dataset, resulting in poor data utility and lower model test set accuracy because they cannot condense the knowledge of the original private dataset. This invention bridges the gap between previous differential privacy dataset distillation and dataset distillation without privacy guarantees, enabling distilled datasets to maintain high data usability even with strict differential privacy guarantees.

[0111] The above embodiments are merely preferred embodiments of the present invention, and their descriptions are quite specific and detailed, but they should not be construed as limiting the scope of the present invention. Those skilled in the art can make partial adjustments without departing from the principles of the present invention, and such adjustments should all be considered within the scope of protection of the present invention.

Claims

1. A method for differential privacy dataset distillation based on image generated data, the method comprising: The method comprises the following steps: (1) performing differential privacy fine-tuning on a pre-trained generator using an original private dataset, or performing differential privacy nearest neighbor voting on randomly generated images of the pre-trained generator to generate a synthetic dataset satisfying Gaussian differential privacy; (2) training a plurality of feature extractors using the synthetic dataset; (3) obtaining an expert model by performing differential privacy fine-tuning on a pre-trained expert model using the original private dataset; (4) generating an initial distillation dataset by randomly sampling or clustering the synthetic dataset according to categories; (5) for samples of the same category in the distillation dataset, iteratively performing the following operations: randomly selecting a feature extractor, respectively performing feature extraction on a subset of the original private dataset and a subset of the distillation dataset under the same category, and optimizing the distillation dataset by adding a differential privacy feature matching loss with Gaussian noise; using the expert model to perform semantic alignment on the output of the distillation dataset under the same category by using the hard label of the output and the soft label of the synthetic dataset under the same category, and obtaining an expert guidance loss to optimize the distillation dataset; traversing all categories; (6) iteratively performing step (5) until a preset number of times is reached, and outputting a final distillation dataset whose total privacy budget satisfies: where μ total is the privacy parameter of Gaussian differential privacy, ∈ and δ are the privacy parameters of (e, δ)-differential privacy, ∈ is the privacy budget and δ is the probability of privacy failure in (e, δ)-differential privacy; μ E represents the privacy parameter of expert model guidance, μ G represents the privacy parameter of synthetic data, μ F represents the privacy parameter of feature matching, and Φ represents the cumulative distribution function of the standard normal distribution.

2. The method of claim 1, wherein, The differential privacy fine-tuning of step (1) refers to performing gradient clipping after computing the gradient of back propagation of the original private data set, injecting Gaussian noise updating the pre-trained generator parameters; wherein, means that the mean is 0, and the covariance is a normal distribution Pre-training generator parameters θ G The update formula is as follows: where η G is the learning rate, B is the batch size, g i is the gradient of the i-th original private dataset sample, σ G is the noise scale of the synthetic data, and ξ is the unit noise, is a Gaussian distribution, I is the identity matrix, clipping(.) is the clipping operation, C G is the clipping threshold of the synthetic data. Using the updated generative model to directly generate synthetic data that satisfies μ G - Synthetic datasets with Gaussian differential privacy.

3. The method of claim 1, wherein, The differential privacy nearest neighbor voting of step (1) comprises: voting on randomly generated images of the pre-trained generator using each sample in the original private dataset to obtain a voting vector; Gaussian noise is injected to the voting vector samples from The voting vector is updated and normalized; wherein, denotes a normal distribution with mean 0 and covariance , and G is the noise scale of the synthetic data, and I is the identity matrix. sampling generated samples with higher probabilities in the voting vector to input the pre-trained generator again, obtaining generated images again and voting, and repeating the iteration until a preset number of rounds is reached, and the generated images in the last round constitute the synthetic dataset.

4. The method of claim 3, wherein, When voting, the similarity of each sample in the original private dataset and the randomly generated image in the CLIP model is calculated, and each sample votes for the randomly generated image with the highest similarity.

5. The method of claim 1, wherein, The pre-trained expert model is obtained by randomly selecting a feature extractor or is trained on the synthetic dataset.

6. The method of claim 1, wherein, The differential privacy fine-tuning of step (1) refers to sampling the original private data set, performing gradient clipping after calculating the gradient of the back propagation of the sampled data, and injecting Gaussian noise updating the expert model parameters; wherein refers to a normal distribution with mean 0 and covariance ​ The update formula of the parameters of the expert model is as follows: where η E is the learning rate, B is the batch size, g i is the i-th original private dataset sample gradient, σ E is the noise scale guided by the expert model, and ξ is the unit noise, is a Gaussian distribution, I is an identity matrix, clipping(.) is a clipping operation, C E is the clipping threshold guided by the expert model.

7. The method of claim 1, wherein, The feature extractor and the expert model adopt a convolutional neural network and are trained based on an image classification loss.

8. The method of claim 1, wherein, The differential privacy feature matching loss is: wherein, denotes a differential privacy feature matching loss, denotes feature cropping, x * denotes an input private image or a distilled image x S , denotes a subset of the original private data mini-batch, denotes a subset of the distilled data mini-batch, φ F is a randomly selected feature extractor, denotes a distribution, C F denotes a cropping threshold for feature matching, σ F denotes a noise scale for feature matching, I denotes an identity matrix; The feature matches the required noise scale s F Computed by the privacy budget allocation formula: where T denotes the number of iterations, denotes the batch subsampling rate, denotes the size of the original private data mini-batch subset denotes the original private data set size.

9. The method of claim 1, wherein, The expert guidance loss comprises: wherein, denotes the total loss, denotes the cross-entropy loss, denotes the KL-divergence loss, and λ denotes the weighting weight, denotes the distilled dataset, y S denotes the distilled image and its label, and φ E denotes the expert model, denotes the reference sample set the predicted probability distribution of the reference sample in the reference sample set, denotes the predicted probability distribution of the distilled image S with label y denotes the KL-divergence, KL denotes the KL-divergence, denotes the average soft label computed over the same-class reference sample set from the synthetic dataset.

10. A system for differentially private dataset distillation based on image generated data for implementing the differentially private dataset distillation method of claim 1, wherein, The system comprises: a differential privacy synthetic data generation module configured to perform differential privacy fine-tuning on a pre-trained generator using an original private dataset, or perform differential privacy nearest neighbor voting on randomly generated images of the pre-trained generator to generate a synthetic dataset satisfying Gaussian differential privacy; a differential privacy feature matching module configured to train a plurality of feature extractors using the synthetic dataset, generate an initial distillation dataset by randomly sampling or clustering the synthetic dataset according to categories, and randomly select a feature extractor, respectively perform feature extraction on a subset of the original private dataset and a subset of the distillation dataset under the same category, and optimize the distillation dataset by adding a differential privacy feature matching loss with Gaussian noise; The differential privacy expert model guidance module is configured to perform differential privacy fine-tuning on a pre-trained expert model using an original private data set to obtain an expert model, and perform semantic alignment on an output of the expert model on a distillation data set of the category using a hard label of the output and a soft label of a synthetic data set of the same category to obtain an expert guidance loss optimized distillation data set. The privacy budget control module is configured to control a total privacy budget to satisfy: where μ total is the privacy parameter of Gaussian differential privacy, ∈ and δ are the privacy parameters of (e, δ)-differential privacy, ∈ is the privacy budget and δ is the probability of privacy failure in (e, δ)-differential privacy; μ E represents the privacy parameter of the expert model guidance, μ G represents the privacy parameter of the synthetic data, μ F represents the privacy parameter of feature matching, and Φ represents the cumulative distribution function of the standard normal distribution.

Images