A labeled data synthesis method and system based on differential privacy and a medium

Abstract

The application claims a label data synthesis method and system based on differential privacy, relating to the technical fields of image processing, computer vision, machine learning, privacy protection and the like. The specific steps are: 1) collecting and sorting image samples containing high privacy information; 2) training a generation model using differential privacy to generate synthetic data; 3) training a student model using knowledge transfer and differential privacy; 4) using the student model to give pseudo-labels to the synthetic data; 5) aligning the synthetic data and pseudo-labels to obtain label data with the same distribution as the private data set. This method releases the original image in the form of a synthetic data set using differential privacy, protecting the privacy of the original data set, and the synthetic data set has high-quality pseudo-labels, which can be applied to real-time scenarios such as face recognition and target recognition, and has practical significance.

Description

TECHNICAL FIELD

[0001] The application relates to a label-based data synthesis method and system based on differential privacy and a medium, and belongs to the technical fields of privacy protection, computer vision, machine learning, etc. BACKGROUND

[0002] With the continuous development of deep learning technology, the achievements made by deep learning have been deeply rooted in people's hearts and are widely used in various fields. Generally, some applications of deep learning, such as the judgment of a certain type of disease, involve a large amount of sensitive data, and the neural network model will implicitly remember these data. Some analysis of the model (member inference attack, model inversion, etc.) may expose some sensitive information. At the same time, training a well-performing deep learning model requires a large amount of training data, which indicates the importance of data in deep learning. In the fields of healthcare and finance, these data sets are usually small in sample size and contain a large amount of private information, which makes deep learning unable to achieve good results in these fields and unable to be implemented on a large scale.

[0003] At the same time, in the core tasks of computer vision technology, such as image classification, object detection, target tracking, semantic segmentation and instance segmentation, the most advanced methods are usually trained based on a large amount of labeled data. In the field of computer vision, there is a large amount of unlabeled data, and adding labels to these data will consume a large amount of human and material resources, so that the size and quality of the available data cannot reach the required level for further improving the results. Generally, labeled data carries more private information than unlabeled data. The motivation of the work proposed in this paper is to transfer the information carried by the private data set to the synthetic data set, while assigning high-quality pseudo-labels to the synthetic data set, while not leaking data privacy from the synthetic labeled data set.

[0004] Differential privacy is a mathematical definition of privacy proposed by Dwork in 2006. Since its introduction, DP has been widely used by the privacy community and has now become the gold standard in the privacy community. DP provides provable protection against attackers with certain prior knowledge and provides explicit quantification of privacy loss. The core idea of differential privacy is to confuse the accurate output result by introducing randomness. Abadi applied differential privacy to deep learning in 2016 and proposed a differential training method (DP-SGD), and by introducing a matrix accountant, it provides a more stringent limit on the privacy loss caused by DP-SGD. This differential privacy-based machine learning ensures that attackers cannot distinguish between models trained with or without specific samples, thereby solving the training set leakage problem in machine learning.

[0005] This training method adds randomness (noise) to the backpropagation gradient of the network, and the amount of noise added is proportional to the size of the gradient. Generally, in large computer vision models with a large number of parameters, a large amount of noise needs to be injected to confuse the output results of the model, which will affect the accuracy of the model, causing this DP-SGD method to not work well for large computer vision tasks.

[0006] A recent model-agnostic private learning considers an alternative method of injecting randomness, which in some cases achieves better accuracy at the same level of privacy compared to the aforementioned DP-SGD. Such methods add randomness to the aggregated results without considering their internal workings, using the sample and aggregation framework. Parnot proposed a PATE method based on transfer learning. This method divides the private dataset into n disjoint datasets, then trains different models independently from these datasets to obtain n "teacher models". When deploying the trained teacher models, we record the prediction results of each teacher model, select the one with the highest number of votes, aggregate the predictions, and add Laplace noise that meets the difference privacy to the aggregated results to protect the privacy information. When the predictions of the aggregated teacher models are highly consistent, the privacy cost remains very small. The aggregated teacher models are then used to annotate public data and transfer knowledge to train student models. The PATE method has strong privacy, but it is not very scalable and cannot solve the problem of small data volume.

[0007] To solve this problem, we use the increasingly developed generative model as a means of data expansion. By drawing a data distribution graph from a small set of training data, we can sample from the distribution graph and generate additional samples for our research. Generative adversarial networks (GANs) and their variants generate high-quality "fake" samples that are indistinguishable from real samples by combining the complexity of deep neural networks with game theory, demonstrating impressive performance in modeling the underlying data distribution.

[0008] After we train the generative model with the private dataset, we input a certain dimension of noise into the generator to obtain a large amount of generated data with the same distribution as the private training set. Since an attacker may reverse the model parameters of the generator to our private dataset, causing a privacy leak, we distinguish the training of the generative model to protect privacy. We use the student model as a label generator for the "fake" samples of the generator, so that we can generate large-scale labeled images for any subsequent tasks.

[0009] CN111628974A, a differential privacy protection method, device, electronic equipment and storage medium. The protection method includes the steps of: using a pre-defined mapping function to map each user response of a user response set to a mapping result of a predetermined word length to form a data set, each mapping result is an element of the data set, the data set is associated with a set differential privacy parameter, the data set is divided into multiple groups according to the distance between elements to classify adjacent elements into the same group, and the group differential privacy parameter corresponding to the group is calculated according to the distance between the elements in the group. The group differential privacy parameter is less than the set differential privacy parameter. In the protection method of the present application, the user response set is mapped into a data set, and adjacent mapping results in the data set are classified into the same group, and the set differential privacy parameter is optimized to generate the group differential privacy parameter according to the mapping result to disturb the mapping result. In this way, the privacy protection degree is improved and the availability of the result is considered.

[0010] This patent mainly reduces the privacy cost while protecting the availability of the result, and improves the privacy protection degree. The main disadvantage of this patent is that the scope of application is not wide, and it can only be applied to specific scenarios.

[0011] One of the advantages of the present application is that it has strong applicability and is applicable in any image field. The present application mainly generates a large amount of labeled synthetic data through a two-stage generation framework, while ensuring the high quality of the synthetic data. SUMMARY

[0012] The present application aims to solve the problems of the above prior art. A differential privacy-based labeled data synthesis method, system and medium are proposed. The technical solution of the present application is as follows:

[0013] A differential privacy-based labeled data synthesis method, comprising the following steps:

[0014] (1) Collect and organize image samples containing high privacy information;

[0015] (2) Use differential privacy to train a generative model for generating synthetic data using private images;

[0016] (3) Train a teacher-student model using knowledge transfer and differential privacy for private images, and publish the student model; the teacher-student model divides the private data set into n data subsets, each data subset trains a network (the benchmark network is a vision-transformer), each network is also called a teacher model in the knowledge distillation framework, and the knowledge of each teacher model is aggregated. A network (benchmark network also uses vision-transformer) is trained again using a public data set, which is called a student model, and then the model is released.

[0017] (4) Use student model to give pseudo-label to synthetic data;

[0018] (5) Align synthetic data and pseudo-label, get labeled synthetic data which is same distribution with private dataset.

[0019] Further, the generation model in step (2) selects a generation model including GAN production adversarial network or VAE variational autoencoder.

[0020] Further, the step (3) trains the teacher-student model using knowledge transfer and differential privacy for private images, and publishes the student model, including the following steps:

[0021] The data holder needs to divide the private dataset, and train the teacher model on each divided data subset; then, aggregate these teacher models;

[0022] When a public unlabeled image query x is input, the teacher model will make a prediction on the image and output a prediction result, vote these prediction results, and calculate the vote number of a certain class j:

[0023] N j (x)=|{i:i∈[n],f i (x)=j}|

[0024] The public dataset has k different labels, x represents a public unlabeled image, i represents the i-th teacher model, n represents the number of teacher models, f represents the query result, N represents the number of votes, f i (x) represents the prediction result of the i-th teacher for the public image x, N j (x) represents the aggregated vote number of class j.

[0025] After getting the label vote N j for the public image x, add Gaussian noise to the aggregated result and analyze the vote result to protect privacy; get the vote result after adding noise

[0026]

[0027] representing Gaussian distribution with mean 0 and variance σ, use adaptive differential privacy voting mechanism to return pseudo-label;

[0028] After inputting some unlabeled public image queries, a series of aggregated teacher model vote results are obtained; when the teacher consistency is greater than the set value, a small amount of noise is added to maintain high precision; when the teacher consistency is lower than the set value, the query is not answered and it is simply discarded;

[0029]

[0030] N j (x) represents the aggregated vote count of each teacher model for the unlabeled public image, argmax represents the index value of the maximum value, that is, if the number of votes for class j is the largest, then j is returned, and f(x) represents the pseudo-label assigned to the unlabeled public dataset x through the differential privacy voting mechanism.

[0031] Furthermore, step (4) assigns pseudo-labels to the synthetic data using the student model, specifically including:

[0032] Then, the student model trained in step (3) is used to assign pseudo-labels to the synthetic data generated by the generator of the differential privacy generation model trained in step (2), so as to obtain high-quality labeled synthetic data, which can maintain utility and protect privacy for any subsequent task.

[0033] Furthermore, the teacher model divides the private dataset into n data subsets, trains a network for each data subset (the baseline network used is vision-transformer), and each network is also called a teacher model in the knowledge distillation framework. Then, the knowledge of each teacher model is aggregated, and another network is trained using the public dataset (the baseline network also uses vision-transformer). This network is called the student model, and then the model is released.

[0034] A medium that stores a computer program internally, which, when read by a processor, executes the method of any one of the preceding claims.

[0035] A differential privacy-based labeled data synthesis system, comprising:

[0036] Collection module: Used to collect and organize image samples containing highly private information;

[0037] Generative model training module: used to train a generative model from private images using differential privacy to generate synthetic data;

[0038] Teacher-Student Model Training Module: This module trains a teacher-student model using knowledge transfer and differential privacy on private images and then publishes the student model. The teacher-student model divides the private dataset into n subsets, trains a network on each subset, and uses the benchmark network Vision-Transformer. Each network is also called a teacher model in the knowledge distillation framework. The knowledge from each teacher model is then aggregated, and another network is trained using the public dataset. The benchmark network also uses Vision-Transformer. This network is called the student model, and then the model is published.

[0039] Aligning module: for giving pseudo labels to the synthetic data using the student model; aligning the synthetic data and the pseudo labels to obtain labeled synthetic data with the same distribution as the private data set.

[0040] The advantages and beneficial effects of the present application are as follows:

[0041] The present application proposes a new method of labeled data synthesis based on differential privacy inspired by PATE. This method is highly versatile, and users do not even need to know the internal details of the model. Under the premise of actual privacy budget, it can well maintain the utility of the synthetic data set.

[0042] The present application uses differential privacy, computer vision, machine learning and other technologies to realize the task of labeled data synthesis. The present application proposes a differential privacy machine learning method based on transfer learning, and then gives pseudo labels to the unlabeled synthetic data synthesized by the differential privacy data synthesis method, thereby obtaining high-quality labeled synthetic data to protect privacy. Compared with existing solutions, the present application has greatly improved efficiency and greatly improved the usability of subsequent tasks. The present application has the following advantages:

[0043] (1) The code is written and run using the pycharm tool, which is low in cost;

[0044] (2) Under the premise of privacy protection, a large amount of high-quality labeled data can be generated, providing more data sources for computer vision tasks.

[0045] (3) It can also balance the trade-off between data quality and differential privacy, and can be flexibly adjusted according to different data application scenarios to meet different needs.

[0046] (4) The transfer learning differential privacy machine learning method proposed by the present application is versatile and can be used with any network and any data set.

[0047] (5) High recognition accuracy can be obtained on various data sets, and in most cases, a higher correct classification percentage than other methods can be obtained.

[0048] (6) The present application has good ease of use. Users do not need any prior knowledge, only need to input noise into the generator, and then input it into the student model to obtain a synthetic data set.

[0049] (7) It can be applied to practical scenarios such as face recognition and target recognition, and has important practical value.

[0050] The main innovation of the present application is the step of claim 3, the present application is the first method to generate labeled synthetic data under the premise of privacy protection, which is suitable for large computer vision tasks now. The present application is mainly a two-stage framework, which can make up for the shortcomings of existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a system flowchart of the present application, which provides a preferred embodiment of the present application;

[0052] Figure 2 is a differential privacy data synthesis flowchart;

[0053] Figure 3 is a differential privacy machine learning flowchart based on transfer learning. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. The described embodiments are only a part of the embodiments of the present application.

[0055] The technical solution of the present application to solve the above technical problems is:

[0056] As Figure 1 shown is a flowchart of a labeled data synthesis method based on differential privacy, comprising the following steps:

[0057] Step 1: Train a generative model (GAN, VAE) and a teacher-student model through differential privacy for the private data set.

[0058] Step 2: Input a certain dimension of noise to the generator of the generative model, such as the generator of the generative adversarial network GAN, to obtain a high-quality unlabeled synthetic data set with the same distribution as the private data set, as shown in Figure 2 .

[0059] Step 3: Publish the student model of the teacher-student model, which is transferred from the teacher model and has strong classification ability. By inputting the high-quality unlabeled data generated in step 2 into the student model, the high-quality synthetic data set is obtained. The specific steps are as follows:

[0060] 1. Train the teacher model

[0061] In order to protect the privacy of the data set, the data holder needs to divide the private data set and train the teacher model on each divided data subset. Then, aggregate these teacher models to obtain a more powerful model-aggregated teacher model.

[0062] 2. Label the public data set

[0063] When we input a public unlabeled image query x, the teacher model will make a prediction on the image and output a prediction result. Since different teacher models may have different advantages and weaknesses, we aggregate these prediction results together to get a more accurate result. Specifically, we can vote on these prediction results and calculate the number of votes for a certain class j:

[0064] N j (x) = | { i: i ∈ [n], f i (x) = j} |

[0065] 3. Add Gaussian noise

[0066] After getting the label votes Nj for the public image x, we add a certain amount of Gaussian noise to the aggregated results and analyze the voting results to protect privacy. Get the vote results after adding noise:

[0067]

[0068] 4. Through some voting strategies, such as the ReportNoiseMax strategy, directly return the class with the most votes after adding noise as the pseudo label of the public image.

[0069]

[0070] 5. Train student model

[0071] The training set of the student model comes from the unlabeled public dataset and the pseudo label assigned to it by the aggregated teacher model. Train the student model with these data to isolate the original data. Attackers can at most attack the student model released by the data holder to obtain the public dataset and the noise label after adding noise. When the student model is trained, the privacy cost is fixed, and any subsequent operation will not increase the privacy cost, which also guarantees the pseudo label assigned to the synthetic data generated by the differential privacy generation module later.

[0072] The specific process is shown in Figure 3 .

[0073] Step 4: Perform image classification or face recognition and other subsequent tasks on the synthetic dataset.

[0074] The systems, apparatuses, modules, or units illustrated in the above examples can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0075] Computer readable media includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carriers.

[0076] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0077] The above examples are to be understood only as illustrative of the application and not a limitation of the scope of protection of the application. After reading the specification, those skilled in the art can make various changes or modifications to the application, and these equivalent changes and modifications also fall within the scope defined by the claims of the application.

Claims

1. A method for synthesizing labeled data based on differential privacy, characterized in that, Includes the following steps: (1) Collect and organize image samples containing highly private information; (2) Use differential privacy to train a generative model from the private images to generate synthetic data; (3) Train a teacher-student model using knowledge transfer and differential privacy on the private images and publish the student model; The teacher-student model divides the private dataset into n data subsets, trains a network for each data subset, and selects the vision-transformer as the benchmark network. Each network is also called the teacher model in the knowledge distillation framework. Then, aggregate the knowledge of each teacher model and train another network using the public dataset. The benchmark network also uses the vision-transformer. This network is called the student model and then the model is published. (4) Use the student model to assign pseudo-labels to the synthetic data; (5) Align the synthetic data and pseudo-labels to obtain labeled synthetic data with the same distribution as the private dataset.

2. The labeled data synthesis method based on differential privacy according to claim 1, characterized in that, The generative model used in step (2) includes generative adversarial networks (GANs) or variational autoencoders (VAEs).

3. The labeled data synthesis method based on differential privacy according to claim 1, characterized in that, Step (3) involves training a teacher-student model using knowledge transfer and differential privacy on the private images, and then publishing the student model, including the following steps: The data holder needs to partition the private dataset and train a teacher model on each partitioned subset of data; then, these teacher models are aggregated. When given a public, unlabeled image query x, the teacher model makes a prediction for that image and outputs a prediction result. These prediction results are then voted on, and the number of votes for a particular category j is calculated. N j (x)=|{i:i∈[n],f i (x)=j}| The public dataset has k distinct labels, where x represents the public unlabeled image, i represents the i-th teacher model, n is the number of teacher models, f represents the query results, and N represents the number of votes. i (x) represents the prediction result of the i-th teacher for the public image x, N j (x) represents the aggregate vote count for category j; After obtaining the label vote N for the public image x j Then, Gaussian noise is added to the aggregated results, and the voting results are analyzed to protect privacy; the result after adding noise is obtained. A Gaussian distribution with a mean of 0 and a variance of σ is used to return pseudo-labels using an adaptive differential privacy voting mechanism; After a query is inputting some unlabeled public images, a series of aggregated teacher model vote results are obtained; when the teacher consistency is greater than a set value, a small amount of noise is added to maintain high accuracy; when the teacher consistency is lower than the set value, the query is not answered and is simply discarded. N j (x) represents the aggregated vote count of each teacher model for the unlabeled public image, argmax represents the index value of the maximum value, that is, if the number of votes for class j is the largest, then j is returned, and f(x) represents the pseudo-label assigned to the unlabeled public dataset x through the differential privacy voting mechanism.

4. The labeled data synthesis method based on differential privacy according to claim 3, characterized in that, Step (4) involves assigning pseudo-labels to the synthetic data using the student model, specifically including: Then, the student model trained in step (3) is used to assign pseudo-labels to the synthetic data generated by the generator of the differential privacy generation model trained in step (2), thus obtaining high-quality labeled synthetic data, which maintains utility and protects privacy for any subsequent task.

5. The labeled data synthesis method based on differential privacy according to claim 3, characterized in that, The teacher model uses the vision-transformer as the base network. The student model also uses the vision-transformer as the baseline network.

6. A medium that internally stores a computer program, characterized in that, When the computer program is read by the processor, it executes the method of any one of claims 1 to 5.

7. A labeled data synthesis system based on differential privacy, characterized in that, include: Collection module: Used to collect and organize image samples containing highly private information; Generative model training module: used to train a generative model from private images using differential privacy to generate synthetic data; Teacher-Student Model Training Module: This module trains a teacher-student model using knowledge transfer and differential privacy on private images and then publishes the student model. The teacher-student model divides the private dataset into n subsets, trains a network on each subset, and uses the benchmark network Vision-Transformer. Each network is also called a teacher model in the knowledge distillation framework. The knowledge from each teacher model is then aggregated, and another network is trained using the public dataset. The benchmark network also uses Vision-Transformer. This network is called the student model, and then the model is published. Alignment module: Used to assign pseudo-labels to synthetic data using the student model; aligns the synthetic data and pseudo-labels to obtain labeled synthetic data with the same distribution as the private dataset.

Images