Knowledge distillation guided privacy data enhancement method and device based on large language model

Abstract

The invention discloses a knowledge distillation guided privacy data enhancement method and device based on a big language model, and relates to the technical field of data enhancement, and the technical key points are as follows: step S1, constructing a public data generator based on the big language model; s2, constructing a discriminator based on differential privacy to check whether a pseudo sample conforms to privacy distribution or not; and S3, constructing label distribution assistant teaching. According to the method, a DP-based text generation task is transferred to a DP-based text classification task, so that the complexity of a DP text generation model is avoided. LLM, students and tutors can be called infinitely to achieve data enhancement, and meanwhile privacy disclosure is limited. Noise required by the discriminator does not change along with the size of the model, and the discriminator can be well combined with a large language model. Meanwhile, the model in the invention is very low in calculation and system overhead consumed in the training process, and has very strong practicability.

Description

Technical Field

[0001] The present invention relates to the technical field of data enhancement, and in particular to a method and device for privacy data enhancement based on a large language model under the guidance of knowledge distillation. Background Art

[0002] There are two main approaches to existing DP-based text generation technology. One is to fine-tune the pre-trained language model using private data and add calibrated noise to the gradient of the model during training to meet DP.

[0003] However, this approach has some disadvantages. First, as the model grows, its computational and system overhead costs will become increasingly higher. Moreover, many large language models (LLMs) currently have excellent performance in text generation tasks, but they are not open source and cannot be fine-tuned in this way. Another way is to train a teacher model on private data (the teacher model is based on a pre-trained language model), and then have the teacher teach the student model, where noise is added to the teacher's output set to meet DP. This type of approach can be combined with LLM, and the noise required does not change with the size of the model, but this method still has the problem of continuous multi-task with a large candidate space in text generation, which greatly increases the noise scale and reduces the quality of the synthesized text. Summary of the invention

[0004] The purpose of the present invention is to solve the above problems and to provide a method and device for privacy data enhancement based on a large language model guided by knowledge distillation.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows:

[0006] The present invention provides a method for privacy data enhancement based on a large language model guided by knowledge distillation, comprising the following steps:

[0007] Step S1: construct a public data generator based on a large language model, and generate a large number of pseudo samples similar to private data as candidate samples by adjusting the prompts;

[0008] Step S2: construct a discriminator based on differential privacy to check whether the pseudo sample conforms to the privacy distribution so that it can serve as a privacy sample. The discriminator is a teacher-student framework with knowledge distillation, which consists of multiple teacher models and a student model. The teacher can access the private data, and the student can only access the perturbed teacher output.

[0009] Step S3: construct a label distribution assistant, the goal of which is to carry the label distribution of private samples. The assistant avoids directly accessing pure private samples and only accesses private information to maintain the protection boundary.

[0010] The present invention is further configured as follows: the specific sub-steps of step S1 are:

[0011] Step S11, designing a prompt template: designing different prompt templates for different types of privacy data; in the classification task, using a large language model to generate input text for each output label; for the large language model trained in the public domain, designing a prompt text to induce the LLM to try to generate private domain samples;

[0012] Step S12, generating a candidate sample set: through the designed prompt template, the large language model synthesizes pseudo samples in each category, thereby obtaining a public data set that does not involve any privacy leakage.

[0013] The present invention is further configured as follows: step S2 comprises the following steps:

[0014] Step S21, training teacher models: multiple teacher models are trained on multiple non-overlapping data sets, where private samples are used as positive samples and public samples generated by a large language model are used as negative samples; the teachers learn to determine whether the samples are from the private set or the public set, and all teachers follow the structure and initial parameters of a pre-trained model, and in order to meet differential privacy, the teacher's data is processed and the teacher is trained through two strategies; first, the private training data needs to contain only unique samples, and repeated private samples should be deleted; second, the shuffled private data is evenly divided into M non-overlapping sets, and each teacher is trained on each subset; at this time, the existence of any specific sample will only affect the output of one teacher, limiting the sensitivity of the teacher's noise distribution, and the average distribution and shuffled distribution of samples ensure the balance and training effect among all teachers;

[0015] Step S22, training the student model: The student model is a discriminator that cannot access the original private data and learns from the teacher through the following steps: (1) Merge: for a given synthetic pseudo sample x, merge the teacher's prediction results in indicates whether the synthetic pseudo sample x comes from a private dataset or a public dataset; (2) Noise addition: according to the differential privacy theory, calibration noise is added to the teacher’s prediction result to obtain represents the teacher’s estimation of the synthetic pseudo sample x. (3) Teaching, using noise output As pseudo labels, the student is taught how to discriminate private samples. For each synthetic pseudo sample x, the teaching process follows the cross-entropy loss.

[0016] The present invention is further configured as follows: in step S3, the teaching assistant collects the noisy label distribution of all private samples. The teaching assistant first obtains the actual label distribution by calculating a total of N training samples, and then adds calibrated Gaussian noise to the label distribution. The above mechanism ensures that the teaching assistant satisfies differential privacy.

[0017] A privacy data enhancement model based on a large language model guided by knowledge distillation is also provided, including:

[0018] Public data generator based on large language model: Generate public data using large language model;

[0019] Knowledge distillation discriminator based on differential privacy: Public samples similar to private samples are selected, and multiple teacher models access disjoint and unique private sets for training; students learn from the teacher output set after noise processing, thus achieving differential privacy;

[0020] Privacy module: Models the noisy private data label distribution and guides the generation of samples at a low privacy cost.

[0021] Compared with the prior art, the beneficial effects of this solution are as follows: the present invention transfers the DP-based text generation task to the DP-based text classification task, avoiding the complexity of the DP text generation model. In addition, obtaining private data through student models and teaching assistants that meet DP does not bring additional losses. Therefore, LLMs, students, and tutors can be called "infinitely" to achieve data enhancement while limiting privacy leakage. And in the present invention, the noise required by the discriminator does not change with the size of the model, so it can be well combined with a large language model. At the same time, the model in the present invention has very little computational and system overhead consumed during training, and is very practical. These advantages make the present invention have broad application prospects in the field of DP-based text generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below in conjunction with the embodiments.

[0025] Example:

[0026] Knowledge distillation-guided privacy data enhancement methods based on large language models, such as Figure 1 ,The specific technical scheme is as follows: Step 1: Construct a public data generator based on LLM, and ,generate a large number of pseudo samples that are as close as possible to ,private data as our candidate samples.

[0027] 1.1 Design prompt templates. Different prompt templates can be designed for different types of privacy data. In the classification task, we use LLM to generate input text for each output label. Even though LLM is trained in the public domain, we designed a prompt text to induce LLM to try to generate private domain samples. Let's take a medical transcription classification dataset as an example, where the prompt follows this template: "You are a professional medical transcriptionist. Please generate a medical transcription of [LABEL] and do not disclose the patient's name. The text length of the medical transcription is about 400 words, and at least 200 words." In the above template, "[LABEL]" represents the classification label c (that is, medical specialty). In this way, LLM generates an input text x with label c, forming a complete pseudo sample.<x,c> .

[0028] 1.2 Generate candidate sample sets. Through the designed prompt template, LLM synthesizes a large number of pseudo samples in each category, thereby obtaining a large public dataset that does not involve any privacy leakage.

[0029] Step 2: Construct a DP-based discriminator to check whether the pseudo sample conforms to the privacy distribution and can act as a private sample. The discriminator is a teacher-student framework with knowledge distillation, consisting of multiple teacher models and a student model. The teacher can access the private data, and the student can only access the perturbed teacher output.

[0030] 2.1 Training Teacher Models. We train multiple teacher models on multiple disjoint datasets, where private samples are used as positive samples and public samples generated by LLM are used as negative samples. The teachers learn to determine whether a sample is from the private set or the public set, and all teachers follow the structure and initial parameters of a pre-trained model. In order to meet DP, we carefully process the teacher data and train the teachers through two strategies. First, the private training data needs to contain only unique samples, and repeated private samples should be removed. The reason is that DP prevents unknown detection by identifying each sample that appears. Repeated samples that appear N times will increase the privacy loss, thereby reducing the bound of privacy protection (expressed in ε). To maintain the bound, the noise scale must be increased by N times or log(N), which is very detrimental to the performance. Second, we evenly divide the shuffled private data into M disjoint sets and train each teacher on each subset separately. In this way, the presence of any specific sample will only affect the output of one teacher, limiting the sensitivity of the teacher's noise distribution. Evenly distributing and shuffling samples ensures the balance and training effect among all teachers.

[0031] 2.2 Training the student model. The student model is a discriminator that cannot access the original private data and learns from the teacher through the following steps: (1) Merge. For a given sample x, we merge the teacher’s prediction results in Indicates whether the sample x comes from the private dataset or the public dataset. (2) Noise addition. According to DP theory, we add calibration noise to the teacher’s prediction result to obtain represents the teacher’s estimate of x. (3) Teaching. We use the noise output As pseudo labels, the student is taught how to identify private samples. For each sample x, the teaching process follows the cross entropy loss.

[0032] Step 3: Construct a label distribution assistant. The goal of the assistant is to carry the label distribution of private samples, because label distribution is very important for generating enhanced data. It avoids directly accessing pure private samples and only accesses a small amount of private information to maintain the protection limit.

[0033] 3.1 The teaching assistant collects the noisy label distribution of all private samples in a statistical way with very little privacy cost. The teaching assistant first obtains the actual label distribution by calculating a total of N training samples. Then, calibrated Gaussian noise is added to the label distribution. The above mechanism ensures that the teaching assistant satisfies DP.

[0034] In summary, the model process design of the present invention is as follows: First, the powerful generation ability of LLM is used to obtain high-quality data as candidate samples. After that, the DP-based discriminator is trained to select LLM-generated samples that are similar to private samples. Finally, the teaching assistant is used to filter the generated samples to ensure that their label distribution matches the private data. It is worth noting that obtaining private data will incur a privacy cost, but obtaining private data through student models and teaching assistants that meet DP will not bring additional losses. Therefore, we can "infinitely" call LLM, students, and tutors to implement DA while limiting privacy leakage.

[0035] like Figure 1 As shown in Figure 1, our device mainly consists of three parts: the generator based on LLM generates public data. The knowledge distillation discriminator based on DP discriminates the public data to obtain the probability that it meets the characteristics of private data. Finally, the label distribution assistant selects the subset with the highest sample probability when it meets the noisy label distribution. The dotted box is the privacy module.

[0036] Datasets and evaluation metrics:

[0037] The method is evaluated on three medical domain datasets (many well-known NLP datasets may have been used for LLM training, and using these datasets may not effectively demonstrate LLM's ability to generate data in the privacy domain): MedicalTrans, Medical Trans-large, and PubMed.

[0038] Medical Transcriptions contains medical transcription samples from 40 different medical specialties. At the same time, more synthetic samples than the original dataset are selected using the method proposed in this patent to obtain MedicalTranscriptions-Large. PubMed, a classic dataset for sequential sentence classification in medical abstracts, is also used, where samples are labeled into five different categories according to their role in the abstract. We use random subsets of these samples for training and evaluation. Basic text processing is performed on the data, converting all text to lowercase and removing punctuation. 75% of the samples are randomly used for training and 25% for testing.

[0039] Metrics include accuracy, precision, recall, and F1 score, (i.e., the harmonic mean of precision and recall).

[0040] Comparison method:

[0041] Two non-DP methods are used as reference:

[0042] (1) Private directly fine-tunes downstream models on private data without protection.

[0043] (2) DAw / Public fine-tunes downstream models on data synthesized by public LLM without accessing private data.

[0044] Three categories of DP-based methods are used:

[0045] (1) DP-FT-Downsteam uses DP-SGD or Ghost (using DP-Adam and ``ghostclipping'') to fine-tune the downstream model on private data.

[0046] (2) DP-FT-Generator: Fine-tune the generator and fine-tune the downstream model using non-private SGD on synthetic text.

[0047] (3) DA w / DP-Discriminator: Train the Discriminator using DP methods (e.g., DP-SGD, Ghost and Ours) on private data, use the Discriminator to score the text synthesized by LLM, and select the synthetic text with higher scores when imitating the noisy label distribution to fine-tune the downstream model using non-private SGD.

[0048] Table 1: Comparison of the results of baselines on three text classification datasets. For each metric, the bold text is the best result of the DP baselines, and the underlined text is the best result of all baselines.

[0049] Table 1

[0050]

[0051] Table 1 shows the overall performance. It can be observed that: Private acts as a performance cap since it fully accesses private data without any private protection. DP-FT-Downtown performs poorly, indicating that downstream models with DPSGD are difficult to fine-tune well under DP noise. In contrast, the post-processing property of DP allows our model to train downstream tasks on (almost unlimited) DP synthetic samples without additional privacy loss, resulting in better downstream models than DP-FT-Downsteam. DP-FT-Generator also performs worse than DA w / DP-Discriminator. This suggests that the synthetic text obtained by DA w / DP-Discriminator is more practical. Compared with DP-FT-Generator, DA w / DP-Discriminator only requires API access to generate synthetic text using powerful closed-source LLM, which is more efficient and convenient. In most cases, DA w / DP-Discriminator outperforms DA w / Public, indicating that the data quality is further improved after the discriminator filtering.

[0052] Ours outperforms other baselines within meaningful privacy protection ranges, and sometimes even outperforms Private. The reason for outperforming Private, which serves as an upper bound, is that the synthetic data of our patent are of higher quality and diversity than private data because they are sampled from LLM, which improves the generalization ability of the classification model when training data is insufficient. Compared with other discriminators,

[0053] It demonstrates more stable and superior performance, proving the effectiveness of the discriminator designed in this patent.

[0054] The above specific embodiments are merely explanations of the present invention and are not limitations of the present invention. After reading this specification, those skilled in the art may make modifications to the embodiments without any creative contribution as needed. However, such modifications are protected by the patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A privacy data enhancement method based on a large language model guided by knowledge distillation, characterized by: The following steps are involved: Step S1: construct a public data generator based on a large language model, and generate a large number of pseudo samples similar to private data as candidate samples by adjusting the prompts; Step S2: construct a discriminator based on differential privacy to check whether the pseudo sample conforms to the privacy distribution so that it can serve as a privacy sample. The discriminator is a teacher-student framework with knowledge distillation, which consists of multiple teacher models and a student model. The teacher can access the private data, and the student can only access the perturbed teacher output. Step S3: construct a label distribution assistant, the goal of which is to carry the label distribution of private samples. The assistant avoids directly accessing pure private samples and only accesses private information to maintain the protection boundary.

2. The method for privacy data enhancement based on a large language model guided by knowledge distillation as claimed in claim 1, characterized in that: The specific sub-steps of step S1 are: Step S11, designing prompt templates: designing different prompt templates for different types of privacy data; in the classification task, using a large language model to generate input text for each output label; For a large language model trained in the public domain, a prompt text is designed to induce the LLM to try to generate private domain samples; Step S12, generating a candidate sample set: through the designed prompt template, the large language model synthesizes pseudo samples in each category, thereby obtaining a public data set that does not involve any privacy leakage.

3. The method for privacy data enhancement based on a large language model guided by knowledge distillation as claimed in claim 1, characterized in that: The step S2 comprises the following steps: Step S21, training teacher models: multiple teacher models are trained on multiple non-overlapping data sets, where private samples are used as positive samples and public samples generated by a large language model are used as negative samples; the teachers learn to determine whether the samples are from the private set or the public set, and all teachers follow the structure and initial parameters of a pre-trained model, and in order to meet differential privacy, the teacher's data is processed and the teacher is trained through two strategies; first, the private training data needs to contain only unique samples, and repeated private samples should be deleted; second, the shuffled private data is evenly divided into M non-overlapping sets, and each teacher is trained on each subset; at this time, the existence of any specific sample will only affect the output of one teacher, limiting the sensitivity of the teacher's noise distribution, and the average distribution and shuffled distribution of samples ensure the balance and training effect among all teachers; Step S22, training the student model: The student model is a discriminator that cannot access the original private data and learns from the teacher through the following steps: (1) Merge: for a given synthetic pseudo sample x, merge the teacher's prediction results in indicates whether the synthetic pseudo sample x comes from a private dataset or a public dataset; (2) Noise addition: according to the differential privacy theory, calibration noise is added to the teacher’s prediction result to obtain represents the teacher’s estimation of the synthetic pseudo sample x. (3) Teaching, using noise output As pseudo labels, the student is taught how to discriminate private samples. For each synthetic pseudo sample x, the teaching process follows the cross-entropy loss.

4. The method for privacy data enhancement based on a large language model guided by knowledge distillation as claimed in claim 1, characterized in that: In step S3, the teaching assistant collects the noisy label distribution of all private samples. The teaching assistant first obtains the actual label distribution by calculating a total of N training samples, and then adds calibrated Gaussian noise to the label distribution. The above mechanism ensures that the teaching assistant satisfies differential privacy.

5. A device for privacy data enhancement based on a large language model guided by knowledge distillation, characterized in that: include: Public data generator based on large language model: Generate public data using large language model; Knowledge distillation discriminator based on differential privacy: Public samples similar to private samples are selected, and multiple teacher models access disjoint and unique private sets for training; students learn from the teacher output set after noise processing, thus achieving differential privacy; Privacy module: Models the noisy private data label distribution and guides the generation of samples at a low privacy cost.

Images