Text classification method and device for protecting data privacy

Abstract

This specification discloses a text and sample classification method and apparatus for protecting data privacy. The text classification method includes: first, based on K subsets of a privacy-preserving text dataset, a low-rank adaptation technique is used to fine-tune a pre-trained base model with a basic parameter set, obtaining K trained teacher models; then, the K teacher models are used to label multiple public texts to obtain a public text dataset, wherein the category label of each public text is determined based on the noisy voting result of the K teacher models for that public text's category, the noisy voting result being obtained by adding differential privacy noise to the original voting result; subsequently, two second parameter groups with a parameter size much smaller than the basic parameter set are updated based on the public text dataset, and the updated parameter combination results of the two second parameter groups are superimposed on the basic parameter set to obtain a trained student model, used for classifying target text.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a text classification method and apparatus for protecting data privacy, a sample classification method and apparatus for protecting data privacy, a computer-readable storage medium, and a computing device. Background Technology

[0002] Text classification is required in many business scenarios. For example, e-commerce platforms need to identify users' levels of interest in products, such as dislike or interest, based on user reviews. With the development of machine learning technology, the idea is to build machine learning models to achieve text classification.

[0003] However, because text data inevitably involves user privacy to varying degrees, the training or use of text classification models carries the risk of leaking user privacy. Therefore, a solution is needed that can protect data privacy while ensuring the usability of text classification models. Summary of the Invention

[0004] This specification describes a text classification method and apparatus for protecting data privacy, which can achieve privacy-preserving and highly available text classification.

[0005] According to the first aspect, a text classification method for protecting data privacy is provided. The method includes:

[0006] Based on K text subsets of a privacy-preserving text dataset, a pre-trained base model with a basic parameter set is fine-tuned to obtain K trained teacher models. This first fine-tuning includes: updating two first parameter groups with significantly fewer parameters than the base parameter set using the i-th text subset, and then adding the updated parameter combinations of these two first parameter groups to the base parameter set to obtain the trained i-th teacher model. The K teacher models are then used to label multiple public texts to obtain a public text dataset. The category label of each public text is determined based on the noisy voting results of the K teacher models for that public text's category. These noisy voting results are obtained by adding differential privacy noise to the original voting results. Finally, two second parameter groups with significantly fewer parameters than the base parameter set are updated based on the public text dataset, and the updated parameter combinations of these two second parameter groups are added to the base parameter set to obtain a trained student model used for classifying target texts.

[0007] In one embodiment, updating two first parameter groups with a parameter count much smaller than the base parameter set using the i-th text subset includes: processing the text in the i-th text subset using the base parameter set and the two first parameter groups to obtain the corresponding text prediction result; determining the training loss based on the text prediction result and the corresponding text category label; and updating the two first parameter groups using the training loss while keeping the base parameter set fixed.

[0008] In one embodiment, the basic parameter set forms a basic parameter matrix, the two first parameter groups form two parameter matrices, and the parameter combination result of the two first parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0009] In one embodiment, the K teacher models are used to label multiple public texts to obtain a public text dataset, including: for each public text, the predicted categories of the K teacher models for that public text are statistically analyzed to obtain the original vote distribution for different predicted categories, which is used as the original voting result; differential privacy noise is added to the original voting result to obtain the noisy voting result; and the predicted category corresponding to the highest number of votes in the noisy voting result is determined as the category label of the public text.

[0010] In one embodiment, the differential privacy noise is Gaussian noise or Laplace noise that conforms to the differential privacy mechanism.

[0011] In one embodiment, the private text dataset belongs to a first domain. Before using the K teacher model to label multiple public texts, the method further includes: obtaining a first number of public texts; filtering out public texts belonging to the first domain from the first number of public texts and classifying them as the multiple public texts.

[0012] In one embodiment, updating two second parameter groups with a parameter count much smaller than the basic parameter set based on the public text dataset includes: processing text in the public text subset using the basic parameter set and the two second parameter groups to obtain corresponding text prediction results; determining a training loss based on the text prediction results and the corresponding text category labels; and updating the two second parameter groups using the training loss while keeping the basic parameter set fixed.

[0013] In one embodiment, the basic parameter set forms a basic parameter matrix, the two second parameter groups form two parameter matrices, and the parameter combination result of the two second parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0014] According to the second aspect, a sample classification method for protecting data privacy is provided. This method includes:

[0015] Based on K subsets of samples from the privacy sample dataset, a first fine-tuning is performed on the pre-trained base model with a basic parameter set to obtain K trained teacher models. This first fine-tuning includes: updating two first parameter groups with significantly fewer parameters than the base parameter set using the i-th sample subset, and then superimposing the updated parameter combinations of these two first parameter groups onto the base parameter set to obtain the trained i-th teacher model. The K teacher models are then used to label multiple public samples to obtain a public sample dataset. The class label of each public sample is determined based on the noisy voting results of the K teacher models for that public sample's class, where the noisy voting results are obtained by adding differential privacy noise to the original voting results. Finally, two second parameter groups with significantly fewer parameters than the base parameter set are updated based on the public sample dataset, and the updated parameter combinations of these two second parameter groups are superimposed onto the base parameter set to obtain a trained student model used for classifying target samples.

[0016] In one embodiment, the privacy sample dataset and the public sample dataset are respectively a privacy image dataset and a public image dataset, or respectively a privacy audio dataset and a public audio dataset.

[0017] According to a third aspect, a text classification device for protecting data privacy is provided, comprising:

[0018] The first fine-tuning module is configured to perform a first fine-tuning on a pre-trained base model with a basic parameter set based on K text subsets of a privacy text dataset, obtaining K trained teacher models. The first fine-tuning includes: updating two first parameter groups with parameter counts significantly smaller than the basic parameter set using the i-th text subset, and then superimposing the updated parameter combinations of these two first parameter groups onto the basic parameter set to obtain the trained i-th teacher model. The labeling module is configured to use the K teacher models to label multiple public texts, obtaining a public text dataset. The category label of each public text is determined based on the noisy voting results of the K teacher models for the category of that public text. The noisy voting results are obtained by adding differential privacy noise to the original voting results. The second fine-tuning module is configured to update two second parameter groups with parameter counts significantly smaller than the basic parameter set based on the public text dataset, and then superimpose the updated parameter combinations of these two second parameter groups onto the basic parameter set to obtain a trained student model used for classifying target texts.

[0019] According to the fourth aspect, a sample classification device for protecting data privacy is provided, comprising:

[0020] The first fine-tuning module is configured to perform a first fine-tuning on a pre-trained base model with a basic parameter set based on K sample subsets of the privacy sample dataset, obtaining K trained teacher models. The first fine-tuning includes: updating two first parameter groups with parameter counts significantly smaller than the basic parameter set using the i-th sample subset, and then superimposing the updated parameter combinations of these two first parameter groups onto the basic parameter set to obtain the trained i-th teacher model. The labeling module is configured to use the K teacher models to label multiple common samples, obtaining a common sample dataset. The class label of each common sample is determined based on the noisy voting results of the K teacher models for that common sample's class, where the noisy voting results are obtained by adding differential privacy noise to the original voting results. The second fine-tuning module is configured to update two second parameter groups with parameter counts significantly smaller than the basic parameter set based on the common sample dataset, and then superimpose the updated parameter combinations of these two second parameter groups onto the basic parameter set to obtain a trained student model used for classifying target samples.

[0021] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first or second aspect.

[0022] According to a sixth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method provided in the first or second aspect.

[0023] In summary, by employing the methods or apparatus disclosed in the embodiments of this specification, and using PATE as the skeleton for fine-tuning a large language model, better performance guarantees and strict privacy and security guarantees (based on differential privacy) are achieved. Simultaneously, the computational cost of fine-tuning a large language model is significantly reduced by utilizing low-rank adaptation techniques. Thus, highly available and privacy-preserving text classification can be achieved. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the architecture of the text classification model training scheme disclosed in the embodiments of this specification;

[0026] Figure 2This is a flowchart illustrating the text classification method for protecting data privacy disclosed in the embodiments of this specification;

[0027] Figure 3 This is a flowchart illustrating the sample classification method for protecting data privacy disclosed in the embodiments of this specification;

[0028] Figure 4 This is a schematic diagram of the structure of the text classification device for protecting data privacy disclosed in the embodiments of this specification;

[0029] Figure 5 This is a schematic diagram of the sample classification device for protecting data privacy disclosed in the embodiments of this specification. Detailed Implementation

[0030] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0031] As mentioned above, a solution is needed that can protect data privacy while ensuring the usability of the text classification model.

[0032] Currently, Large Language Models (LLMs) have demonstrated superior performance in understanding natural language text and have become a research hotspot in artificial intelligence. Therefore, this paper proposes to fine-tune pre-trained LLMs using privacy-preserving text data from a specified domain (or a specific business scenario), thereby enabling the fine-tuned models to be deployed in practical applications.

[0033] However, while large language models offer extremely high performance, they also raise concerns about unintended memory, which poses a privacy risk to fine-tuning of large language models in specific application domains. Furthermore, fine-tuning updates all parameters of the pre-trained model, leading to excessive computational costs.

[0034] Based on the above observations and analysis, this specification discloses a training scheme for a text classification model. It proposes to use the Private Aggregation of Teacher Ensembles (PATE) as the algorithm skeleton of the training scheme to ensure performance stability and achieve privacy protection. At the same time, by applying the Low-rank adaptation (LoRA) technique, the number of fine-tuning parameters is much smaller than the number of parameters in the original large language model, effectively reducing the fine-tuning cost.

[0035] Figure 1 This is a schematic diagram of the architecture of the text classification model training scheme disclosed in the embodiments of this specification. Figure 1As shown, the training scheme consists of two phases. In the first phase, K (≥2) teacher models are constructed using a pre-trained large language model and privacy-preserving text data. Specifically, the LoRA trick is applied to each of the K sub-data sets to freeze the parameters of the large language model, and a separate LoRA model is trained for each sub-data set. Figure 1 In simple terms, this is represented as LoRA. Since the number of parameters that need to be updated in a LoRA model is much smaller than the number of parameters in a large language model (e.g., less than one ten-thousandth), the cost of training K (e.g., 5 or 10) LoRA models is perfectly acceptable. After training K LoRA models, they are integrated as K teacher models into a privacy-preserving PATE model.

[0036] In the second stage, the PATE model integrated in the first stage is queried using publicly available text data to obtain privacy-preserving soft labels for the publicly available text. These PATE labels already integrate classification information from the privacy-preserving text data while protecting its privacy. Then, the LoRA technique is used to train a student model, which serves as the final text classification model for deployment.

[0037] The following is combined Figure 2 The specific implementation steps of the above training scheme are described in further embodiments. Figure 2 This is a flowchart illustrating a text classification method for protecting data privacy disclosed in the embodiments of this specification. The entity executing the method can be any device, server, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the method includes the following steps:

[0038] Step S210: Based on K text subsets of the privacy text dataset, perform the first fine-tuning on the pre-trained base model with the basic parameter set to obtain the K trained teacher models.

[0039] It should be understood that the aforementioned privacy text dataset includes multiple privacy texts and corresponding text category labels for each privacy text. Generally, privacy text datasets belong to a specific domain; for example, the dataset may be constructed based on data collected under a predetermined business scenario.

[0040] In one embodiment, conversation data from online customer service scenarios can be collected as private text, and the user intent corresponding to the conversation data can be used as a text category label. For example, the user's conversation data in a single customer service session can be used as a private text, and the user's standard question corresponding to the conversation data (e.g., how to activate a payment QR code) can be used as the corresponding text category label.

[0041] In another embodiment, user comment text from e-commerce platforms can be collected as private text, and the corresponding sentiment category (such as support, neutrality, and resistance) of the private text can be used as a text category label to obtain a private text dataset.

[0042] For the aforementioned K text subsets, generally, each of the K text subsets is mutually exclusive, and the union of the K text subsets is the aforementioned private text dataset. K is an integer greater than 1, and its specific value can be set by staff based on actual circumstances or experience; for example, K = 8 can be set.

[0043] There are several ways to divide a privacy-preserving text dataset into K text subsets. For example, the privacy-preserving text dataset can be divided into K equal text subsets. Specifically, the privacy-preserving text dataset can be randomly sampled without replacement until the sampled text reaches a predetermined number of equal parts, which are then classified into a text subset, until K text subsets are obtained.

[0044] It's understandable that in text classification scenarios, the foundation model in this step is a large language model. The text refers to the set of model parameters from this large language model as the foundation parameter set. This set can be on the order of hundreds of billions (or more) of parameters, trained on massive amounts of text data, typically crawled from the internet, covering various domains and thus possessing a very rich dataset. For example, one can directly obtain open-source, pre-trained large language models online, such as LLaMa and BLOOM, or alternatively, the currently closed-source GPT-3, and so on.

[0045] The above introduces K text subsets and the base model. Therefore, each text subset can be used individually to perform a first fine-tuning of the base model to obtain a trained teacher model, and thus obtain K teacher models. It should be understood that the "first" in "first fine-tuning," as well as similar terms like "first," "second," etc., used elsewhere in the text, are for distinguishing similar items and do not have any ranking or other limiting function.

[0046] The first fine-tuning described above employs the LoRA technique, which introduces low-rank parameters and freezes the parameters of the original large model, resulting in a significantly smaller number of fine-tuned parameters compared to the original large model. It is understood that those skilled in the art have a clear and unified understanding of "significantly smaller," and differences in magnitude, such as a difference of 1-2 orders of magnitude or even more, can be considered significant.

[0047] The first fine-tuning described above will be introduced below, taking the acquisition of any i-th teacher model among K teacher models as an example.

[0048] Specifically, the first fine-tuning includes: updating two first parameter groups of a basic parameter set with a much smaller number of parameters than the basic model using the i-th text subset, and superimposing the parameter combination result of the two updated first parameter groups on the basic parameter set to obtain the trained i-th teacher model.

[0049] Without loss of generality, the basic parameter set can be represented in the form of a matrix, denoted as the basic parameter matrix The two first parameter groups are respectively denoted as the first parameter matrix and the second parameter matrix It can be observed that the product matrix between the first parameter matrix and the second parameter matrix has the same dimension as the basic parameter matrix. However, by setting r i to be a value much smaller than d or t, for example, setting r i = 1, at this time, W1 and W2 are low-rank matrices (the rank refers to the number of vectors in the maximal linearly independent group of the matrix), and the number of parameters of W1 and W2 is much smaller than the number of parameters in W0, that is, d*r i + r i *t << d*t. It should be understood that the value of r i can be the same or different in different fine-tuning rounds. For example, set r1 = 1 and r2 = 2.

[0050] Based on this, the text in the i-th text subset can be processed using the basic parameter matrix, as well as the first parameter matrix and the second parameter matrix to obtain the prediction result of the text. However, when adjusting the parameters, only the parameters in the two parameter matrices are adjusted, and the basic parameter matrix is frozen, so that the number of parameters to be fine-tuned is much smaller than the number of parameters of the basic model.

[0051] Exemplarily, processing a certain text x in the i-th text subset using the basic parameter matrix W0, the first parameter matrix W1, and the second parameter matrix W2 can be simply expressed as the following formula:

[0052] where ΔW = W1 * W2(1)

[0053] It should be understood that formula (1) is only a simple illustration and not the calculation formula in actual prediction. For example, it does not illustrate the use of an activation function for non-linear transformation, nor does it illustrate the use of parameters in different network layers for forward propagation calculation.

[0054] In the actual prediction process, according to the mapping relationship between the parameters at the same positions in W0 and ΔW, after superimposing the parameters at the same position in ΔW on each parameter in W0 of the basic model, the text x is processed to obtain the prediction result of the text x This refers to the estimation of the category label y of text x. Additionally, for the initialization of W1 and W2, for example, W1 can be initialized as a zero matrix, and W2 can be randomly initialized using a Gaussian distribution.

[0055] Furthermore, after obtaining the text prediction results Then, based on the text prediction results Given the corresponding text category label y, determine the training loss, and update the first parameter matrix W1 and the second parameter matrix W2 using the training loss while keeping the basic parameter matrix W0 fixed.

[0056] From the above, through multiple rounds of iterative updates until the convergence criterion is reached, the updated first parameter matrix W1 and second parameter matrix W2 can be obtained; then, based on the basic model, the product matrix of the first parameter matrix W1 and the second parameter matrix W2 is superimposed on its basic parameter matrix W0 to obtain the trained i-th teacher model.

[0057] The above describes how to obtain any i-th teacher model through the first fine-tuning, and so on, to obtain K teacher models.

[0058] Then, in step S220, the K teacher models are used to label multiple public texts to obtain a public text dataset. The category label of each public text is determined based on the noisy voting result of the K teacher models on the category of the public text. The noisy voting result is obtained by adding differential privacy noise to the original voting result.

[0059] It should be noted that the data used to train the student model is publicly available and therefore not sensitive or private. Thus, even if the student model memorizes the training data during training, it is still non-private data and poses no risk of privacy breach.

[0060] In one embodiment, publicly available text crawled from the web can be directly used as the aforementioned plurality of publicly available texts. In another embodiment, considering that the student model is trained using publicly available data, which may have a distributional bias compared to the aforementioned private text dataset, an intuitive approach is proposed: the collected publicly available texts are filtered based on prior information about the domain or business scenario to which the private text dataset belongs. Specifically, if the private text dataset is known to belong to a first domain (such as risk control), then publicly available texts belonging to the first domain can be filtered from the initial number of publicly available texts already obtained, and used as the aforementioned plurality of publicly available texts.

[0061] The above describes the selection of multiple public texts. Next, taking any first public text as an example, we will introduce how to determine the category label of a public text.

[0062] Specifically, the first public text is predicted K times using the K pre-trained teacher models, resulting in K predicted categories. It should be understood that any two predicted categories may be the same or different.

[0063] Next, the votes for the K prediction categories are tallied to obtain the original vote distribution (or original voting results) for each prediction category. For example, the vote count for each candidate text category can be obtained. Assuming the total number of candidate text categories is C, the vote count for the j-th candidate text category can be denoted as n. j Therefore, the original vote distribution can be denoted as: (n1, n2, ..., n C ).

[0064] Then, differential privacy noise is added to the original vote distribution to obtain a noisy vote distribution (or noisy voting result). In one embodiment, noise conforming to the Differential Privacy (DP) mechanism can be added to each vote in the original vote distribution. Thus, based on the inheritance of the DP mechanism, the subsequently determined public text labels possess differential privacy characteristics, effectively preventing query attacks. That is, even if there is a difference of one data point in the query dataset, the change in output cannot be distinguished probabilistically, making it impossible to determine whether a certain data point is in the training dataset.

[0065] In one example, a Gaussian difference privacy mechanism is employed. Accordingly, Gaussian noise is added; for instance, C values ​​randomly sampled from a Gaussian distribution are added to C counts, which can be denoted as n. j +N(0,σ 2 In another example, a Laplace differential privacy mechanism is employed, which involves adding Laplace noise accordingly.

[0066] In another embodiment, noise conforming to differential privacy mechanisms can be added to a predetermined number (<C) of votes in the original vote distribution.

[0067] Next, the predicted category corresponding to the highest number of votes in the noisy voting results can be determined and used as the category label for the first public text. For example, c* = argmax can be selected. {j＝1,2,...,C} (n j +N(0,σ 2 On the other hand, in one embodiment, in the event of a tie, one of the category labels corresponding to the highest vote can be randomly selected as the category label for the corresponding public text, or the public text can be discarded.

[0068] The above method enables the labeling of the first public text. By analogy, multiple public texts can be labeled, thus forming a public text dataset with the category labels corresponding to each public text.

[0069] Then, in step S230, two second parameter groups with a parameter count much smaller than the basic parameter set are updated based on the public text dataset, and the parameter combination results of the updated two second parameter groups are superimposed on the basic parameter set to obtain a trained student model for classifying target text.

[0070] It should be noted that in this step, the base model is fine-tuned a second time using a public text dataset to obtain a trained student model. For an explanation of this step, please refer to the aforementioned description of step S210. The main differences are: compared to the first fine-tuning which uses a subset of text, the second fine-tuning uses a public text dataset; and compared to the two first parameter sets tuned in the first fine-tuning, the second fine-tuning uses two second parameter sets. In fact, the two initialized first parameter sets can be directly used as the two initialized second parameter sets, or the two second parameter sets can be initialized separately, or the number of parameters in the two second parameter sets may differ from the number of parameters in the two first parameter sets.

[0071] By performing this step, you can obtain a trained student model, which can be used as the final text classification model for classifying text (or target text) in actual business scenarios.

[0072] In summary, this paper proposes a text classification method for protecting data privacy, using PATE as the skeleton for fine-tuning a large language model. This achieves better performance guarantees and strict privacy protection (based on differential privacy). Furthermore, by utilizing low-rank adaptation techniques, the computational cost of fine-tuning the large language model is significantly reduced. Thus, highly available and strongly privacy-protected text classification can be achieved.

[0073] The above describes a text classification method for protecting data privacy. Furthermore, the applicant proposes that this method could be extended to process other types of samples besides text, such as images or audio.

[0074] Figure 3 This is a flowchart illustrating the sample classification method for protecting data privacy disclosed in the embodiments of this specification. The entity executing the method can be any device, platform, server, or equipment cluster with computing and processing capabilities. Figure 3 As shown, the method includes the following steps:

[0075] Step S310: Based on K sample subsets of the privacy sample dataset, perform a first fine-tuning on the pre-trained basic model with a basic parameter set to obtain K trained teacher models. The first fine-tuning includes: using the i-th sample subset to update two first parameter groups with a parameter count much smaller than the basic parameter set, and superimposing the updated parameter combination results of the two first parameter groups onto the basic parameter set to obtain the trained i-th teacher model.

[0076] It should be noted that the base model in this step is adapted to the sample content. For example, if the sample is an image, the pre-training data for the base model will use massive amounts of image data, and the implementation algorithm of the base model can also be adaptively selected, such as using Convolutional Neural Networks (CNN). As another example, if the sample is audio, the pre-training data for the base model will use massive amounts of audio data, and the implementation algorithm of the base model can be adaptively selected as a temporal network, such as the Transformer.

[0077] In one embodiment, the above-mentioned use of the i-th sample subset to update two first parameter groups with a parameter quantity much smaller than that of the basic parameter set includes: processing the samples in the i-th sample subset using the basic parameter set and the two first parameter groups to obtain the corresponding sample prediction results; determining the training loss based on the sample prediction results and the corresponding sample category labels; and updating the two first parameter groups using the training loss while keeping the basic parameter set fixed.

[0078] In one embodiment, the aforementioned set of basic parameters forms a basic parameter matrix, the two first parameter groups form two parameter matrices, and the result of the parameter combination of the two first parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0079] Step S320: Label multiple public samples using the K teacher models to obtain a public sample dataset. The category label of each public sample is determined based on the noisy voting result of the K teacher models on the category of the public sample. The noisy voting result is obtained by adding differential privacy noise to the original voting result.

[0080] In one embodiment, this step may include: for each public sample, statistically analyzing the predicted categories of the K teacher models for that public sample to obtain the original vote distribution for different predicted categories, which serves as the original voting result; adding the differential privacy noise to the original voting result to obtain the noisy voting result; and determining the predicted category corresponding to the highest number of votes in the noisy voting result as the category label for that public sample.

[0081] In one embodiment, the differential privacy noise is Gaussian noise conforming to the differential privacy mechanism, or Laplace noise.

[0082] In one embodiment, the privacy sample dataset belongs to a first domain. Before labeling multiple public samples using the K teacher models, the method further includes: obtaining a first number of public samples; and filtering out public samples belonging to the first domain from the first number of public samples, classifying them as the multiple public samples.

[0083] Step S330: Based on the public sample dataset, update two second parameter groups with a parameter count much smaller than the basic parameter set, and superimpose the updated parameter combination results of the two second parameter groups onto the basic parameter set to obtain a trained student model for classifying target samples.

[0084] In one embodiment, updating two second parameter groups based on the public sample dataset with a parameter count much smaller than that of the basic parameter set includes: processing samples in the public sample subset using the basic parameter set and the two second parameter groups to obtain corresponding sample prediction results; determining training loss based on the sample prediction results and the corresponding sample category labels; and updating the two second parameter groups using the training loss while keeping the basic parameter set fixed.

[0085] In one embodiment, the basic parameter set forms a basic parameter matrix, the two second parameter groups form two parameter matrices, and the parameter combination result of the two second parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0086] In addition, for Figure 3 The steps are explained below; you can also refer to the... Figure 2 The steps are described below.

[0087] In summary, this specification proposes a sample classification method for protecting data privacy, using PATE as the skeleton for fine-tuning large models. This achieves better performance guarantees and strict privacy and security guarantees (based on differential privacy). Furthermore, by utilizing low-rank adaptation techniques, the computational cost of fine-tuning large models is significantly reduced. Thus, highly available and strongly privacy-protected sample classification can be achieved.

[0088] Corresponding to the above-described classification method for protecting data privacy, embodiments of this specification also disclose a classification device.

[0089] Figure 4 This is a schematic diagram of the structure of the text classification device for protecting data privacy disclosed in the embodiments of this specification. Figure 4As shown, the text classification device 400 includes:

[0090] The first fine-tuning module 410 is configured to perform a first fine-tuning on a pre-trained base model with a basic parameter set based on K text subsets of a privacy text dataset, obtaining K trained teacher models. The first fine-tuning includes: updating two first parameter groups with parameter counts significantly smaller than the basic parameter set using the i-th text subset, and then superimposing the updated parameter combination results of the two first parameter groups onto the basic parameter set to obtain the trained i-th teacher model. The labeling module 420 is configured to use the K teacher models to label multiple public texts, obtaining a public text dataset. The category label of each public text is determined based on the noisy voting results of the K teacher models for the category of that public text. The noisy voting results are obtained by adding differential privacy noise to the original voting results. The second fine-tuning module 430 is configured to update two second parameter groups with parameter counts significantly smaller than the basic parameter set based on the public text dataset, and then superimpose the updated parameter combination results of the two second parameter groups onto the basic parameter set to obtain a trained student model used for classifying target texts.

[0091] In one embodiment, the first fine-tuning module 410 is specifically configured to: process the text in the i-th text subset using the basic parameter set and the two first parameter groups to obtain the corresponding text prediction result; determine the training loss based on the text prediction result and the corresponding text category label; and update the two first parameter groups using the training loss while keeping the basic parameter set fixed.

[0092] In one embodiment, the basic parameter set forms a basic parameter matrix, the two first parameter groups form two parameter matrices, and the parameter combination result of the two first parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0093] In one embodiment, the labeling module 420 is specifically configured to: for each public text, statistically analyze the predicted categories of the K teacher models for the public text to obtain the original vote distribution for different predicted categories, which is used as the original voting result; add the differential privacy noise to the original voting result to obtain the noisy voting result; and determine the predicted category corresponding to the highest number of votes in the noisy voting result as the category label of the public text.

[0094] In one embodiment, the differential privacy noise is Gaussian noise conforming to the differential privacy mechanism, or Laplace noise.

[0095] In one embodiment, the privacy text dataset belongs to a first domain, and the text classification device 400 further includes a text filtering module 440 configured to: acquire a first number of public texts; filter out public texts belonging to the first domain from the first number of public texts and classify them into the plurality of public texts.

[0096] In one embodiment, the second fine-tuning module 430 is specifically configured to: process the text in the public text subset using the basic parameter set and the two second parameter sets to obtain the corresponding text prediction results; determine the training loss based on the text prediction results and the corresponding text category labels; and update the two second parameter sets using the training loss while keeping the basic parameter set fixed.

[0097] In one embodiment, the basic parameter set forms a basic parameter matrix, the two second parameter groups form two parameter matrices, and the parameter combination result of the two second parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0098] Figure 5 This is a schematic diagram of the sample classification device for protecting data privacy disclosed in the embodiments of this specification. Figure 5 As shown, the sample classification device 500 includes:

[0099] The first fine-tuning module 510 is configured to perform a first fine-tuning on a pre-trained base model with a basic parameter set based on K sample subsets of the privacy sample dataset, obtaining K trained teacher models. The first fine-tuning includes: updating two first parameter groups with parameter counts significantly smaller than the basic parameter set using the i-th sample subset, and then superimposing the updated parameter combination results of the two first parameter groups onto the basic parameter set to obtain the trained i-th teacher model. The labeling module 520 is configured to label multiple public samples using the K teacher models, obtaining a public sample dataset. The category label of each public sample is determined based on the noisy voting results of the K teacher models for the category of that public sample. The noisy voting results are obtained by adding differential privacy noise to the original voting results. The second fine-tuning module 530 is configured to update two second parameter groups with parameter counts significantly smaller than the basic parameter set based on the public sample dataset, and then superimpose the updated parameter combination results of the two second parameter groups onto the basic parameter set to obtain a trained student model for classifying target samples.

[0100] In one embodiment, the privacy sample dataset and the public sample dataset are respectively a privacy image dataset and a public image dataset, or respectively a privacy audio dataset and a public audio dataset.

[0101] In one embodiment, the first fine-tuning module 510 is specifically configured to: process the samples in the i-th sample subset using the basic parameter set and the two first parameter groups to obtain the corresponding sample prediction results; determine the training loss based on the sample prediction results and the corresponding sample category labels; and update the two first parameter groups using the training loss while keeping the basic parameter set fixed.

[0102] In one embodiment, the basic parameter set forms a basic parameter matrix, the two first parameter groups form two parameter matrices, and the parameter combination result of the two first parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0103] In one embodiment, the labeling module 520 is specifically configured to: for each public sample, statistically analyze the predicted categories of the K teacher models for the public sample to obtain the original vote distribution for different predicted categories, which is used as the original voting result; add the differential privacy noise to the original voting result to obtain the noisy voting result; and determine the predicted category corresponding to the highest number of votes in the noisy voting result as the category label of the public sample.

[0104] In one embodiment, the differential privacy noise is Gaussian noise conforming to the differential privacy mechanism, or Laplace noise.

[0105] In one embodiment, the privacy sample dataset belongs to a first domain, and the sample classification device 500 further includes a sample screening module 540 configured to: acquire a first number of public samples; and screen out public samples belonging to the first domain from the first number of public samples, classifying them into the plurality of public samples.

[0106] In one embodiment, the second fine-tuning module 530 is specifically configured to: process samples in the public sample subset using the basic parameter set and the two second parameter sets to obtain corresponding sample prediction results; determine training loss based on the sample prediction results and the corresponding sample category labels; and update the two second parameter sets using the training loss while keeping the basic parameter set fixed.

[0107] In one embodiment, the basic parameter set forms a basic parameter matrix, the two second parameter groups form two parameter matrices, and the parameter combination result of the two second parameter groups is the product matrix of the two parameter matrices; the basic parameter matrix and the product matrix have the same matrix dimension.

[0108] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 2 The method described.

[0109] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 2 The method described.

[0110] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0111] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text classification method for protecting data privacy, comprising: performing first fine-tuning on a base model with a base parameter set to obtain K trained teacher models based on K text subsets of a privacy text dataset, wherein the first fine-tuning comprises: updating two first parameter groups with a parameter amount far less than the base parameter set by using an i-th text subset, and adding a parameter combination result of the two updated first parameter groups to the base parameter set to obtain an i-th trained teacher model; performing labeling on a plurality of public texts by using the K teacher models to obtain a public text dataset, wherein a class label of each public text is determined based on a noisy voting result of the K teacher models on a class of the public text, and the noisy voting result is obtained by adding differential privacy noise to an original voting result; updating two second parameter groups with a parameter amount far less than the base parameter set based on the public text dataset, and adding a parameter combination result of the two updated second parameter groups to the base parameter set to obtain a trained student model for classifying a target text.

2. The method of claim 1, wherein, updating the two first parameter groups with a parameter amount far less than the base parameter set by using the i-th text subset comprises: processing texts in the i-th text subset by using the base parameter set and the two first parameter groups to obtain corresponding text prediction results; determining a training loss based on the text prediction results and corresponding text class labels; updating the two first parameter groups by using the training loss under the premise of fixing the base parameter set.

3. The method of claim 1 or 2, wherein, The base parameter set forms a base parameter matrix, the two first parameter groups form two parameter matrices, and the parameter combination result of the two first parameter groups is a product matrix of the two parameter matrices; The base parameter matrix and the product matrix have the same matrix dimension.

4. The method of claim 1, wherein, The labeling on the plurality of public texts by using the K teacher models comprises: for each public text, counting prediction classes of the public text by the K teacher models to obtain an original vote distribution for different prediction classes as the original voting result; adding the differential privacy noise to the original voting result to obtain the noisy voting result; determining a prediction class corresponding to the highest vote in the noisy voting result as the class label of the public text.

5. The method of claim 1 or 4, wherein, The differential privacy noise is Gaussian noise or Laplace noise conforming to a differential privacy mechanism.

6. The method of claim 1, wherein, The privacy text dataset belongs to a first field, and before the labeling on the plurality of public texts by using the K teacher models, the method further comprises: obtaining a first number of public texts; selecting public texts belonging to the first field from the first number of public texts as the plurality of public texts.

7. The method of claim 1, wherein, The updating of the two second parameter groups with a parameter amount far less than the base parameter set based on the public text dataset comprises: processing texts in the public text subset by using the base parameter set and the two second parameter groups to obtain corresponding text prediction results; determine a training loss based on the text prediction result and the corresponding text category label; update the two second parameter groups using the training loss while fixing the base parameter set.

8. The method of claim 1 or 7, wherein, The base parameter set forms a base parameter matrix, and the two second parameter groups form two parameter matrices, and the parameter combination result of the two second parameter groups is a product matrix of the two parameter matrices. The base parameter matrix and the product matrix have the same matrix dimension.

9. A sample classification method for protecting data privacy, comprising: performing first fine-tuning on a base model having a base parameter set after pre-training based on K sample subsets of a private sample dataset to obtain K trained teacher models, wherein the first fine-tuning comprises: updating two first parameter groups with a parameter quantity much smaller than the base parameter set using an i-th sample subset, and superimposing a parameter combination result of the two updated first parameter groups onto the base parameter set to obtain a trained i-th teacher model; performing labeling on a plurality of public samples using the K teacher models to obtain a public sample dataset, wherein a category label of each public sample is determined based on a noisy voting result of the K teacher models on the category of the public sample, and the noisy voting result is obtained by adding differential privacy noise to an original voting result; updating two second parameter groups with a parameter quantity much smaller than the base parameter set based on the public sample dataset, and superimposing a parameter combination result of the two updated second parameter groups onto the base parameter set to obtain a trained student model for classifying target samples. The private sample dataset and the public sample dataset are a private image dataset and a public image dataset, or are a private audio dataset and a public audio dataset.

10. A text classification device for protecting data privacy, comprising: a first fine-tuning module configured to perform first fine-tuning on a base model having a base parameter set after pre-training based on K text subsets of a private text dataset to obtain K trained teacher models, wherein the first fine-tuning comprises: updating two first parameter groups with a parameter quantity much smaller than the base parameter set using an i-th text subset, and superimposing a parameter combination result of the two updated first parameter groups onto the base parameter set to obtain a trained i-th teacher model; a labeling module configured to perform labeling on a plurality of public texts using the K teacher models to obtain a public text dataset, wherein a category label of each public text is determined based on a noisy voting result of the K teacher models on the category of the public text, and the noisy voting result is obtained by adding differential privacy noise to an original voting result; a second fine-tuning module configured to update two second parameter groups with a parameter quantity much smaller than the base parameter set based on the public text dataset, and superimpose a parameter combination result of the two updated second parameter groups onto the base parameter set to obtain a trained student model for classifying target texts.

11. A sample classification device for protecting data privacy, comprising: The first fine-tuning module is configured to fine-tune the base model with the base parameter set to obtain K teacher models based on K sample subsets of the privacy sample dataset, wherein the fine-tuning includes: updating two first parameter groups with a parameter quantity far less than the base parameter set by using the i-th sample subset, and superimposing a parameter combination result of the two first parameter groups after the update on the base parameter set to obtain the i-th teacher model after the training; The labeling module is configured to label a plurality of public samples by using the K teacher models to obtain a public sample dataset, wherein a class label of each public sample is determined based on a noisy voting result of the K teacher models on the class of the public sample, and the noisy voting result is obtained by adding differential privacy noise to an original voting result; The second fine-tuning module is configured to update two second parameter groups with a parameter quantity far less than the base parameter set based on the public sample dataset, and superimpose a parameter combination result of the two second parameter groups after the update on the base parameter set to obtain a student model after the training, which is used for classifying target samples. The privacy sample dataset and the public sample dataset are respectively a privacy picture dataset and a public picture dataset, or are respectively a privacy audio dataset and a public audio dataset.

12. A computer readable storage medium having stored thereon a computer program, wherein, When the computer program is executed in the computer, the computer is caused to execute the method in any one of claims 1-9.

13. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-9.

Images