Method, device, equipment and storage medium for privacy protection of input data

Abstract

The present application relates to a privacy protection method, apparatus, device and storage medium for input data. The method comprises: obtaining initial input data, converting the initial input data through a prompt embedding matrix to obtain embedded input data; wherein the embedded input data comprises multiple input batches, each input batch comprises multiple input samples, and each input sample comprises multiple embedding units; determining a perturbation candidate unit from each input sample based on the importance of each embedding unit in each input sample; determining a perturbation similar unit set corresponding to the perturbation candidate unit based on the similarity between each embedding unit and the perturbation candidate unit, and calculating a perturbation replacement unit corresponding to the perturbation candidate unit based on the perturbation similar unit set; replacing the corresponding perturbation candidate unit based on the perturbation replacement unit to obtain the target input data of the replaced input batch. The method can effectively achieve data privacy protection.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, apparatus, device and storage medium for privacy protection of input data. Background Art

[0002] Currently, generative large language models (LLMs) have demonstrated remarkable capabilities by combining various natural language processing tasks into a comprehensive text generation framework. These models, such as OpenAI's GPT-4, Anthropic's Claude 2, Meta's Llama 2, etc., have had a significant impact on understanding and generating human language in recent years.

[0003] In addition to performance improvements, the size of language model input data has also increased with model size. These models are not only fed with task-specific annotated text data, but also consume vast amounts of publicly available text data from the internet. Unlike carefully curated annotated data, free-form text data extracted from the internet is of poor quality and prone to inadvertently leaking personal information. Research has shown that LLMs remember a significant portion of the input data, which can be extracted using appropriately crafted prompts. This extraction poses privacy risks for contributors of the input data. For example, simple interactions with the model can lead to the unintended dissemination of personally identifiable information (PII). To mitigate these privacy risks, existing techniques often directly replace private data in the original input data. However, direct reconstruction of the input data, coupled with the general training loss function of causal language modeling (LLM), can render the design unsuitable for more general scenarios.

[0004] Currently, no effective solution has been proposed for the problem of how to effectively protect the privacy of input data in large language models in existing technologies. Summary of the Invention

[0005] Based on this, it is necessary to provide a privacy protection method, device, equipment and storage medium for input data to address the above technical issues.

[0006] In a first aspect, the present application provides a method for protecting the privacy of input data. The method comprises:

[0007] Obtaining initial input data and inputting the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units;

[0008] Determine the perturbation candidate units from each embedding unit in each input sample according to the importance of each embedding unit in each input sample;

[0009] Based on the similarity between each embedded unit and the perturbation candidate unit, a perturbation similar unit set corresponding to the perturbation candidate unit is determined, and a perturbation replacement unit corresponding to the perturbation candidate unit is calculated based on the perturbation similar unit set;

[0010] The disturbance replacement unit replaces the corresponding disturbance candidate unit to obtain target input data of each input batch after replacement.

[0011] In one embodiment, determining a perturbation candidate unit from each embedding unit in each input sample includes:

[0012] Obtaining a preset perturbation ratio corresponding to each input sample;

[0013] Taking one input sample as a unit, calculate the in-degree importance and out-degree importance of the embedding unit, and calculate the importance score corresponding to each embedding unit based on the in-degree importance and out-degree importance;

[0014] In the same input sample, the corresponding embedding units are sorted based on the importance score to obtain an important unit sequence, and the benchmark perturbation unit is determined in the important unit sequence according to the perturbation ratio;

[0015] In the same input sample, the embedding unit with an importance score smaller than the baseline perturbation unit is judged as a perturbation candidate unit.

[0016] In one embodiment, determining a perturbation candidate unit from each embedding unit in each input sample includes:

[0017] Get the preset effective length for each input sample;

[0018] Determine a valid mask in the corresponding input sample based on the valid length, and determine a valid unit in the input sample through the valid mask;

[0019] The perturbation ratio of the effective unit in each input sample is obtained, and the perturbation candidate unit in the effective unit is calculated according to the perturbation ratio and the importance score of the effective unit.

[0020] In one embodiment, determining a set of disturbance-similar units corresponding to a candidate disturbance unit includes:

[0021] Calculate the similarity between each embedding unit and other embedding units, and determine the first similar unit corresponding to each embedding unit according to the similarity calculation result;

[0022] Perform clustering calculations on all embedded units to obtain at least two cluster partitions, calculate the similarity between each embedded unit and other embedded units in the same cluster partition, and determine the second similar unit corresponding to each embedded unit in each cluster partition based on the similarity calculation results;

[0023] Determine a similar unit set corresponding to each embedded unit based on the first similar unit and the second similar unit;

[0024] The perturbation candidate unit is determined in the embedding unit, and the corresponding perturbation similar unit set is obtained based on the perturbation candidate unit.

[0025] In one embodiment, calculating the first similarity unit includes:

[0026] Obtain an embedding matrix tensor of the initial input data and perform normalization on the embedding matrix tensor; wherein the embedding matrix tensor includes an embedding vector of an embedding unit;

[0027] Perform matrix multiplication on the normalized embedding matrix tensor to obtain the cosine similarity index matrix, where the cosine similarity index matrix represents the similarity relationship between each embedding unit;

[0028] Determining a first similar unit corresponding to each embedding unit based on the first quantity and the cosine similarity index matrix;

[0029] Calculating the second similarity unit, including:

[0030] Perform clustering calculation on the embedding matrix tensor to obtain at least two cluster partitions, and classify and store the embedding units according to the cluster partitions;

[0031] The similarity between each embedding unit and other embedding units in the same cluster partition is calculated, and based on the second quantity and the similarity calculation result, a second similar unit corresponding to each embedding unit in each cluster partition is determined.

[0032] In one embodiment, calculating a disturbance replacement unit corresponding to a disturbance candidate unit based on a disturbance similar unit set includes:

[0033] Determine a preset context window, and determine the context representation of the perturbation candidate unit in the corresponding input sample based on the context window;

[0034] Calculate the cosine similarity score vector based on the context representation and the set of perturbation similar units corresponding to the perturbation candidate unit;

[0035] The cosine similarity score vector is converted into a similarity data distribution in the form of a probability distribution, and high cosine similarity sampling is performed based on the similarity data distribution to obtain the perturbation replacement unit corresponding to the perturbation candidate unit.

[0036] In one embodiment, the cosine similarity score vector is converted into a similarity data distribution in the form of a probability distribution, and high cosine similarity sampling is performed based on the similarity data distribution to obtain a perturbation replacement unit corresponding to the perturbation candidate unit, including:

[0037] A preset temperature parameter is obtained, and a perturbation replacement unit is determined based on the temperature parameter and the cosine similarity score vector.

[0038] In a second aspect, the present application also provides a privacy protection device. The device includes:

[0039] An acquisition module is configured to acquire initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units;

[0040] a calculation module, configured to determine a candidate perturbation unit from each embedding unit in each input sample according to the importance of each embedding unit in each input sample; determine a set of perturbation similar units corresponding to the candidate perturbation unit based on the similarity between each embedding unit and the candidate perturbation unit, and calculate a perturbation replacement unit corresponding to the candidate perturbation unit based on the set of perturbation similar units;

[0041] The generation module is used to replace the corresponding perturbation candidate unit based on the perturbation replacement unit to obtain the target input data of each input batch after replacement.

[0042] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0043] Obtaining initial input data and inputting the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units;

[0044] Determine the perturbation candidate units from each embedding unit in each input sample according to the importance of each embedding unit in each input sample;

[0045] Based on the similarity between each embedded unit and the perturbation candidate unit, a perturbation similar unit set corresponding to the perturbation candidate unit is determined, and a perturbation replacement unit corresponding to the perturbation candidate unit is calculated based on the perturbation similar unit set;

[0046] The disturbance replacement unit replaces the corresponding disturbance candidate unit to obtain target input data of each input batch after replacement.

[0047] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0048] Obtaining initial input data and inputting the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units;

[0049] Determine the perturbation candidate units from each embedding unit in each input sample according to the importance of each embedding unit in each input sample;

[0050] Based on the similarity between each embedded unit and the perturbation candidate unit, a perturbation similar unit set corresponding to the perturbation candidate unit is determined, and a perturbation replacement unit corresponding to the perturbation candidate unit is calculated based on the perturbation similar unit set;

[0051] The disturbance replacement unit replaces the corresponding disturbance candidate unit to obtain target input data of each input batch after replacement.

[0052] The above-mentioned input data privacy protection method, device, equipment and storage medium obtains initial input data and inputs the initial input data into a pre-trained large language model to obtain embedded input data. Then, based on the importance of each embedding unit in each input sample, a perturbation candidate unit is determined from each embedding unit in each input sample. Based on the similarity between each embedding unit and the perturbation candidate unit, a perturbation similar unit set corresponding to the perturbation candidate unit is determined, and a perturbation replacement unit corresponding to the perturbation candidate unit is calculated based on the perturbation similar unit set; finally, the perturbation candidate unit is replaced based on the perturbation replacement unit to obtain the target input data of each input batch after replacement. Through this application, the privacy data in the input data can be effectively replaced, and the personal privacy security can be effectively guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A diagram of an application environment of a privacy protection method in one embodiment;

[0054] Figure 2 1 is a flowchart of a privacy protection method according to an embodiment;

[0055] Figure 3 is a structural block diagram of a privacy protection device in one embodiment;

[0056] Figure 4 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0058] The privacy protection method for input data provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. First, the initial input data is obtained, and the initial input data is input into the pre-trained large language model to obtain embedded input data, and the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units; then, according to the importance of each embedding unit in each input sample, the perturbation candidate unit is determined from each embedding unit in each input sample, and based on the similarity between each embedding unit and the perturbation candidate unit, the perturbation similar unit set corresponding to the perturbation candidate unit is determined, and the perturbation replacement unit corresponding to the perturbation candidate unit is calculated according to the perturbation similar unit set; finally, the corresponding perturbation candidate unit is replaced based on the perturbation replacement unit to obtain the target input data of each input batch after replacement. Terminal 102 may include, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers.

[0059] In one embodiment, Figure 2 As shown, a privacy protection method for input data is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0060] Step S210: Obtain initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units.

[0061] Specifically, initial input data is obtained. In practical applications, after shielding the private information therein, a pre-trained large language model can be trained based on the processed initial input data. This application provides a method for replacing the private information therein. The initial input data generally includes multiple input batches, each of which contains B batch-sized input sample sequences.

[0062] Similarly, the embedded input data includes multiple input batches, the input batches include multiple input samples, and each input sample includes multiple embedding unit tokens, wherein the input batches of the initial input data correspond one-to-one to the input batches of the embedded input data, and the input samples of the initial input data correspond one-to-one to the input samples of the embedded input data. Furthermore, the above-mentioned embedding unit token is an element included in the embedded input data. In practical applications, the corresponding embedded input data can be obtained by obtaining the token vector space information stored in the input-embedding layer of the model. In essence, it is the original word embedding matrix of the model, and each row represents the embedding vector of a token in the vocabulary, which stores the representation of the corresponding token in the model vector space.

[0063] In step S220 , a candidate perturbation unit is determined from the embedding units in each input sample according to the importance of each embedding unit in each input sample.

[0064] The importance of each embedding unit is calculated for each input sample. The importance of an embedding unit represents the degree of attention paid to it by other embedding units, as well as the degree of attention paid by this embedding unit to other embedding units. In practical applications, the multi-head attention weight matrix A∈R can be obtained from the last layer of the pre-trained large language model. B ×H×S×S , where B is the batch size, H is the number of attention heads, and S is the sequence length, i.e., the length of the input sample. To improve computational efficiency, only the attention weights of the last layer can be extracted because this layer contains the richest semantic information. The shape of the attention weight matrix is ​​[batch_size, num_heads, seq_len, seq_len], which records the degree of attention of each embedding unit token to all other tokens in the input sample. It can be understood that if an embedding unit token pays attention to a large number of other tokens, and / or an embedding unit token is paid attention to by a large number of other tokens, it means that the embedding unit token has a higher importance. In summary, the importance of each embedding unit can be calculated through the multi-head attention weight matrix.

[0065] After obtaining the importance of each embedding unit, a perturbation candidate unit can be determined from each input sample, wherein the perturbation candidate unit is an embedding unit that needs to be perturbed and the data needs to be replaced. In this embodiment, it is preferred to replace the embedding units with low importance, because the tokens at high importance positions have a greater impact on the training effect of the subsequent large language model. Therefore, in this embodiment, only the embedding units with low importance are replaced or modified, and the embedding units at high importance positions remain unchanged. In summary, the perturbation candidate unit can be determined in each input sample based on the importance of the embedding unit. In some embodiments, the embedding units whose importance is lower than the preset importance threshold can be used as perturbation candidate units.

[0066] Step S230 : Based on the similarity between each embedding unit and the perturbation candidate unit, a perturbation similar unit set corresponding to the perturbation candidate unit is determined, and a perturbation replacement unit corresponding to the perturbation candidate unit is calculated based on the perturbation similar unit set.

[0067] Specifically, the similarity between the candidate perturbation unit and all remaining embedded units is calculated, where the similarity can preferably be cosine similarity. A set of perturbation-similar units corresponding to the candidate perturbation unit is then determined. This perturbation-similar unit set is a set of embedded units that are relatively similar to the candidate perturbation unit. In practical applications, this includes, but is not limited to, adding embedded units with a similarity greater than a preset threshold to the perturbation-similar unit set, or selecting the top-k embedded units most similar to the candidate perturbation unit as the perturbation-similar unit set.

[0068] After obtaining the perturbation-similar unit set, a calculation is performed based on the embedded units in the perturbation-similar unit set to obtain a perturbation replacement unit corresponding to the candidate perturbation unit. Specific calculation methods include, but are not limited to, calculating the mean between the embedded units in the perturbation-similar unit set to calculate another perturbation replacement unit that is similar to, but not identical to, the candidate perturbation unit.

[0069] In some preferred embodiments, pre-calculation may be performed first, that is, a similar unit set corresponding to each embedding unit is calculated first, and the similar unit set is saved. Then, after the perturbation candidate unit is determined, the corresponding perturbation similar unit set is extracted based on the perturbation candidate unit.

[0070] Step S240 : performing a replacement process on the corresponding candidate disturbance units based on the disturbance replacement units to obtain target input data of each input batch after replacement.

[0071] Specifically, the perturbation replacement unit replaces the corresponding candidate perturbation unit, thereby obtaining the replaced target input data. In practical applications, the obtained target input data can be used to train the large language model, namely performing subsequent training steps such as backpropagation loss calculation, thereby effectively avoiding the potential privacy leakage issues that may occur during the training process.

[0072] In summary, after the perturbation candidate unit is replaced with the perturbation replacement unit, the replaced input data can be input as training data into the pre-trained large language model to perform gradient training parameter updates.

[0073] Through steps S210 to S240, privacy data that may exist in the input data can be protected, effectively preventing the leakage of personal information. In addition, this application can complete data protection at a lower disturbance rate, and effectively takes into account both privacy protection and performance utility during subsequent training of large language models.

[0074] In some embodiments, determining a perturbation candidate unit from each embedding unit in each input sample includes:

[0075] Obtaining a preset perturbation ratio corresponding to each input sample;

[0076] Taking one input sample as a unit, calculate the in-degree importance and out-degree importance of the embedding units included in an input batch, and calculate the importance score corresponding to each embedding unit based on the in-degree importance and out-degree importance;

[0077] In the same input sample, the corresponding embedding units are sorted based on the importance score to obtain an important unit sequence, and the benchmark perturbation unit is determined in the important unit sequence according to the perturbation ratio;

[0078] In the same input sample, the embedding unit with an importance score smaller than the baseline perturbation unit is judged as a perturbation candidate unit.

[0079] Specifically, this embodiment provides a method for determining the perturbation candidate units in each input sample. First, the above-mentioned multi-head attention weight matrix is ​​obtained (it can be understood that according to the above, the multi-head attention matrix includes the degree of attention of each embedding unit token to other tokens in the input sample). Furthermore, in practical applications, the input data in this application is training data for a large language model. Therefore, after the input data is input into the pre-trained language model, the above-mentioned multi-head attention weight matrix can be obtained, and wherein, in this embodiment, the large language model includes a frozen model original parameter matrix and a soft prompt embedding matrix, which is integrated before the model input embedding layer for soft prompt tuning Soft-Prompt-Tunning. Calculate the in-degree importance of each embedding unit, that is, the degree to which the current token is paid attention to by other tokens, and obtain it by averaging the attention matrix in the head dimension and the source token dimension:

[0080] ;

[0081] Among them, h is the number of attention, A h,j,i It refers to the sum of the attention weights of token i to all attention heads h and all other token j in a sample sequence, t i It refers to the in-degree importance of token i.

[0082] In-degree importance reflects the amount of information a token receives in the entire input sample. A high in-degree value indicates that the token is followed by a large number of other tokens.

[0083] And calculate the out-degree importance of each token, that is, calculate the degree of attention of the current token to other tokens in a training sample, and obtain it by averaging the attention matrix in the head dimension and the target token dimension:

[0084] ;

[0085] Among them, out-degree importance reflects the amount of information divergence of a token in the training sample. A high out-degree value indicates that the token pays attention to a large number of other tokens.

[0086] Then the importance score of the embedded unit is calculated by combining the out-degree importance and the in-degree importance, specifically:

[0087] ;

[0088] The in-degree weight α and the out-degree weight β can be set in advance by relevant technical personnel.

[0089] In some preferred embodiments, to facilitate subsequent calculations, the importance scores may be normalized to ensure that all scores are within the range of [0, 1]:

[0090] ;

[0091] Among them, t i refers to the i-th token (embedding unit) in the input data, min j I(t j ) refers to the minimum importance score of all tokens in the current input batch, max j I(t j ) refers to the maximum importance score of all tokens in the current input batch, and ε refers to the privacy budget.

[0092] Obtain the perturbation ratio γ corresponding to each input sample. In practical applications, the perturbation ratio of the perturbation token to the input sample in a specific scenario can be determined based on different task types, different training dataset styles, different training dataset sizes, different types of large language models (the trend change of the training data memorized by the large language model mainly depends on the size of the model parameters), etc. (for example, the perturbation ratio of models with less than 3B parameters can be controlled within 10%, and the perturbation ratio of models with more than 10B~100B can be controlled within 15%). The perturbation ratio γ can be set in advance by relevant technical personnel.

[0093] Then, after obtaining the perturbation ratio of each input sample and the importance score corresponding to each embedding unit, each input sample is sorted according to the importance score of the embedding unit token, obtaining the above-mentioned importance unit sequence, and selecting the (s·γ)th value as the threshold. The embedding unit corresponding to the (s·γ)th value is the above-mentioned baseline perturbation unit, where s is the number of embedding unit tokens in the input sample. Finally, in the same input sample, the embedding units with an importance score smaller than the baseline perturbation unit are selected as the perturbation candidate units that need to be perturbed.

[0094] In practical applications, after determining the perturbation candidate unit, a noise mask M can be created noise , identifies the location of the candidate perturbation unit that needs to be added with noise, that is, the location of the candidate perturbation unit that needs to be perturbed, so as to facilitate the replacement of the candidate perturbation unit in the subsequent steps.

[0095] Through this application, the perturbation ratio can be determined according to actual needs, and the number of perturbation candidate units required in an input sample can be flexibly determined based on the set perturbation ratio, thereby flexibly adapting to various different scenarios.

[0096] In some embodiments, determining a perturbation candidate unit from each embedding unit in each input sample includes:

[0097] Get the preset effective length for each input sample;

[0098] Determine a valid mask in the corresponding input sample based on the valid length, and determine a valid unit in the input sample through the valid mask;

[0099] The perturbation ratio of the effective unit in each input sample is obtained, and the perturbation candidate unit in the effective unit is calculated according to the perturbation ratio and the importance score of the effective unit.

[0100] Specifically, a preset valid length L is obtained for each input sample. The valid length L is the length of the embedding unit considered valid in the input sample. It can be understood that an invalid embedding unit is the location of the embedding unit token in the input sample that is marked as not requiring subsequent loss calculation. A corresponding validity mask is then created based on the valid length, and the valid tokens, or valid units, of the input sample can be determined based on the validity mask.

[0101] In some preferred embodiments, a method for creating a valid mask is provided, comprising: calculating the loss only for the suffix sequence (i.e., the suffix part of the input sample), marking the prefix sequence as invalid and excluding it from the interference candidate consideration, and creating a valid token mask M. valid =(L≠no_valid) (i.e. valid token mask M valid (i.e., the position where the label is not equal to the invalid mark no_valid / the position that is not marked as invalid by the mark "no_valid"), where L represents the token label or ID of each position, that is, the identification information of each token. This is because in practical applications, common data extraction attack scenarios include scenarios where only the suffix content is extracted, without involving the sequence prefix. Therefore, in this embodiment, the prefix sequence is marked invalid, that is, the prefix sequence is not included in the scope of consideration of the perturbation candidate unit, and only the suffix sequence of the input sample is taken as a valid unit, and the perturbation candidate unit is determined in the valid unit. In summary, the setting of the valid unit in this embodiment can also be determined by relevant technical personnel based on actual conditions to choose which part of the token needs to be invalidated, thereby achieving selective perturbation of specific tokens and providing more refined control capabilities for the privacy protection of large language models.

[0102] It is understandable that after determining the valid units of the input sample, the method for determining the candidate perturbation units in the valid units is the same as the method described above, that is, determining the perturbation ratio of the valid units of the input sample and calculating the importance score of the valid units. Then, the valid units are sorted according to the importance score to obtain the importance unit sequence of the valid units, and the baseline perturbation unit is determined in the importance unit sequence of the valid units according to the perturbation ratio. The valid units with an importance score less than the baseline perturbation unit are judged as candidate perturbation units. Through this embodiment, the valid units in the input sample can be determined first, and then the candidate perturbation units can be determined in the valid units, thereby ensuring the calculation effect while reducing the computational burden during subsequent model training.

[0103] In some embodiments, determining a set of disturbance-similar units corresponding to a candidate disturbance unit includes:

[0104] Calculate the similarity between each embedding unit and other embedding units, and determine the first similar unit corresponding to each embedding unit according to the similarity calculation result;

[0105] Perform clustering calculations on all embedded units to obtain at least two cluster partitions, calculate the similarity between each embedded unit and other embedded units in the same cluster partition, and determine the second similar unit corresponding to each embedded unit in each cluster partition based on the similarity calculation results;

[0106] Determine a similar unit set corresponding to each embedded unit based on the first similar unit and the second similar unit;

[0107] The perturbation candidate unit is determined in the embedding unit, and the corresponding perturbation similar unit set is obtained based on the perturbation candidate unit.

[0108] Specifically, in practical applications, the word embedding matrix E∈R can be extracted from the pre-trained language model V×D , where V is the vocabulary size and D is the embedding dimension. The pre-trained language model is the large language model described above that needs to be trained based on the input data described above in this application. The word embedding matrix obtained carries the mapping of each embedding unit token in the input data input to the pre-trained language model to a low-dimensional vector.

[0109] Each word vector in the word embedding matrix can be L2 normalized (L2 Norm Normalization) to facilitate the subsequent calculation of cosine similarity. The normalization calculation is as follows:

[0110] ;

[0111] The normalized embedding similarity matrix is ​​calculated as ;

[0112] Then calculate the similarity between each embedded unit i and other embedded units, and select the top k most similar similar units. The top k most similar similar units are the above-mentioned first similar units. k can be set by relevant technical personnel according to actual needs. Record the index of the first similar unit corresponding to the current embedded unit. Through the above method, all embedded units and their corresponding first similar units can be calculated and recorded, and then generate a mapping M based on the embedded first similar units. emb :

[0113] ;

[0114] Among them, S i is the similarity score of all embedding unit tokens in the row where the i-th embedding unit token is located in the cosine similarity matrix. It can be understood that the row where the i-th embedding unit token is located stores the cosine similarity between the i-th embedding unit token and all other embedding unit tokens in the vocabulary. It can be understood that by calculating the method of each embedding unit and its corresponding first similar unit in this embodiment, it is possible to ensure the similarity query of embedding unit tokens that are close to each other in the basic functionalism-oriented model word embedding vector space. The purpose of its design is to maintain the rationality of the basic vector space, ensure that the replacement tokens generated based on the similar units are close to the original embedding vector (original token) in the word vector space, maintain the local grammatical structure, and maintain syntactic correctness through word form similarity, maintain the perplexity (PPL) of the language model, control the replacement word frequency distribution to be close to the original distribution, and ensure the minimum support role.

[0115] Then, the embedding space is partitioned based on the K-means clustering algorithm, that is, clustering calculation is performed to obtain N clusters Clustering partition results (also known as semantic clusters):

[0116] ;

[0117] Furthermore, C i Represents the i-th embedding unit token i The partition ID to which it belongs is the cluster ID. The formula C represents the calculation formula for all cluster IDs. E is the above word embedding matrix.

[0118] Generate similar embedding units based on cluster partitioning for each embedding unit, that is, a mapping M of similar tokens semThat is, after clustering each embedding unit token into multiple cluster partitions, the similarity between each embedding unit and other embedding units in the same cluster partition is calculated, and the top k most similar embedding units are used as the second similar units of the embedding unit. This method can be used to calculate each embedding unit and its corresponding second similar unit in each cluster partition. At this time, we have:

[0119] M sem (i)={j |C_ j = C_i , j≠i};

[0120] Among them, i represents the token index of the current embedding unit currently being calculated, which is a specific tokenID in the vocabulary, j is the token index of another embedding unit in the same cluster partition as i, and the value range of j is all integers from 0 to the vocabulary size V-1. As an element in the set, j needs to meet the following conditions, C_i is the cluster ID to which the above token i belongs, which is the cluster identifier assigned to token i after the word embedding space is divided by the K-means clustering algorithm, and C_j is similar, indicating the cluster ID to which token j belongs. In summary, the meaning of the above formula is that token i and token j belong to the same semantic cluster, and token j is not token i itself. In summary, these token j that meet the requirements constitute a token set M that is semantically similar to token i. sem (i).

[0121] In summary, semantic cluster-based similarity mapping completes the candidate search based on semantic vector space similarity. Its role is to break through the limitations of simple proximity semantic vector space similarity and find candidate tokens with potential semantic category similarity based on the context. This adapts to the search for similar token candidates when the same token is transferred to different contexts. For example, based on the basic candidate similar token sequence for "apple-fruit" in the semantic vector space, it breaks through to the search for candidate tokens for "apple-phone" based on the contextual semantic category association.

[0122] In summary, the above method can be used to calculate the similar unit set corresponding to any embedding unit, namely the first similar unit and the second similar unit. Furthermore, in some preferred embodiments, if the number of tokens in the similar unit set corresponding to the embedding unit is too large, the intersection can be further sorted to retain the most similar k1 tokens, where k1 can be set by relevant technicians:

[0123] ;

[0124] At this time, M combinedThis is the set of similar units mentioned above.

[0125] In practical applications, the calculation method of the similar units is pre-calculated. Therefore, when it is necessary to determine the set of disturbance similar units corresponding to the disturbance unit, the set of disturbance similar units corresponding to the disturbance unit can be determined from the pre-calculated set.

[0126] In some embodiments, calculating the first similarity unit includes:

[0127] Obtain an embedding matrix tensor of the initial input data and perform normalization on the embedding matrix tensor; wherein the embedding matrix tensor includes an embedding vector of an embedding unit;

[0128] Perform matrix multiplication on the normalized embedding matrix tensor to obtain the corresponding cosine similarity index matrix, where the cosine similarity index matrix represents the similarity relationship between each embedding unit;

[0129] Determine the first similar unit corresponding to each embedding unit based on the cosine similarity index matrix;

[0130] Calculating the second similarity unit, including:

[0131] Perform clustering calculation on the embedding matrix tensor to obtain at least two cluster partitions, and classify and store the embedding units according to the cluster partitions;

[0132] The similarity between each embedding unit and other embedding units in the same cluster partition is calculated, and the similarity calculation results are used to determine the second similar unit corresponding to each embedding unit in each cluster partition.

[0133] Specifically, this embodiment provides a method for calculating a first similarity unit and a second similarity unit. Calculating the first similarity unit includes:

[0134] Obtain the aforementioned word embedding matrix, process the embedding layer encapsulation of the language pre-trained model, and obtain the model's original embedding matrix tensor. Normalize the original embedding layer one by one to obtain the unit embedding matrix. Then, perform matrix multiplication on the unit embedding matrix to obtain the aforementioned cosine similarity index matrix. Matrix multiplication on the unit embedding matrix refers to calculating the dot product of the normalized word embedding matrix and its transpose to form the aforementioned similarity matrix. This calculation method allows the similarity between all tokens to be calculated simultaneously through a single matrix multiplication, which is much more efficient than pairwise calculation. The cosine similarity index matrix represents the similarity relationship between each embedding unit. Then, based on the cosine similarity index matrix, each embedding unit and its corresponding k most similar other embedding units are determined and stored, so as to obtain each embedding unit and its corresponding first similar unit. In practical applications, the embedding unit (that is, a specific token) and its corresponding first similar unit can be converted into a dictionary structure: {token_id:[similar_ids]}, where token_id is the unit index corresponding to the embedding unit, and similar_ids is the unit index corresponding to each first similar unit corresponding to the embedding unit.

[0135] Furthermore, this embodiment also provides a specific method for calculating the second similar unit, including:

[0136] Using MiniBatch K-means clustering, each original word embedding matrix is ​​clustered into multiple cluster partitions. Each embedding unit is traversed, grouped and stored by cluster, and the corresponding token index is stored separately, resulting in a dictionary structure {cluster_id: [token_ids]}, where cluster_id refers to the id index of the cluster partition and token_ids is the index id of the embedding unit stored in that cluster partition. The similarity between each embedding unit and other embedding units in the same cluster partition is calculated and sorted in descending order of similarity. The k most similar embedding units are selected to obtain the aforementioned second similar unit. At this point, {token_id: [similar_ids} is the unit index corresponding to the embedding unit, and similar_ids is the unit index corresponding to each second similar unit corresponding to the embedding unit.

[0137] In practical applications, the first similar unit and the second similar unit corresponding to each embedded unit can be calculated first, and the results can be stored. After the perturbation candidate unit is determined, the first similar unit and the second similar unit corresponding to the perturbation candidate unit can be directly called to obtain the perturbation similar unit set corresponding to the perturbation candidate unit. At this time, the first similar unit of the perturbation candidate unit is obtained from the pre-calculated result: C emb =Memb (L i,j ), where M emb (·) is a mapping from tokenID (i.e., the perturbation candidate unit currently being calculated) to a list of similar tokenIDs (i.e., other embedding units similar to the perturbation candidate unit), L i,j is the tokenID of the jth position in the i-th input batch, that is, the original perturbation candidate unit that needs to be perturbed, C emb is the first similar unit candidate obtained by similarity search; similarly, the second similar units of the perturbation candidate unit obtained from the pre-calculated results are: C sem =M sem (L i,j ), where M sem is a mapping from tokenID to similar tokenID list based on semantic clustering, L i,j Same as above, it is the tokenID of the jth position in the i-th input batch, C sem is the set of candidate tokens obtained by semantic clustering similarity search, that is, the second similarity unit mentioned above. These tokens belong to the same semantic category as the original perturbation candidate tokens.

[0138] In some preferred embodiments, the intersection of the first similar unit and the second similar unit corresponding to the perturbation candidate unit can be used as the perturbation similar unit set corresponding to the perturbation candidate unit. In this case,

[0139] C combined =(C emb ∩C sem )∪C emb '∪C sem ';

[0140] Among them, C emb ' indicates C emb C sem The set of elements remaining after the intersection of sem 'Similarly, C sem C emb The set of elements remaining after the intersection of .

[0141] In some embodiments, calculating a disturbance replacement unit corresponding to a candidate disturbance unit based on a set of disturbance similar units includes:

[0142] Determine a preset context window, and determine the context representation of the perturbation candidate unit in the corresponding input sample based on the context window;

[0143] Calculate the cosine similarity score vector based on the context representation and the set of perturbation similar units corresponding to the perturbation candidate unit;

[0144] The cosine similarity score vector is converted into a similarity data distribution in the form of a probability distribution, and high cosine similarity sampling is performed based on the similarity data distribution to obtain the perturbation replacement unit corresponding to the perturbation candidate unit.

[0145] First, determine the context window N(j) for a perturbation candidate unit token j in the corresponding input sample. This context window can be preset to the distance of four embedded unit tokens. If the perturbation candidate unit token j is at position i in the input sample, its corresponding context window should be the i-2th token position, the i-1th token position, the i-th token (i.e., the perturbation candidate unit token j itself), the i+1th token position, and the i+2th token position. In practical applications, the size of the context window can be adjusted according to actual application requirements. By calculating and learning the context representation, it can be ensured that the final calculated perturbation replacement unit and the perturbation candidate unit still maintain the same or similar basic data structure.

[0146] The computation context can be expressed as:

[0147] ;

[0148] Among them, ctx j It refers to the context embedding representation of token j, which is the average of all token embedding vectors in the context window of token j. N(j) represents the context window of token j, and |N(j)| represents the number of tokens in the context window. is the embedding vector of the i-th token in the context window, represents the sum of the embedding vectors of all tokens in the context window, That is, it is the average of the sum of the embedding vectors of all tokens in the context window.

[0149] Then calculate the perturbation similarity unit set corresponding to the perturbation candidate unit token j, and determine each perturbation similarity unit token c (token c∈C combined ), and the cosine similarity score vector between the context representation of the perturbed candidate unit:

[0150] ;

[0151] Among them, the cosine similarity score vector sim cDenotes the cosine similarity between the embedding vector of the candidate token c and the current context embedding representation of the corresponding perturbation candidate unit token j, E c Refers to the embedding vector of the perturbed similar unit token c, obtained from the embedding layer of the pre-trained model.

[0152] The cosine similarity score vector is converted into a probability distribution using the softmax function to obtain the similarity data distribution. Based on the similarity data distribution, high cosine similarity sampling is performed to calculate the perturbation replacement unit. At this time, we have:

[0153] ;

[0154] Among them, p c is the probability value of a similar unit in the perturbation similar unit set. The sum of the probabilities of all perturbation similar units corresponding to a perturbation candidate unit tokenj is 1. Each perturbation similar unit has a corresponding p c Value, P c It refers to the probability that the corresponding perturbation similar unit is selected as the perturbation replacement unit, that is, the perturbation similar unit set includes the perturbation replacement unit, sim c’ To represent the cosine similarity between each perturbed similarity unit token c' and the contextual embedding representation of the perturbed candidate unit token j when traversing the tokens in the perturbed similarity unit set, c' is a variable that takes different values ​​when traversing the entire similarity set. T is a temperature parameter. This embodiment introduces the temperature parameter to prevent the computational behavior of the perturbation replacement unit from approaching argmax selection, thus avoiding overly consistent replacement of tokens with similar meanings. In other words, it prevents the perturbation replacement unit from being too similar to the perturbation candidate unit, thereby ensuring a certain balance between privacy and utility.

[0155] In one embodiment, the cosine similarity score vector is converted into a similarity data distribution in the form of a probability distribution, and high cosine similarity sampling is performed based on the similarity data distribution to obtain a perturbation replacement unit corresponding to the perturbation candidate unit, including:

[0156] A preset temperature parameter is obtained, and a perturbation replacement unit is determined based on the temperature parameter and the cosine similarity score vector.

[0157] Specifically, this embodiment provides a method for calculating similarity data distribution, which includes:

[0158] ;

[0159] Among them, p cis the similarity data distribution corresponding to a perturbation similarity unit. The similarity data distribution is a probability vector, which represents the probability of each candidate token c being selected as a replacement token (i.e., perturbation replacement unit) under a given context. From an implementation perspective, p c is the output of the softmax function after applying the normalized similarity score, and c' is the sum of the similarity scores from the combined candidate set C combined The element taken out is used as the index variable in the summation process, sim c’ The cosine similarity between the context of each candidate token c' and token j is calculated, and T is the temperature parameter. The perturbed similarity unit corresponding to the target similarity data distribution with the highest value among all similarity data distributions is then used as the perturbation replacement unit. This embodiment introduces the temperature parameter to prevent the computational behavior of the perturbation replacement unit from approaching argmax selection, thus avoiding overly consistent replacement of tokens with similar meanings. This prevents the perturbation replacement unit from being too similar to the candidate perturbation unit, thus ensuring a certain balance between privacy and utility.

[0160] The present application also provides a preferred embodiment of a method for protecting privacy of input data.

[0161] First, get the initial input data, which usually contains some personal sensitive information, such as: "My credit card number is , valid until 05 / 24" (it can be understood that this example is an input sample in the input data).

[0162] Then use the multi-head attention mechanism to calculate the importance score of each embedding unit token in the input sample, and then we have:

[0163] Position 1: "My credit card number is" has an average importance score of 0.78 (high importance).

[0164] position2: card number" The average importance score of " is 0.21 (low importance);

[0165] Position 3: "Valid until 05 / 24" has an average importance score of 0.65 (medium-high importance).

[0166] The importance threshold is further determined. The importance threshold can be determined based on a preset perturbation ratio and the importance scores corresponding to each embedded unit. The importance threshold can also be preset in advance. For example, in this preferred embodiment, the importance threshold can be set to 0.4. At this time, the perturbation candidate unit in the input sample that needs to be perturbed is identified (i.e., the above-mentioned position 2).

[0167] Then, a set of perturbation-similar units similar to the perturbation candidate unit can be determined from the input data, wherein the perturbation candidate unit and the set of perturbation-similar units corresponding to the perturbation candidate unit are pre-calculated, for example:

[0168] For "4532", find similar digital format tokens such as "7891", "3256", etc.

[0169] For "7891", find similar numeric format tokens such as "4532", "6745", and so on.

[0170] Based on the context window, the context information corresponding to the perturbation candidate unit is obtained: "credit card number is" to ensure that the selected replacement token still maintains the digital format and basic structure of the credit card number (16 digits, divided into 4 groups).

[0171] Based on the extracted perturbation similar unit set and the above context information, the perturbation replacement unit corresponding to the perturbation candidate unit is calculated: ", replace the original perturbation candidate unit with the perturbation replacement unit, and you can get the replaced target input data: "My credit card number is , valid until 05 / 24".

[0172] The pre-trained language model is then trained using the replaced target input data. Consequently, the language model only learns the concept and format of a "credit card number" and is unable to capture or memorize the actual card number sequence. This application effectively protects privacy, and training the language model using the adjusted training data does not significantly alter the semantic interpretation capabilities of the language model (LLM).

[0173] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0174] Based on the same inventive concept, embodiments of the present application also provide a privacy protection device for implementing the aforementioned privacy protection method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more privacy protection device embodiments provided below can be found in the limitations of the privacy protection method above and will not be repeated here.

[0175] In one embodiment, Figure 3 As shown, a privacy protection device is provided, including: an acquisition module 31, a calculation module 32 and a generation module 33, wherein:

[0176] An acquisition module 31 is configured to acquire initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units;

[0177] A calculation module 32 is configured to determine a candidate perturbation unit from each embedding unit in each input sample based on the importance of each embedding unit in each input sample; determine a set of perturbation similar units corresponding to the candidate perturbation unit based on the similarity between each embedding unit and the candidate perturbation unit, and calculate a perturbation replacement unit corresponding to the candidate perturbation unit based on the set of perturbation similar units;

[0178] The generating module 33 is configured to replace the corresponding candidate disturbance units based on the disturbance replacement units to obtain target input data of each input batch after replacement.

[0179] Each module in the above-mentioned privacy protection device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0180] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store privacy protection-related computing data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, it implements a privacy protection method.

[0181] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0182] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0183] Obtaining initial input data and inputting the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units;

[0184] Determine the perturbation candidate units from each embedding unit in each input sample according to the importance of each embedding unit in each input sample;

[0185] Based on the similarity between each embedded unit and the perturbation candidate unit, a perturbation similar unit set corresponding to the perturbation candidate unit is determined, and a perturbation replacement unit corresponding to the perturbation candidate unit is calculated based on the perturbation similar unit set;

[0186] Based on the disturbance replacement unit, the corresponding disturbance candidate unit is replaced to obtain the target input data of each input batch after replacement.

[0187] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0188] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processors (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0189] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A privacy protection method for input data, characterized in that: The method comprises: Obtaining initial input data, and inputting the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes multiple input batches, each of the input batches includes multiple input samples, and each of the input samples includes multiple embedding units; Determining a perturbation candidate unit from each of the embedding units in each of the input samples according to the importance of each of the embedding units in each of the input samples; Based on the similarity between each of the embedding units and the perturbation candidate unit, a perturbation similar unit set corresponding to the perturbation candidate unit is determined, and a perturbation replacement unit corresponding to the perturbation candidate unit is calculated according to the perturbation similar unit set; The corresponding candidate disturbance units are replaced based on the disturbance replacement units to obtain target input data of each input batch after replacement.

2. The method according to claim 1, characterized in that The determining of a candidate perturbation unit from each of the embedding units in each of the input samples comprises: Obtaining a preset disturbance ratio corresponding to each of the input samples; Taking one input sample as a unit, calculating the in-degree importance and out-degree importance of each embedding unit, and obtaining the importance score corresponding to each embedding unit according to the in-degree importance and out-degree importance; In the same input sample, the corresponding embedding units are sorted based on the importance scores to obtain an importance unit sequence, and a reference perturbation unit is determined in the importance unit sequence according to the perturbation ratio; In the same input sample, the embedding unit having an importance score smaller than that of the reference perturbation unit is determined as the candidate perturbation unit.

3. The method according to claim 1, characterized in that The determining of a candidate perturbation unit from each of the embedding units in each of the input samples comprises: Obtaining a preset effective length for each of the input samples; Determining a valid mask in the corresponding input sample based on the valid length, and determining a valid unit in the input sample through the valid mask; The perturbation ratio of each valid unit in the input sample is obtained, and the perturbation candidate unit in the valid unit is calculated according to the perturbation ratio and the importance score of the valid unit.

4. The method according to claim 1, wherein The determining a set of disturbance similar units corresponding to the candidate disturbance unit includes: Calculating the similarity between each embedding unit and other embedding units, and determining the first similar unit corresponding to each embedding unit according to the similarity calculation result; Performing clustering calculation on all the embedding units to obtain at least two cluster partitions, calculating the similarity between each embedding unit and other embedding units in the same cluster partition, and determining the second similar unit corresponding to each embedding unit in each cluster partition based on the similarity calculation result; Determine a similar unit set corresponding to each of the embedding units based on the first similar unit and the second similar unit; The perturbation candidate unit is determined in the embedding unit, and the corresponding perturbation similar unit set is obtained based on the perturbation candidate unit.

5. The method according to claim 4, characterized in that Calculating the first similarity unit includes: Obtaining an embedding matrix tensor of the initial input data and performing normalization processing on the embedding matrix tensor; wherein the embedding matrix tensor includes the embedding vector of the embedding unit; Performing matrix multiplication on the normalized embedding matrix tensor to obtain a corresponding cosine similarity index matrix, wherein the cosine similarity index matrix represents the similarity relationship between each of the embedding units; Determining the first similarity unit corresponding to each of the embedding units based on the cosine similarity index matrix; Calculating the second similarity unit includes: Performing cluster calculation on the embedding matrix tensor to obtain at least two cluster partitions, and classifying and storing the embedding units according to the cluster partitions; The similarity between each embedding unit and other embedding units in the same cluster partition is calculated, and based on the similarity calculation result, the second similar unit corresponding to each embedding unit in each cluster partition is determined.

6. The method according to claim 1, characterized in that The step of calculating, based on the set of disturbance similar units, a disturbance replacement unit corresponding to the candidate disturbance unit comprises: Determining a preset context window, and determining a context representation of the candidate perturbation unit in the corresponding input sample based on the context window; Calculating a cosine similarity score vector based on the context representation and a set of perturbation similar units corresponding to the perturbation candidate unit; The cosine similarity score vector is converted into a similarity data distribution in a probability distribution form, and high cosine similarity sampling is performed based on the similarity data distribution to obtain the perturbation replacement unit corresponding to the perturbation candidate unit.

7. The method according to claim 6, characterized in that The converting the cosine similarity score vector into a similarity data distribution in a probability distribution form, and performing high cosine similarity sampling based on the similarity data distribution to obtain the disturbance replacement unit corresponding to the disturbance candidate unit, comprises: A preset temperature parameter is obtained, and the disturbance replacement unit is determined based on the temperature parameter and the cosine similarity score vector.

8. A privacy protection device, characterized in that: The device comprises: an acquisition module, configured to acquire initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes a plurality of input batches, each of the input batches includes a plurality of input samples, and each of the input samples includes a plurality of embedding units; a calculation module, configured to determine, based on the importance of each embedding unit in each input sample, a candidate perturbation unit from each embedding unit in each input sample; determine, based on the similarity between each embedding unit and the candidate perturbation unit, a set of perturbation similar units corresponding to the candidate perturbation unit; and calculate, based on the set of perturbation similar units, a perturbation replacement unit corresponding to the candidate perturbation unit; A generating module is configured to perform a replacement process on the corresponding candidate disturbance unit based on the disturbance replacement unit to obtain target input data of each input batch after replacement.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images