Privacy protection method and device for input data, equipment and storage medium

Abstract

The invention relates to a privacy protection method and device for input data, equipment and a storage medium. Comprising the following steps: acquiring initial input data, and performing conversion processing on the initial input data by prompting an embedded matrix to obtain embedded input data; wherein the embedded input data comprises a plurality of input batches, each input batch comprises a plurality of input samples, and each input sample comprises a plurality of embedded units; determining a disturbance candidate unit from each input sample according to the importance of each embedded unit in each input sample; determining a disturbance similar unit set corresponding to the disturbance candidate unit based on the similarity between each embedding unit and the disturbance candidate unit, and calculating a disturbance replacement unit corresponding to the disturbance candidate unit according to the disturbance similar unit set; and performing replacement processing on the corresponding disturbance candidate unit based on the disturbance replacement unit to obtain target input data of the replaced input batch. By adopting the method, the privacy protection of the data can be effectively completed.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method, apparatus, device, and storage medium for privacy protection of input data. Background Art

[0002] Currently, generative large language models (LLMs) have demonstrated remarkable capabilities by combining various natural language processing tasks into a comprehensive text generation framework. These models, such as GPT-4 by OpenAI, Claude 2 by Anthropic, Llama 2 by Meta, etc., have had a significant impact on understanding and generating human language in recent years.

[0003] In addition to performance improvements, the scale of the input data for language models has also increased with the expansion of the model scale. These models not only take input on annotated text data for specific tasks but also consume a large amount of publicly available text data from the Internet. Different from carefully curated annotated data, the free-form text data extracted from the Internet has poor quality and is prone to inadvertently revealing personal information. Research shows that LLMs can remember a large portion of the input data, and this data can be extracted using appropriately crafted prompts, and such extraction poses a privacy risk to the contributors of the input data. For example, a simple interaction with the model may lead to the accidental dissemination of personally identifiable information (PII). Correspondingly, to reduce the above privacy risk, the prior art often directly replaces the privacy data in the original input data. However, directly reconstructing the input data may cause the design to be unable to adapt to more general scenarios for the general training loss function of causal language modeling LLMs.

[0004] Currently, in view of the problem in the prior art of how to effectively complete the privacy protection of input data in large language models, no effective solution has been proposed. Summary of the Invention

[0005] Based on this, in order to solve the above technical problems, it is necessary to provide a method, apparatus, device, and storage medium for privacy protection of input data.

[0006] In a first aspect, this application provides a method for privacy protection of input data. The method includes: Obtain initial input data, and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein, the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units; Determine perturbation candidate units from the embedding units in each input sample according to the importance of each embedding unit; Based on the similarity between each embedding unit and the perturbation candidate units, determine the set of perturbation similar units corresponding to the perturbation candidate units, and calculate the perturbation replacement units corresponding to the perturbation candidate units according to the set of perturbation similar units; Perform replacement processing on the perturbation candidate units corresponding to the perturbation replacement units to obtain the target input data of each input batch after replacement.

[0007] In one embodiment, determining perturbation candidate units from the embedding units in each input sample includes: Obtain the preset perturbation ratio corresponding to each input sample; Taking one input sample as a unit, calculate the in-degree importance and out-degree importance of the embedding units, and calculate the importance score corresponding to each embedding unit according to the in-degree importance and out-degree importance; In the same input sample, sort the corresponding embedding units based on the magnitude of the importance scores to obtain the importance unit sequence, and determine the reference perturbation unit in the importance unit sequence according to the perturbation ratio; In the same input sample, determine the embedding units with importance scores less than the reference perturbation unit as perturbation candidate units.

[0008] In one embodiment, determining perturbation candidate units from the embedding units in each input sample includes: Obtain the preset effective length for each input sample; Based on the effective length, determine the effective mask in the corresponding input sample, and determine the effective units in the input sample through the effective mask; Obtain the perturbation ratio of the effective units in each input sample, and calculate the perturbation candidate units in the effective units according to the perturbation ratio and the importance scores of the effective units.

[0009] In one embodiment, determining the set of perturbation similar units corresponding to the perturbation candidate units includes: Calculate the similarity between each embedding unit and other embedding units, and determine the first similar unit corresponding to each embedding unit according to the similarity calculation result; Perform clustering calculation on all the embedding units to obtain at least two clustering partitions, calculate the similarity between each embedding unit and other embedding units in the same clustering partition, and determine the second similar unit corresponding to each embedding unit in each clustering partition according to the similarity calculation result; Determine the set of similar units corresponding to each embedding unit based on the first similar unit and the second similar unit; Determine perturbation candidate units in the embedding units, and obtain a corresponding set of perturbation similar units based on the perturbation candidate units.

[0010] In one embodiment, calculating the first similar units includes: Obtain the embedding matrix tensor of the initial input data, and perform normalization processing on the embedding matrix tensor; wherein, the embedding matrix tensor includes the embedding vectors of the embedding units; Perform matrix multiplication on the normalized embedding matrix tensor to obtain a cosine similarity index matrix, where the cosine similarity index matrix characterizes the similarity relationship between each embedding unit; Based on the first quantity and the cosine similarity index matrix, determine the first similar units corresponding to each embedding unit; Calculate the second similar units, including: Perform clustering calculation on the embedding matrix tensor to obtain at least two clustering partitions, and classify and store the embedding units according to the clustering partitions; Calculate the similarity between each embedding unit and other embedding units in the same clustering partition, and based on the second quantity and the similarity calculation result, determine the second similar units corresponding to each embedding unit in each clustering partition.

[0011] In one embodiment, calculating the perturbation replacement unit corresponding to the perturbation candidate unit according to the set of perturbation similar units includes: Determine a preset context window, and determine the context representation of the perturbation candidate unit in the corresponding input sample based on the context window; Based on the context representation and the set of perturbation similar units corresponding to the perturbation candidate unit, calculate a cosine similarity score vector; Convert the cosine similarity score vector into a similarity data distribution in the form of a probability distribution, and perform high cosine similarity sampling based on the similarity data distribution to obtain the perturbation replacement unit corresponding to the perturbation candidate unit.

[0012] In one embodiment, converting the cosine similarity score vector into a similarity data distribution in the form of a probability distribution, and performing high cosine similarity sampling based on the similarity data distribution to obtain the perturbation replacement unit corresponding to the perturbation candidate unit includes: Obtain a preset temperature parameter, and determine the perturbation replacement unit based on the temperature parameter and the cosine similarity score vector.

[0013] In a second aspect, the present application also provides a privacy protection device. The device includes: An acquisition module, configured to acquire initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein, the embedded input data includes a plurality of input batches, each input batch includes a plurality of input samples, and each input sample includes a plurality of embedding units; A calculation module, configured to determine perturbation candidate units from the embedding units in each input sample according to the importance of each embedding unit in each input sample; determine a perturbation similarity unit set corresponding to the perturbation candidate units based on the similarity between each embedding unit and the perturbation candidate units, and calculate a perturbation replacement unit corresponding to the perturbation candidate units according to the perturbation similarity unit set; A generation module, configured to perform a replacement process on the corresponding perturbation candidate units based on the perturbation replacement units to obtain target input data for each input batch after replacement.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: Acquire initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein, the embedded input data includes a plurality of input batches, each input batch includes a plurality of input samples, and each input sample includes a plurality of embedding units; Determine perturbation candidate units from the embedding units in each input sample according to the importance of each embedding unit in each input sample; Determine a perturbation similarity unit set corresponding to the perturbation candidate units based on the similarity between each embedding unit and the perturbation candidate units, and calculate a perturbation replacement unit corresponding to the perturbation candidate units according to the perturbation similarity unit set; Perform a replacement process on the corresponding perturbation candidate units based on the described perturbation replacement units to obtain target input data for each input batch after replacement.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented: Acquire initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein, the embedded input data includes a plurality of input batches, each input batch includes a plurality of input samples, and each input sample includes a plurality of embedding units; Determine perturbation candidate units from the embedding units in each input sample according to the importance of each embedding unit in each input sample; Determine a set of perturbation similarity units corresponding to the perturbation candidate units based on the similarity between each embedding unit and the perturbation candidate units, and calculate a perturbation replacement unit corresponding to the perturbation candidate units according to the set of perturbation similarity units; Perform a replacement process on the perturbation candidate units corresponding to the perturbation replacement unit pair to obtain the target input data of each input batch after replacement.

[0016] The above privacy protection method, device, equipment and storage medium for input data obtain the initial input data, input the initial input data into a pre-trained large language model to obtain the embedded input data, and then determine the perturbation candidate units from each embedding unit in each input sample according to the importance of each embedding unit. Determine a set of perturbation similarity units corresponding to the perturbation candidate units based on the similarity between each embedding unit and the perturbation candidate units, and calculate a perturbation replacement unit corresponding to the perturbation candidate units according to the set of perturbation similarity units; finally, perform a replacement process on the perturbation candidate units based on the perturbation replacement units to obtain the target input data of each input batch after replacement. Through this application, the privacy data in the input data can be effectively replaced, effectively ensuring personal privacy security. Description of the Drawings

[0017] Figure 1 It is an application environment diagram of the privacy protection method in an embodiment; Figure 2 It is a schematic flowchart of the privacy protection method in an embodiment; Figure 3 It is a structural block diagram of the privacy protection device in an embodiment; Figure 4 It is an internal structure diagram of a computer device in an embodiment. Detailed Implementation Modes

[0018] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] The privacy protection method for input data provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or can be placed on the cloud or other network servers. First, obtain the initial input data, and input the initial input data into the pre-trained large language model to obtain the embedded input data, and the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units; then, according to the importance of each embedding unit in each input sample, determine the perturbation candidate units from each embedding unit in each input sample, and based on the similarity between each embedding unit and the perturbation candidate units, determine the perturbation similar unit set corresponding to the perturbation candidate units, and calculate the perturbation replacement unit corresponding to the perturbation candidate units according to the perturbation similar unit set; finally, perform replacement processing on the corresponding perturbation candidate units based on the perturbation replacement units to obtain the target input data of each input batch after replacement. Among them, the terminal 102 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0020] In one embodiment, as Figure 2 shown, a privacy protection method for input data is provided. Taking the server in Figure 1 as an example, the method includes the following steps: Step S210, obtain the initial input data, and input the initial input data into the pre-trained large language model to obtain the embedded input data; wherein, the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units.

[0021] Specifically, to obtain the initial input data, in practical applications, after masking the privacy information therein, the pre-trained large language model can be trained based on the processed initial input data. This application provides a method for replacing the privacy information therein. Among them, the initial input data generally includes multiple input batches batchs, and there are B input samples sequence of batch size in each input batch batch.

[0022] Similarly, the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedded unit tokens. Among them, the input batches of the initial input data correspond one-to-one with the input batches of the embedded input data, and the input samples of the initial input data correspond one-to-one with the input samples of the embedded input data. Further, the above-mentioned embedded unit tokens are elements included in the embedded input data. In practical applications, the corresponding embedded input data can be obtained by acquiring the token vector space information stored in the input-embedding layer of the model, which is essentially the original word embedding matrix of the model. Each row represents the embedding vector of a token in the vocabulary, storing the representation of the corresponding token in the vector space of the model.

[0023] Step S220, determine the perturbation candidate units from each embedded unit in each input sample according to the importance of each embedded unit.

[0024] Taking each input sample as a unit, calculate the importance of each embedded unit. Among them, the importance of the embedded unit characterizes the degree of attention of this embedded unit by other embedded units and the degree of attention of this embedded unit to other embedded units. In practical applications, the multi-head attention weight matrix A ∈ R can be obtained from the last layer of the pre-trained large language model B ×H×S×S , where B is the batch size, H is the number of attention heads, S is the sequence length, that is, the length of the input sample. To improve the calculation efficiency, only the attention weights of the last layer can be extracted because this layer contains the richest semantic information. The shape of the attention weight matrix is [batch_size, num_heads, seq_len, seq_len], which records the degree of attention of each embedded unit token to all other tokens in the input sample. It can be understood that if a certain embedded unit token pays attention to a large number of other tokens, and / or a certain embedded unit token is paid attention to by a large number of other tokens, it means that this embedded unit token has a higher importance. In summary, the importance of each embedded unit can be calculated through the multi-head attention weight matrix.

[0025] After obtaining the importance of each embedding unit, perturbation candidate units can be determined from each input sample. Here, the perturbation candidate units are the embedding units that need to be perturbed and whose data needs to be replaced. In this embodiment, it is preferably to perform a replacement operation on the embedding units with low importance because the tokens at high-importance positions have a greater impact on the training effect of the subsequent large language model. Therefore, in this embodiment, only the embedding units with low importance are replaced or modified, and the embedding units at high-importance positions remain unchanged. In summary, perturbation candidate units can be determined in each input sample according to the importance of the embedding units. In some embodiments, the embedding units with importance lower than a preset importance threshold can be used as perturbation candidate units.

[0026] Step S230: Based on the similarity between each embedding unit and the perturbation candidate units, determine a set of perturbation similar units corresponding to the perturbation candidate units, and calculate a perturbation replacement unit corresponding to the perturbation candidate unit according to the set of perturbation similar units.

[0027] Specifically, calculate the similarity between the perturbation candidate unit and all the other embedding units. Here, the similarity is preferably the cosine similarity. Then determine the set of perturbation similar units corresponding to the perturbation candidate unit. The set of perturbation similar units is a set of embedding units that are relatively similar to the perturbation candidate unit. In practical applications, including but not limited to, the embedding units with a similarity greater than a preset similarity threshold can be added to the above set of perturbation similar units, or the top-k embedding units that are most similar to the perturbation candidate unit can be used as the above set of perturbation similar units.

[0028] After obtaining the set of perturbation similar units, calculate according to the embedding units in the set of perturbation similar units to obtain a perturbation replacement unit corresponding to the perturbation candidate unit. The specific calculation methods include but are not limited to calculating the mean value among the embedding units in the set of perturbation similar units to calculate another perturbation replacement unit that is similar but different from the perturbation candidate unit.

[0029] In some preferred embodiments, pre-computation can be performed first, that is, calculate the set of similar units corresponding to each embedding unit first and save the set of similar units. Then, after determining the perturbation candidate units, extract the corresponding set of perturbation similar units according to the perturbation candidate units.

[0030] Step S240: Perform a replacement process on the corresponding perturbation candidate unit based on the perturbation replacement unit to obtain the target input data of each input batch after replacement.

[0031] Specifically, the corresponding perturbation candidate units are replaced by the perturbation replacement units, and then the replaced target input data is obtained. In practical applications, the obtained target input data can be used to train the large language model, that is, to perform subsequent training steps such as backpropagation to calculate the loss, thereby effectively avoiding the problem of personal privacy leakage that is prone to occur during the training process.

[0032] In summary, after replacing the perturbation candidate units with the perturbation replacement units, the replaced input data can be used as training data and input into the pre-trained large language model to update the gradient training parameters.

[0033] Through steps S210 to S240, the privacy data that may exist in the input data can be protected, effectively preventing the leakage of personal information. Moreover, this application can complete data protection under a relatively low perturbation rate, effectively balancing privacy protection and the performance utility during subsequent training of the large language model.

[0034] In some embodiments, determining the perturbation candidate units from each embedding unit in each input sample includes: Obtaining the preset perturbation ratio corresponding to each input sample; Taking one input sample as a unit, calculating the in-degree importance and out-degree importance of the embedding units included in one input batch, and calculating the importance score corresponding to each embedding unit according to the in-degree importance and out-degree importance; In the same input sample, sorting the corresponding embedding units based on the magnitude of the importance scores to obtain the importance unit sequence, and determining the benchmark perturbation unit in the importance unit sequence according to the perturbation ratio; In the same input sample, the embedding units with importance scores less than the benchmark perturbation unit are determined as perturbation candidate units.

[0035] Specifically, this embodiment provides a method for determining perturbation candidate units in each input sample. First, after obtaining the above-mentioned multi-head attention weight matrix (it can be understood that according to the above description, this multi-head attention matrix includes the degree of attention of each embedding unit token to other tokens in the input sample). Further, in practical applications, the input data in this application is training data for a large language model. Therefore, after inputting the input data into the pre-trained language model, the above-mentioned multi-head attention weight matrix can be obtained. And among them, the large language model in this embodiment includes a frozen model original parameter matrix and a soft prompt embedding matrix. The soft prompt embedding matrix is integrated before the model input embedding layer for soft prompt tuning Soft-Prompt-Tunning. Calculate the in-degree importance of each embedding unit, that is, the degree to which the current token is concerned by other tokens, which can be obtained by averaging the attention matrix in the head dimension and the source token dimension: ; where h is the number of attentions, and A h,j,i refers to the sum of the attention weights of token i to all attention heads h and all other tokens j in a sample sequence, and t i refers to the in-degree importance of token i.

[0036] The in-degree importance reflects the amount of information received by a token in the entire input sample. A high in-degree value indicates that the token is concerned by a large number of other tokens.

[0037] And calculate the out-degree importance of each token, that is, calculate the degree to which the current token is concerned about other tokens in a training sample, which is obtained by averaging the attention matrix in the head dimension and the target token dimension: ; where the out-degree importance reflects the amount of information divergence of a token in the training sample. A high out-degree value indicates that the token is concerned about a large number of other tokens.

[0038] Then, comprehensively calculate the importance score of the embedding unit based on the out-degree importance and the in-degree importance. Specifically: ; where the in-degree weight α and the out-degree weight β can be set in advance by relevant technicians.

[0039] In some preferred embodiments, for the convenience of subsequent calculations, the importance scores can be normalized to ensure that all scores are in the range of [0,1]: ; Among them, t i refers to the i-th token (embedding unit) in the input data, min j I(t j ) refers to the minimum value among the importance scores of all tokens in the current input batch, max j I(t j ) refers to the maximum value among the importance scores of all tokens in the current input batch, and ε refers to the privacy budget.

[0040] Obtain the perturbation ratio γ corresponding to each input sample. In practical applications, according to different task types, different styles of training datasets, different sizes of training datasets, different types of large language models (the trend of the large language model memorizing training data mainly depends on the size of the model parameters), etc., determine the perturbation ratio of the perturbed tokens in the input sample in the specific scenario (for example, the perturbation ratio of a model with a parameter size of 3B or less can be controlled within 10%, and the perturbation ratio of a model with more than 10B - 100B can be controlled within 15%). This perturbation ratio γ can be set in advance by relevant technical personnel.

[0041] Then, after obtaining the perturbation ratio of each input sample and the importance scores corresponding to each embedding unit, each input sample sorts the included embedding units in sequence according to the size of the importance scores of the embedding unit tokens to obtain the above-mentioned importance unit sequence, and selects the (s·γ)-th value as the threshold, and the embedding unit corresponding to the (s·γ)-th value is the above-mentioned reference perturbation unit, where s is the number of embedding unit tokens in the input sample. Finally, in the same input sample, the embedding units with importance scores less than the reference perturbation unit are used as the perturbation candidate units to be perturbed.

[0042] In practical applications, after determining the perturbation candidate units, a noise mask M noise can be created to identify the positions of the perturbation candidate units that need to add noise, that is, need to be perturbed, facilitating the replacement of the perturbation candidate units in the subsequent steps.

[0043] Through this application, the perturbation ratio can be determined according to actual needs, and the number of perturbation candidate units required in an input sample can be flexibly determined according to the set perturbation ratio, thereby flexibly adapting to various different scenarios.

[0044] In some of these embodiments, determining the perturbation candidate units from each embedding unit in each input sample includes: Obtain the preset effective length for each input sample; Determine the valid mask in the corresponding input sample based on the valid length, and determine the valid units in the input sample through the valid mask; Obtain the perturbation ratio of the valid units in each input sample, and calculate the perturbation candidate units in the valid units according to the perturbation ratio and the importance score of the valid units.

[0045] Specifically, obtain the preset valid length L corresponding to each input sample. The valid length L is the length of the embedding units regarded as valid in the input sample. It can be understood that the invalid embedding units are the positions where the tokens of the embedding units marked as not requiring loss calculation subsequently in the input sample are located. Then, a corresponding valid mask can be created based on the valid length, so that the valid tokens, that is, the valid units, in the input sample can be determined according to the valid mask.

[0046] In some preferred embodiments, a method for creating a valid mask is provided, including: calculating the loss only for the suffix sequence (i.e., the suffix part of the input sample), marking the prefix sequence as invalid, and not including it in the range of consideration for perturbation candidates, to create a valid token mask M valid =(L≠no_valid) (i.e., the valid token mask M valid is the position where the label is not equal to the invalid label no_valid / the position not marked as invalid by the "no_valid" label), where L represents the token label or ID at each position, that is, the identification information of each token. This is because in the actual application scenarios of common data extraction attacks, there are scenarios where only the suffix content is extracted, without involving the sequence prefix. Therefore, in this embodiment, the prefix sequence is marked as invalid, that is, the prefix sequence is not included in the range of consideration for perturbation candidate units, and only the suffix sequence of the input sample is used as the valid unit, and the perturbation candidate units are determined in the valid units. In summary, in this embodiment, the setting of the valid units can also be selected by those skilled in the art according to the actual situation which part of the tokens need to be invalidated, realizing the selective perturbation of specific tokens, and providing a more refined control ability for the privacy protection of large language models.

[0047] It can be understood that after determining the valid units of the input sample, the method for determining the perturbation candidate units in the valid units is the same as the method described above, that is, determining the perturbation ratio of the valid units of the input sample, calculating the importance scores of the valid units, then sorting the valid units according to the magnitude of the importance scores to obtain the importance unit sequence of the valid units, and determining the benchmark perturbation unit in the importance unit sequence of the valid units according to the perturbation ratio, and determining the valid units with importance scores less than the benchmark perturbation unit as the perturbation candidate units. Through this embodiment, the valid units in the input sample can be determined first, and then the perturbation candidate units can be determined in the valid units, thereby reducing the computational burden during subsequent model training while ensuring the computational effect.

[0048] In some of these embodiments, determining the perturbation similarity unit set corresponding to the perturbation candidate units includes: Calculating the similarity between each embedding unit and other embedding units, and determining the first similar unit corresponding to each embedding unit according to the similarity calculation result; Performing clustering calculation on all the embedding units to obtain at least two clustering partitions, calculating the similarity between each embedding unit and other embedding units in the same clustering partition, and determining the second similar unit corresponding to each embedding unit in each clustering partition according to the similarity calculation result; Determining the similarity unit set corresponding to each embedding unit based on the first similar unit and the second similar unit; Determining the perturbation candidate units in the embedding units, and obtaining the corresponding perturbation similarity unit set based on the perturbation candidate units.

[0049] Specifically, in practical applications, the word embedding matrix E∈R can be extracted from the pre-trained language model V×D , where V is the vocabulary size and D is the embedding dimension. The above pre-trained language model is the large language model that needs to be trained based on the input data described above in this application. Among them, the obtained word embedding matrix carries the vectors mapped to low dimensions corresponding to each embedding unit token in the input data input to the pre-trained language model.

[0050] The L2 normalization (L2 Norm Normalization) can be performed on each word vector in the word embedding matrix first for subsequent calculation of cosine similarity. The normalization calculation is as follows: ; The calculated similarity matrix of the normalized embeddings has ; Then, calculate the similarity between each embedding unit i and other embedding units, and select the top k most similar units. These top k most similar units are the above-mentioned first similar units, and k can be set by those skilled in the art according to actual needs. Record the index of the first similar unit corresponding to the current embedding unit. By the above method, all embedding units and their corresponding first similar units can be calculated and recorded, and then a mapping M based on the first similar units of the embeddings is generated. emb : ; Where S i is the similarity score of all embedding unit tokens in the row where the i-th embedding unit token is located in the cosine similarity matrix. It can be understood that the row where the i-th embedding unit token is located stores the cosine similarity between the i-th embedding unit token in the vocabulary and all other embedding unit tokens. It can be understood that by the method of calculating each embedding unit and its corresponding first similar unit in this embodiment, it can ensure the similarity query of embedding unit tokens with relatively close distances in the model word embedding vector space oriented by basic functionalism. Its design purpose is to maintain the rationality of the basic vector space, ensure that the replacement tokens generated based on the similar units are close to the original embedding vectors (original tokens) in the word vector space, can maintain the local syntactic structure, and can also maintain syntactic correctness through word form similarity, maintain the language model perplexity (Perplexity, PPL), control the replacement word frequency distribution to be close to the original distribution, and ensure the minimum support effect.

[0051] Then, partition the embedding space based on the K-means clustering algorithm, that is, perform clustering calculation to obtain N clusters clustering partition results (i.e., semantic clusters): ; Further, C i represents the partition ID to which the i-th embedding unit token i belongs, that is, the cluster ID. This formula for C represents the general calculation formula for all cluster IDs, and E is the above-mentioned word embedding matrix.

[0052] Generate similar embedding units (i.e., similar token mappings M sem ) for each embedding unit based on the clustering partition. That is, after clustering each embedding unit token into multiple clustering partitions, taking each clustering partition as a unit, calculate the similarity between each embedding unit and other embedding units in the same clustering partition, and use the top k most similar embedding units as the second similar unit of this embedding unit. By this method, in each clustering partition, each embedding unit and its corresponding second similar unit can be calculated. At this time, there is: M sem (i) = {j | C_j = C_i, j ≠ i}; Wherein, i represents the token index of the currently calculated current embedding unit, which is a specific token ID in the vocabulary, j is the token index of some other embedding unit in the same clustering partition as i, the value range of j is all integers from 0 to the vocabulary size V - 1, and as an element in the set, j needs to meet the following conditions. C_i is the cluster ID to which the above token i belongs, which is the cluster identifier assigned to token i after partitioning the word embedding space by the K-means clustering algorithm. Similarly, C_j represents the cluster ID to which token j belongs. In summary, the meaning of the above formula is that token i and token j belong to the same semantic cluster, and token j is not token i itself. In summary, these token j that meet the requirements constitute the token set M that is semantically similar to token i sem (i).

[0053] In summary, based on the similar mapping of semantic clusters, the candidates based on the similarity of the semantic vector space are completed. Its function is to break through the limitation of the similarity of the simple adjacent semantic vector space, search for potential candidate tokens for the similarity of semantic categories that may exist based on the context, so as to adapt to the search for similar token candidates for the migration of the same token in different context environments. For example, the basic candidate similar token sequence of "apple - fruit" in the semantic vector space breaks through to the candidate token search of "apple - mobile phone" related to the semantic category association of the context

[0054] In summary, the similar unit set corresponding to any embedding unit can be calculated by the above method, that is, the first similar unit and the second similar unit. Further, in some preferred embodiments, if the number of tokens in the similar unit set corresponding to the embedding unit is too large, the intersection can be further sorted to retain the k1 most similar tokens, where k1 can be set by those skilled in the relevant art: ; At this time, where M combined is the above similar unit set

[0055] In practical applications, the above method for calculating similar units is pre-calculated. Therefore, when it is necessary to determine the perturbed similar unit set corresponding to the perturbed unit, the perturbed similar unit set corresponding to the perturbed unit can be determined from the pre-calculated set

[0056] In some of these embodiments, calculating the first similar unit includes: Obtain the embedding matrix tensor of the initial input data and normalize the embedding matrix tensor; wherein, the embedding matrix tensor includes the embedding vectors of the embedding units; Perform matrix multiplication on the normalized embedding matrix tensor to obtain the corresponding cosine similarity index matrix, where the cosine similarity index matrix characterizes the similarity relationship between each embedding unit; Based on the cosine similarity index matrix, determine the first similar unit corresponding to each embedding unit; Calculate the second similar unit, including: Perform clustering calculation on the embedding matrix tensor to obtain at least two clustering partitions, and classify and store the embedding units according to the clustering partitions; Calculate the similarity between each embedding unit and other embedding units in the same clustering partition, and based on the similarity calculation results, determine the second similar unit corresponding to each embedding unit in each clustering partition.

[0057] Specifically, this embodiment provides a method for calculating the first similar unit and the second similar unit. Calculating the first similar unit includes: Obtain the above word embedding matrix, process the encapsulation of the embedding layer of the language pre-training model to obtain the original embedding matrix tensor of the model, perform normalization on the original embedding layer one by one to obtain the unit embedding matrix, and then perform matrix multiplication on the self-unit embedding matrix to obtain the above cosine similarity index matrix. Among them, performing matrix multiplication on the self-unit embedding matrix refers to calculating the dot product of the normalized word embedding matrix and its transpose to form the above similarity matrix. Through this calculation method, the similarity between all tokens can be calculated simultaneously through one matrix multiplication, which is much more efficient than pairwise calculation. This cosine similarity index matrix characterizes the similarity relationship between each embedding unit. Then, based on the cosine similarity index matrix, for each embedding unit, its k most similar other embedding units are determined and stored, so as to obtain each embedding unit and its corresponding first similar unit. In practical applications, the embedding unit (i.e., a specific token) and its corresponding first similar unit can be converted into a dictionary structure: {token_id:[similar_ids]}, where token_id is the unit index corresponding to the embedding unit, and similar_ids are the unit indices corresponding to each first similar unit of the embedding unit.

[0058] Furthermore, this embodiment also gives a specific method for calculating the second similar unit, including: Using MiniBatch K-means clustering calculation, each original word embedding matrix is clustered into multiple clustering partitions, and each embedding unit is traversed, grouped and classified by cluster for storage, and the corresponding token indexes are stored separately, so as to obtain a dictionary structure {cluster_id: [token_ids]}, where cluster_id refers to the id index of the clustering partition, and token_ids is the index id of the embedding units stored in the clustering partition. Calculate the similarity between each embedding unit and other embedding units in the same clustering partition, sort them in descending order of similarity, and take the k most similar embedding units to obtain the above-mentioned second similar units. At this time, there is {token_id: [similar_ids]}, where token_id is the unit index corresponding to the embedding unit, and similar_ids is the unit index corresponding to each second similar unit of the embedding unit.

[0059] In practical applications, the first similar unit and the second similar unit corresponding to each embedding unit can be calculated preferentially and the results can be stored. Then, after determining the perturbation candidate units, the first similar unit and the second similar unit corresponding to the perturbation candidate units can be directly called, that is, the perturbation similar unit set corresponding to the perturbation candidate units is obtained. At this time, to obtain the first similar unit of the perturbation candidate unit from the pre-computed results: C emb =M emb (L i,j ), where M emb (·) is a mapping from the tokenID (i.e., the currently calculated perturbation candidate unit) to a list of similar tokenIDs (i.e., other embedding units similar to the perturbation candidate unit), and L i,j is the tokenID at the j-th position in the i-th input batch, that is, the original perturbation candidate unit to be perturbed, and C emb is the candidate first similar unit obtained through similarity search; similarly, to obtain the second similar unit of the perturbation candidate unit from the pre-computed results: C sem =M sem (L i,j ), where M sem is a mapping from the tokenID to a list of similar tokenIDs based on semantic clustering, and L i,j is the same as the tokenID at the j-th position in the i-th input batch, and C sem is the candidate token set obtained through semantic clustering similarity search, that is, the above-mentioned second similar units, and these tokens belong to the same semantic category as the original perturbation candidate token.

[0060] In some preferred embodiments, the intersection between the first similar unit and the second similar unit corresponding to the perturbation candidate unit may be used as the set of perturbation similar units corresponding to the perturbation candidate unit. At this time, there is: C combined =(C emb ∩C sem )∪C emb ’∪C sem ’; Among them, C emb ’ represents the set of remaining elements in C emb after excluding the intersection with C sem . Similarly, C sem ’ represents the set of remaining elements in C sem after excluding the intersection with C emb .

[0061] In some of these embodiments, calculating the perturbation replacement unit corresponding to the perturbation candidate unit based on the set of perturbation similar units includes: Determine a preset context window and determine the context representation of the perturbation candidate unit in the corresponding input sample based on the context window; Based on the context representation and the set of perturbation similar units corresponding to the perturbation candidate unit, calculate a cosine similarity score vector; Convert the cosine similarity score vector into a similarity data distribution in the form of a probability distribution, and perform high-cosine similarity sampling based on the similarity data distribution to obtain the perturbation replacement unit corresponding to the perturbation candidate unit.

[0062] First, determine the context window N(j) of a certain perturbation candidate unit token j in the corresponding input sample. Among them, the context window can be preset to a distance of 4 embedding unit tokens. If the perturbation candidate unit token j is at the i-th position in the input sample, its corresponding context window should be the (i - 2)-th token position, the (i - 1)-th token position, the i-th token (i.e., the perturbation candidate unit token j itself), the (i + 1)-th token position, and the (i + 2)-th token position. In practical applications, the size of the context window can be adjusted according to actual application requirements. By calculating and learning the context representation, it can be ensured that the finally calculated perturbation replacement unit and the perturbation candidate unit still maintain the same or similar basic data structure.

[0063] The calculation of the context can be expressed as: ; Among them, ctx jrefers to the context embedding representation of token j, which is the average of all token embedding vectors within the context window of token j. N(j) represents the context window of token j, and |N(j)| represents the number of tokens in the context window. is the embedding vector of the i-th token in the context window. represents the sum of the embedding vectors of all tokens in the context window. is the average of the sum of the embedding vectors of all tokens in the context window.

[0064] Then, calculate the cosine similarity score vector between each perturbation similar unit token c (token c ∈ C combined ) in the set of perturbation similar units corresponding to the perturbation candidate unit token j and the context representation of the perturbation candidate unit: ; where the cosine similarity score vector sim c represents the cosine similarity between the embedding vector of candidate token c and the current context embedding representation of the corresponding perturbation candidate unit token j. E c refers to the embedding vector of the perturbation similar unit token c, which is obtained from the embedding layer of the pre-trained model.

[0065] Use the softmax function to convert the above cosine similarity score vector into a probability distribution, obtain the above similarity data distribution, and perform high cosine similarity sampling based on this similarity data distribution to calculate the above perturbation replacement unit. At this time, there is: ; where p c is the probability value of a certain similar unit in the set of perturbation similar units. The sum of the probabilities of all perturbation similar units corresponding to a perturbation candidate unit token j is 1, and each perturbation similar unit has a corresponding p c value. P c refers to the probability that the corresponding perturbation similar unit is selected as the perturbation replacement unit, that is, the set of perturbation similar units includes this perturbation replacement unit. sim c’To represent the cosine similarity between the context embedding representations of each perturbed similar unit token c' and the perturbed candidate unit token j when traversing the tokens in the set of perturbed similar units, where c' is a variable that takes different values when traversing the entire similar set, and T is the temperature parameter. In this embodiment, the temperature parameter is introduced to avoid the computational behavior of the perturbed replacement unit approaching the argmax selection, avoiding overly consistent replacements of tokens with similar meanings, that is, avoiding the perturbed replacement unit being too similar to the perturbed candidate unit, and ensuring a certain balance between privacy and utility.

[0066] In one embodiment, the cosine similarity score vector is converted into a similarity data distribution in the form of a probability distribution, and high-cosine similarity sampling is performed based on the similarity data distribution to obtain a perturbed replacement unit corresponding to the perturbed candidate unit, including: Obtain a preset temperature parameter, and determine the perturbed replacement unit based on the temperature parameter and the cosine similarity score vector.

[0067] Specifically, this embodiment provides a method for calculating the similarity data distribution, specifically as follows: ; where p c is the similarity data distribution corresponding to a certain perturbed similar unit, and this similarity data distribution is a probability vector, representing the probability that each candidate token c is selected as the replacement token (i.e., the perturbed replacement unit) under the given context. From an implementation perspective, p c is the output of applying the softmax function to the normalized similarity scores, c' is an element taken from the combined candidate set C combined and serves as an index variable during the summation process, sim c’ is the cosine similarity between each candidate token c' and the context of token j, and T is the temperature parameter. Then, the perturbed similar unit corresponding to the target similarity data distribution with the highest value among all similarity data distributions is used as the above-mentioned perturbed replacement unit. In this embodiment, the temperature parameter is introduced to avoid the computational behavior of the perturbed replacement unit approaching the argmax selection, avoiding overly consistent replacements of tokens with similar meanings, that is, avoiding the perturbed replacement unit being too similar to the perturbed candidate unit, and ensuring a certain balance between privacy and utility.

[0068] This application also provides a preferred embodiment of a privacy protection method for input data.

[0069] First, obtain the initial input data. Generally, the initial input data will contain some text with personal sensitive information, such as: "My credit card number is , and the expiration date is 05 / 24" (it can be understood that this example is an input sample in the input data).

[0070] Then, the multi - head attention mechanism is used to calculate the importance scores of each embedding unit (token) in the input sample. At this time, there are: The average importance score of "My credit card number is" at position1 is 0.78 (high importance); position2: The card number " " has an average importance score of 0.21 (low importance); The average importance score of "Expiry date: 05 / 24" at position3 is 0.65 (medium - high importance); Further determine the importance threshold. According to the preset perturbation ratio and the importance scores corresponding to each embedding unit, the importance threshold can be determined, or the importance threshold can be preset in advance. For example, in this preferred embodiment, the importance threshold can be set to 0.4. At this time, the units in the above - mentioned input sample that need to be perturbed are identified, that is, the perturbation candidate units in the above - mentioned input sample (i.e., the above - mentioned position2).

[0071] Then, a set of perturbation - similar units similar to the perturbation candidate unit can be determined from the input data. Among them, the perturbation candidate unit and the set of perturbation - similar units corresponding to the perturbation candidate unit are pre - calculated. For example: For "4532", find similar digit - formatted tokens such as "7891", "3256", etc.; For "7891", find similar digit - formatted tokens such as "4532", "6745", etc., and so on.

[0072] Based on the context window, obtain the context information corresponding to the perturbation candidate unit: "The credit card number is" to ensure that the selected replacement token still maintains the digit format and the basic structure of the credit card number (16 digits, divided into 4 groups).

[0073] Based on the extracted set of perturbation - similar units and the above - mentioned context information, calculate the perturbation replacement unit corresponding to the perturbation candidate unit: " ", and replace the position where the original perturbation candidate unit is located with this perturbation replacement unit, then the target input data after replacement can be obtained: "My credit card number is ,Expiry date: 05 / 24".

[0074] Then, the pre - trained language model is trained with the target input data after replacement. Furthermore, the language model will only learn the concept and format of "credit card number", and cannot obtain and remember the actual card number sequence. Through this application, privacy protection can be effectively achieved, and when the language model is trained with the training data adjusted by this application, the semantic interpretation ability of the language model LLM will not change significantly.

[0075] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are sequentially shown according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0076] Based on the same inventive concept, an embodiment of the present application also provides a privacy protection device for implementing the privacy protection method described above. The solution provided by this device to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the privacy protection device provided below can refer to the limitations on the privacy protection method in the above text, and will not be repeated here.

[0077] In one embodiment, as Figure 3 shown, a privacy protection device is provided, including: an acquisition module 31, a calculation module 32, and a generation module 33, where: The acquisition module 31 is configured to acquire initial input data and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein, the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units; The calculation module 32 is configured to determine perturbation candidate units from each embedding unit in each input sample according to the importance of each embedding unit; determine a perturbation similarity unit set corresponding to the perturbation candidate units based on the similarity between each embedding unit and the perturbation candidate units, and calculate a perturbation replacement unit corresponding to the perturbation candidate units according to the perturbation similarity unit set; The generation module 33 is configured to perform a replacement process on the corresponding perturbation candidate units based on the perturbation replacement units to obtain the target input data of each input batch after replacement.

[0078] Each module in the above privacy protection device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0079] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structural diagram may be as shown in Figure 4 . The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store privacy protection-related calculation data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a privacy protection method.

[0080] Those skilled in the art can understand that Figure 4 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0081] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: Obtain initial input data, and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein, the embedded input data includes multiple input batches, each input batch includes multiple input samples, and each input sample includes multiple embedding units; According to the importance of each embedding unit in each input sample, determine perturbation candidate units from each embedding unit in each input sample; Based on the similarity between each embedding unit and the perturbation candidate units, determine a perturbation similarity unit set corresponding to the perturbation candidate units, and calculate a perturbation replacement unit corresponding to the perturbation candidate units according to the perturbation similarity unit set; Based on the perturbation replacement unit, perform replacement processing on the corresponding perturbation candidate unit to obtain the target input data of each input batch after replacement.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0083] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0084] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0085] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A privacy protection method for input data, characterized in that: The method comprises: Acquire initial input data, and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes a plurality of input batches, each of the input batches includes a plurality of input samples, and each of the input samples includes a plurality of embedding units; Determine a candidate perturbation unit from each of the embedding units in each of the input samples according to the importance of each of the embedding units in each of the input samples; Based on the similarity between each of the embedding units and the candidate perturbation unit, a set of perturbation similar units corresponding to the candidate perturbation unit is determined, and a perturbation replacement unit corresponding to the candidate perturbation unit is calculated according to the set of perturbation similar units; The corresponding candidate disturbance units are replaced based on the disturbance replacement units to obtain target input data of each input batch after replacement.

2. The method according to claim 1, characterized in that The determining of a candidate perturbation unit from each of the embedding units in each of the input samples comprises: Obtaining a preset disturbance ratio corresponding to each of the input samples; Taking an input sample as a unit, calculating the in-degree importance and out-degree importance of each embedding unit, and obtaining the importance score corresponding to each embedding unit according to the in-degree importance and out-degree importance; In the same input sample, the corresponding embedding units are sorted based on the importance scores to obtain an importance unit sequence, and a reference perturbation unit is determined in the importance unit sequence according to the perturbation ratio; In the same input sample, the embedding unit whose importance score is smaller than the reference perturbation unit is determined as the perturbation candidate unit.

3. The method according to claim 1, characterized in that The determining of a candidate perturbation unit from each of the embedding units in each of the input samples comprises: Obtaining a preset effective length for each of the input samples; Determine a valid mask in the corresponding input sample based on the valid length, and determine a valid unit in the input sample through the valid mask; The perturbation ratio of the valid unit in each of the input samples is obtained, and the perturbation candidate unit in the valid unit is calculated according to the perturbation ratio and the importance score of the valid unit.

4. The method according to claim 1, characterized in that: The determining a disturbance similar unit set corresponding to the disturbance candidate unit includes: Calculate the similarity between each embedding unit and other embedding units, and determine the first similar unit corresponding to each embedding unit according to the similarity calculation result; Performing cluster calculation on all the embedding units to obtain at least two cluster partitions, calculating the similarity between each embedding unit and other embedding units in the same cluster partition, and determining the second similar unit corresponding to each embedding unit in each cluster partition according to the similarity calculation result; Determine a similar unit set corresponding to each of the embedding units based on the first similar unit and the second similar unit; The disturbance candidate unit is determined in the embedding unit, and the corresponding disturbance similar unit set is obtained based on the disturbance candidate unit.

5. The method according to claim 4, characterized in that Calculating the first similar unit includes: Acquire an embedding matrix tensor of the initial input data, and perform normalization processing on the embedding matrix tensor; wherein the embedding matrix tensor includes an embedding vector of the embedding unit; Performing matrix multiplication on the normalized embedding matrix tensor to obtain a corresponding cosine similarity index matrix, wherein the cosine similarity index matrix represents the similarity relationship between each of the embedding units; Determine the first similar unit corresponding to each of the embedding units based on the cosine similarity index matrix; Calculating the second similar unit includes: Performing cluster calculation on the embedding matrix tensor to obtain at least two cluster partitions, and classifying and storing the embedding units according to the cluster partitions; The similarity between each of the embedding units and other embedding units in the same cluster partition is calculated, and based on the similarity calculation result, the second similar unit corresponding to each of the embedding units in each cluster partition is determined.

6. The method according to claim 1, characterized in that The step of calculating, according to the disturbance similar unit set, a disturbance replacement unit corresponding to the disturbance candidate unit comprises: Determining a preset context window, and determining a context representation of the candidate perturbation unit in a corresponding input sample based on the context window; Calculating a cosine similarity score vector based on the context representation and a set of perturbation similar units corresponding to the perturbation candidate unit; The cosine similarity score vector is converted into a similarity data distribution in the form of a probability distribution, and high cosine similarity sampling is performed based on the similarity data distribution to obtain the disturbance replacement unit corresponding to the disturbance candidate unit.

7. The method according to claim 6, characterized in that The step of converting the cosine similarity score vector into a similarity data distribution in a probability distribution form, and performing high cosine similarity sampling based on the similarity data distribution to obtain the disturbance replacement unit corresponding to the disturbance candidate unit includes: A preset temperature parameter is obtained, and the disturbance replacement unit is determined based on the temperature parameter and the cosine similarity score vector.

8. A privacy protection device, characterized in that: The device comprises: An acquisition module, configured to acquire initial input data, and input the initial input data into a pre-trained large language model to obtain embedded input data; wherein the embedded input data includes a plurality of input batches, each of the input batches includes a plurality of input samples, and each of the input samples includes a plurality of embedding units; A calculation module, configured to determine a perturbation candidate unit from each of the embedding units in each of the input samples according to the importance of each of the embedding units in each of the input samples; determine a perturbation similar unit set corresponding to the perturbation candidate unit based on the similarity between each of the embedding units and the perturbation candidate unit, and calculate a perturbation replacement unit corresponding to the perturbation candidate unit according to the perturbation similar unit set; A generating module is used to perform a replacement process on the corresponding candidate disturbance unit based on the disturbance replacement unit to obtain target input data of each input batch after replacement.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images