Large language model privacy protection method and device based on adaptive semantic perception

Abstract

The invention discloses an adaptive semantic perception large language model privacy protection method and device, and the method comprises the steps: enabling a large language model service provider to decompose a large language model into a bottom embedded layer model and other parts, and respectively deploying the bottom embedded layer model and other parts at a client side and a server side; in a pre-training stage, a client trains a generator and a sampler. The generator comprises a learnable text embedding layer and a gating loop unit, and can generate a text sequence similar to an original text and semantically close to the original text. A sampler adaptively selects a text position needing to be replaced and a replacement word of the text position, and semantic embedding of an original text is gradually approached; in the enhanced training stage, a pre-trained generator and a discriminator based on Transform carry out adversarial training, a sampler generates a text with similar semantics, and the generator is guided to generate a more real sample conforming to semantics; the trained generator generates disturbance embedding, and the server side receives the disturbance embedding and carries out fine tuning or reasoning and other operations.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a privacy protection method and device for a large language model with adaptive semantic perception. Background Art

[0002] The emergence of pre-trained language models (PLM) has significantly promoted the development of natural language processing (NLP) and has made significant breakthroughs in the performance of related downstream tasks. Generative pre-trained transformers (GPT) and other large language models (LLMs) have been widely used in finance, medicine, protein molecule prediction, physics, database systems and other fields, opening up new ways to improve data management efficiency and accessibility of scientific research.

[0003] However, fine-tuning PLMs with billions or even trillions of parameters remains a major challenge for users who lack high-performance computing infrastructure. This problem has led to the development of the Model-as-a-Service (MaaS) model, where service providers provide users with pre-trained models, computing resources, and database management capabilities. Fine-tuning under the MaaS model has many advantages, such as accelerating model customization and reducing local computing burdens, but it also raises concerns about data security, especially the potential risks of data leakage and unauthorized access.

[0004] Users' private data often contain sensitive information, such as personal identity and property information. If these data are directly transmitted to the service provider, they may not only be eavesdropped during the transmission process, but also may be leaked due to the service provider's breach of trust. Therefore, how to fully utilize the advantages of the MaaS model while ensuring data security has become an important issue that needs to be solved urgently.

[0005] In this context, several studies have proposed solutions to the above challenges, mainly divided into two major directions: cryptographic methods and perturbation protection techniques. Cryptography-based methods typically employ homomorphic encryption algorithms. Although this approach can theoretically provide strong privacy protection, it faces significant computational complexity issues in practical applications. Especially in large language models based on the Transformer architecture, homomorphic encryption can lead to a substantial increase in time overhead on both the server side and the client side, thus limiting the feasibility of its practical application. Additionally, how to maintain the performance of the LLM while ensuring privacy protection remains a key challenge. This is because it is particularly important to protect the semantic integrity of the original data without revealing personally identifiable information (PII).

[0006] In contrast, perturbation-based methods can effectively protect data privacy at the cost of a moderate sacrifice in accuracy, especially suitable for resource-constrained clients. For example, Qu et al. utilized χ differential privacy (DP) to perturb sensitive data for privacy protection. SAP-CTI optimized on this basis by splitting the pre-trained language model (PLM) into the bottom embedding layer and other parts, deploying them on the client side and the server side respectively, and introducing a Contributing-Token Identification mechanism to enhance the preservation of the original semantics of the perturbed text. RAPT also adopted χ to perturb sensitive data and optimized the prompt fine-tuning process through Privatized Token Reconstruction technology.

[0007] However, the application of differential privacy methods to text data faces some challenges. First, for structured text data, defining "adjacent" datasets is relatively complex because the differences in text are difficult to precisely quantify. Second, differential privacy methods are difficult to adaptively generate appropriate perturbations for different application scenarios, which may lead to a large semantic deviation between the perturbed text embeddings and the original text, thereby affecting the accuracy of model fine-tuning. Currently, no adaptive differential privacy method for large language model fine-tuning has been applied in practice.

[0008] Split Learning (SL) is a distributed learning technique that divides a model into parts held by different parties, enabling collaborative training without exposing the original data. In methods like SplitNN, the client's data is first processed into an intermediate representation and transmitted to the server. The server then continues the forward propagation, processes these intermediate features, and sends the gradients back to the client to update its parameters. Although SL itself provides a certain degree of privacy protection, research has shown that it is still vulnerable to privacy leakage. To mitigate these risks, SL typically combines encryption techniques and perturbation algorithms to enhance its privacy protection capabilities.

[0009] Generative networks can synthesize data consistent with real-world training sets and have wide applications in fields such as computer vision, natural language processing, and audio synthesis. Variational Autoencoders (VAEs) use probabilistic methods to minimize the reconstruction error, ensuring a stable training process and an interpretable latent space. Diffusion Models have become increasingly popular in recent years. They generate data by learning to reverse the process of gradually denoising. However, high computational requirements limit the application of these models on resource-constrained devices. In contrast, Generative Adversarial Networks (GANs) support precise, input-driven data generation and can maintain high relevance at a relatively low computational cost. Significant progress has also been made in the application of GANs in privacy protection.

[0010] Despite the significant progress in the field of generative models, current models mostly focus on specific application domains and have not been specifically optimized for the privacy protection of large language models (LLMs). In particular, for secure data encryption methods that need to maintain semantic integrity and model usability, existing research is still insufficient.

[0011] Therefore, future research should focus on developing privacy-preserving generative models suitable for LLMs, with an emphasis on addressing how to ensure semantic consistency of the generated data and model performance while protecting privacy. This requires not only interdisciplinary collaboration but also exploring new theoretical and technical frameworks to tackle this complex challenge. Summary of the Invention

[0012] To at least to some extent solve one of the technical problems existing in the prior art, an object of the present invention is to provide a privacy protection method, device, and medium for large language models with adaptive semantic awareness.

[0013] The first technical solution adopted by the present invention is:

[0014] A privacy protection method for large language models with adaptive semantic perception, comprising the following steps:

[0015] The server splits the pre-trained large language model into a bottom embedding layer and the remaining layers, downloads the bottom embedding layer to the client, and retains the remaining layers on the server;

[0016] The client receives the original input text and uses the bottom embedding layer to convert the input text into word embeddings;

[0017] In the pre-training stage, the client trains a generator and a sampler; the generator is used to receive word embeddings and generate perturbed word embeddings; the sampler is used to select the words to be replaced and their replacement probabilities according to the learned probability distribution;

[0018] In the enhanced training stage, the pre-trained generator is adversarially trained with a Transformer-based discriminator, and the sampler generates semantically approximate texts to guide the generator to generate more realistic and semantically compliant samples;

[0019] The perturbed text generated by the generator after enhanced training is passed through the word embedding layer of the large language model (LLM) to generate perturbed embeddings, and then transmitted to the server for fine-tuning and inference tasks.

[0020] Furthermore, the perturbation form of the client is in an adaptive manner. For the perturbation pattern in the split learning scenario, the parameter update process of the global PLM is as follows:

[0021]

[0022] In the formula, θ is the parameter of the PLM, θ * is the parameter updated by the PLM; ε represents the embedding layer, δ represents the privacy protection operation, represents the loss function inside the general large model; D is the dataset held by the client, d i is the i-th data in the dataset D;

[0023] The client receives the original input text and uses the bottom embedding layer to convert the input text into word embeddings, including:

[0024] For the input text X = {x 1 , x 2 ,..., x n}, each piece of data in the text X is marked as x i , then the word embeddings generated by a piece of input text passing through the bottom embedding layer are marked as Φ(x i ).

[0025] Furthermore, the working mode of the generator is:

[0026] The input text index is passed through the embedding layer of the generator, generating the embedding φ g (x i );

[0027] The embedding φ g (x i ) is concatenated with the initial hidden state and the word embedding Φ(x i ), and input into the gated recurrent unit (GRU) of the generator, where h represents the dimension of the hidden layer; the calculation process is as follows:

[0028]

[0029] In the formula, represents the hidden state input of the gated recurrent unit at time step t; W gg , b gg represent the learnable weights and biases of the gated recurrent unit in the generator;

[0030] The output of the gated recurrent unit is projected into the complete vocabulary space through a linear layer for sequence-to-sequence text generation. The expression is as follows:

[0031]

[0032] In the formula, Φ G (x i ) represents the output of the generator, and W gl , b gl represent the learnable weights and biases of the internal linear layer of the generator;

[0033] The generated perturbation sequence is calculated as:

[0034]

[0035] Furthermore, in the pre-training stage, the sampler is used to train and replace tokens according to the input text to sample samples related to semantic similarity; the replacement probability of each token position of the sampler is Pr p , and the probability of the replacement word of the token at each position is Pr r , and its calculation process is as follows:

[0036]

[0037] In the formula, and respectively represent the outputs of the last layers in the locator and replacer GRUs; respectively represent the learnable weights and biases inside the one-dimensional convolution; W srl , bsrl respectively represent learnable weights and biases; h is the dimension of the hidden layer;

[0038] Based on Pr r and Pr p The process of identifying and replacing the Top-k tokens in each input text x i is as follows:

[0039] x i ′[j] = M[x i [j]][argmax(Pr r [x i [j]])]

[0040] j = Top-k(Pr p )

[0041] where M is obtained by calculating and sorting the cosine similarity between the embedding layer matrix of the LLM and itself.

[0042] Furthermore, the training loss function of the generator in the pre-training stage is:

[0043]

[0044] In the formula, v represents the vocabulary size; tr represents the training set; X ic represents the corresponding token;

[0045] The training loss function of the sampler in the pre-training stage is:

[0046]

[0047] In the formula, l represents the sequence length, Φ(x i ) represents the original embedding of x i , Φ(x i ′ ) represents the embedding of x i after being sampled by the sampler and passing through the embedding layer.

[0048] Furthermore, the discriminator within the client is used to enhance the generation quality of the generator; the working method of the discriminator is:

[0049] Denote the output of the Transformer encoder as z dt , and the linear layer projects z dt onto a two-dimensional space to distinguish whether the input is real or generated:

[0050]

[0051] In the formula, W dl , b dlRepresents the learnable weights and biases of the linear layers within the discriminator.

[0052] Furthermore, the adversarial loss of the discriminator is:

[0053]

[0054] In the formula, Represents the true data distribution, Represents the data distribution of the perturbed text output by the generator, Φ D (x i ) represents the text embedding of x within the discriminator i .

[0055] Furthermore, the sampler generates texts with semantic approximations to guide the generator to generate more realistic and semantically consistent samples, including:

[0056] The sampler pre-computes Pr p and Pr r through the locator and replacer, and determines the number of tokens N to be replaced;

[0057] According to the highest value in Pr p , select the top N tokens for replacement to form the replacement index set

[0058] For each sample j (from 1 to m), perform the following operations:

[0059] A1. Create a clone s i of the text data x i ;

[0060] A2. For each token at each position

[0061] A21. Retrieve the k most similar tokens from the similarity matrix M to

[0062] A22. Sample a replacement token r r from x isim according to the replacement probability in Pr i ;

[0063] A23. Replace with r i ;

[0064] A3. Add the modified sequence s i to the sampled token set S.

[0065] Further, the sampler trained by the client is used to replace the tokens at the relevant positions of the original text to generate samples with similar semantics to the original text; the sampler consists of a specified number of token replacement operations based on the original input to form a set S; the guiding loss function of the sampler for the generator is as follows:

[0066]

[0067] In the formula, m is the number of samples, v is the vocabulary length, and S jc is the element corresponding to the set S; Φ G (x i ) jc represents the embedding of x i generated by the corresponding generator;

[0068] In the enhancement stage, the loss function of the generator is as follows:

[0069]

[0070] In the formula, λ 1 , λ 2 , λ 3 are weight coefficients.

[0071] The second technical solution adopted by the present invention is:

[0072] An electronic device, the electronic device includes a processor and a memory, and at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement an adaptive semantic perception large language model privacy protection method as described above.

[0073] The third technical solution adopted by the present invention is:

[0074] A computer-readable storage medium, at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an adaptive semantic perception large language model privacy protection method as described above.

[0075] The fourth technical solution adopted by the present invention is:

[0076] A computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions so that the computer device executes the above method.

[0077] The beneficial effects of the present invention are as follows: The present invention can adaptively add perturbations according to the perturbed text, making the perturbations more in line with the original semantics, thereby maintaining a high semantic similarity while protecting privacy. The present invention eliminates the process of matching each perturbed text with its nearest neighbor in the existing differential privacy (DP) method, significantly improving the perturbation speed. The present invention drives the perturbed text generated by the generator to approximate the sampling samples of the sampler, improving the diversity and generalization ability of the perturbed text. Using a deep learning network for perturbation increases the difficulty for attackers to reverse-engineer sensitive information of the original text, further balancing the privacy protection ability and semantic integrity. Through the above improvements, the SaPGAN framework significantly enhances the performance of the LLM under perturbation-based privacy protection, making it more suitable for practical application scenarios. Description of the Drawings

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following introduces the accompanying drawings of the related technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings in the following introduction are only for conveniently and clearly presenting some embodiments of the technical solutions in the present invention. For those skilled in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0079] Figure 1 It is a flowchart of a privacy protection method for an adaptive semantic-aware large language model in an embodiment of the present invention;

[0080] Figure 2 It is a flowchart of a generator placed on the client side in an embodiment of the present invention;

[0081] Figure 3 It is a flowchart of a discriminator placed on the client side in an embodiment of the present invention;

[0082] Figure 4 It is a training flowchart of a sampler placed on the client side in an embodiment of the present invention;

[0083] Figure 5 It is a sampling flowchart of a sampler placed on the client side in an embodiment of the present invention. Detailed Embodiments

[0084] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation of the present invention. For the step numbers in the following embodiments, they are only set for the convenience of elaboration and explanation, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0085] In the description of the present invention, it should be understood that for the orientation description, such as the orientation or positional relationship indicated by up, down, front, back, left, right, etc., is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0086] In the description of the present invention, the meaning of several is one or more, the meaning of multiple is two or more, greater than, less than, exceeding, etc. are understood not to include the original number, and above, below, within, etc. are understood to include the original number. If there is a description of first and second, it is only for the purpose of distinguishing technical features and should not be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features.

[0087] In the description of the present invention, unless otherwise clearly defined, words such as setting, installing, connecting, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meaning of the above words in the present invention in combination with the specific content of the technical solution.

[0088] To better meet the privacy fine-tuning requirements of large language models (LLMs) and achieve a balance between the semantic similarity of perturbed text and privacy protection benefits, the present invention proposes an Adaptive Semantic-Aware Generative Adversarial Network framework (SaPGAN). This is the first framework that uses a sequence-to-sequence generative model and a Transformer-based discriminator for adversarial training to generate perturbed text. This innovative method not only ensures privacy protection but also highly preserves the semantic similarity of the original text. The SaPGAN framework contains an adaptive sampler that can identify the tokens to be replaced, enabling the generator to effectively balance privacy protection and semantic integrity during the generation process. The SaPGAN framework is applicable to the fine-tuning and inference stages of LLMs, providing a new solution for the privacy protection of large models.

[0089] Embodiment 1

[0090] As Figure 1As shown in the figure, this embodiment provides a privacy protection method for large language models with adaptive semantic perception, including the following steps:

[0091] Step S1: The server splits the pre-trained large language model into the bottom embedding layer and the remaining layers, downloads the bottom embedding layer to the client, and retains the remaining layers in the server.

[0092] This embodiment focuses on the privacy risks associated with data transmission between the client and the server. In this scenario, the client holds a private, labeled dataset D = {d 1 , d 2 ,..., d n}.

[0093] Following the split learning framework, due to the resource limitations of the client, the embedding layer of the PLM is deployed on the client, and the remaining layers reside on the server side. On the client side, given the embedding layer ε and the privacy protection operation δ applied to the input text, the perturbed global PLM optimization can be expressed as:

[0094]

[0095] where θ * is the parameter updated by the PLM.

[0096] Step S2: The client receives the original input text and uses the bottom embedding layer of the pre-trained large language model to convert the input text into word embeddings.

[0097] For the input text X = {x 1 , x 2 ,..., x n}, where each piece of data is labeled as x i , then the word embedding generated by a piece of input text passing through the bottom embedding layer is labeled as Φ(x i ).

[0098] Step S3: In the pre-training stage, the generator receives the word embeddings and generates perturbed word embeddings; the sampler selects the words to be replaced and their replacement probabilities according to the learned probability distribution.

[0099] Specifically, the generator includes a learnable text embedding layer and a gated recurrent unit, which can generate a text sequence similar to the original text and semantically close. The sampler adaptively selects the text positions to be replaced and their replacement words, gradually approaching the semantic embedding of the original text.

[0100] This example introduces a sequence-to-sequence privacy-protecting text generator. This generator reduces the risks of information loss and semantic distortion common in DP methods and can maintain good semantic integrity even after privacy protection. The framework of the generator is as Figure 2As shown. Given the input text The embedding layer of the LLM generates token embeddings The input text index is passed through the embedding layer of the generator, generating the SaPG embedding φ g (x i ) ∈ R l×d , where l and d represent the sequence length and dimension of the embedding respectively. The embedding φ g (x i ) is concatenated with the initial hidden state and Φ(x i ) and input into the gated recurrent unit (GRU) of the generator, where h represents the dimension of the hidden layer. The main calculation process is as follows:

[0101]

[0102] In the formula represents the hidden state input of the GRU at time step t. W gg , b gg represent the learnable weights and biases of the GRU in the generator. The output of the GRU passes through a linear layer to project the embedding probability into the complete vocabulary space for sequence-to-sequence text generation. The transformation is given by:

[0103]

[0104] where Φ G (x i ) ∈ R l×v represents the output of the generator, and v represents the size of the vocabulary. The generated sequence index is

[0105] The sampler identifies which positions in the input should be replaced to adaptively minimize the semantic gap. This process is as Figure 4 shown. The final hidden state output of the generator is input into the locator of the sampler, which generates the index replacement probability Pr p ∈ R l . Pr p represents the possibility of each token in the input text being replaced. The hidden state h slg ∈ R l×h of the locator is also input into the GRU inside the locator, which learns the semantic features retained between the generator layers. The calculation process is as follows:

[0106]

[0107] where Denotes the output of the t-th layer in the GRU. The output of the last GRU layer in the locator Is further processed using Conv1D convolution and Max-pooling to transform the embedded dimension to the token level:

[0108]

[0109] where b slc Denotes the bias term in the locator Conv1D convolutional layer. Max-pooling enables the sampler to focus on the most prominent features in the generator hidden state, thereby improving the overall quality of token position selection.

[0110] Meanwhile, Is passed to the replacer of the sampler, and the replacer is responsible for learning which alternative words are semantically most appropriate at each identified replacement position. The replacer consists of a GRU and a linear layer, and this linear layer can compute the token replacement probability Pr r :

[0111]

[0112] where b srl ∈R h Denote the learnable weights and biases respectively, which are used to project the hidden state into the similarity token space defined by the similarity matrix . M is obtained by calculating and sorting the cosine similarity of the embedding layer matrix of the LLM with itself, and d sim Denotes the dimension of the replacement word space. Combining Pr p , Represents the probability distribution for selecting the most appropriate replacement word for each token.

[0113] To train the sampler to balance privacy protection and high semantic protection, SaPGAN iteratively adjusts token replacement. Specifically, the sampler identifies and replaces the top-k tokens in each input text x r and Pr p : i :

[0114] x i ′[j] = M[x i [j]][argmax(Pr r [x i [j]])]

[0115] where j = Top-k(Pr p ).

[0116] Step S4: The generator and the sampler update their internal learnable parameters using their respective loss functions.

[0117] To enable the generator to systematically learn the semantic structure of the input text during pre-training, SaPGAN uses cross-entropy loss to optimize the generated embeddings:

[0118]

[0119] The loss of the sampler during pre-training is defined as the mean squared error (MSE) between the embeddings of the input text x i and the modified text x i ′, and the calculation process is as follows:

[0120]

[0121] Step S5: In the enhanced training phase, the discriminator takes the perturbed text and the real text as inputs and outputs discrimination probabilities to guide the generator to generate perturbed texts that are more in line with the real distribution.

[0122] As Figure 3 shown, the embodiment of the present invention uses a discriminator based on a Transformer encoder for adversarial training to optimize the performance of the generator. In this setting, the token indices in the real text sequence and the generated text sequence are randomly shuffled and then input into the discriminator. Specifically, these shuffled token indices first generate token embeddings through the embedding layer of the discriminator

[0123] Since the Transformer encoder lacks an inherent mechanism to capture position information in the sequence, the positional encoding of the input text is added to the token embeddings. The result of adding the embeddings and the positional encoding is input into the Transformer encoder to extract the key features required to distinguish between the real text sequence and the generated text sequence. The process is described as follows:

[0124]

[0125] q = HW q , k = HW k , ν = HW v

[0126]

[0127] where H represents the combined embedding, and z dt represents the output of the Transformer encoder. The linear layer projects z dt onto a two-dimensional space to distinguish whether the input is real or generated:

[0128]

[0129] The adversarial loss of the discriminator is given by:

[0130]

[0131] Step S6: The sampler samples samples to make the output of the generator tend to the text distribution of the samples.

[0132] See Figure 5 The sampler pre-computes Pr through the locator and the replacer p and Pr r to determine the number of tokens to be replaced

[0133] According to the highest value in Pr p the top N tokens are selected for replacement to form a replacement index set For each sample j (from 1 to m), perform the following operations:

[0134] 1) Create a clone s i of the text data x i .

[0135] 2) For each token at each position

[0136] 2.1) Retrieve the k most similar tokens from the similarity matrix M to

[0137] 2.2) Sample a replacement token r r from x isim according to the replacement probability in Pr i .

[0138] 2.3) Replace with r i .

[0139] 3) Add the modified sequence s i to the sampled token set S.

[0140] The adversarial loss calculation of the generator in the enhanced training phase is:

[0141]

[0142] To further guide the output of the generator to tend to the samples generated by the sampler, the sampling loss is defined as:

[0143]

[0144] During the enhanced training process, the overall loss of the generator is guided by a combined loss function:

[0145]

[0146] where λ 1 , λ 2 , λ 3 balances each component. Here, is the same as the objective in the pre-training stage to enable the generator to continue semantic learning while undergoing adversarial training.

[0147] Step S7: The client transmits the perturbed text generated by the generator after pre-training and enhanced training to the server for fine-tuning and inference tasks after generating perturbed embeddings through the word embedding layer of the LLM.

[0148] That is, the generator can produce the final perturbed text after training After inputting it into this embedding layer of the LLM, a perturbed vector Φ′(x i ) is generated, and this perturbed vector can be uploaded to the server for tasks such as fine-tuning and inference.

[0149] Embodiment 2

[0150] The embodiment of the present invention also provides an electronic device, which includes a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement Figure 1 an adaptive semantic-aware large language model privacy protection method as shown.

[0151] It can be understood that the memory may include a random access memory (RAM), or may also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0152] The processor may include one or more processing cores. The processor uses various interfaces and circuits to connect various parts within the entire server. By running or executing instructions, programs, code sets, or instruction sets stored in the memory, and by invoking the data stored in the memory, it performs various functions of the server and processes data. Optionally, the processor may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor may integrate a combination of one or several of a central processing unit (CPU) and a modem, etc. Among them, the CPU mainly processes the operating system and application programs, etc.; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor and may be implemented separately by a single chip.

[0153] Since this electronic device is the electronic device corresponding to an adaptive semantic perception large language model privacy protection method in an embodiment of the present invention, and the principle by which this electronic device solves problems is similar to that of this method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be elaborated.

[0154] Embodiment 3

[0155] An embodiment of the present invention further provides a computer-readable storage medium, in which at least one instruction, at least one segment of program, code set, or instruction set is stored, and the at least one instruction, the at least one segment of program, the code set, or the instruction set is loaded and executed by a processor to implement Figure 1 an adaptive semantic perception large language model privacy protection method as shown.

[0156] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disc memories, tape memories, or any other computer-readable medium capable of carrying or storing data.

[0157] Since this storage medium is the storage medium corresponding to an adaptive semantic perception large language model privacy protection method of an embodiment of the present invention, and the principle of the storage medium to solve problems is similar to that of this method, the implementation of this storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be elaborated.

[0158] Embodiment 4

[0159] In some possible implementation manners, each aspect of the method of the embodiment of the present invention can also be implemented in the form of a program product, which includes program code. When the program product runs on a computer device, the program code is used to cause the computer device to execute the steps of an adaptive semantic perception large language model privacy protection method according to various exemplary implementation manners described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0160] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0161] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0162] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the art to understand the content of the present invention and implement it accordingly, and cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the essence of the content of the present invention should be covered within the protection scope of the present invention.

Claims

1. An adaptive semantic-aware large language model privacy protection method, characterized in that: The following steps are involved: The server splits the pre-trained large language model into the bottom embedding layer and the remaining layers, and downloads the bottom embedding layer to the client. The remaining layers are kept on the server side; The client receives the raw input text and uses the bottom embedding layer to convert the input text into word embeddings; In the pre-training phase, the client trains a generator and a sampler; the generator is used to receive word embeddings and generate perturbed word embeddings; The sampler is used to select the words to be replaced and their replacement probabilities according to the learned probability distribution; In the enhanced training phase, the pre-trained generator is trained against the Transformer-based discriminator, and the sampler generates semantically similar text to guide the generator to generate more realistic and semantically consistent samples; The perturbed text generated by the enhanced trained generator is embedded in the word embedding layer of the large language model and then transmitted to the server for fine-tuning and inference tasks.

2. According to claim 1, the privacy protection method of a large language model with adaptive semantic awareness is characterized in that: The perturbation form of the client is adaptive. For the perturbation mode in the split learning scenario, the parameter update process of the global PLM is: Where θ is the parameter of PLM, θ * are the parameters updated by PLM; ε represents the embedding layer, δ represents the privacy protection operation, represents the loss function of the large model parameter update; D is the dataset held by the client, d o is the i-th data in the data set D; The client receives raw input text and converts the input text into word embeddings using the bottom embedding layer, including: For input text X = {x1, x2, ..., x n }, each piece of data in the text X is marked as x i , then the word embedding generated by an input text through the bottom embedding layer is marked as Φ(x i ).

3. According to claim 1, the privacy protection method of a large language model with adaptive semantic awareness is characterized in that: The generator works as follows: The input text index is passed through the embedding layer of the generator, producing the embedding φ g (x i ); Embedding φ g (x i ) and the initial hidden state and word embedding Φ(x i ) are connected together and input into the gated recurrent unit of the generator, where h represents the dimension of the hidden layer; the calculation process is: In the formula, represents the hidden state input of the gated recurrent unit at time step t; W gg ,b gg represents the learnable weights and biases of the gated recurrent unit in the generator; Output of the gated recurrent unit Through a linear layer, the embedding probability is projected into the complete vocabulary space for sequence-to-sequence text generation, expressed as follows: In the formula, Φ G (x i ) represents the output of the generator, W gl 、b gl represents the learnable weights and biases of the linear layer inside the generator; The resulting perturbation sequence is calculated as:

4. According to claim 1, the privacy protection method of a large language model with adaptive semantic awareness is characterized in that: In the pre-training stage, the sampler is used to train and replace tokens according to the input text to sample samples with semantic similarity; the replacement probability of each token position of the sampler is Pr p , the probability of replacing a word with a token at each position is Pr r , the calculation process is: In the formula, and Represent the output of the last layer in the locator and replacer GRU respectively; Respectively represent the learnable weights and biases inside the one-dimensional convolution; W srl ,b srl They represent the learnable weights and biases in the linear layer respectively; h is the hidden layer dimension; Based on Pr r and Pr p Identify and replace each input text x i The process of Top-k token is as follows: x i ′[j]=M[x i [j]][argmax(Pr r [x i [j]])] j=Top-k(Pr p , Among them, M is obtained by calculating and sorting the cosine similarity between the embedding layer matrix of LLM and itself.

5. According to claim 1, the privacy protection method of a large language model with adaptive semantic awareness is characterized in that: The training loss function of the generator in the pre-training phase is: Where v is the vocabulary size; tr is the training set; X ic Indicates the corresponding token; The training loss function of the sampler in the pre-training stage is: In the formula, l represents the sequence length, Φ(x i ) represents x i The original embedding of Φ(x i ′ ) represents x i The embedding layer samples the sampler and then embeds it.

6. According to claim 1, the privacy protection method of a large language model with adaptive semantic awareness is characterized in that: The discriminator in the client is used to enhance the quality of the generator; the discriminator works as follows: Let the output of Transformer encoder be z dt , the linear layer converts z dt Project to 2D space to distinguish whether the input is real or generated: Where W dl , b dl represents the learnable weights and biases of the linear layer in the discriminator.

7. The privacy protection method of a large language model with adaptive semantic awareness according to claim 1, characterized in that: The adversarial loss of the discriminator is: In the formula, represents the real data distribution, represents the data distribution of the perturbed text output by the generator, Φ D (x i ) represents the discriminator x i Text embedding.

8. The privacy protection method of a large language model with adaptive semantic awareness according to claim 1, characterized in that: The sampler generates semantically similar text to guide the generator to generate more realistic and semantically consistent samples, including: The sampler pre-calculates Pr through the locator and replacer p and Pr r , determine the number of tokens N that need to be replaced; According to Pr p The highest value in , select the first N tokens to replace, and form a replacement index set For each sample j, do the following: A1. Create text data x i Clones i ; A2. For each position Token A21. Retrieve from similar word matrix M The k most similar tokens A22. According to Pr r The probability of replacement in x isim Sample a replacement tokenr i ; A23. Replace with r i ; A3. The modified sequence s i Add to the sampling token set S.

9. The privacy protection method of a large language model with adaptive semantic awareness according to claim 1, characterized in that: The sampler trained by the client is used to replace the tokens at relevant positions in the original text to generate samples with similar semantics to the original text; the sampler performs a specified number of token replacement operations based on the original input to form a set S; the sampler's guidance loss function for the generator is: In the formula, m is the number of samples, v is the length of the vocabulary, and S jc is the element of the corresponding set S; Φ G (x i ) jc Represents x generated by the corresponding generator i Embedding In the enhancement phase, the loss function of the generator is: Where λ1, λ2, λ3 are weight coefficients.

10. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 9.

Images