An adaptive semantic-aware large language model privacy protection method and device

Abstract

The application discloses a self-adaptive semantic perception large language model privacy protection method and device, wherein the method comprises the following steps: a large language model service provider decomposes a large language model into a bottom embedding layer model and other parts, and respectively deploys the bottom embedding layer model and the other parts on a client and a server; in a pre-training stage, the client trains a generator and a sampler. The generator comprises a learnable text embedding layer and a gated recurrent unit, and can generate a text sequence similar to an original text and close to the semantic of the original text. The sampler adaptively selects a text position to be replaced and a replacement word thereof, and gradually approaches the semantic embedding of the original text; in an enhanced training stage, the pre-trained generator and a discriminator based on a transformer are subjected to adversarial training, and meanwhile, the sampler generates a text close in semantic, so as to guide the generator to generate a more real and semantic sample; and the trained generator generates a perturbation embedding, and the server end receives the perturbation embedding to perform fine-tuning or reasoning and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a self-adaptive semantic perception large language model privacy protection method and device. BACKGROUND

[0002] The emergence of pre-trained language models (PLMs) has significantly promoted the development of the natural language processing (NLP) field, enabling significant breakthroughs in the performance of related downstream tasks. Generative pre-trained transformers (GPT) and other large language models (LLMs) have been widely applied in finance, medicine, protein molecule prediction, physics, database systems and other fields, opening up new avenues for improving data management efficiency and scientific research accessibility.

[0003] However, for users lacking high-performance computing infrastructure, fine-tuning PLMs with billions or even trillions of parameters remains a major challenge. This difficulty has given rise to the development of the model-as-a-service (MaaS) model, in which service providers provide users with pre-trained models, computing resources and database management functions. Fine-tuning under the MaaS model has many advantages, such as accelerating model customization and reducing local computing burden, but it also raises concerns about data security, particularly the potential risks of data leakage and unauthorized access.

[0004] Users' private data often contains sensitive information, such as personal identity and property information. If these data are directly transmitted to service providers, they may be intercepted during transmission and may also be at risk of data leakage due to the untrustworthiness of service providers. Therefore, how to fully utilize the advantages of the MaaS model while ensuring data security has become an important problem to be solved.

[0005] In this context, several studies have proposed solutions to the above challenges, mainly divided into two directions: cryptographic methods and perturbation protection techniques. Cryptographic-based methods usually employ homomorphic encryption algorithms. Although this method can provide strong privacy protection in theory, it faces significant computational complexity problems in practical applications. Especially on large language models based on the Transformer architecture, homomorphic encryption will cause a significant increase in time overhead on both the server and client sides, thus limiting its practical application feasibility. In addition, how to maintain the performance of LLM while ensuring privacy protection is still a key challenge. This is because, under the premise of not revealing personal identifiable information (PII), it is particularly important to protect the semantic integrity of the original data.

[0006] In contrast, perturbation-based methods can effectively protect data privacy with moderate sacrifice of accuracy, especially for resource-constrained clients. For example, Qu et al. used d χ Differential privacy (DP) to perturb sensitive data to achieve privacy protection. SAP-CTI optimized this by splitting the pre-trained language model (PLM) into a bottom embedding layer and other parts, deployed on the client and server sides, respectively, and introduced a contributing token identification mechanism to enhance the preservation of the original semantics of the perturbed text. RAPT also used d χ to perturb sensitive data and optimized the prompt fine-tuning process through privatized token reconstruction technology.

[0007] However, the application of differential privacy methods on text data faces some challenges. First, for structured text data, it is complex to define "adjacent" data sets because the differences between texts are difficult to quantify accurately. Second, differential privacy methods are difficult to adaptively generate appropriate perturbations for different application scenarios, which may result in a large semantic deviation between the perturbed text embedding and the original text, thus affecting the accuracy of model fine-tuning. Currently, there is no adaptive differential privacy method for large language model fine-tuning applied in practice.

[0008] Split Learning (SL) is a distributed learning technique that divides a model into parts held by different parties, enabling collaborative training without exposing raw data. In methods such as SplitNN, client data is first processed into intermediate representations and transmitted to the server side. The server continues to propagate forward, processes these intermediate features, and sends gradients back to the client to update its parameters. Although SL itself provides some degree of privacy protection, research shows that it is still vulnerable to privacy leaks. To mitigate these risks, SL is often combined with encryption techniques and perturbation algorithms to enhance its privacy protection capabilities.

[0009] Generative networks can synthesize data consistent with real training sets, with wide applications in computer vision, natural language processing, and audio synthesis. Variational Autoencoders (VAEs) use probabilistic methods to minimize reconstruction error, ensuring stable training processes and interpretable latent spaces. Diffusion Models have gained popularity in recent years, generating data by learning to reverse a gradually denoising process. However, high computational demands limit the application of these models on resource-constrained devices. In contrast, Generative Adversarial Networks (GANs) support precise, input-driven data generation, maintaining high relevance at relatively low computational cost. GANs have also made significant progress in privacy protection.

[0010] Despite significant progress in generative models, current models mostly focus on specific application areas and have not been optimized specifically for privacy protection of large language models (LLMs). In particular, for secure data encryption methods that need to maintain semantic integrity and model utility, existing research is still insufficient.

[0011] Therefore, future research should focus on developing privacy-protecting generative models suitable for LLMs, focusing on how to protect privacy while ensuring the semantic consistency of generated data and model performance. This not only requires interdisciplinary collaboration, but also requires exploring new theoretical and technical frameworks to address this complex challenge. SUMMARY

[0012] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a self-adaptive semantic-aware large language model privacy protection method, device and medium.

[0013] The first technical solution adopted by the present application is:

[0014] A self-adaptive semantic-aware large language model privacy protection method, comprising the following steps:

[0015] The server side splits the pre-trained large language model into a bottom embedding layer and the remaining layers, downloads the bottom embedding layer to the client, and retains the remaining layers on the server side;

[0016] The client receives the original input text, and converts the input text into word embedding using the bottom embedding layer;

[0017] In the pre-training phase, the client trains a generator and a sampler; the generator is used to receive word embedding and generate perturbed word embedding; the sampler is used to select the word to be replaced and its replacement probability according to the learned probability distribution;

[0018] In the enhanced training phase, the pre-trained generator is adversarially trained with a discriminator based on Transformer, and the sampler generates semantically similar text to guide the generator to generate more realistic and semantically consistent samples;

[0019] The perturbed text generated by the enhanced generator is passed through the word embedding layer of the large language model (LLM) to generate perturbed embedding, and then transmitted to the server side for fine-tuning and inference tasks.

[0020] Further, the perturbation form of the client is adaptive, and for the perturbation mode in the split learning scenario, the parameter update process of the global PLM is:

[0021]

[0022] In the formula, θ is the parameter of the PLM, θ * is the updated parameter of the PLM; ε represents the embedding layer, δ represents the privacy protection operation, represents the loss function inside the general large model; D is the data set held by the client, d i is the i-th data in the data set D;

[0023] The client receives the original input text, converts the input text into word embedding using the bottom embedding layer, comprising:

[0024] For input text X={x1,x2,...,x n}, each data in the text X is marked as x i , then the word embedding generated by the bottom embedding layer for an input text is marked as Φ(x i ).

[0025] Further, the generator works as follows:

[0026] The input text index is passed through the embedding layer of the generator to generate embedding φg (x i );

[0027] embedding φ g (x i ) and the initial hidden state and word embedding Φ(x i ) are concatenated and input into the gated recurrent unit (GRU) of the generator, where h represents the dimension of the hidden layer; the calculation process is as follows:

[0028]

[0029] wherein, represents the hidden state input of the gated recurrent unit at time step t; W gg ,b gg represent the learnable weights and biases of the gated recurrent unit in the generator;

[0030] The output of the gated recurrent unit The embedding probability is projected into the complete vocabulary space through a linear layer for sequence-to-sequence text generation, and the expression is as follows:

[0031]

[0032] wherein, Φ G (x i ) represents the output of the generator, W gl ,b gl represent the learnable weights and biases of the linear layer inside the generator;

[0033] The generated perturbation sequence is calculated as:

[0034]

[0035] Further, in the pre-training stage, the sampler is used to train and replace tokens according to the input text to sample samples related to semantic similarity; the replacement probability of each token position of the sampler is Pr p , and the probability of replacing words at each position token is Pr r , and the calculation process is as follows:

[0036]

[0037] wherein, and represent the outputs of the last layer of the locator and the replacer GRU, respectively; represent the learnable weights and biases inside the one-dimensional convolution, respectively; W srl ,b srl represent the learnable weights and biases, respectively; h is the dimension of the hidden layer.

[0038] based on Pr r and Pr p The process of identifying and replacing the Top-k tokens in each input text x i is as follows:

[0039] x i ′[j]=M[x i [j]][argmax(Pr r [x i [j]])]

[0040] j=Top-k(Pr p )

[0041] where M is obtained by calculating and sorting the cosine similarity of the embedding layer matrix of the LLM with itself.

[0042] Further, the training loss function of the generator in the pre-training stage is:

[0043]

[0044] where v represents the size of the vocabulary; tr represents the training set; X ic represents the corresponding token;

[0045] The training loss function of the sampler in the pre-training stage is:

[0046]

[0047] where l represents the sequence length, Φ(x i ) represents the original embedding of x i , and Φ(x i ′ ) represents the embedding of x i after being sampled by the sampler and then passing through the embedding layer.

[0048] Further, the discriminator within the client is used to enhance the generation quality of the generator; the working mode of the discriminator is:

[0049] Let the output of the Transformer encoder be z dt , and the linear layer projects z dt into a two-dimensional space to distinguish whether the input is real or generated:

[0050]

[0051] where W dl and b dl represent the learnable weights and biases of the linear layer within the discriminator.

[0052] Further, the adversarial loss of the discriminator is:

[0053]

[0054] where, represents the real data distribution, represents the data distribution of the perturbed text output by the generator, Φ D (x i ) represents the text embedding of x i inside the discriminator.

[0055] Further, the sampler generates semantically similar texts to guide the generator to generate more realistic and semantically consistent samples, including:

[0056] The sampler pre-computes Pr p and Pr r through the locator and the replacer, determines the number of tokens N that need to be replaced;

[0057] According to the highest value in Pr p , the first N tokens are selected for replacement to form a replacement index set

[0058] For each sample j (from 1 to m), the following operations are performed:

[0059] A1, create a clone s i of the text data x i ;

[0060] A2, for each token at position ,

[0061] A21, retrieve the k tokens most similar to from the similar word matrix M

[0062] A22, according to the replacement probability in Pr r , sample a replacement token r isim from x i ;

[0063] A23, replace with r i ;

[0064] A3, add the modified sequence s i to the sampled token set S.

[0065] Further, the client trained sampler is used to replace the token of the original text related position to generate a sample with similar semantics to the original text; the sampler performs a specified number of token replacement operations based on the original input to form a set S; the guidance loss function of the sampler to the generator is:

[0066]

[0067] In the formula, m is the sample quantity, v is the vocabulary length, S jc is the element corresponding to set S; Phi G (x i ) jc represents the embedding of x i generated by the corresponding generator;

[0068] In the enhancement stage, the loss function of the generator is:

[0069]

[0070] In the formula, lambda1, lambda2, lambda3 are weight coefficients.

[0071] The second technical solution adopted by the application is:

[0072] An electronic device, the electronic device includes a processor and a memory, the memory has at least one instruction, at least one program, a code set or instruction set stored therein, the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to realize an adaptive semantic-aware large language model privacy protection method as described above.

[0073] The third technical solution adopted by the application is:

[0074] A computer readable storage medium, the storage medium has at least one instruction, at least one program, a code set or instruction set stored therein, the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to realize an adaptive semantic-aware large language model privacy protection method as described above.

[0075] The fourth technical solution adopted by the application is:

[0076] A computer program product or computer program, the computer program product or computer program includes computer instructions stored in a computer readable storage medium. The processor of the computer equipment can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer equipment executes the above method.

[0077] The beneficial effects of the present application are: the present application can adaptively add perturbations according to the perturbed text, so that the perturbations are more consistent with the original semantics, thereby maintaining high semantic similarity while protecting privacy. The present application eliminates the process of matching each perturbed text with the nearest neighbor in the existing differential privacy (DP) method, significantly improving the perturbation speed. The present application drives the generator to generate perturbed text that approximates the sampling sample of the sampler, improving the diversity and generalization ability of the perturbed text. Using a deep learning network for perturbation increases the difficulty for attackers to reverse the sensitive information of the original text, further balancing the privacy protection ability and semantic integrity. Through the above improvements, the SaPGAN framework significantly improves the performance of LLM under privacy protection based on perturbation, making it more suitable for practical application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0079] Figure 1 The flowchart of the adaptive semantic-aware large language model privacy protection method in the embodiment of the present application;

[0080] Figure 2 The flowchart of the generator placed in the client in the embodiment of the present application;

[0081] Figure 3 The flowchart of the discriminator placed in the client in the embodiment of the present application;

[0082] Figure 4 The training flowchart of the sampler placed in the client in the embodiment of the present application;

[0083] Figure 5 The sampling flowchart of the sampler placed in the client in the embodiment of the present application. DETAILED DESCRIPTION

[0084] Embodiments of the present application are described below in detail with reference to the accompanying drawings, wherein the same or similar components or components having the same or similar functions are denoted by the same or similar reference numerals throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and description, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0085] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or component referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0086] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two. Greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0087] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0088] To better adapt to the privacy fine-tuning needs of large language models (LLMs) and strike a balance between the semantic similarity of perturbed texts and privacy protection benefits, the present application proposes a self-adaptive semantic-aware generative adversarial network framework (SaPGAN). This is the first framework that uses sequence-to-sequence generative models and Transformer-based discriminators for adversarial training to generate perturbed texts. This innovative approach not only ensures privacy protection but also highly preserves the semantic similarity of the original text. The SaPGAN framework includes an adaptive sampler that can identify tokens that need to be replaced, allowing the generator to effectively balance privacy protection and semantic integrity during the generation process. The SaPGAN framework is suitable for the fine-tuning and inference stages of LLMs, providing a new solution for privacy protection of large models.

[0089] Example 1

[0090] As Figure 1As shown, the embodiment provides a self-adaptive semantic-aware large language model privacy protection method, which includes the following steps:

[0091] Step S1, the server side splits the pre-trained large language model into the bottom embedding layer and the remaining layers, downloads the bottom embedding layer to the client, and retains the remaining layers in the server.

[0092] The embodiment focuses on the privacy risks related to data transmission between the client and the server. In this scenario, the client holds a private, labeled data set D = {d1, d2,..., d n}.

[0093] Following the split learning framework, due to the resource limitations of the client, the embedding layer of the PLM is deployed on the client, and the remaining layers reside on the server side. On the client side, given the embedding layer ε and the privacy protection operation δ applied to the input text, the perturbed global PLM optimization can be expressed as:

[0094]

[0095] where θ * is the updated parameter of the PLM.

[0096] Step S2, the client receives the original input text and converts the input text into word embeddings using the bottom embedding layer of the pre-trained large language model.

[0097] For an input text X = {x1, x2,..., x n}, each piece of data is labeled as x i , then the word embedding generated by the bottom embedding layer for an input text is labeled as Φ(x i ).

[0098] Step S3, in the pre-training stage, the generator receives the word embedding and generates a perturbed word embedding; the sampler selects the word to be replaced and its replacement probability according to the learned probability distribution.

[0099] Specifically, the generator includes a learnable text embedding layer and a gated recurrent unit, which can generate a text sequence similar to the original text and close in semantics. The sampler adaptively selects the text position to be replaced and its replacement word, gradually approaching the semantic embedding of the original text.

[0100] This example introduces a sequence-to-sequence privacy-protected text generator. This generator reduces the risk of information loss and semantic distortion commonly seen in DP methods, maintaining good semantic integrity even after privacy protection. The framework of the generator is as shown in Figure 2 . Given an input text LLM's embedding layer generates a labeled embedding The input text indices are passed through the generator’s embedding layer, producing SaPG embeddings φ g (x i )∈R l×d , l and d represent the sequence length and dimension of the embeddings, respectively. The embeddings φ g (x i ) are concatenated with the initial hidden state and Φ(x i ) are input into the generator’s gated recurrent unit (GRU), where h represents the dimension of the hidden layer. The main computation is:

[0101]

[0102] where represents the hidden state input of the GRU at time step t. W gg ,b gg represent the learnable weights and biases of the GRU in the generator. The output of the GRU is passed through a linear layer to project the embedding probabilities into the full vocabulary space for sequence-to-sequence text generation. The transformation is given by:

[0103]

[0104] where Φ G (x i )∈R l×v represents the output of the generator, and v represents the size of the vocabulary. The generated sequence indices are

[0105] The sampler identifies which positions in the input should be replaced to adaptively minimize the semantic gap. This process is illustrated in Figure 4 . The final hidden state of the generator is output to the sampler’s locator, which generates index replacement probabilities Pr p ∈R l . Pr p represents the likelihood of replacing each token in the input text. The hidden state h slg ∈R l×h of the locator is also input into a GRU within the locator, which learns to preserve the semantic features retained by the generator layers. The computation is as follows:

[0106]

[0107] where represents the output of the t-th layer in the GRU. The output of the last GRU layer in the locator Further processing using Conv1D convolutions and Max-pooling converts the dimension of the embedding to the token level:

[0108]

[0109] where b slc denotes the bias term in the positioner Conv1D convolution layer. Max-pooling makes the sampler focus on the most salient features in the generator hidden state, thus improving the overall quality of token position selection.

[0110] Meanwhile, The replacer passed to the sampler is responsible for learning which alternative words are semantically most appropriate at each identified replacement position. The replacer consists of a GRU and a linear layer that can compute the token replacement probability Pr r :

[0111]

[0112] where b srl ∈R h denote the learnable weights and biases, respectively, for projecting the hidden state into a similar token space defined by a similarity matrix M computed and sorted by cosine similarity of the embedding layer matrix of the LLM with itself. d sim denotes the dimension of the replacement word space. Combining Pr p , represents the probability distribution for selecting the most appropriate replacement word for each token.

[0113] To train the sampler to balance privacy protection and high semantic protection, SaPGAN iteratively adjusts token replacement. Specifically, the sampler identifies and replaces the top-k tokens in each input text x r based on Pr p and Pr i :

[0114] x i ′[j] = M[x i [j]] [argmax(Pr r [x i [j]])]

[0115] where j = Top-k(Pr p ).

[0116] Step S4, the generator and the sampler update the internal learnable parameters using the corresponding loss functions, respectively.

[0117] To enable the generator to systematically learn the semantic structure of the input text during pre-training, SaPGAN uses cross-entropy loss to optimize the generation embedding:

[0118]

[0119] The loss definition of the sampler during pre-training is the MSE (Mean Squared Error) between the input text x i and the modified text x i ′ embedding, which is calculated as follows:

[0120]

[0121] Step S5, in the enhanced training phase, the input of the discriminator is the perturbed text and the real text, and the output is the discrimination probability, which guides the generator to generate perturbed text more consistent with the real distribution.

[0122] As Figure 3 shown, the embodiment of the present application adopts a discriminator based on a Transformer encoder to perform adversarial training to optimize the performance of the generator. In this setting, the token indices in the real text sequence and the generated text sequence are randomly shuffled and then input into the discriminator. Specifically, these shuffled token indices first generate token embeddings

[0123] Since the Transformer encoder lacks an inherent mechanism to capture position information in the sequence, the position encoding of the input text is added to the token embedding. The result of adding the embedding and the position encoding is input into the Transformer encoder to extract the key features needed to distinguish the real text sequence and the generated text sequence. The flow is described as follows:

[0124]

[0125] q = Hw q , k = Hw k , v = Hw v

[0126]

[0127] where H represents the combined embedding, z dt represents the output of the Transformer encoder. The linear layer projects z dt onto a two-dimensional space to distinguish whether the input is real or generated:

[0128]

[0129] The adversarial loss of the discriminator is given by the following formula:

[0130]

[0131] Step S6, the sampler samples the sample, prompting the output of the generator to tend to the text distribution of the sample.

[0132] Referring to Figure 5 The sampler pre-calculates Pr p and Pr r through the locator and the replacer

[0133] According to the highest value in Pr p , the first N tokens are selected for replacement to form a replacement index set For each sample j (from 1 to m), the following operations are performed:

[0134] 1) Create a clone s i of the text data x i .

[0135] 2) For each token at position

[0136] 2.1) Retrieve the k most similar tokens from the similar word matrix M

[0137] 2.2) According to the replacement probability in Pr r , sample a replacement token r isim from x i .

[0138] 2.3) Replace with r i .

[0139] 3) Add the modified sequence s i to the sampled token set S.

[0140] The adversarial loss calculation of the generator in the enhancement training stage is:

[0141]

[0142] To further guide the output of the generator to tend to the samples generated by the sampler, the sampling loss is defined as:

[0143]

[0144] ​​In the enhanced training process, the overall loss of the generator is guided by a combined loss function:

[0145]

[0146] where λ1, λ2, λ3balance each component. Here, The same as the goal of the pre-training stage , so that the generator continues to perform semantic learning while performing adversarial training.

[0147] Step S7, the client transmits the perturbation text generated by the pre-trained and enhanced trained generator to the server side for fine-tuning and inference tasks after the perturbation text is generated by the word embedding layer of the LLM to generate perturbation embedding.

[0148] That is, the generator can generate the final perturbation text i after training , which is input into the embedding layer of the LLM to generate a perturbation vector Φ'(x i ), which can be uploaded to the server side for fine-tuning and inference tasks.

[0149] Embodiment 2

[0150] The embodiments of the application also provide an electronic device, which includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement a self-adaptive semantic-aware large language model privacy protection method as shown in Figure 1 .

[0151] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing each of the above methods, etc.; and the data storage area can store data created according to the use of the server, etc.

[0152] The processor can include one or more processing cores. The processor connects various parts within the entire server by various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Alternatively, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with one or several combinations of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a single chip.

[0153] Since the electronic device is an electronic device corresponding to the adaptive semantic perception large language model privacy protection method of the embodiments of the present application, and the principle of solving the problem of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be repeated.

[0154] Embodiment 3

[0155] The embodiments of the present application also provide a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement an adaptive semantic perception large language model privacy protection method as shown in Figure 1

[0156] ​Those skilled in the art can understand that all or part of the steps of various methods of the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0157] Since the storage medium is a storage medium corresponding to the adaptive semantic-aware large language model privacy protection method of the embodiments of the present application, and the principle of solving the problem is similar to the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described again.

[0158] Embodiment 4

[0159] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the adaptive semantic-aware large language model privacy protection method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Among them, the executable computer program code or "code" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0160] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so forth.

[0161] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0162] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the essence of the present application should be covered within the protection scope of the present application.

Claims

1. An adaptive semantic-aware large language model privacy protection method, characterized in that, The method comprises the following steps: The server splits the pre-trained large language model into a bottom embedding layer and the remaining layers, downloads the bottom embedding layer to the client, and retains the remaining layers on the server side; The client receives the original input text, converts the input text into word embeddings using the bottom embedding layer; In the pre-training phase, the client trains a generator and a sampler; the generator is used to receive word embeddings and generate perturbed word embeddings; The sampler is used to select the words to be replaced and their replacement probabilities according to the learned probability distribution; In the enhanced training phase, the pre-trained generator is adversarially trained with a Transformer-based discriminator, and the sampler generates semantically similar text to guide the generator to generate more realistic and semantically consistent samples; The perturbed text generated by the enhanced generator is transmitted to the server side for fine-tuning and inference tasks after being converted into perturbed embeddings by the word embedding layer of the large language model; The sampler generates semantically similar text to guide the generator to generate more realistic and semantically consistent samples, comprising: The sampler pre-computes through the locator and the replacer and determines the number of tokens that need to be replaced ; wherein, is the probability of replacement for each token position, is the probability of each position token replacement word; According to the highest value in , the first tokens are selected for replacement, forming a replacement index set ; For each sample the following is performed: A1, creating text data of the clone ; A2. For each position at , the following is performed: A21. retrieving from the similarity matrix the most similar ;​​ A22. The method of claim 21, wherein the replacement probability is based on a number of times the first character has been replaced in the past. ;​​ A23, to replace ; A3. The modified sequence Added to the sampled token set .

2. The method of claim 1, wherein, The perturbation form of the client is adaptive, and for the perturbation mode in the split learning scenario, the parameter update process of the global PLM is: wherein, is a parameter of the PLM, is a parameter of the PLM update; represents an embedding layer, represents a privacy protection operation, represents a loss function of the large model parameter update; is a dataset held by the client, is a dataset is the data in the dataset The client receives the original input text and converts it into word embeddings using the bottom embedding layer, comprising: For an input text , each piece of data in the text is tagged as , then a word embedding produced by the bottom embedding layer for a piece of input text is tagged as .

3. The method of claim 1, wherein, The working mode of the generator is: The input text indices are passed through the embedding layer of the generator, producing embeddings ; embedding with the initial hidden state and word embeddings are concatenated together and input into the gated recurrent unit of the generator, where denotes the dimension of the hidden layer; the computation is wherein denotes the hidden state input of the gated recurrent unit at time step ; denotes the learnable weights and biases of the gated recurrent unit in the generator; Output of the gated recurrent unit The embedding probabilities are projected into the full vocabulary space by a linear layer for sequence-to-sequence text generation, expressed as follows: wherein denotes the output of the generator, denotes the learnable weights and biases of the linear layers within the generator; The generated perturbation sequence is calculated as: 。 4. The method of claim 1, wherein, In the pre-training stage, the sampler is used to train and replace tokens according to input texts to sample samples related to semantic similarity; the replacement probability of each token position of the sampler is The probability of replacing words at each position token is The calculation process is as follows: wherein and denote the output of the last layer in the positioner and replacer GRU, respectively; denote the learnable weights and bias within a one-dimensional convolution, respectively; denote the learnable weights and bias within a linear layer, respectively; is the hidden layer dimension; Based on and The process of identifying and replacing the Top-k tokens in each input text is as follows: wherein, By computing and sorting the cosine similarity of the embedding layer matrix of the LLM with itself.

5. The method of claim 1, wherein, The training loss function of the generator in the pre-training phase is: In the formula, denotes the vocabulary size; denotes the training set; denotes the corresponding token; The training loss function of the sampler in the pre-training phase is: wherein denotes the sequence length, denotes the original embedding of denotes the embedding of after sampling by the sampler and passing through the embedding layer.

6. The method of claim 1, wherein, The discriminator in the client is used to enhance the generation quality of the generator; the working mode of the discriminator is: The output of the Transformer encoder is noted , and a linear layer projects into a two-dimensional space to distinguish between real and generated inputs: In the formula, denote the learnable weights and biases of the linear layers within the discriminator.

7. The method of claim 1, wherein, The adversarial loss of the discriminator is: wherein, represents the real data distribution, represents the data distribution of the perturbed text output by the generator, represents the text embedding within the discriminator, the discriminator.

8. The method of claim 1, wherein, The client trained sampler is used to replace tokens of relevant positions in the original text to generate samples with similar semantics to the original text; the sampler performs a specified number of token replacement operations based on the original input to form a set The guidance loss function of the sampler for the generator is: wherein, is the number of samples, is the vocabulary size, is an element of the corresponding set ; denotes the embedding of generated by the corresponding generator; In the enhancement phase, the loss function of the generator is: wherein are weight coefficients, denotes the data distribution of the perturbed text output by the generator.

9. An electronic device, comprising: The electronic device comprises a processor and a memory, the memory stores at least one program, the processor loads and executes the at least one program to implement the method of any one of claims 1 to 8.

Images