A language model gradient inversion method and device for federated learning

Abstract

The application discloses a language model gradient inversion method and device for federated learning. Firstly, a character embedding vector with minimum reconstruction loss is selected as an optimization object. For the optimization object, continuous optimization with the minimum reconstruction loss minimization as the target is performed to obtain a character embedding vector after continuous optimization. Then, the character embedding vector after continuous optimization is mapped into a character sequence. A heuristic function is constructed by combining the gradient reconstruction loss of the character embedding vector and the perplexity of the character sequence. Discrete optimization of sequence beam search is performed to obtain a character embedding vector and a character sequence with the lowest heuristic function score. Iteration of alternating continuous optimization and discrete optimization is performed, and finally, a reconstructed text is obtained. The method can mine the potential relationship between sequences, thereby realizing efficient search of a sequence space.

Description

Technical Field

[0001] This application belongs to the field of gradient inversion technology, and in particular relates to a method and apparatus for gradient inversion of language models for federated learning. Background Technology

[0002] Federated learning (FL), as a distributed machine learning paradigm, has enormous potential and significant advantages in protecting privacy and improving data security. Its core idea is to allow multiple clients to train a model through distributed collaboration without directly sharing private datasets. This distributed collaboration approach ensures high data confidentiality without reducing model training efficiency. A typical federated learning implementation is as follows: a central server distributes model parameters to each participating client; each client calculates and returns gradient update information based on its local data; finally, the central server aggregates the gradient update information returned by all clients to update the global model. Thus, federated learning enhances privacy by storing data on local devices. Considering its privacy advantages, federated learning has enormous research and application potential in privacy-sensitive fields such as healthcare, transportation, and personal communication.

[0003] The rise of federated learning has led to a surge in research focused on its security. Recent studies have revealed inherent risks in federated learning, where attackers can directly recover the original training data from shared gradient update information. If privacy-sensitive image or text data can be reconstructed through attacks, the privacy advantages of federated learning will be severely challenged. In-depth research into gradient inversion methods serves as a warning to researchers and practitioners in the field of federated learning to develop more efficient safeguards to ensure more comprehensive privacy protection for user data during sharing. Furthermore, gradient inversion methods can serve as a privacy auditing tool to assess the amount of privacy-sensitive data that attackers can obtain, contributing to the continuous improvement of data protection efficiency and effectiveness, and the design of future privacy protection mechanisms.

[0004] Existing gradient inversion methods integrate Language Model Priors (LAMP) into discrete optimization to achieve sequence transformation of character sequences. Building upon this, an alternating continuous-discrete optimization method has been proposed, demonstrating excellent performance in character recovery and text reconstruction. However, existing techniques neglect the potential relationships between sequences, leading to insufficient search of the sequence space and thus difficulty in recovering character order and long texts. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for gradient inversion of language models for federated learning, which overcomes the problem of insufficient sequence space search, uncovers the potential relationships between sequences, and thus improves the search efficiency of the sequence space.

[0006] To achieve the above objectives, the technical solution of this application is as follows:

[0007] A language model gradient inversion method for federated learning includes:

[0008] Step 1: Using Gaussian distribution, randomly initialize the first number of character embedding vectors. For each initialized character embedding vector, calculate the reconstruction loss corresponding to the target language model, and select the character embedding vector with the minimum reconstruction loss as the optimization object.

[0009] Step 2: For the optimization object, perform continuous optimization with the goal of minimizing reconstruction loss to obtain the continuously optimized character embedding vector;

[0010] Step 3: Map the continuously optimized character embedding vectors into character sequences. Construct a heuristic function by combining the gradient reconstruction loss of the character embedding vectors and the perplexity of the character sequences. Perform discrete optimization of sequence beam search to obtain the character embedding vectors and character sequences with the lowest heuristic function scores.

[0011] Step 4: Determine if the set number of first iterations has been reached. If so, output the character sequence with the lowest heuristic function score as the inverted text. Otherwise, use the character embedding vector with the lowest heuristic function score as the optimization object and return to step 2 to continue iterating.

[0012] Furthermore, the heuristic function is expressed by the following formula:

[0013]

[0014] Where h(x,t) is the heuristic function score, x is the character embedding vector, and t is the character sequence corresponding to the character embedding vector x; Let α be the gradient reconstruction loss of the character embedding vector x with respect to the target language model. lm As a weighting factor to assist the language model, Let t be the perplexity of the character sequence t for the auxiliary language model.

[0015] Furthermore, the perplexity of the character sequence t for the auxiliary language model The calculation formula is as follows:

[0016]

[0017] Where n is the number of characters in the character sequence, P(t) l+1|t1,t2,...t l For a given character sequence t1, t2, ... t l The next character t l+1 The probability of occurrence.

[0018] Furthermore, the discrete optimization of the execution sequence beam search yields the character embedding vector and character sequence with the lowest heuristic function score, including:

[0019] Step F1: Initialize a min-heap H of capacity B, where B is the beam size. Map the continuously optimized character embedding vector x to a character sequence t. Insert the character embedding vector x, the character sequence t, and the corresponding heuristic function score h(x,t) into the min-heap H as initial sentence candidates. The character embedding vector x and the character sequence t are set to the optimal character embedding vector x. best and the best character sequence t best The initial state;

[0020] Step F2: Extract all sentence candidates from the min-heap H, and perform a preset number of discrete sequence transformations on each sentence candidate;

[0021] Step F3: Sort all the sentence candidates after the transformation of the discrete sequences from low to high according to their respective heuristic function scores, and select the top B sentence candidates to re-insert into the min-heap H;

[0022] Step F4: Select the sentence candidate with the lowest heuristic function score in the current min-heap H, and combine its heuristic score with the current best character embedding vector x. best and character sequence t best The corresponding heuristic function scores are compared; if the former is lower than the latter, the current best character embedding vector x is updated. best and the best character sequence t best The sentence with the lowest heuristic function score in H is the candidate.

[0023] Step F5: Determine if the second iteration has been reached. If so, output the optimal character embedding vector x. best and the best character sequence t best Otherwise, return to step F2 to continue the iteration.

[0024] Furthermore, the preset number of discrete sequence transformations includes at least two different discrete sequence transformations.

[0025] This application also proposes a language model gradient inversion device for federated learning, including a processor and a memory storing a number of computer instructions, which, when executed by the processor, implement the steps of the above-described language model gradient inversion method for federated learning.

[0026] This application proposes a gradient inversion method and apparatus for language models in federated learning, and introduces a novel Sequence Beam Search (Seq-BS) mechanism to extend the traditional alternating continuous-discrete optimization structure. Compared with previous studies, the ILAMP method proposed in this application can uncover the latent relationships between sequences, thereby achieving efficient searching of the sequence space. Attached Figure Description

[0027] Figure 1 This is a flowchart of a language model gradient inversion method for federated learning proposed in this application.

[0028] Figure 2 This is a flowchart illustrating the discrete optimization process for sequence beam search in an embodiment of this application.

[0029] Figure 3 This is a schematic diagram of discrete sequence transformation in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0031] Existing gradient inversion methods integrate language model priors into discrete optimization to achieve character sequence transformation. Building upon this, we propose an alternating continuous-discrete optimization method, which employs a Language Model Prior (LAMP) and demonstrates excellent performance in character recovery and text reconstruction. In the discrete optimization phase, sequence transformation is performed iteratively on a single candidate text, generating a series of candidate texts. An auxiliary language model is then used to evaluate and select the optimal text.

[0032] The continuous optimization can be represented by equations (1) and (2):

[0033]

[0034]

[0035] Among them, reconstruction losses It consists of two parts: gradient reconstruction loss. and regularization term λ is the learning rate, α reg It is the regularization weight factor.

[0036] Specifically The cosine distance is used to measure the distance between gradients, as shown in equation (3). The reconstruction process is regularized using embedding regularization, as shown in equation (4).

[0037]

[0038] Where x = (x1, x2, ..., x n ), where n is the number of characters in the character embedding vector x, V is the size of the vocabulary, and e1, e2, ..., e V This represents the character embedding vector in the target language model embedding matrix. It is a virtual gradient Compared with the original gradient The inner product of. This is the virtual gradient obtained by inputting the character embedding vector x into the target language model. The vocabulary is a list containing all words and symbols learned by the language model during pre-training. Each word or symbol in the vocabulary corresponds to a character embedding vector, which are stored in the embedding matrix.

[0039] This application proposes an improved LAMP method (ILAMP), extending the alternating continuous-discrete optimization structure of the LAMP method. It employs a novel heuristic mechanism deployed in the discrete optimization stage—Sequence Beam Search (Seq-BS). Unlike earlier beam searches, Seq-BS is a heap-based heuristic mechanism capable of simultaneously transforming multiple candidate texts into sequences. The Seq-BS mechanism embeds a heuristic function based on gradient reconstruction loss using character embedding vectors and perplexity of the character sequence, used to evaluate and select the best candidate text. Therefore, the ILAMP method in this application can uncover latent relationships between sequences through the Seq-BS mechanism, thereby improving the search efficiency of the sequence space.

[0040] One embodiment of this application, such as Figure 1 As shown, a method for gradient inversion of language models for federated learning is provided, including:

[0041] Step S1: Using Gaussian distribution, randomly initialize the first number of character embedding vectors. For each initialized character embedding vector, calculate the reconstruction loss corresponding to the target language model, and select the character embedding vector with the minimum reconstruction loss as the optimization object.

[0042] In this embodiment, the character embedding vector x is obtained by performing n0 random initializations using a Gaussian distribution N(0,1), resulting in n0 character embedding vectors x.

[0043] For each character embedding vector x obtained from initialization, the reconstruction loss is calculated according to formula (1). The character embedding vector with the minimum reconstruction loss is selected as the optimization target.

[0044] It should be noted that this application uses the language model undergoing federated training as the target language model, that is, the language model used as the inversion target, which is referred to as the target language model in this application.

[0045] When performing gradient inversion on the target language model, the model structure and its parameters f can be obtained. θ and loss function and the original gradient calculated by the client on the private dataset. Therefore, the reconstruction loss can be calculated based on these known parameters and loss functions, which will not be elaborated further below.

[0046] Step S2: For the optimization object, perform continuous optimization with the goal of minimizing reconstruction loss to obtain the continuously optimized character embedding vector.

[0047] In this embodiment, for each initialized character embedding vector x, the reconstruction loss is calculated according to formula (1). The character embedding vector with the minimum reconstruction loss is selected as the optimization object, and continuous optimization is performed with the goal of minimizing the gradient reconstruction loss.

[0048] During the continuous optimization process, gradient descent was performed on x to minimize the reconstruction loss. Continuous optimization execution n c In each round, the optimized character embedding vector obtained in each round is used to continue the optimization process in the next round.

[0049] Continuous optimization is a relatively mature technology in this field, and will not be elaborated on here.

[0050] Step S3: Map the continuously optimized character embedding vectors into character sequences, construct a heuristic function by combining the gradient reconstruction loss of the character embedding vectors and the perplexity of the character sequences, perform discrete optimization of sequence beam search, and obtain the character embedding vectors and character sequences with the lowest heuristic function scores.

[0051] The Sequence Beam Search Discrete Optimization (Seq-BS) in this embodiment is a heap-based heuristic mechanism used to search for the optimal character embedding vector x that minimizes the heuristic function. best and character sequence t bestThe Seq-BS mechanism employs a min-heap data structure to improve efficiency and performance. First, a min-heap H of capacity B is initialized, where B is the beam size. The continuously optimized character embedding vectors are mapped to character sequences. The character embedding vector x, the character sequence t, and the corresponding heuristic function score h(x,t) are inserted into H as initial sentence candidates. That is, (h(x,t), x,t) is a sentence candidate, where x and t are set to x... best and t best The initial state.

[0052] This embodiment uses an embedding-to-sequence method to embed the input characters into a vector x = x1, x2, ..., x. n Embedding matrix W of the target language model embed The characters e1, e2, ..., e V By comparing the characters, we find the one with the smallest cosine distance, and then map x to the character sequence t = t1, t2. 2,... ,t n .

[0053] In one specific embodiment, a heuristic function is constructed by combining the gradient reconstruction loss of the character embedding vector and the perplexity of the character sequence.

[0054] In this embodiment, a heuristic function h(·) is used to evaluate the generated sentence candidates, as shown in equation (5). This function incorporates the gradient reconstruction loss of the character embedding vectors. Perplexity of character sequences Where α lm It is a weight factor for the auxiliary language model.

[0055]

[0056] This embodiment uses a pre-trained large-scale language model as an auxiliary language model to evaluate the perplexity of the generated text. Given a character sequence t = t1, t2, ... t n , where n is the length of the sequence. The perplexity of a character sequence t is expressed as . It can be calculated using an auxiliary language model, and its calculation formula is shown in equation (6):

[0057]

[0058] Wherein, P(t) l+1 |t1,t2,...t l () is for a given character sequence t1, t2, ... t l The next character t l+1The probability of occurrence. For example, the GPT-2 model can be used as an auxiliary language model. GPT-2 is an open-source language model based on the Transformer architecture, developed by OpenAI. It can effectively measure the perplexity of text, and perplexity is an important indicator for evaluating the coherence and predictability of the model for text.

[0059] In the discrete optimization of this embodiment, K iterations will be performed. In each iteration, first, all sentence candidates are extracted from the minimum heap H. Then, for the character embedding x and character sequence t of each candidate, four different discrete sequence transformations (for example, T1, T2, T3, T4) are performed to generate four different candidates. For all the transformed candidates, they are ranked from low to high according to the scores of their heuristic functions, the top B candidates are retained, and reinserted into the minimum heap H for the next iteration, while the other candidates are discarded. In this process, the size of the minimum heap H does not exceed B to retain the top B candidates, and the other candidates will be discarded. Repeat the K iterations, and select the x best ,t best ) with the lowest heuristic function score h(x best and t best as the best output, and denoted as x best and t best .

[0060] As Figure 2 shown, the process of each iteration is as follows:

[0061] Step F2: Extract all sentence candidates from the minimum heap H, and perform a preset number of different discrete sequence transformations on each sentence candidate respectively.

[0062] In this embodiment, the following four discrete sequence transformations are applied to the character embedding vector x and character sequence t. Taking the sequence transformation of the character embedding vector x as an example, as Figure 3 shown, where x = x1, x2, x3, x4, x5, x6.

[0063] The first sequence transformation T1: Randomly select two subscripts i and j (assuming i < j), and then swap the positions of x i and x j .

[0064] Assume i = 2 and j = 4, the modified sequence is: x' = x1, x4, x3, x2, x5, x6.

[0065] The second sequence transformation T2: Randomly select three subscripts i, j, and k (assuming i < j < k), and then swap the subsequence from subscript i to j - 1 with the subsequence from subscript j + 1 to k.

[0066] Assume i = 2, j = 4, and k = 6. The modified sequence is: x' = x1, x5, x6, x4, x2, x3.

[0067] The third sequence transformation T3: Randomly select two indices i and j (assume i < j), and then move x i to the position after index j.

[0068] Assume i = 2 and j = 5. The modified sequence is: x' = x1, x3, x4, x5, x2, x6.

[0069] The fourth sequence transformation T4: Randomly select three indices i, j, and k (assume i < j < k), and then move the subsequence from index i to j to the position after index k.

[0070] Assume i = 2, j = 3, and k = 5. The modified sequence is: x' = x1, x4, x5, x2, x3, x6.

[0071] It should be noted that for each sentence candidate, the above four transformation methods need to be executed to generate four sentence candidates after discrete sequence transformation. This embodiment is not limited to the number of types of sequence transformation, and at least two or more are required.

[0072] Step F3: Sort all the sentence candidates after discrete sequence transformation in ascending order according to their heuristic function scores, and select the first B sentence candidates to reinsert into the minimum heap H.

[0073] Calculate the corresponding scores according to the constructed heuristic function, then sort them in ascending order, select the first B sentence candidates to reinsert into the minimum heap H. The size of the minimum heap H will not exceed B to retain the first B candidates, while other candidates will be discarded.

[0074] Step F4: Adopt the sentence candidate with the lowest heuristic function score in the current minimum heap H, and compare its heuristic score with the current best character embedding vector x best and the heuristic function score corresponding to the character sequence t best If the former is lower than the latter, update the current best character embedding vector x best and the best character sequence t best to the sentence candidate with the lowest heuristic function score in H.

[0075] It should be noted that if the former is not lower than the latter, it means that the heuristic function scores corresponding to the original best character embedding vector x best and the character sequence t best are relatively low and remain unchanged.

[0076] Step F5: Determine if the second iteration has been reached. If so, output the optimal character embedding vector x. best and the best character sequence t best Otherwise, return to step F1 to continue the iteration.

[0077] For example, let the number of the second iteration be K, execute the loop K times, repeat steps F1-F3, and after the iteration is complete, select the heuristic function score h(x). best ,t best The lowest x best and t best As output.

[0078] Step S4: Determine whether the set number of first iterations has been reached. If so, output the character sequence with the lowest heuristic function score as the inverted text. Otherwise, use the character embedding vector with the lowest heuristic function score as the optimization object and return to step S2 to continue iterating.

[0079] In this embodiment, the first iteration number n is set. i After each iteration, it is determined whether the set first iteration count has been reached. Before reaching the set first iteration count, the character embedding vector with the lowest heuristic function score is selected as the optimization target for the next iteration. This process is repeated n times. i The last iteration yields t. best This is the text reconstructed in this embodiment.

[0080] This application also validates the above method through experiments using three binary text classification datasets: CoLA (Corpus of Linguistic Acceptability), SST-2 (Stanford Sentiment Treebank), and RottenTomatoes. These datasets have varying sequence lengths and complexities. The CoLA dataset consists of grammatically correct English sentences, with text lengths ranging from 5 to 9 words. The SST-2 dataset contains sentiment-labeled sentences from movie reviews, with text lengths ranging from 3 to 13 words. The RottenTomatoes dataset, from a popular movie review aggregation website, has text lengths ranging from 14 to 27 words and was also used for sentiment classification analysis of movie reviews.

[0081] The experiment compared the ILAMP method proposed in this application with three gradient inversion methods: DLG (Deep Leakage from Gradients), TAG (Transformer Attack from Gradient), and LAMP. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) was selected as the evaluation metric to assess the similarity between the generated text and the reference text. The following are the three metrics included in ROUGE:

[0082] ROUGE-1: Used to measure the degree of overlap of unary word segmentation, reflecting the keyword recovery ability.

[0083] ROUGE-2: Used to measure the degree of overlap in bigram segmentation, reflecting the model's ability to accurately recover the token order.

[0084] ROUGE-L: Evaluates the longest subsequence between the generated text and the reference text, measuring their overall structural consistency.

[0085] The experiment selected a specific version of BERT (Bidirectional Encoder Representations from Transformers). base The model serves as the target model for gradient inversion, while GPT-2 (Generative Pre-trained Transformer 2) is used as an auxiliary language model to ensure consistency with the experimental conditions of LAMP.

[0086] With the batch size of the language model set to 1, ILAMP was comprehensively evaluated against other benchmark methods on three datasets (CoLA, SST-2, and RottenTomatoes). As shown in Table 1, on the ROUGE-1 and ROUGE-L metrics, ILAMP significantly outperformed other benchmark methods on the CoLA and SST-2 datasets, only slightly lagging behind ILAMP on the RottenTomatoes dataset. On the ROUGE-2 metric, ILAMP significantly outperformed other benchmark methods on all three datasets, exceeding them by 7.92%, 10.46%, and 10.03%, respectively. This demonstrates that Seq-BS performs exceptionally well in restoring character order. Furthermore, as the text to be reconstructed becomes more complex and longer (i.e., the text length in CoLA increases from 5 to 9, in SST-2 from 3 to 13, and in RottenTomatoes from 14 to 27), ILAMP's performance improvement over benchmark methods becomes even more significant, validating the excellent performance of Seq-BS in reconstructing long texts.

[0087] Table 1

[0088]

[0089] Next, the performance of all methods was evaluated on the CoLA dataset at different batch sizes. As shown in Table 2, ILAMP outperforms other benchmark methods on all metrics, especially the ROUGE-2 metric. Notably, for the ROUGE-2 metric, ILAMP outperforms LAMP by 7.92%, 13.58%, and 18.97% for batch sizes of 1, 2, and 4, respectively. This demonstrates that as the text reconstruction difficulty increases, ILAMP's performance improvement over benchmark methods becomes more significant.

[0090] Table 2

[0091]

[0092] The experiment also compared the performance of three variant models of ILAMP to determine the impact of different components of ILAMP on overall performance. The following are the three variant models designed for this purpose:

[0093] ILAMP(B=1,K=50): This is a variant of ILAMP with beam size B set to 1 and iteration count K set to 50.

[0094] w / o Seq-BS: This is an ILAMP variant model that does not use the Seq-BS mechanism, meaning there is no discrete optimization.

[0095] w / o lm: This is an ILAMP variant that does not use an auxiliary language model.

[0096] Next, an ablation study was conducted on ILAMP and three variant models, with the batch size for the language model set to 1. As shown in Table 3, the following conclusions can be drawn:

[0097] (1) ILAMP (B=1, K=50) performed worse than ILAMP, indicating that performing sequence transformation on multiple sentence candidates at the same time can improve ILAMP’s ability to search the sequence space.

[0098] (2) w / o Seq-BS performed worse than ILAMP, indicating that the Seq-BS mechanism can significantly improve ILAMP’s ability to reconstruct text.

[0099] (3) Both w / o Seq-BS and w / o lm performed worse than ILAMP, indicating that the combination of Seq-BS and language model prior is effective.

[0100] Table 3

[0101]

[0102] ILAMP(B=1,K=50)87.732 40.120 72.958 82.443 48.793 71.157 60.51812.339 41.533w / o Seq-BS 86.924 31.621 70.430 74.284 33.680 63.344 54.70910.771 39.472

[0103]

[0104] This application proposes a novel gradient inversion method called ILAMP to extract text from gradients, demonstrating the possibility and effectiveness of attacking leaked gradient information and providing a novel privacy auditing tool for federated learning. This method extends the attack structure of alternating continuous-discrete optimization by combining Seq-BS and LAMP. In particular, a Seq-BS mechanism is proposed in the discrete optimization to mine potential relationships between sequences, enabling efficient exploration of the sequence space. By using heuristic sequence beam search to transform multiple sentence candidates, ILAMP's ability to recover character order and reconstruct long texts is significantly improved.

[0105] In another embodiment, this application also provides a language model gradient inversion apparatus for federated learning, including a processor and a memory storing a plurality of computer instructions, which, when executed by the processor, implement the steps of the above-described language model gradient inversion method for federated learning.

[0106] Specific limitations regarding the language model gradient inversion device for federated learning can be found in the limitations of the language model gradient inversion method for federated learning mentioned above, and will not be repeated here. The aforementioned language model gradient inversion device for federated learning can be implemented entirely or partially through software, hardware, or a combination thereof. It can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0107] The memory and processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory stores a computer program that can run on the processor, which implements the method in the embodiments of the present invention by running the computer program stored in the memory.

[0108] The memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory stores the program, and the processor executes the program upon receiving an execution instruction.

[0109] The processor may be an integrated circuit chip with data processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0110] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A language model gradient inversion method for federated learning, characterized in that, The language model gradient inversion method for federated learning includes: Step 1: Using Gaussian distribution, randomly initialize the first number of character embedding vectors. For each initialized character embedding vector, calculate the reconstruction loss corresponding to the target language model, and select the character embedding vector with the minimum reconstruction loss as the optimization object. Step 2: For the optimization object, perform continuous optimization with the goal of minimizing reconstruction loss to obtain the continuously optimized character embedding vector; Step 3: Map the continuously optimized character embedding vectors into character sequences. Construct a heuristic function by combining the gradient reconstruction loss of the character embedding vectors and the perplexity of the character sequences. Perform discrete optimization of sequence beam search to obtain the character embedding vectors and character sequences with the lowest heuristic function scores. Step 4: Determine if the set number of first iterations has been reached. If so, output the character sequence with the lowest heuristic function score as the inverted text. Otherwise, use the character embedding vector with the lowest heuristic function score as the optimization object and return to step 2 to continue iterating. The discrete optimization of the execution sequence beam search yields the character embedding vector and character sequence with the lowest heuristic function score, including: Step F1: Initialize a min-heap H of capacity B, where B is the beam size. Map the continuously optimized character embedding vector x to a character sequence t. Then, assign the character embedding vector x and the character sequence t to the corresponding heuristic function scores. Insert them into the min-heap H as initial sentence candidates, where the character embedding vector x and the character sequence t are set to the optimal character embedding vector. and the best character sequence The initial state; Step F2: Extract all sentence candidates from the min-heap H, and perform a preset number of discrete sequence transformations on each sentence candidate; Step F3: Sort all the sentence candidates after the transformation of the discrete sequences from low to high according to their respective heuristic function scores, and select the top B sentence candidates to re-insert into the min-heap H; Step F4: Select the sentence candidate with the lowest heuristic function score in the current min-heap H, and combine its heuristic score with the current best character embedding vector. and character sequence The corresponding heuristic function scores are compared; if the former is lower than the latter, the current best character embedding vector is updated. and the best character sequence The sentence with the lowest heuristic function score in H is the candidate. Step F5: Determine if the second iteration has been reached. If so, output the optimal character embedding vector. and the best character sequence Otherwise, return to step F2 to continue the iteration.

2. The language model gradient inversion method for federated learning according to claim 1, characterized in that, The heuristic function is expressed by the following formula: ； in, The heuristic function score is given by x, where x is the character embedding vector and t is the character sequence corresponding to the character embedding vector x. Let x be the gradient reconstruction loss of the character embedding vector with respect to the target language model. As a weighting factor to assist the language model, Let t be the perplexity of the character sequence t for the auxiliary language model.

3. The language model gradient inversion method for federated learning according to claim 2, characterized in that, The perplexity of the character sequence t for the auxiliary language model The calculation formula is as follows: ； Where n is the number of characters in the character sequence. For a given character sequence The next character The probability of occurrence.

4. The language model gradient inversion method for federated learning according to claim 1, characterized in that, The preset number of discrete sequence transformations includes at least two different discrete sequence transformations.

5. A language model gradient inversion device for federated learning, comprising a processor and a memory storing a plurality of computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 4.

Images