Large language model delearning method based on bootstrap regression

Abstract

The invention provides a large language model delearning method based on bootstrap regression, and the method comprises the steps: obtaining a data set and an irrelevant set, and dividing the data set into a forgetting set and a reserved set; identifying whether each basic unit in the forgotten set text data is a named entity, and constructing a corresponding binary mask; obtaining a pre-trained original large language model, and adding the LoRA trainable module to the original large language model to obtain a de-learning model; respectively inputting the forgetting set into the original large language model and the de-learning model for training, and calculating a loss function between the original large language model and the de-learning model; respectively inputting the irrelevant set into the original large language model and the de-learning model for training, and calculating a loss function between the original large language model and the de-learning model; and weighting the calculated loss function to obtain a final loss function. According to the method, the large language model can forget the specified training set content, and the effects of protecting privacy and preventing information leakage are achieved.

Description

Technical Field

[0001] This invention relates to the field of computer natural language processing technology, and in particular to a method for learning large language models based on bootstrap regression. Background Technology

[0002] With the increasing prevalence of large-scale pre-trained language models, the potential presence of sensitive information, misinformation, or content requiring removal within these models has become a critical issue requiring urgent attention. Traditional model update methods typically require complete retraining, which is not only computationally expensive but also often impractical in real-world applications. With the rapid development of artificial intelligence, machine forgetting learning offers a more efficient and practical solution to this problem compared to simple model replacement strategies. Machine forgetting learning aims to enable models to selectively "forget" the impact of specific data while retaining the ability to learn from other data. This is crucial for meeting data privacy regulations (such as the "right to be forgotten" under GDPR) and enhancing model security.

[0003] In recent years, research on machine forgetting learning has developed rapidly. For example, in-depth explorations have been conducted on exact forgetting, approximate forgetting, and targeted knowledge editing. For lightweight forgetting tasks, traditional parameter fine-tuning and knowledge distillation methods perform well in handling simple forgetting requests. Regarding forgetting learning for large-scale language models, current state-of-the-art methods can perform a variety of tasks, from the exact removal of individual facts to the adjustment of the entire data distribution. Instance-level forgetting tasks aim to eliminate model memories for specific input-output pairs; however, semantic-level forgetting requires more challenging criteria, involving complex and multi-layered knowledge representation structures within the model. In large-scale language model tasks, state-of-the-art forgetting algorithms can effectively remove specific sensitive information or erroneous knowledge. However, the forgetting effectiveness of these methods drops sharply when faced with highly interconnected knowledge networks and complex semantic dependencies. Furthermore, existing forgetting techniques often lead to a significant reduction in overall model performance, especially when it is necessary to keep other model functions unaffected. Summary of the Invention

[0004] The purpose of this invention is to provide a learning method based on a large language model with bootstrap regression, which does not require the construction of additional positive samples and achieves efficient knowledge forgetting by precisely adjusting the model to predict the logit value.

[0005] To achieve the above objectives, this invention provides a method for learning large language models based on bootstrapping regression, the method comprising: S11. Obtain the dataset and irrelevant set, and divide the dataset into the forget set and the retention set according to preset rules; S12. Identify whether each basic unit in the forgotten set text data is a named entity, and construct the corresponding binary mask based on the identification result; S13. Obtain the original large language model that has been pre-trained on the forget set and the retention set, add the LoRA trainable module to the original large language model, and obtain the delearning model. S14. Input the forgotten set into the original large language model and the learning model respectively for training, and calculate the loss function between the original large language model and the learning model. S15. Input the irrelevant set into the original large language model and the delearning model respectively for training, and calculate the loss function between the original large language model and the delearning model. S16. Weight the loss functions calculated in steps S14 and S15 to obtain the final loss function; S17. Update the parameters of the large language model by optimizing the algorithm to minimize the final loss function, and evaluate the performance of the updated large language model on a preset validation set.

[0006] Furthermore, step S12 specifically includes: S21. Identify whether each basic unit in the forgotten set text data is a named entity by using named entity recognition, and obtain the type label of each named entity and its position information in the text. S22. Based on the location information, generate a binary mask corresponding to the forgotten set text data; S23. Establish a mapping relationship between the binary mask, the named entity, and the forgotten set text data.

[0007] Furthermore, in step S22, the binary mask is used to indicate whether each basic unit in the forgotten set text data belongs to the identified named entity. If the basic unit in the forgotten text data is a named entity, then the binary mask at the corresponding position is 1; If the basic unit in the forgotten set text data is a non-named entity, then the binary mask at the corresponding position is 0.

[0008] Furthermore, the LoRA trainable module can be turned on or off. When the LoRA trainable module is turned on, a learnable model is obtained. When the LoRA trainable module is turned off, the original large language model remains unchanged.

[0009] Furthermore, step S14 specifically includes: S31. Input the forget set into the original large language model for training to obtain the original logit; S32. Input the forget set into the learning model for training, and obtain the learned logit; S33. Modify the logit value of the corresponding label in the original logit to the average value of the current basic unit logit to obtain the modified logit; S34. The loss function between the logit and the modified logit is learned by calculating the weighted squared error, and the bootstrap regression loss function is obtained and minimized.

[0010] Furthermore, step S14 also includes: On the forget set, if the current basic unit is a named entity, then calculate the loss function between learning the logit and modifying the logit; If the current basic unit is a non-named entity, then calculate the loss function between the learned logit and the original logit.

[0011] Furthermore, step S15 specifically includes: S41. Input the irrelevant set into the original large language model for training to obtain the original irrelevant logit; S42. Input the irrelevant set into the learning model to train it, and obtain the learning irrelevant logit. S43. The irrelevant loss function is obtained by calculating the loss function between the original irrelevant logit and the learned irrelevant logit using weighted squared error.

[0012] Furthermore, the performance of the updated large language model is evaluated on a pre-defined validation set, specifically including: If the performance of the large language model parameters on the retention set remains within a preset range and the performance on the forgetting set is below a preset threshold, then the delearning of the large language model is complete. If the preset conditions are not met, but the preset maximum number of iterations has been reached, then the learning of the large language model is completed.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a learning method for large language models based on bootstrapping regression. Its key feature is the precise control of output probabilities across all label dimensions through a bootstrapping logit regression mechanism. This effectively addresses the technical bottleneck of existing technologies that constrain only a single erroneous label, leading to excessive forgetting. This ensures the model maintains the integrity of the overall knowledge structure while forgetting target knowledge. By accurately identifying and distinguishing between named and unnamed entity words, and performing learning operations only on key information, the semantic representation of function words is effectively preserved, significantly improving the grammatical correctness and fluency of language generation, fundamentally avoiding unnecessary loss of model utility. Furthermore, by employing a model alignment technique based on irrelevant sets, the invention overcomes the dependence of traditional methods on the retained set data. Model parameter optimization is achieved by constructing a third-party dataset unrelated to both the forgotten and retained sets, ensuring effective forgetting while completely avoiding the access restrictions of the retained set. This significantly expands the applicability of this method in privacy-sensitive scenarios and data-constrained environments. This invention enables large language models to forget specified training set content, thus protecting privacy and preventing information leakage. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Figure 1 A comparison diagram of the present invention and the prior art provided for embodiments of the present invention; Figure 2 This invention provides a schematic diagram of a method for learning large language models based on bootstrap regression, as provided in an embodiment of the invention. Figure 3 A flowchart illustrating a framework for a large language model learning method based on bootstrap regression, provided in this embodiment of the invention; Figure 4 Comparison chart of experimental results provided for embodiments of the present invention; Figure 5 The diagram illustrates the generation effect of the present invention and the comparative method on a 10% partitioned forgetting set, as provided in the embodiments of the present invention. Figure 6 The diagram illustrates the generation effect of the present invention and the comparison method on a 10% reserved set, as provided in the embodiments of the present invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0016] The core objective of this invention is to significantly reduce the model's performance on the forget set (denoted as...). While generating the probability of unwanted answers on the model, the goal is to maximize the retention of the model within the retain set (denoted as ). Model utility refers to the model's ability to generate accurate and coherent responses in various natural language processing tasks. The forget set refers to the specific subset of data in the pre-training dataset that the user specifies to be forgotten, while the retention set refers to the training data excluding the forget set.

[0017] This invention uses the TOFU dataset as a benchmark platform. The TOFU dataset contains 200 virtual characters, each configured with 20 specific question-answer pairs. In the evaluation system, the performance retention of the model on the retention set is quantitatively evaluated using standard metrics provided by the TOFU dataset; simultaneously, the Unlearn Quality metric is introduced to objectively measure the forgetting effect of the model on the forgetting set.

[0018] Reference Figure 1 This invention records three key shortcomings in existing technologies and proposes innovative solutions to address them, as follows: 1. Analysis of the defects of existing technology Logit loss level defects: such as Figure 1 As shown in (a), existing gradient-ascending forgetting methods only constrain the output probability of a single "incorrect" label, without effectively constraining the remaining labels, leading to over-forgetting of the model, which manifests as a sharp decline in model utility.

[0019] Sequence-level defects: such as Figure 1 As shown in (b), sentence generation involves multi-word sequence prediction, in which some function words (such as prepositions and articles) do not need to be forgotten. However, existing methods cannot distinguish between key information and non-key information, resulting in unnecessary information loss.

[0020] Data-level deficiencies: such as Figure 1 As shown in (c), existing methods require direct access to the reserved set for model alignment, which is difficult to implement in scenarios with strict privacy protection or restricted data access.

[0021] 2. Innovation of this invention To address the aforementioned shortcomings, this invention proposes a three-level collaborative optimization framework: Logit loss optimization: An innovative bootstrap logit regression mechanism is introduced. During training on the forgotten dataset, for each predicted word, the value of the corresponding target dimension in the original model's logit vector is adjusted to the average of all dimensions; the student model is initialized, and its output distribution is aligned with the modified logit value, effectively preventing excessive forgetting.

[0022] Sequence-level precise forgetting: Integrating named entity recognition (NER) technology, it automatically identifies and marks non-named entity words; during the forgetting process, it only performs learner operations on named entity-related words, retaining the original semantic representation of function words and ensuring the grammatical correctness of language generation.

[0023] Data access optimization: An irrelevant set is creatively used instead of the retained set for model alignment. The irrelevant set is defined as a third-party dataset that belongs neither to the forgotten set nor the retained set. This design fundamentally solves the problem of restricted access to the retained set, significantly improving the method's practicality and applicability.

[0024] Reference Figure 2 and Figure 3 This embodiment provides a method for learning large language models based on bootstrap regression, the method including: S11. Obtain the dataset and irrelevant set, and divide the dataset into the forgotten set and the retained set according to preset rules.

[0025] This embodiment downloads the TOFU dataset and uses the officially provided partitioning method to obtain the forgetting and retention sets. Based on different forgetting rates, three forgetting and retention sets are constructed for 1%, 5%, and 10% forgetting percentages, respectively. Each data point consists of a question and its corresponding answer, used for model training and evaluation. For example: Question: [INST] What was one of the challenges Hsiao Yun-Hwa faced in herearly writing career? [ / INST] Answer: In her early career, Hsiao Yun-Hwa faced challenges to be recognized as a credible author in the leadership genre due to her young age and perceived lack of experience. Obtain the irrelevant set (denoted as...) This invention downloads and uses the Cosmos QA dataset as an irrelevant set for aligning the model to be learned with the original model. This can be represented as:

[0026] Introducing irrelevant sets is to avoid access to retained data, thus adapting to more demanding application scenarios. Setting multiple forgetting ratios can simulate the forgetting requirements of various situations in real-world applications, better reflecting the performance of the learning algorithm under different levels of difficulty.

[0027] S12. Identify whether each basic unit in the forgotten set text data is a named entity, and construct a corresponding binary mask based on the identification result, specifically including: S21. Identify whether each basic unit in the forgotten set text data is a named entity by using named entity recognition, and obtain the type label of each named entity and its position information in the text.

[0028] S22. Based on location information, generate a binary mask corresponding to the forgotten set text data, wherein the binary mask is used to indicate whether each basic unit in the forgotten set text data belongs to the identified named entity. If the basic unit in the forgotten text data is a named entity, then the binary mask at the corresponding position is 1; If the basic unit in the forgotten set text data is a non-named entity, then the binary mask at the corresponding position is 0.

[0029] S23. Establish a mapping relationship between the binary mask, the named entity, and the forgotten set text data.

[0030] In this embodiment, for the data in the forgetting set, the Named Entity Recognition (NER) module in the NLTK toolkit is used to automatically identify named entities in the answers. The automatic identification of named entities in the answers is integrated into the dataset loading process, resulting in a rapid response and requiring no additional processing time. A binary mask is constructed based on the identification results: if the basic unit (word) belongs to a named entity, the corresponding mask value is 1, indicating that the word needs to be learned; if it is a non-named entity word, the mask value is 0, indicating that it does not need to be learned, and its original semantic information is preserved. For example: Answer: In her early career, Hsiao Yun-Hwafacedchallengesto be recognized as a credible author in the leadership genre due to her young age and perceived lack of experience. MASK=[0,0,0,0,0,1,1,0,1,0,0,0,0,0,0,1,0,0,1,1,0,0,0,0,1,0,0,1,0,1,0] Bold words are marked as named entities, with a corresponding MASK value of 1. Named entity recognition can accurately identify words with actual semantic meaning in forgotten data. Learning from named entities can prevent other words from being mistakenly forgotten, improving the algorithm's retention of the original model's utility and its robustness.

[0031] S13. Obtain the original large language model that has been pre-trained on the forget set and the retention set, add the LoRA trainable module to the original large language model, and obtain the delearning model.

[0032] In a preferred embodiment, the LoRA trainable module can be turned on or off. When the LoRA trainable module is turned on, a delearning model is obtained. When the LoRA trainable module is turned off, the original large language model remains unchanged.

[0033] In this embodiment, the original model weights are downloaded, and the weights of the TOFU official pre-trained LLAMA-2-7B and PHI-1.5B models are obtained through Hugging Face's AutoModel interface, denoted as the original large language model. The original large language model was implemented on a complete corpus (including the forgetting set). and Reserved set Pre-training is performed on the LoRA module, fully preserving the knowledge from all training data. The LoRA trainable module is then added to the original large language model. The LoRA trainable module can be enabled or disabled; enabling LoRA allows for model learning, denoted as... .

[0034] S14. Input the forget set into the original large language model and the learning model respectively for training, and calculate the loss function between the original large language model and the learning model, specifically including: S31. Input the forget set into the original large language model for training to obtain the original logit; S32. Input the forget set into the learning model for training, and obtain the learned logit; S33. Modify the logit value of the corresponding label in the original logit to the average value of the current basic unit logit to obtain the modified logit; S34. The loss function between the logit and the modified logit is learned by calculating the weighted squared error, and the bootstrap regression loss function is obtained and minimized.

[0035] In this embodiment, the probability distribution Logit generated by the original large language model when inferring a word is a vector of length equal to the dictionary, and the sum of the dimensions of Logit is 1. During inference, data from the forget set is used... Inputting the raw large language model without LoRA activation yields the raw logit, denoted as... The expression is:

[0036] After enabling LoRA, the unlearned model learns the logit, denoted as... The expression is:

[0037] Revise The logit value corresponding to the label is adjusted to the average logit value of the current word. The modified logit is denoted as . The expression is:

[0038] in, This is a custom modification function.

[0039] Calculated using WSE (Weighted Squared Error) and The loss between them is expressed as:

[0040] Where V is the dimension of the logit. For the input logit, For the target logit, label This indicates the index of the current word in the dictionary in the correct answer. This is a hyperparameter used to balance the contributions of modified and unmodified labels to the loss function. Because the dictionary of a language model is very large, the loss value of a single label can easily be overwhelmed by the loss values ​​of a large number of other labels.

[0041] The expression for minimizing the error of the current word using WSE is:

[0042] This invention abandons the traditional gradient ascent loss function. The gradient ascent loss function only requires the logit value of the corresponding label to be as low as possible, without constraining the other labels, leading to excessive forgetting. By specifying an ideal logit value for each label, excessive forgetting is avoided. Furthermore, by forcing the unlearned model to learn a logit that approximates the modified logit of the original large language model, compared to traditional supervised fine-tuning (SFT) which uses one-hot vectors as supervision signals, this method inherently carries more information and helps the model train more stably.

[0043] In a preferred embodiment, step S14 further includes: On the forget set, if the current basic unit is a named entity, then calculate the loss function between learning the logit and modifying the logit.

[0044] If the current basic unit is a non-named entity, then calculate the loss function between the learned logit and the original logit.

[0045] In this embodiment, the traditional training is improved by combining step S12 (named entity recognition) and step S14 (bootstrapping logit regression) to achieve fine-grained unlearning control. Specifically, in the logit dimension, the unlearned logit of the unlearned model is aligned with the modified logit of the original large language model. In the sequence dimension, unlearning is performed only on named entity words, while other words are retained. Specifically, in the forget set, if the current word's MASK=1, unlearning is performed, and the WSE loss between the unlearned logit and the modified logit is calculated. Otherwise, the WSE loss between the unlearned logit and the original logit is calculated, expressed as:

[0046] in, Let T represent the loss function applied to the forgetting set, and let T represent the number of words in the answer. This represents the result of named entity recognition for the i-th word.

[0047] S15. Input the irrelevant set into the original large language model and the delearning model respectively for training, and calculate the loss function between the original large language model and the delearning model, specifically including: S41. Input the irrelevant set into the original large language model for training to obtain the original irrelevant logit.

[0048] S42. Input the irrelevant set into the learning model for training to obtain the learning irrelevant logit.

[0049] S43. The irrelevant loss function is obtained by calculating the loss function between the original irrelevant logit and the learned irrelevant logit using weighted squared error.

[0050] In this embodiment, most existing methods align the unlearn model and the original large language model on the retained set. However, in some cases, the retained set data is unavailable. Therefore, an irrelevant set is introduced as the alignment dataset. The data from the irrelevant set... Input the original large language model Training is performed to obtain the original, unrelated logit, denoted as... Input unlearn model In the middle, we learn unrelated logit, denoted as The expression is:

[0051]

[0052] Then, WSE is used to calculate the loss between the original irrelevant logit and the learned irrelevant logit, expressed as:

[0053] S16. Weight the loss functions calculated in steps S14 and S15 to obtain the final loss function, which is used for updating the large language model.

[0054] In this embodiment, the loss functions calculated in steps S14 and S15 are weighted to obtain the final loss function used for model updating, denoted as... The expression is:

[0055] Using the adamW optimizer, the learning rate was set to 0.0001, and fine-tuning was performed for 32 epochs. This embodiment provides two unique hyperparameters. Setting it to 1 is used to balance the forgetting loss value and the alignment loss value. Set to 2048 to balance the weights of the modified Label and other label loss values.

[0056] S17. Update the parameters of the large language model through an optimization algorithm to minimize the final loss function, and evaluate the performance of the updated large language model on a pre-defined validation set, specifically including: If the performance of the large language model parameters on the retention set remains within a preset range and the performance on the forgetting set is below a preset threshold, then the delearning of the large language model is complete. If the preset conditions are not met, but the preset maximum number of iterations has been reached, then the learning of the large language model is completed.

[0057] Figure 4 This paper presents experimental results obtained by this invention under three different splitting ratios on the TOFU dataset. An ideal model refers to a model obtained by fully retraining on the retained set, which exhibits the best learning performance. Figure 4 The term "invention" in this invention refers to the method proposed in this patent, and the bolded text indicates the best performance of the corresponding column.

[0058] Figure 5 and Figure 6 This is an example of the generation effect of the present invention and the comparative method on the 10% forget set and retain set. On the forget set, the method of the present invention can accurately forget the named entity "Hsiao Yun-Hwa", answer the question about "Hwei", and is unrelated to the original answer. Other methods, on the other hand, have problems such as invalid duplicate answers, evasive answers, and answering the original content.

[0059] While the content generated by this invention differs somewhat from the original answer in the retention set, the meaning remains consistent. Other methods suffer from problems such as failing to provide correct answers, providing repetitive answers, providing meaningless answers, and avoiding answers. In comparison, the method of this invention has a better forgetting effect and retains the model utility of the original model well.

[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A large language model unlearning method based on bootstrap regression, characterized in that, The method comprises: S11, acquiring a data set and an irrelevant set, and dividing the data set into a forgetting set and a retaining set through a preset rule; S12, identifying whether each basic unit in the text data of the forgetting set is a named entity, and constructing a corresponding binary mask based on the identification result; S13, acquiring an original large language model pre-trained on the forgetting set and the retaining set, adding a LoRA trainable module to the original large language model to obtain a forgetting learning model; S14, inputting the forgetting set into the original large language model and the forgetting learning model respectively for training, and calculating a loss function between the original large language model and the forgetting learning model; S15, inputting the irrelevant set into the original large language model and the forgetting learning model respectively for training, and calculating a loss function between the original large language model and the forgetting learning model; S16, weighting the loss functions calculated in steps S14 and S15 to obtain a final loss function; S17, updating the parameters of the large language model through an optimization algorithm to minimize the final loss function, and evaluating the performance of the updated large language model on a preset validation set.

2. The bootstrap regression-based large language model unlearning method according to claim 1, characterized in that, Step S12 specifically comprises: S21, identifying whether each basic unit in the text data of the forgetting set is a named entity through named entity recognition, and acquiring the type label of each named entity and its position information in the text; S22, generating a binary mask corresponding to the text data of the forgetting set based on the position information; S23, establishing a mapping relationship between the binary mask, the named entity and the text data of the forgetting set.

3. The bootstrap regression-based large language model unlearning method according to claim 2, characterized in that, In step S22, the binary mask is used to indicate whether each basic unit in the text data of the forgetting set belongs to the identified named entity; If the basic unit in the text data of the forgetting set is a named entity, the binary mask at the corresponding position is 1; If the basic unit in the text data of the forgetting set is a non-named entity, the binary mask at the corresponding position is 0.

4. The bootstrap regression-based large language model unlearning method according to claim 1, wherein, The LoRA trainable module can be turned on or off. When the LoRA trainable module is turned on, the forgetting learning model is obtained. When the LoRA trainable module is turned off, the original large language model remains unchanged.

5. The bootstrap regression-based large language model unlearning method according to claim 1, wherein, Step S14 specifically comprises: S31, inputting the forgetting set into the original large language model for training to obtain an original logit; S32, inputting the forgetting set into the forgetting learning model for training to obtain a forgetting learning logit; S33, modifying the logit value corresponding to the label in the original logit to the average value of the current basic unit logit to obtain a modified logit; S34, calculating the loss function between the forgetting learning logit and the modified logit through weighted squared error to obtain a bootstrap regression loss function, and minimizing the bootstrap regression loss function.

6. The bootstrap regression-based large language model unlearning method according to claim 5, characterized in that, Step S14 further comprises: On the forgetting set, if the current basic unit is a named entity, the loss function between the forgetting learning logit and the modified logit is calculated; If the current basic unit is a non-named entity, the loss function between the forgetting learning logit and the original logit is calculated.

7. The bootstrap regression-based large language model unlearning method according to claim 1, characterized in that, Step S15 specifically comprises: S41, inputting the irrelevant set into the original large language model for training to obtain an original irrelevant logit; S42, inputting the irrelevant set into the unlearning model for training to obtain an unlearning irrelevant logit; S43, calculating a loss function between the original irrelevant logit and the unlearning irrelevant logit through a weighted squared error to obtain an irrelevant loss function.

8. The bootstrap regression-based large language model unlearning method according to claim 1, characterized in that, The performance of the updated large language model is evaluated on a preset verification set, specifically including: If the performance of the large language model parameters on the reserved set remains within a preset range and the performance on the forgetting set is lower than a preset threshold, the unlearning of the large language model is completed; If the preset conditions are not met, but the preset maximum iteration number has been reached, the unlearning of the large language model is completed.

Images