Forgetting learning method and device for large language model

Abstract

The embodiment of the invention discloses a forgetting learning method and device for a large language model. The method comprises the steps that firstly, a first large language model and a forgetting sample set are obtained, the first large language model is initially a fine-tuning model obtained by fine-tuning a fine-tuning sample set, and the forgetting sample set is a subset of the fine-tuning sample set; then, for any first forgotten sample, a plurality of similar samples are determined based on a second large language model, and each similar sample and the first forgotten sample have similar semantics but different expressions; processing the first forgotten sample by using a first large language model to obtain a first hidden layer representation; then, training loss is determined, the training loss is negatively correlated to the distance between the first hidden layer representation and the hidden layer representation of each similar sample and is positively correlated to the distance between the first hidden layer representation and the random vector, and the hidden layer representation of each similar sample is obtained based on a fine tuning model; and then training the first large language model by using the training loss to realize forgetting learning.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of machine learning, and in particular to a large language model unlearning method and device. BACKGROUND

[0002] In recent years, large language models (LLMs) have been widely applied in many fields due to their excellent ability in natural language understanding and generation, including intelligent customer service, machine translation, content creation, knowledge question answering, etc.

[0003] However, the large amount of data consumed by LLMs in the training process usually contains private or sensitive information, thereby causing key problems in privacy, copyright and data security, thus giving rise to the research of large language model unlearning (LLMU).

[0004] LLMU is essentially a model information erasure method, and the core goal is to accurately remove the influence of target data (a small amount of training samples) without significantly affecting the overall performance of the model, thereby avoiding the high cost of training the model from scratch. The training data that needs to be forgotten usually includes sensitive data related to privacy, harmful speech, and data that needs to be deleted due to copyright issues, etc.

[0005] However, the current implementation scheme of LLMU is difficult to meet the higher requirements in actual applications, for example, more effectively removing the information that needs to be erased while ensuring the availability of the model to improve the unlearning effect. SUMMARY

[0006] Embodiments of the present specification describe a large language model unlearning method and device that can meet the higher requirements in the above actual applications.

[0007] According to a first aspect, a large language model unlearning method is provided. The method comprises:

[0008] The first large language model and a forgetting sample set are obtained, the first large language model is initially a fine-tuned model obtained by fine-tuning a fine-tuning sample set, and the forgetting sample set is a subset of the fine-tuning sample set. For any first forgetting sample, a plurality of similar samples are determined based on a second large language model, wherein each similar sample has similar semantics but different expressions with the sample input of the first forgetting sample. The first forgetting sample is processed by using the first large language model to obtain a first hidden layer representation. A training loss is determined, which is negatively correlated with the distance between the first hidden layer representation and the hidden layer representation of each similar sample, and is positively correlated with the distance between the first hidden layer representation and a random vector; the hidden layer representation of each similar sample is obtained based on the fine-tuned model. The first large language model is trained by using the training loss to achieve forgetting learning.

[0009] In one embodiment, for any first forgetting sample, a plurality of similar samples are determined based on a second large language model, comprising: generating similar samples of the first forgetting sample by using a third large language model to obtain a plurality of first candidate samples; and determining the plurality of similar samples based on the plurality of first candidate samples.

[0010] Further, in one specific embodiment, the plurality of similar samples are determined based on the plurality of first candidate samples, comprising: performing target processing by taking the first forgetting sample and each sample in the plurality of first candidate samples as a target sample respectively: performing multiple prediction processing on the target sample by using the third large language model to obtain a plurality of prediction results corresponding thereto; and determining an index value for measuring the concentration degree of the plurality of prediction results. For each first candidate sample, if the index value corresponding thereto is greater than or equal to the index value corresponding to the first forgetting sample, it is included in a second candidate sample set. The plurality of similar samples are determined based on the second candidate sample set.

[0011] Further, in one example, the index value for measuring the concentration degree of the plurality of prediction results is determined, comprising: for each prediction result, determining the prediction probability of the prediction result based on the prediction probability of each token therein; and accumulating and summing the prediction probabilities of the plurality of prediction results, and taking the sum value obtained as the index value.

[0012] In one example, the plurality of similar samples are determined based on the second candidate sample set, comprising: filtering the plurality of similar samples from the second candidate sample set by using a clustering algorithm.

[0013] In one embodiment, the plurality of similar samples are determined based on the plurality of first candidate samples, comprising: filtering the plurality of similar samples from the plurality of first candidate samples by using a clustering algorithm.

[0014] Further, in one example, the filtering the plurality of first candidate samples to obtain the similar samples comprises: determining an encoding vector corresponding to a sample input of each of the plurality of first candidate samples; performing adaptive clustering on the plurality of encoding vectors, and selecting a first candidate sample corresponding to a vector closest to a clustering center as a similar sample.

[0015] In one embodiment, the first hidden layer representation and the hidden layer representation of each similar sample each comprises an output of an lth hidden layer in a respective corresponding model.

[0016] In one embodiment, before determining the training loss, the method further comprises: processing a first reserved sample by using the first large language model to obtain a second hidden layer representation; the first reserved sample belongs to a difference set between the fine-tuning sample set and the forgetting sample set. Wherein, the training loss is also positively related to a distance between the second hidden layer representation and a third hidden layer representation, wherein the third hidden layer representation is obtained by processing the first reserved sample by using the fine-tuning model.

[0017] According to a second aspect, a forgetting learning device of a large language model is provided. The device comprises:

[0018] a model data obtaining module configured to obtain a first large language model and a forgetting sample set, the first large language model being initially a fine-tuning model obtained by fine-tuning a fine-tuning sample set, and the forgetting sample set being a subset of the fine-tuning sample set; a sample determining module configured to determine, for any first forgetting sample, a plurality of similar samples based on a second large language model, wherein each similar sample has similar semantics but different expressions with a sample input of the first forgetting sample; a sample processing module configured to process the first forgetting sample by using the first large language model to obtain a first hidden layer representation; a loss determining module configured to determine a training loss, which is negatively related to a distance between the first hidden layer representation and a hidden layer representation of each similar sample, and positively related to a distance between the first hidden layer representation and a random vector; the hidden layer representation of each similar sample being obtained based on the fine-tuning model; and a model training module configured to train the first large language model by using the training loss to achieve forgetting learning.

[0019] According to a third aspect, a computer-readable storage medium is provided, which stores a computer program, when the computer program is executed in a computer, the computer program causes the computer to execute the method provided in the first aspect.

[0020] According to a fourth aspect, a computing device is provided, which comprises a memory and a processor, the memory stores executable code, and the processor executes the executable code to implement the method provided in the first aspect.

[0021] To sum up, by using the above method and device disclosed in the embodiments of the present specification, the contrast learning technology is first used to solve the large language model unlearning problem, specifically, the latent representation (hidden layer representation) of the sample to be forgotten is pulled away from the similar sample in distance, and the latent representation of the sample to be forgotten is aligned with a random vector, thereby effectively improving the erasing effect and robustness of the model to specific information. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description are briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 An example graph of the forgetting effect of the forgetting model on the original input and similar input;

[0024] Figure 2 An experimental data schematic diagram showing the relationship between CLP and EUB representing the forgotten sample;

[0025] Figure 3 A schematic diagram of the method flow steps for screening high-quality similar samples disclosed in the embodiments of the present specification;

[0026] Figure 4 A schematic diagram of the flow steps of the large language model forgetting learning method disclosed in the embodiments of the present specification;

[0027] Figure 5 A schematic diagram of the functional structure of the large language model forgetting learning device disclosed in the embodiments of the present specification. DETAILED DESCRIPTION

[0028] Next, the scheme provided by the present specification is described in combination with the drawings.

[0029] As described above, the current LLMU scheme is difficult to meet the higher requirements in actual application. Specifically, the existing LLMU scheme mainly refers to the model training idea, and some reverse optimization objectives or optimization methods are constructed in the loss space to achieve the purpose of unlearning (or called unlearning, forgetting learning). For example, the gradient-based optimization method as the mainstream method, the gradient ascent is applied to remove the knowledge learned from specific data.

[0030] However, these methods are difficult to effectively remove the information to be erased from the LLM, that is, it is difficult to guarantee the forgetting effect of the LLM.

[0031] Further, the applicant finds that when unlearning for a specific training sample, even if the LLM successfully forgets the information of the specific training sample, it cannot be guaranteed that the LLM will not leak key information in the content generated when processing queries related to the sample. The following illustrates this problem through a specific example.

[0032] As shown in Figure 1 , the sample input "What is the full name of the author born in * year * month * day * town?" is selected from the forgetting data set to query the LLM after forgetting learning, and the result shows that when the original query is used, the model output (B) does not match the correct answer (A). However, when using the query "What is the full name of the novelist born in * year * month * day * town?" with the same semantics but slightly different expression, the LLM outputs the correct answer.

[0033] This phenomenon shows that even if the LLM shows unlearning effect for a specific training sample, it does not mean that the model has truly forgotten the sample information - because when using related queries, the model can still leak unexpected information.

[0034] On the other hand, contrastive learning has been proven to be a powerful representation learning framework. Specifically, contrastive learning is a self-supervised learning method, the core idea of which is to learn to distinguish similar and dissimilar data, thereby learning meaningful feature representations.

[0035] Based on the above observations and analysis, the applicant proposes an improvement scheme, which, when performing unlearning, can also construct similar samples that are semantically similar (or the same) but have different expressions than the specific training sample, thereby using contrastive learning to increase the gap between the feature representations (or called hidden layer representations, latent representations) of the specific training sample and the similar samples, and to reduce the gap between the feature representations of the specific training sample and random samples, to train the LLM, thereby ensuring the effective removal of the target information and achieving high-quality unlearning.

[0036] The following describes the specific implementation of the above improvement scheme.

[0037] For intuitive description, first give the mathematical symbols mainly used in the introduction of the improved LLMU scheme. The goal of LLMU is to remove specific knowledge from the LLM while maintaining its overall performance. Specifically, given a model M fine-tuned on the downstream task on the training data set w , the model (or simply the forgetting model) M θ to be unlearned is initialized as M w , and M θThe model parameters enable it to perform well on the forgotten dataset after forgetting learning is complete. For all samples (q, y), the response to query q is different from the corresponding response to y. Meanwhile, M... θ For the preserved dataset (satisfy Queries in the ) maintain high-quality responses.

[0038] In short, LLMU requires a base model M that has been fine-tuned. w The new forgetting model M is obtained through adjustment. θ This makes M θ Acting as if nothing was used Trained, and at the same time Maintain with M w Comparable performance.

[0039] The above outlines some of the mathematical symbols primarily involved in the LLMU scheme. Next, we will first introduce the construction of similar samples, and then explain how to construct a contrastive loss using similar samples, which is then used to train the model to achieve forgetting learning.

[0040] I. Construction of Similar Samples

[0041] For forgotten datasets (or forgotten sample sets) For any first forgotten sample, several similar samples need to be constructed for it. It should be noted that the "first" in "first forgotten sample," and similar terms like "first," "second," etc., used elsewhere in the text, are for distinguishing similar items and do not have any ranking or other limiting function. "Several" in the text refers to one or more. Furthermore, each similar sample and its corresponding forgotten sample have the same or similar semantics in their sample input but different expressions. This means that in fine-tuning model M... w The output is consistent or the core information is the same.

[0042] In one construction method, it is proposed that similar samples can be constructed manually, and correspondingly, similar samples can be obtained by receiving similar samples input by staff.

[0043] In another construction approach, it is proposed to use LLM to construct similar samples in order to reduce the construction cost.

[0044] Specifically, for the first forgotten sample, this sample and the task description are jointly input into the second large language model F. θ This yields similar samples. It should be noted that the second largest language model, F... θ It can be used with fine-tuning model M w They can be the same, or they can be different.

[0045] Exemplarily, the task description can be: please rephrase the given sample so that the rephrased sample has the same or similar semantics as the original sample in the sample input part but different expressions, and the same expected output part, and the number of rephrased samples is not less than 10.

[0046] The above method of generating similar samples by LLM hopes to construct as many similar samples as possible to ensure the forgetting effect. However, the overall quality of the similar samples thus constructed is limited, which may lead to redundancy and excessive computational overhead in the forgetting learning process. Therefore, the applicant proposes to screen high-quality samples from the similar samples generated by LLM for forgetting learning, thereby reducing the computational overhead while ensuring the forgetting learning effect.

[0047] The following introduces two screening methods, which will screen the similar samples generated by the second large language model F θ The generated multiple similar samples are used as multiple first candidate samples, and similar samples for constructing contrastive loss are screened. The two screening methods can be used independently or in combination.

[0048] 1) Screening method based on the concentration of predicted output

[0049] The core idea of this screening method is: if the forgetting effect of the model after forgetting learning on a certain sample is limited (i.e., the model does not completely "forget" the sample and still retains some memory or familiarity), the prediction output of the model on the sample will be more confident, which is manifested as lower perplexity, i.e., more concentrated prediction probability distribution.

[0050] In order to verify this idea, the applicant conducted experimental analysis, and specifically calculated the following indexes of the model after forgetting learning on different samples: 1) Cumulative Log-Probability (CLP): the sum of the logarithm of the prediction probability of the model on the sample, reflecting the prediction confidence of the model on the sample. 2) Expectation Upper Bound (EUB): an upper bound index that measures the concentration of the prediction result, the higher the EUB, the more concentrated the prediction distribution.

[0051] Referring to Figure 2 , the experimental results show that the CLP and EUB of different forgetting samples are significantly positively correlated, i.e.: the higher the CLP (the higher the prediction confidence) → the higher the EUB (the more concentrated the prediction result). Conversely, the lower the CLP (the lower the prediction confidence) → the lower the EUB (the more dispersed the prediction result).

[0052] This correlation supports the above core idea: when the forgetting effect of the model on a certain sample is limited (CLP is high), its prediction is more concentrated (EUB is high), and the perplexity is lower; when the forgetting effect of the model on a certain sample is better (CLP is low), its prediction is more dispersed (EUB is low), and the perplexity is higher.

[0053] Based on the above core idea, the following is proposed Figure 3 The screening process steps are shown in FIG. 1. It should be noted that Figure 3 The execution subject of step S1 can be any device, platform, or device cluster with computing and processing capabilities. Figure 3 The steps include the following steps:

[0054] Step S31, taking each sample in the first forgetting sample and the plurality of first candidate samples as a target sample, performing target processing: using a third large language model to perform multiple prediction processing on the target sample to obtain a plurality of corresponding prediction results; determining an index value for measuring the concentration of the plurality of prediction results.

[0055] It should be noted that the third large language model can be the same as the fine-tuned model M w or the second large language model F θ , or a large model different from both.

[0056] The calculation method of the above concentration index can be flexibly set. In one embodiment, first, for each prediction result in the plurality of prediction results, the result prediction probability of the prediction result is determined based on the token prediction probability of each token in the prediction result. It should be understood that the token prediction probability of each token can be directly extracted from the output layer of the third large language model; in addition, the token prediction probability of each token can be multiplied to obtain the multiplication result as the result prediction probability. Further, the plurality of result prediction probabilities corresponding to the plurality of prediction results are summed as the index value corresponding to the target sample.

[0057] In another embodiment, the similarity (such as cosine similarity, Jaccard similarity, etc.) between each two prediction results in the plurality of prediction results can be calculated, and the average value is taken as the concentration index value corresponding to the target sample.

[0058] From the above, the concentration index value corresponding to each sample in the first forgetting sample and the plurality of first candidate samples can be obtained.

[0059] Step S32, for each first candidate sample, if the index value corresponding thereto is greater than or equal to the index value corresponding to the first forgetting sample, it is classified as a similar sample for constructing a contrastive loss.

[0060] It can be understood that, assuming that a certain first candidate sample has a greater concentration index value than the first forgetting sample, in combination with the foregoing core idea, it can be known that the LLM has a deep memory for the first candidate sample, and therefore it can be selected as a high-quality similar sample to participate in forgetting learning.

[0061] The above Figure 3 The similar sample screening method based on the above core idea is introduced. Further, a complete example from constructing candidate samples to screening similar samples is introduced below, which can be specifically referred to Algorithm 1 shown below.

[0062]

[0063] It should be noted that in Algorithm 1 shown above: the fine-tuning M w is used as the third large language model described above, the forgetting sample q corresponds to the first forgetting sample described above, the K candidate questions correspond to the plurality of first candidate samples described above, and the m answers correspond to the plurality of prediction results; lines 1-4 are for calculating the cumulative sampling probability corresponding to q (corresponding to the concentration index value described above), lines 6-9 are for calculating the cumulative sampling probability corresponding to each candidate question, lines 10-11 correspond to the screening of similar samples by comparing the cumulative sampling probabilities in step S32 described above, and line 14 returns the is used to construct the contrastive loss.

[0064] The above introduces the screening method based on the concentration of the prediction output.

[0065] 2) Screening method based on clustering algorithm

[0066] Specifically, the encoding vectors corresponding to the sample inputs of the plurality of first candidate samples are first determined. In an embodiment, a semantic embedding model can be used to encode the sample input. Exemplarily, the semantic embedding model can adopt a general embedding model (BAAI General Embedding, BGE).

[0067] Then, the plurality of encoding vectors are clustered to obtain a plurality of class clusters, so that the first candidate samples corresponding to the encoding vectors determined as the cluster centers or closest to the cluster centers are classified into a plurality of similar samples.

[0068] In an embodiment, a Dirichlet Process Mixture Model (DPMM) is used to adaptively cluster the plurality of encoding vectors, and the first candidate sample corresponding to the vector closest to the cluster center is selected to be classified into the plurality of similar samples.

[0069] It should be noted that the purpose of selecting DPMM is to utilize its adaptive clustering capability. DPMM does not need to specify the number of clustering centers in advance, and can adaptively determine the optimal number of clusters in the clustering process. This feature is suitable for the screening scenario of similar samples, because in this scenario, the number of clustering centers required after clustering processing may be different for different candidate sample sets corresponding to different forgetting samples.

[0070] The above introduces the screening method based on clustering algorithm.

[0071] The above two screening methods can be used alone or in combination. For example, the samples retained by the screening method based on the concentration of predicted output can be determined as a plurality of second candidate samples, and then the screening method based on the clustering algorithm is applied to the plurality of second candidate samples, thereby obtaining a number of similar samples participating in the forgetting learning training.

[0072] The above introduces the construction of corresponding similar samples for any one forgetting sample, and then introduces the construction of contrastive loss using similar samples and the like for training the model to achieve forgetting learning.

[0073] II. Constructing contrastive loss using similar samples to achieve forgetting learning

[0074] Figure 4 The flowchart of the method of the large language model forgetting learning method disclosed in the embodiments of the present specification. The execution subject of the method can be any device, platform, server or device cluster with computing and processing capability. As shown in the figure, the method includes the following steps: Figure 4

[0075] Step S410, obtaining a forgetting model (or a first large language model) M θ and a forgetting sample set wherein the forgetting model M θ is initialized as a fine-tuned model M obtained by fine-tuning the fine-tuning sample set w , and the forgetting sample set is a subset of the fine-tuning sample set .

[0076] It should be understood that the fine-tuning sample set is adapted to a specific downstream task. For example, assuming that the downstream task is a text classification task, at this time, the sample input of the fine-tuning sample can include the text published by a user on a social platform, and the expected output can be a specific text category, such as entertainment news, sports news, etc.

[0077] The forgetting sample set ​The forgotten sample contained therein can be artificially selected, or selected based on sensitive words, etc. It should be understood that the number of general forgotten samples is small.

[0078] Step S420, for the forgotten sample set Any first forgotten sample, based on the second large language model F θ Determine a number of similar samples, wherein each similar sample has similar (including the same) semantics but different expressions between the sample input of the first forgotten sample.

[0079] It should be noted that the introduction of this step can refer to the related description in the aforementioned "I. Construction of similar samples".

[0080] Step S430, using the forgotten model M θ Process the first forgotten sample to obtain a first hidden layer representation.

[0081] Specifically, the output of one or more hidden layers in the forgotten model M θ The first hidden layer representation can be determined based on the first forgotten sample.

[0082] Exemplarily, the output M θ The (l)th hidden layer in M θ,l (q) can be selected as the first hidden layer representation, which includes the hidden layer encoding M θ,l (q t ) of each token in the sample input q of the first forgotten sample, wherein q t represents the tth token in q.

[0083] From the above, the first hidden layer representation corresponding to the first forgotten sample can be obtained.

[0084] Step S440, determine a training loss, which is negatively related to the distance between the first hidden layer representation and the hidden layer representation of each similar sample, and is positively related to the distance between the first hidden layer representation and a random vector; The hidden layer representation of each similar sample is obtained based on the fine-tuning model.

[0085] It should be noted that the construction of the training loss is based on the idea of contrastive learning, which essentially pushes the distance between the first forgotten sample and its similar samples, and pulls the distance between the first forgotten sample and the random vector, so that the model can completely forget the target knowledge, and the query of the first forgotten sample can only output meaningless content.

[0086] The following introduces each loss term in the training loss.

[0087] 1) a first loss term negatively related to the distance between the first hidden layer representation and the hidden layer representation of each similar sample. In this way, the distance between the first forgetting sample and its similar samples in the embedding space of the forgetting model M θ can be pushed away.

[0088] For the hidden layer representation of each similar sample, a fine-tuned model M w can be used to process each similar sample respectively. It should be understood that the main difference in determining the hidden layer representation of the similar samples compared to the first forgetting sample is the use of different models, and the selection and processing of the hidden layer output are the same.

[0089] For the distance between the hidden layer representations, the Euclidean distance, cosine distance, etc. can be used.

[0090] Exemplarily, the calculation of the first loss term can be calculated using the following formula:

[0091]

[0092] where τ is an edge hyperparameter, q represents the sample input of the first forgetting sample, represents the set formed by the similar samples corresponding to q, |q| represents the number of tokens in q, q t represents the t-th token in q, x t represents the t-th token in the similar sample x, M θ,l (q t ) represents the output of the l-th hidden layer in M θ for the token q t , M w,l (x t ) represents the output of the l-th hidden layer in M w for the token x t .

[0093] 2) a second loss term positively related to the distance between the first hidden layer representation and a random vector. In this way, the hidden layer representation can be pulled closer to a random vector, ensuring that the model generates meaningless output for the forgetting sample rather than recalling the original memorized knowledge.

[0094] Exemplarily, the calculation of the second loss term can use the following formula:

[0095]

[0096] where r represents a random unit vector uniformly sampled from [0, 1).

[0097] The weighted sum of the above first loss term and the second loss term can be determined as the overall forgetting loss Exemplarily:

[0098]

[0099] Experiments show that introducing the first loss term can significantly improve the forgetting performance of the model compared to using only the second loss term as the forgetting loss.

[0100] In one embodiment, it can be This is determined as the training loss. In another embodiment, the training loss may also include a third loss term.

[0101] 3) Third loss item

[0102] Third loss item Its positive correlation is with the forgetting model M respectively. θ and fine-tuning model M w The distance between the hidden layer representations obtained after processing the first retained sample. The first retained sample belongs to the fine-tuning sample set. and forgotten sample set Difference between Thus, by preserving the dataset Top-aligned forgetting model M θ and the original fine-tuning model M w Feature representations can help maintain the forgetting model M θ Availability.

[0103] For example, the third loss item The calculation can be performed using the following formula:

[0104]

[0105] Among them, D r z represents the reserved sample set. t M represents the t-th word in the sample input z of the retained sample. θ,l (z t ) and M w,l (z t ) represent the forgetting model M respectively θ and fine-tuning model M w The l-th hidden layer targets word z. t Output hidden layer representation.

[0106] Based on a specific example, the training loss described above can be calculated using the following formula:

[0107]

[0108] Where λ is a hyperparameter, and its value range is (0,1).

[0109] Based on the above, the training loss can be determined.

[0110] Step S440, training the forgetting model M using the training loss θ to achieve forgetting learning.

[0111] Specifically, the forgetting model M can be adjusted using the training loss by gradient descent method θ to achieve forgetting learning. The adjustment of the model parameters can refer to the existing related technologies, and will not be described here.

[0112] From the above, forgetting learning can be achieved.

[0113] In summary, the forgetting learning method of the large language model disclosed in the embodiments of the present specification first proposes to use the contrast learning technology to solve the problem of large language model anti-learning, specifically to pull away the distance between the latent representation (hidden layer representation) of the sample to be forgotten and the similar sample, and to align the latent representation of the sample to be forgotten with a random vector, effectively improving the erasing effect and robustness of the model to specific information.

[0114] Corresponding to the forgetting learning method described above, the embodiments of the present specification also disclose a forgetting learning device. Referring to Figure 5 wherein the forgetting learning device 500 comprises the following functional modules:

[0115] The model data acquisition module 510 is configured to acquire a first large language model and a forgetting sample set, the first large language model being initially a fine-tuned model obtained by fine-tuning a fine-tuning sample set, and the forgetting sample set being a subset of the fine-tuning sample set. The sample determination module 520 is configured to determine, for any first forgetting sample, a plurality of similar samples based on a second large language model, wherein each similar sample has similar semantics but different expressions between the sample input of the first forgetting sample. The sample processing module 530 is configured to process the first forgetting sample using the first large language model to obtain a first hidden layer representation. The loss determination module 540 is configured to determine a training loss, which is negatively related to the distance between the first hidden layer representation and the hidden layer representation of each similar sample, and positively related to the distance between the first hidden layer representation and a random vector; the hidden layer representation of each similar sample is obtained based on the fine-tuned model. The model training module 550 is configured to train the first large language model using the training loss to achieve forgetting learning.

[0116] In one embodiment, the sample determination module 520 is specifically configured to generate similar samples of the first forgetting sample using a third large language model to obtain a plurality of first candidate samples; and determine the plurality of similar samples based on the plurality of first candidate samples.

[0117] Further, in one specific embodiment, the sample determination module 520 is configured to determine the several similar samples based on the plurality of first candidate samples, specifically comprising:

[0118] performing target processing on each sample in the first forgetting sample and the plurality of first candidate samples as a target sample: performing multiple prediction processing on the target sample by using the third large language model to obtain a plurality of prediction results corresponding thereto; determining an index value for measuring the concentration degree of the plurality of prediction results. For each first candidate sample, if its corresponding index value is greater than or equal to the index value corresponding to the first forgetting sample, it is classified into the second candidate sample set. Based on the second candidate sample set, the several similar samples are determined.

[0119] Further, in one example, the sample determination module 520 is configured to determine the index value for measuring the concentration degree of the plurality of prediction results, specifically comprising: for each prediction result, determining the prediction probability of the prediction result based on the prediction probability of each token therein; and accumulating and summing the prediction probabilities of the plurality of prediction results, and taking the obtained sum value as the index value.

[0120] In one example, the sample determination module 520 is configured to determine the several similar samples based on the second candidate sample set, comprising: filtering the several similar samples from the second candidate sample set by using a clustering algorithm.

[0121] In another specific embodiment, the sample determination module 520 is configured to determine the several similar samples based on the plurality of first candidate samples, specifically comprising: filtering the several similar samples from the plurality of first candidate samples by using a clustering algorithm.

[0122] Further, in one example, the sample determination module 520 is configured to filter the several similar samples from the plurality of first candidate samples by using a clustering algorithm, specifically comprising: determining the encoding vectors corresponding to the sample inputs of the plurality of first candidate samples; adaptively clustering the plurality of encoding vectors, and selecting the first candidate sample corresponding to the vector closest to the clustering center to be classified into the several similar samples.

[0123] In one embodiment, the first hidden layer representation and the hidden layer representation of each similar sample each include the output of the lth hidden layer in the model corresponding thereto.

[0124] In an embodiment, the sample processing module 530 is further configured to process a first reserved sample using the first large language model to obtain a second hidden layer representation; the first reserved sample belongs to a difference set between the fine-tuning sample set and the forgetting sample set. Wherein the training loss is also positively related to the distance between the second hidden layer representation and a third hidden layer representation, wherein the third hidden layer representation is obtained by processing the first reserved sample using the fine-tuning model.

[0125] It should be noted that the above-mentioned functional units can also be referred to the related description of the process method in the foregoing embodiments.

[0126] In this specification, the large language model can also be referred to as a large model. The large language model is a natural language processing model based on deep learning technology, and its parameter order is usually up to tens of billions to hundreds of billions or even higher, with strong language understanding and generation capabilities. The large language model can use the Transformer architecture or its variants (such as GPT, BERT, etc.), which uses attention mechanism to realize global modeling of sequence data, and can efficiently handle long-distance dependencies, thus performing well in natural language tasks. The large language model is pre-trained on a large-scale corpus to learn the statistical characteristics and semantic relevance of language, so that it has excellent generalization ability. The core capabilities of the large language model include but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage methods usually include direct inference and fine-tuning. In the direct inference mode, users guide the large language model to generate specific output by designing prompts. The prompt can be a text form of task description or instruction to stimulate the semantic understanding and generation ability of the large language model. In the fine-tuning mode, the large language model is further trained on a small-scale dataset in a specific domain to optimize its performance on specific tasks. The powerful generalization ability and flexibility of the large language model make it an important tool in the field of artificial intelligence technology, providing efficient and accurate solutions for automated text generation and understanding.

[0127] In some embodiments, the large language model can also have understanding and generation capabilities for other modalities (such as visual, audio, etc.) data, in which case the large language model can also be referred to as a multimodal large language model (MLLM). MLLMs provide a more rich and natural interactive experience by integrating text, images, sounds, and other types of inputs and outputs. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse these information to complete complex tasks. For example, MLLMs can analyze a picture and generate descriptive text, or generate corresponding images according to text descriptions. This cross-modal understanding and generation capability makes MLLMs have wide application prospects in many fields.

[0128] It should be noted that the key technologies of the large language model can be referred to the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, publication time: March 11, 2025, publication link: https: / / doi.org / 10.48550 / arXiv.2303.18223), which will not be repeated here.

[0129] According to another aspect, embodiments also provide a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform Figure 3 or Figure 4 the described method.

[0130] According to still another aspect, embodiments also provide a computing device comprising a memory and a processor, the memory having stored therein executable code which, when executed by the processor, implements Figure 3 or Figure 4 the described method.

[0131] Those skilled in the art should be aware that, in one or more examples described above, the functions described in the present disclosure can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0132] The above detailed description of the specific embodiments of the present application has been given to illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application shall be included in the protection scope of the present application.

Claims

1. A method for forgetting learning of a large language model, comprising: obtaining a first large language model and a forgetting sample set, the first large language model being initially a fine-tuned model obtained by fine-tuning a fine-tuning sample set, and the forgetting sample set being a subset of the fine-tuning sample set; for any first forgetting sample, determining a plurality of similar samples based on a second large language model, wherein each similar sample has similar semantics but different expression with the sample input of the first forgetting sample; processing the first forgetting sample by the first large language model to obtain a first hidden layer representation; determining a training loss negatively related to the distance between the first hidden layer representation and the hidden layer representation of each similar sample, and positively related to the distance between the first hidden layer representation and a random vector, wherein the hidden layer representation of each similar sample is obtained based on the fine-tuned model; training the first large language model by the training loss to achieve forgetting learning.

2. The method of claim 1, wherein, For any first forgetting sample, determining a plurality of similar samples based on a second large language model, comprising: generating similar samples of the first forgetting sample by a third large language model to obtain a plurality of first candidate samples; determining the plurality of similar samples based on the plurality of first candidate samples.

3. The method of claim 2, wherein, Determining the plurality of similar samples based on the plurality of first candidate samples, comprising: performing target processing by taking the first forgetting sample and each sample in the plurality of first candidate samples as a target sample respectively: performing multiple prediction processing on the target sample by the third large language model to obtain a plurality of prediction results corresponding to the target sample; and determining an index value for measuring the concentration degree of the plurality of prediction results; for each first candidate sample, if the index value corresponding to the first candidate sample is greater than or equal to the index value corresponding to the first forgetting sample, the first candidate sample is classified into a second candidate sample set; determining the plurality of similar samples based on the second candidate sample set.

4. The method of claim 3, wherein, Determining an index value for measuring the concentration degree of the plurality of prediction results, comprising: for each prediction result, determining a prediction probability of the prediction result based on the prediction probability of each token in the prediction result; cumulatively summing the prediction probabilities of the plurality of prediction results, and taking the sum value obtained as the index value.

5. The method of claim 3, wherein, Determining the plurality of similar samples based on the second candidate sample set, comprising: filtering the plurality of similar samples from the second candidate sample set by using a clustering algorithm.

6. The method of claim 2, wherein, Determining the plurality of similar samples based on the plurality of first candidate samples, comprising: filtering the plurality of similar samples from the plurality of first candidate samples by using a clustering algorithm.

7. The method of claim 6, wherein, Filtering the plurality of similar samples from the plurality of first candidate samples by using a clustering algorithm, comprising: determining the encoding vectors corresponding to the sample inputs of the plurality of first candidate samples; performing adaptive clustering on the plurality of encoding vectors, and selecting the first candidate sample corresponding to the vector closest to the clustering center into the plurality of similar samples.

8. The method of claim 1, wherein, The first hidden layer representation and the hidden layer representation of each similar sample each comprise the output of the lth hidden layer in the corresponding model.

9. The method of claim 1, wherein, Before determining the training loss, the method further comprises: The first reserved sample is processed by using the first large language model to obtain a second hidden layer representation; the first reserved sample belongs to a difference set between the fine-tuning sample set and the forgetting sample set; The training loss is also positively related to a distance between the second hidden layer representation and a third hidden layer representation, where the third hidden layer representation is obtained by processing the first reserved sample by using the fine-tuning model.

10. A forgetting learning device of a large language model, comprising: A model data acquisition module configured to acquire a first large language model and a forgetting sample set, the first large language model being initially a fine-tuning model obtained by fine-tuning a fine-tuning sample set, and the forgetting sample set being a subset of the fine-tuning sample set; A sample determination module configured to determine, for any first forgetting sample, a plurality of similar samples based on a second large language model, wherein each similar sample has similar semantics but different expressions with the sample input of the first forgetting sample; A sample processing module configured to process the first forgetting sample by using the first large language model to obtain a first hidden layer representation; A loss determination module configured to determine a training loss, which is negatively related to a distance between the first hidden layer representation and a hidden layer representation of each similar sample, and positively related to a distance between the first hidden layer representation and a random vector; the hidden layer representation of each similar sample is obtained based on the fine-tuning model; A model training module configured to train the first large language model by using the training loss to achieve forgetting learning.

11. A computer readable storage medium having stored thereon a computer program, wherein, When the computer program is executed in the computer, the computer is caused to perform the method of any one of claims 1-9.

12. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-9.

Images