A machine forgetting learning method based on information retrieval and large language model

Abstract

The present invention discloses a machine forgetting learning method based on information retrieval and driven by a large language model, which is used for forgetting learning of a knowledge-rich large language model in the legal field. For a given user query, the forgetting learning task is redefined, and the model parameters are not changed. By adding permissions to the model that prohibit access to the forgetting set, the model refuses to answer information about the forgetting set, thereby achieving the forgetting learning goal. First, the user query is subjected to information retrieval in the forgetting set corpus through a hybrid retrieval mechanism. Then, based on the reranker designed by the present invention, the retrieved paragraphs are reranked according to relevance. The top n most relevant paragraphs are then input into a verifier based on a large language model for verification to further clarify whether the user query involves information in the forgetting set. The method of the present invention has a high overall performance in the forgetting learning task, and can achieve accurate and reliable forgetting learning while taking into account efficiency and cost.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a machine forgetting learning method based on information retrieval and driven by a large language model. Background Art

[0002] Over the past few decades, with the continuous growth of data and advancements in machine learning algorithms, artificial intelligence has achieved tremendous success in various fields. The quality and quantity of data are crucial to the performance of machine learning models. However, as AI becomes increasingly data-dependent, it also presents challenges in privacy protection. Models inevitably learn from data collected from individuals, which often involves sensitive private information such as medical records or personal emails. Data governance frameworks such as the EU's General Data Protection Regulation (GDPR) and the US's California Consumer Privacy Act (CCPA) include the right to be forgotten as a key principle, allowing individuals to request the deletion of personal data under certain conditions. This is intended to address the potential for long-term harm to individuals caused by the long-term existence of online information. Machine forgetting learning is an emerging field in machine learning that focuses on eliminating the influence of a specific subset of training examples (called the "forgotten set") in a trained model. The goal is to develop algorithms that can effectively eliminate the influence of these examples while preserving other desirable model properties, such as accuracy on the remaining training data and generalization to new, unseen examples.

[0003] Leveraging large language models (LLMs) has become a mainstream paradigm for various natural language processing applications. LLMs memorize extensive knowledge when pre-trained or fine-tuned on large amounts of textual data. However, this data may contain sensitive information, such as names, phone numbers, and email addresses. Extensive research has shown that LLMs can generate private information. Therefore, the ability to easily remove data related to individual users from LLMs without degrading their predictive quality after deletion is increasingly important. However, previous work on forgetting learning has primarily focused on computer vision scenarios and largely overlooked the nature of forgetting in natural language processing, as textual data contains more explicit and sensitive personal information than images. Work on forgetting learning for LLMs is even more scarce. Current work on forgetting learning for LLMs focuses on forcing the LLM to replace harmful outputs (i.e., text that leaks private information) with harmless outputs. For example, this is achieved by reinforcing the model to replace specific expressions in the target data with generic counterparts. Another approach, for example, employs gradient ascent (GA) strategies and reinforcement learning methods to directly update model parameters in the opposite direction of the harmful token gradient, reducing the probability of the model generating harmful outputs. However, these methods essentially instruct the model to produce irrelevant outputs to cover up the information in the forgotten set data. However, this approach is unacceptable in some fields with high credibility requirements, such as law and medicine. The model outputs irrelevant text for specific instances, which will lead to model hallucinations. Text output that deviates from facts, is logically confusing, or is fabricated out of thin air will seriously mislead users.

[0004] Traditional forgetting learning methods are divided into exact machine unlearning and approximate forgetting. Exact forgetting refers to retraining model parameters from scratch on the remaining dataset. Although retraining from scratch is optimal for forgetting learning, it incurs a large computational overhead. This problem can be alleviated by approximate forgetting learning, which is an easy-to-compute alternative to retraining. It improves computational efficiency at the expense of certain forgetting learning performance and aims to enable the LLM to forget specific parts of its training data without the need for complete retraining. The technical routes of approximate forgetting learning mainly include fine-tuning, gradient ascent, Fisher forgetting, and influence unlearning. Since these methods all require adjustment of model parameters, the forgetting effect is strongly related to the size of the forgetting set. The forgetting effect can only be guaranteed when the size of the forgetting set accounts for a small proportion of the original model training set (generally ≤10%). If the forgetting set ratio is large, not only will a large number of forgetting requests likely lead to frequent retraining or parameter adjustments, significantly increasing computing resources and time costs, but as the proportion of forgotten samples increases, the model may need to frequently adapt to data changes, which may not only undermine model stability, but also lead to a decline in model generalization ability and affected accuracy.

[0005] Furthermore, while traditional forgetting learning has primarily focused on classification models, forgetting learning for LLMs is significantly more challenging for the following reasons. (1) Compared to classification tasks, the output space of LLMs is extremely broad, much larger than the label categories in classification. In classification, forgetting learning is more clearly defined: as long as the sample is classified (or not classified) into a specific category. However, in the context of natural language output, the definition of behavior is much more ambiguous. (2) Given the scale of LLMs, any computationally expensive forgetting method is not feasible. The forgetting process must be efficient to accommodate the size of the model and the requirements of practical applications, and efficiency is a significant consideration. (3) The behaviors of LLMs are diverse and highly unstructured. Unlike simple classification, the model needs to be able to forget these specific undesirable behavior patterns while maintaining useful functionality.

[0006] To address these challenges, this paper investigates how legal LLMs (LLMs) can achieve forgetting learning for specific instances. This paper proposes a forgetting learning method driven by information retrieval and a large language model. The paper argues that, for the purpose of protecting private data, the essence of forgetting learning is to erase traces of specific instance data from the original model, thereby causing the model to behave as if it had never seen such data. To achieve this goal, the paper avoids modifying the model's parameters, as is done in previous forgetting learning techniques. Instead, the paper attempts to achieve forgetting by instructing the model to refuse to respond to relevant information in the forgetting set. Therefore, the proposed forgetting learning method, driven by information retrieval and LLM, eliminates the need to train or adjust the original model. Instead, it searches for information within the forgetting set based on user queries and introduces a language model-based verifier to dynamically guide the search process. If the verification results indicate that the user query information involves content in the forgetting set, the LLM directly refuses to respond to the user's query and denies the right to discuss instances related to the forgetting set. If the verification results indicate that the user query is not related to the forgetting set, the LLM can respond normally to the user's question. In addition, in previous methods, the forgetting sets during forgetting learning are collected in advance. If new forgotten data is added subsequently, the model needs to be re-learned for forgetting. In vertical fields, requests for data forgetting may increase dynamically at any time. The method of the present invention only needs to add new forgetting requests to the forgetting set, and no additional optimization of the model is required. Therefore, this shortcoming of previous static forgetting learning is solved, and the forgetting learning effect for the dynamic forgetting set is efficiently achieved. Summary of the Invention

[0007] Technical Problem: The technical problem to be solved by this invention is that traditional forgetting learning methods require adjustment of model parameters. The forgetting effect is strongly correlated with the size of the forgetting set. As the proportion of forgotten samples increases, the model's generalization ability decreases and its accuracy is affected. Traditional forgetting learning mainly focuses on classification models, while forgetting learning for large language models is more challenging and computationally expensive. Current methods for forgetting learning using large language models essentially instruct the model to produce irrelevant outputs to mask the information in the forgetting set data, resulting in model hallucination and being unapplicable in the legal field. Current forgetting learning methods use pre-collected forgetting sets, and subsequent addition of forgotten data requires re-learning the model. This invention provides a machine forgetting learning method based on information retrieval and driven by a large language model. This method is mainly used in the legal field to perform forgetting learning on knowledge-rich large language models, filter sensitive information, and protect personal privacy. For a given user query, this invention redefines the forgetting learning task. By adding permissions to the model that prohibit access to the forgetting set without changing the model parameters, the model refuses to answer questions related to the forgetting set, achieving the forgetting learning goal. First, a hybrid retrieval mechanism is used to perform information retrieval on the forgetting set corpus. Then, based on the re-ranker designed by the present invention, the retrieved paragraphs are re-ranked according to relevance. The top n most relevant paragraphs are then input into the verifier based on the large language model for verification, to further clarify whether the user query involves information in the forgotten set. In the verification process, the present invention introduces an outline guidance mechanism, which guides the verification scope through the outline, and combines the user's conversational dynamic update of supplementary information to gradually determine the user's intention, thereby improving the verification accuracy, and instructing the large model to make a corresponding response. If the forgotten set information is not involved, the large model will respond normally, otherwise it will refuse to answer. Finally, the present invention constructed a forgetting learning dataset for theft cases in the private legal domain and demonstrated the effectiveness of the method through a large number of experiments. A series of experiments show that the method of the present invention has a high overall performance in forgetting learning tasks, and can achieve accurate and reliable forgetting learning while taking into account efficiency and cost.

[0008] Technical Solution: The present invention addresses its technical problems through a machine learning method for forgetting, based on information retrieval and driven by a large language model. This method redefines the forgetting task for a given user query. By adding a permission to the model that prohibits access to the forgetting set, the model refuses to answer questions related to the forgetting set, achieving the forgetting learning goal without modifying model parameters. First, a hybrid retrieval mechanism is used to retrieve information from the forgetting set corpus. Then, a reranker designed based on the present invention reranks the retrieved paragraphs by relevance. The top n most relevant paragraphs are then input into a verifier based on the large language model for verification, further clarifying whether the user query involves information in the forgetting set. During the verification process, the present invention introduces a summary-guided mechanism. This guides the verification scope using the summary, and combined with user-interactive dynamic updates and supplementary information, gradually determines user intent, thereby improving verification accuracy and instructing the large model to respond accordingly. If no information in the forgetting set is involved, the large model responds normally; otherwise, it refuses to respond. Finally, the present invention constructs a dataset of forgetting learning cases from the private legal domain and demonstrates the effectiveness of the method through extensive experiments.

[0009] The machine forgetting learning method based on information retrieval and large language model driving of the present invention comprises the following steps:

[0010] 1) For a given user query, redefine the forgetting learning task;

[0011] 2) Design a retriever that uses a hybrid retrieval mechanism to perform information retrieval on the user query in the forgotten set corpus to further improve search performance;

[0012] 3) For the first K paragraphs output by the retriever in step 2), use the reranker to adjust the order of these paragraphs to ensure that the most relevant paragraphs are ranked better;

[0013] 4) The n most relevant paragraphs in the forgotten set obtained in step 3) are input into the verifier based on the large language model for verification to further determine whether the user query involves information in the forgotten set and instruct the large model to respond accordingly;

[0014] 5) Construct a forgetting learning benchmark dataset of theft cases in the legal field to verify the effectiveness of the method proposed in this paper.

[0015] As a further improvement of the present invention, in step 1), considering that the existing method requires adjustment of model parameters, the forgetting effect is closely affected by the size of the forgetting set, and it is difficult to maintain the normal performance of the model on non-forgetting set data, the present invention decides to adopt a method without changing the model parameters. By adding the permission to prohibit access to the forgetting set to the model, the model refuses to answer information about the forgetting set, thereby achieving the forgetting learning goal of erasing the data traces of the forgetting set in the model and the model showing that it is unaware of it. The forgetting learning task is formally defined as: for the forgetting set D f ={}y1,y2,...,y f ,...}, where y f It is harmful text that will leak private information, and the pre-trained language model θ o Processing Tips x f When y is generated f The harmful output of . Denote the forgetting learning mechanism as U, such that

[0016]

[0017] Among them, when the model θ o Process normal prompts x that do not involve the contents of the forget set n When , it should be treated as a normal question and generate a normal response y n ; When the model θ o Encountered hints about the contents of the Oblivion Set x f When it is, it should be identified and a rejection response y should be generated reject ,in

[0018] As a further improvement of the present invention, in step 2), in order to achieve zero-sample retrieval of user queries, from the perspective of efficiency and retrieval performance, the present invention believes that the dictionary-based retrieval method BM25 is an ideal choice for implementing the retriever. BM25 only relies on dictionary overlap and considers the dictionary's term and document frequency, which is simple to implement and has good retrieval performance. BM25 considers factors such as term frequency (TF) and inverse document frequency (IDF) in the document, first parses the user query, and obtains each morpheme q i , then for each search document D, calculate each morpheme q i The correlation score with D, and finally the morphemes q i The weighted sum of the relevance scores relative to D is used to obtain the relevance score of the user query and the search document D. BM25 .

[0019] As a further improvement of the present invention, in step 2), in order to further improve the search performance, the present invention designs a hybrid retrieval mechanism, which combines the dictionary-based BM25 with the vector similarity retrieval, and integrates the retrieval results of the two to achieve hybrid retrieval.

[0020] The present invention selects BGE-M3 as the vector retrieval model of the retriever. The user query Q will first be converted into a hidden state H based on the text encoder. Q And normalized: e Q =norm(H Q ). Similarly, the embedding vector of the candidate document paragraph is expressed as: e D =norm(H D ). Therefore, the relevance score between the user query Q and the paragraph D is measured by the inner product between the two embeddings:

[0021] Score dense (D,Q)= <e D ,e Q >

[0022] Then, the final retrieval results are re-ranked according to the comprehensive BM25 score and vector similarity retrieval score:

[0023] Score(D,Q)=Score BM25 +k*Score dense

[0024] Where k is a hyperparameter used to adjust the importance weights of the two retrieval scores. Finally, the top K paragraphs with the highest comprehensive scores are selected and entered into the re-ranker.

[0025] As a further improvement of the present invention, in step 3), the present invention represents the first K paragraphs as: The reorderer proposed in the present invention calculates P(p i |Q) to measure the relevance score of each paragraph relative to the user query to achieve re-ranking. Specifically, the present invention uses the Bayesian formula to calculate P(p i |Q) for analysis:

[0026]

[0027] Among them, P(p i ) is the prior probability of the retrieved paragraph, and log P(Q) is related to p i is irrelevant, so for any p i , which has the same value and is a common constant. Further, inspired by the existing work, the present invention assumes that the prior probability P(p i ) is uniformly distributed (i.e. for all paragraphs pi are the same), so they can be ignored when reordering. Therefore, the above formula can be simplified to:

[0028] log P(p i |Q)∝log P(Q|p i )

[0029] So, the reordering is based on P(Q|p i ) size.

[0030] As a further improvement of the present invention, in the step 3), the present invention selects mT5 as the basic model of the re-ranker, and fine-tunes it using a question-answering dataset in the legal field. mT5 adopts a sequence-to-sequence (Seq2Seq) architecture, in which the encoder encodes the input sequence into a hidden layer representation, and the decoder generates an output sequence based on the hidden layer representation. In the fine-tuning process, the present invention uses the answer sequence as input and the question sequence as the target output, and learns to rearrange the candidate paragraphs by learning the mapping relationship between these sequences. During the training process, the cross-entropy loss is used to measure the gap between the answer generated by the model and the real answer, thereby optimizing the parameters. After fine-tuning, the present invention obtains a re-ranker based on a pre-trained language model, which has low implementation cost and high re-ranking efficiency, providing strong support for the subsequent verification process.

[0031] As a further improvement of the present invention, in step 4), in order to clarify whether the user's query intent involves the forgotten set, the present invention designs a verifier based on a large language model. The large language model has a strong contextual understanding ability and can better understand the user's query intent in the context of multiple rounds of dialogue between the user and the system. Secondly, the large language model exhibits a strong instruction-following ability and can well complete instructions and realize the judgment of the relevance between the query and the forgotten set. Therefore, the present invention is designed to achieve this goal, which is formally expressed as follows:

[0032]

[0033] Among them, goal(Q) represents the query intent of the user query. If the query intent of the user query is irrelevant to the forgotten set, the judgment result of the verifier based on the large language model is 0; if the query intent of the user query is related to the forgotten set, the judgment result of the verifier based on the large language model is 1.

[0034] As a further improvement to the present invention, in step 4), inspired by conversational search models and combining the characteristics of legal cases, the present invention designs a theft case outline to assist the large language model-based verifier in determining and clarifying the user's query intent, thereby preventing misidentification of intent. The outline is an ontological model of the forgotten set corpus information, characterizing the common information of the forgotten set data. In this work, the forgotten set of the present invention is the legal theft case data. Therefore, the present invention models the outline into four attributes describing the key case information: "time," "place," "defendant," and "stolen items." The large language model-based verifier uses these attributes to determine whether the related case C has related cases, effectively improving the efficiency and accuracy of verification.

[0035] As a further improvement to the present invention, in step 4), a dynamic interaction model is established between the user and the system. If the large language model-based verifier, given the current outline, finds multiple cases relevant to the user's query, the user will be asked to provide additional unknown key information about the case to update the outline. Through a series of back-and-forth exchanges, the system responds to user queries and clarifies their information needs, instantly updating the outline. This guides the large language model-based verifier to gradually narrow its review of relevant cases, retrieving relevant cases that align with the latest outline, thereby improving the accuracy of the final judgment.

[0036] As a further improvement of the present invention, in step 5), in order to verify the effectiveness of the method described in the present invention, a forgetting learning benchmark dataset for theft cases in the legal field was constructed. In the dataset constructed by the present invention, each data item consists of a user query related to the case content, a text describing the case facts, and an outline of the case. The outline contains four attributes: "time", "place", "defendant", and "stolen items", stored in the form of key-value pairs. The initial user query only contains information related to "defendant". The present invention divides the dataset into a forgetting set and a non-forgetting set, totaling 2039 samples.

[0037] Beneficial effects:

[0038] Compared to existing technologies, the present invention has the following advantages: 1) Existing forgetting learning methods require adjustment of model parameters, and the forgetting effect is strongly correlated with the size of the forgetting set. As the proportion of forgotten samples increases, the model's generalization ability decreases, affecting accuracy. The present invention, however, proposes achieving the forgetting effect by causing the model to refuse to answer relevant information in the forgetting set, without changing the model's original parameters. 2) Existing methods primarily focus on classification models, making forgetting learning for LLMs more challenging and computationally expensive. The present invention significantly reduces the cost of forgetting learning for LLMs by amortizing the cost of forgetting learning over each questioning session. 3) Current methods using large language models for forgetting learning essentially instruct the model to produce irrelevant outputs to mask the information in the forgetting set data, leading to model hallucinations and making them unsuitable for legal applications. The present invention, by causing the model to refuse to answer relevant information in the forgetting set, fundamentally eliminates the generation of text that deviates from the facts and is fabricated, making it applicable to the legal field. 4) Existing methods use pre-collected forgetting sets, and subsequent addition of forgotten data requires re-learning the model for forgetting. The present invention designs a forgetting learning framework that clarifies the user's information needs through a dynamic interaction mode and can effectively handle dynamically increasing forgetting learning requests without having to re-perform forgetting learning on the model.

[0039] Experimental analysis has proved that the machine forgetting learning method based on information retrieval and large language model proposed in this method has improved the forgetting learning model, greatly improved the model performance, and can achieve accurate and reliable forgetting learning while taking into account efficiency and cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a basic process schematic diagram of the present invention;

[0041] Figure 2 It is a model framework diagram of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0043] Example:

[0044] See also Figure 1 、 Figure 2 The machine forgetting learning method based on information retrieval and large language model driving of the present invention includes the following steps:

[0045] 1) For a given user query, redefine the forgetting learning task. Assume that the large-scale language model θ oThe training has been done on the dataset D. The present invention hopes that the model can forget the information of some data in D. This part of the data belongs to a subset of D, which is called the forgetting set D in the present invention. f The goal of forgetting learning is to approximately simulate the r =D\D f The effect of retraining the model is to get the new model θ u , let θ u To D f The data is like the unseen data, but while retaining θ o Processing D f In addition, forgetting the learned θ u Should still maintain D r The utility on the non-forgetting set, θ u The ability should be related to θ o The closer the better. One easy way to do this is to r Retraining the LLM on the fly is too slow and expensive to be an acceptable approach.

[0046] The present invention aims to adjust the model parameters to obtain the θ after forgetting learning in consideration of the existing methods. u This approach will make the forgetting effect closely affected by the size of the forgetting set, and it is difficult to maintain the normal performance of the model on non-forgetting set data. This paper decides to adopt a method without changing the model parameters. By adding the permission to prohibit access to the forgetting set to the model, the model refuses to answer questions about the forgetting set, achieving the forgetting learning goal of erasing the data traces of the forgetting set in the model and making the model appear unaware of it. The formal definition is: for the forgetting set D f ={y1,y2,...,y f ,...}, where y f It is harmful text that will leak private information, and the pre-trained language model θ o Processing Tips x f When y is generated f The harmful output of . Denote the forgetting learning mechanism as U, such that

[0047]

[0048] Among them, when the model θ o Process normal prompts x that do not involve the contents of the forget set n When , it should be treated as a normal question and generate a normal response y n ; When the model θ o Encountered hints about the contents of the Oblivion Set x f When it is, it should be identified and a rejection response y should be generated refect ,in

[0049] 2) Design a retriever that performs information retrieval on user queries in the forgotten set corpus through a hybrid retrieval mechanism to further improve search performance. In order to achieve zero-sample retrieval of user queries, from the perspective of efficiency and retrieval performance, the present invention believes that the dictionary-based retrieval method BM25 is an ideal choice for implementing a retriever. BM25 only relies on dictionary overlap and considers the dictionary's term and document frequency, which is simple to implement and has good retrieval performance. BM25 takes into account factors such as word frequency (TF) and inverse document frequency (IDF) in the document. First, the user query is parsed to obtain each morpheme q i , then for each search document D, calculate each morpheme q i The correlation score with D, and finally the morphemes q i The relevance score of the user query and the search document D is obtained by weighted summation relative to the relevance score of D:

[0050]

[0051] Where avgdl is the average length of all documents, k1 and b are adjustable parameters, N is the total number of documents in the document collection, and n(q i ) is a string containing the word q i The number of documents.

[0052] In addition to probabilistic retrieval models like BM25, commonly used information retrieval models also include vector space models. Vector similarity-based search engines represent documents as vectors in the document space through feature selection and weight calculation, and then measure the similarity between documents by calculating the similarity between vectors. Therefore, to further improve search performance, this paper combines the dictionary-based BM25 with vector similarity search, integrating the search results of the two to achieve hybrid search.

[0053] This paper selects BGE-M3 as the vector retrieval model of the retriever. It is a semantic vector model newly proposed by BAAI. It has the characteristics of multilingualism, multi-functionality and multi-granularity, and has achieved leading performance in a wide range of information retrieval tasks. The user query Q is first converted into a hidden state H based on the text encoder. Q And normalized: e Q =norm(H Q ). Similarly, the embedding vector of the candidate document paragraph is expressed as: e D =norm(H D ). Therefore, the relevance score between the user query Q and the paragraph D is measured by the inner product between the two embeddings:

[0054] Score dense (D,Q)= <eD ,e Q >

[0055] Then, the final retrieval results are re-ranked according to the comprehensive BM25 score and vector similarity retrieval score:

[0056] Score(D,Q)=Score BM25 +k*Score dense

[0057] Where k is a hyperparameter used to adjust the importance weights of the two retrieval scores. Finally, the top K paragraphs with the highest comprehensive scores are selected and entered into the re-ranker.

[0058] 3) For the first K paragraphs output by the retriever in step 2), the reranker is used to adjust the order of these paragraphs to ensure that the most relevant paragraphs are ranked better. The present invention represents these first K paragraphs as: By calculating P(p i |Q) to measure the relevance score of each paragraph relative to the user query and achieve re-ranking.

[0059] Specifically, the present invention uses the Bayesian formula to calculate P(p i |Q) for analysis:

[0060]

[0061] Among them, P(p i ) is the prior probability of the retrieved paragraph, and log P(Q) is related to p i is irrelevant, so for any p i , which has the same value and is a common constant. Further, inspired by the existing work, the present invention assumes that the prior probability P(p i ) is uniformly distributed (i.e. for all paragraphs p i are the same), so they can be ignored when reordering. Therefore, the above formula can be simplified to:

[0062] log P(p i |Q)∝log P(Q|p i )

[0063] So, the reordering is based on P(Q|p i ). The present invention uses a pre-trained language model (PLM) to calculate it:

[0064]

[0065] Where Θ represents the parameters of the pre-trained language model. The conditional probability of the entire output sequence Q is calculated by multiplying the conditional probabilities of all time steps, where P(Qt |Q ＜t ,p i ; Θ) is the conditional probability of the model generating the current tag at each time step, which can be expressed as:

[0066] P(Q t |Q ＜t ,p i ; Θ)=Softmax(f(Q ＜t ,p i ;Θ))

[0067] Among them, f(Q ＜t ,p i ; Θ) is the output of the model, which represents the generation of the current tag Q t The score is converted into a probability distribution through the Softmax function. In this work, the present invention selects mT5 as the basic model of the re-ranker and fine-tunes it using a question-answering dataset in the legal field. mT5 adopts a sequence-to-sequence (Seq2Seq) architecture, in which the encoder encodes the input sequence into a hidden layer representation, and the decoder generates an output sequence based on the hidden layer representation. In the fine-tuning process, the present invention uses the answer sequence as input and the question sequence as the target output, and learns to rearrange the candidate paragraphs by learning the mapping relationship between these sequences. During the training process, the cross entropy loss is used to measure the gap between the answer generated by the model and the real answer, thereby optimizing the parameters. After fine-tuning, the present invention obtains a re-ranker based on a pre-trained language model, which has low implementation cost and high re-ranking efficiency, providing strong support for the subsequent verification process.

[0068] 4) The n most relevant paragraphs in the forgotten set obtained in step 3) are input into the verifier based on the large language model for verification to further clarify whether the user query involves information in the forgotten set and instruct the large model to make a corresponding response. The present invention regards the top n paragraphs after re-ranking as relevant cases potentially related to the user query, represented by C = {c1, c2, ..., c n Next, the present invention needs to determine whether the user query has an exact related case in the related case C, that is, to clarify whether the query intention of the user query involves the forgotten set. LLM has a strong context understanding ability and can better understand the user's query intention in the context of multiple rounds of dialogue between the user and the system. Secondly, LLM shows a strong instruction following ability and can complete the instruction well to realize the judgment of the relevance between the query and the forgotten set. Therefore, the present invention designs a verifier based on a large language model to achieve this goal, which is formally expressed as follows

[0069]

[0070] Among them, goal (Q) represents the query intention of the user's query. If the query intention of the user's query is irrelevant to the forgotten set, the judgment result of the verifier based on the large language model is 0; if the query intention of the user's query is related to the forgotten set, the judgment result of the verifier based on the large language model is 1. Inspired by the conversational search model, the present invention combines the characteristics of cases in the legal field and designs an outline of the theft case to assist the verifier based on the large language model in determining and clarifying the user's query intention, thereby preventing misjudgment of intent recognition. At the same time, the present invention constructs a dynamic interaction mode between the user and the system. Through a series of back-and-forth exchanges, it responds to user queries and clarifies the user's information needs, and updates the outline in real time to guide the verifier based on the large language model to gradually narrow the scope of review of related cases C, to retrieve related cases that are consistent with the latest outline, and thus improve the accuracy of the final judgment.

[0071] like Figure 2 As shown, the prompts input to the verifier based on the large language model include three parts: related cases C, instructions, and outlines. First, the related case C is provided to the verifier based on the large language model as the judgment object of the user query, and then the instructions are used to instruct the verifier based on the large language model to judge whether there are related cases in the above-mentioned related cases C based on the information provided by the outline. The outline is an ontology-level modeling of the forgotten set corpus information, which describes the common information of the forgotten set data. In this work, the forgotten set of the present invention is the theft case data in the legal field. Therefore, the present invention models the outline into four attributes describing the key information of the case, namely "time", "place", "defendant", and "stolen items". The verifier based on the large language model judges whether there are related cases in the related case C based on them, which can effectively improve the efficiency and accuracy of verification.

[0072] When the large language model-based verifier believes that there is no case related to the user query in the related cases C, that is, the user query does not involve the forgotten set information, so the LLM can normally answer the relevant questions of the user query. If the large language model-based verifier believes that there are related cases according to the existing outline, the IDs of the corresponding related cases will be returned in the form of a list, such as Figure 2As shown. If the number of related cases is 1 at this time, the large language model-based verifier determines that the query target of the user query touches the information in the forgotten set. In order to prevent LLM from leaking relevant information, the LLM is instructed to refuse to answer. If the large language model-based verifier finds that there are multiple cases related to the user query under the given current outline, the user will be asked to further provide the remaining unknown key information about the case to update the outline. The updated outline and the related cases C and instructions form a new prompt and are re-input into the large language model-based verifier to perform a new round of verification. If there are still empty attributes in the outline, the above process is repeated until the large language model-based verifier determines that the number of related cases is 0 or 1; if all the attribute information in the outline is clear, the large language model-based verifier still believes that there are multiple related cases, then it is determined that the user query involves forgotten set information and the LLM is instructed to refuse to answer.

[0073] 5) Construct a forgetting learning benchmark dataset of theft cases in the legal field to verify the effectiveness of the method proposed in the present invention. In the dataset constructed by the present invention, each data consists of a user query related to the content of the case, a text describing the facts of the case and an outline of the case, wherein the outline contains four attributes: "time", "place", "defendant" and "stolen items", which are stored in the form of key-value pairs. The initial user query only contains information related to the "defendant". The present invention divides the dataset into a forgetting set and a non-forgetting set, with a total of 2039 samples. In the experiment, the present invention quantifies the forgetting learning effect by evaluating the accuracy rate of accurately identifying whether the case facts involved in the user query exist in the forgetting set.

[0074] The above embodiments are only preferred implementations of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and equivalent substitutions can be made without departing from the principles of the present invention. These technical solutions after improvements and equivalent substitutions to the claims of the present invention all fall within the scope of protection of the present invention.

Claims

1. A machine forgetting learning method based on information retrieval and large language model, characterized by: The method comprises the following steps: 1) For a given user query, redefine the forgetting learning task; 2) Design a retriever that uses a hybrid retrieval mechanism to perform information retrieval on the user query Q in the forgotten set corpus to further improve search performance; 3) For the first K paragraphs output by the retriever in step 2), use the reranker to adjust the order of these paragraphs. To ensure that the most relevant paragraphs are ranked better; 4) The n most relevant paragraphs in the forgotten set obtained in step 3) are input into the verifier based on the large language model for verification to further determine whether the user query involves information in the forgotten set and instruct the large model to respond accordingly; 5) Construct a forgetting learning benchmark dataset for theft cases in the legal field to verify the effectiveness of the learning method; Among them, in the step 1), the model parameters are not changed. By adding the permission of prohibiting access to the forgotten set to the model, the model refuses to answer information about the forgotten set, so as to erase the data traces of the forgotten set in the model and make the model appear to be unaware of it. The forgetting learning task is formally defined as: for the forgotten set D f ={y1,y2,...,y f ,...}, where y f It is harmful text that will leak private information, and the pre-trained language model θ o Processing Tips x f When y is generated f The harmful output of the forgetting learning mechanism is represented as U, so that Among them, when the model θ o Process normal prompts x that do not involve the contents of the forget set n When , it should be treated as a normal question and generate a normal response y n ; When the model θ o Encountered hints about the contents of the Oblivion Set x f When it is, it should be identified and a rejection response y should be generated reject ,in In step 4), a dynamic interaction mode between the user and the system is established. If the verifier based on the large language model finds that there are multiple cases related to the user query under the given current outline, the user will be required to further provide the remaining unknown key information about the case to update the outline. Through a series of back-and-forth exchanges, the user query is responded to and the user's information needs are clarified. The outline is updated in real time, thereby guiding the verifier based on the large language model to gradually narrow the scope of review of relevant cases to retrieve relevant cases that are consistent with the latest outline.

2. The machine forgetting learning method based on information retrieval and large language model driving according to claim 1 is characterized in that: In step 2), the dictionary-based retrieval method BM25 is an ideal choice for implementing the retriever. BM25 takes into account the word frequency TF and inverse document frequency IDF factors in the document. First, the user query is parsed to obtain each morpheme q i , then for each search document D, calculate each morpheme q i The correlation score with D, and finally the morphemes q i The weighted sum of the relevance scores relative to D is used to obtain the relevance score of the user query and the search document D. BM25 (D,Q).

3. The machine forgetting learning method based on information retrieval and large language model driving according to claim 2 is characterized in that: In step 2), a hybrid search mechanism is designed to combine the dictionary-based search method BM25 with the vector similarity search, and integrate the search results of the two to achieve hybrid search. BGE-M3 is selected as the vector retrieval model of the retriever. The user query Q is first converted into a hidden state H based on the text encoder. Q And normalized: e Q =norm(H Q ), similarly, the embedding vector of the candidate document paragraph is expressed as: e D =norm(H D ), therefore, the relevance score between the user query Q and the paragraph D is measured by the inner product between the two embeddings: Score dense (D,Q)=<e D ,e Q > Then, the BM25 score and vector similarity retrieval score are combined to re-rank the final retrieval results: Score(D,Q)=Score BM25 (D,Q)+k*Score dense (D,Q) Among them, k is a hyperparameter used to adjust the importance weights of the two retrieval scores. Finally, the top K paragraphs with the highest comprehensive scores are screened out and enter the reranker.

4. The machine forgetting learning method based on information retrieval and large language model driving according to claim 1 is characterized in that: In step 3), mT5 is selected as the base model of the reranker and is fine-tuned using a question-answering dataset in the legal field. mT5 adopts a sequence-to-sequence (Seq2Seq) architecture, in which the encoder encodes the input sequence into a latent representation, and the decoder generates an output sequence based on the latent representation. During the fine-tuning process, the answer sequence is used as input and the question sequence as the target output. By learning the mapping relationship between these sequences, the reranking of candidate paragraphs is learned. During the training process, cross-entropy loss is used to measure the gap between the answers generated by the model and the actual answers, thereby optimizing the parameters. After fine-tuning, a reranker based on the pre-trained language model is obtained.

5. The machine forgetting learning method based on information retrieval and large language model driving according to claim 1 is characterized in that: In step 4), in order to clarify whether the user's query intent involves the forgotten set, a verifier based on a large language model is designed, which is expressed as follows: Here, goal(Q) represents the query intent of user query Q. If the query intent of the user query is irrelevant to the forgotten set C, the verifier based on the large language model will judge the result as 0; if the query intent of the user query is related to the forgotten set, the verifier based on the large language model will judge the result as 1.

6. The machine forgetting learning method based on information retrieval and large language model driving according to claim 1 is characterized in that: In step 4), a theft case outline is designed to assist the large language model-based verifier in determining and clarifying the user's query intent. The forgotten set is theft case data in the legal field. The outline is modeled into four attributes describing key case information: "time," "place," "defendant," and "stolen items." The large language model-based verifier uses these attributes to determine whether related cases exist.

7. The machine forgetting learning method based on information retrieval and large language model driving according to claim 1 is characterized in that: In step 5), each piece of data in the constructed dataset consists of a user query related to the case content, a text describing the case facts, and an outline of the case. The outline contains four attributes: "time", "place", "defendant", and "stolen items", and is stored in the form of key-value pairs. The initial user query only contains information related to "defendant". The dataset is divided into a forgotten set and a non-forgotten set, with a total of 2039 samples.

Images