A method and system for entity disambiguation and forgetting based on a large language model

Abstract

The present invention discloses a method and system for entity disambiguation and forgetting based on a large language model, which relates to the technical field of entity disambiguation in natural language processing, including: determining an entity disambiguation dataset and a forgetting dataset, constructing a contrastive learning sample and performing data preprocessing; extracting features using an LLaMA3 model, adding a projection layer and a contrastive learning module; constructing a loss function, for each sample, using cosine similarity to calculate the similarity between its feature representation and positive and negative samples, and using contrastive loss to measure the effectiveness of the model in distinguishing positive and negative samples; based on the loss value, updating the model parameters through back propagation until the model converges. The present invention uses the LLaMA3 model infrastructure and a contrastive learning method to enhance the model's discrimination ability, while providing a way to implement a forgetting mechanism. The present invention provides new ideas for the research and application of models in the field of natural language processing.

Description

Technical Field

[0001] The present invention relates to the technical field of entity disambiguation in natural language processing, and in particular to a method and system for entity disambiguation and forgetting based on a large language model. Background Art

[0002] With the explosive growth of internet information, text data contains a large number of ambiguous entities. Accurate entity disambiguation is crucial for applications such as information retrieval, question-answering systems, and text summarization. Modern applications such as search engines, knowledge graph construction, question-answering systems, and virtual assistants rely on the precise identification and disambiguation of entities to provide users with the accurate information they need. Entity disambiguation has become a pressing need for models to accurately understand and remember entities. However, the disambiguation task is challenging, especially when dealing with homonymous entities (e.g., company and place names) and polysemous words (the same word has different meanings). Furthermore, during entity disambiguation, models may process entities containing personal privacy information (e.g., names of people and locations). Under data privacy regulations, users have the right to request the deletion of their personal data. Entity disambiguation and forgetting methods based on large language models provide an efficient solution for identifying and distinguishing entities with multiple meanings in large amounts of text, while forgetting specific entity information. This approach is particularly important in search engines, knowledge graph construction, question-answering systems, and virtual assistants, ensuring the accuracy and completeness of the information users need while complying with data privacy regulations, and has a wide range of application scenarios.

[0003] Previous machine learning-based entity disambiguation methods mostly used machine learning models such as support vector machines (SVMs) and decision trees to encode entities and their context as features for classification. These methods suffer from the following drawbacks: 1. Classic machine learning models typically require extensive manual feature engineering to extract effective features for training. This is not only time-consuming and labor-intensive, but the quality of feature design directly impacts model performance. Manual features struggle to capture complex semantics and contextual information, limiting the model's expressive power. 2. Compared to modern large language models, classic machine learning models have significant shortcomings in understanding and utilizing contextual information, making it difficult to capture long-range dependencies and complex contextual relationships. 3. For highly ambiguous entities, classic models often struggle to effectively distinguish their specific meanings. This is because classic models rely on predefined features and lack an understanding of the underlying semantics of entities, resulting in poor performance when faced with complex and ambiguous entities. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide an entity disambiguation and forgetting technology based on a large language model. With the LLaMA3 model as the basic architecture, it fundamentally solves the many defects exposed by the previous classic machine learning models (SVM, decision tree), and the present invention combines contrastive learning. The model automatically learns how to bring different contextual representations of the same entity closer and push the representations of different entities farther apart, and the disambiguation ability is significantly enhanced. In addition, contrastive learning enhances the model's distinction between these entities by constructing positive and negative sample pairs. It can dynamically learn new entity features, adapt to new contexts, and achieve effective forgetting of specific entity information.

[0005] To achieve the above technical objectives, this application provides an entity disambiguation and forgetting method based on a large language model, comprising the following steps:

[0006] Based on the entity disambiguation dataset and the forgetting dataset, after constructing the comparative learning samples, data preprocessing was performed to remove irrelevant information. The vocabulary of the LLaMA3 model was used to segment and encode the text to generate the dataset.

[0007] Based on the LLaMA3 model, it is improved by adding a projection layer and a contrastive learning module. Based on the improved LLaMA3 model, model training is performed according to the dataset to construct a large language model. The constructed large language model has the ability to both entity disambiguation and forgetting.

[0008] Preferably, in the process of constructing comparative learning samples, the comparative learning samples include positive samples and negative samples, wherein the positive samples include the text of the target entity, which is encoded using the LLaMA3 model to obtain its representation; the negative samples include two types, type one is the text containing the entity that needs to be forgotten, as the object whose similarity the model should reduce, and type two is the text containing other entities that are easily confused with the target entity.

[0009] Preferably, in the process of improving the LLaMA3 model, a projection layer is added after the output layer of the LLaMA3 model to map the high-dimensional text representation to the contrastive learning space, and integrate the modules required for contrastive learning, including contrastive loss calculation and positive and negative sample matching.

[0010] Preferably, during the model training process, for each sample, the cosine similarity is used to calculate the similarity between its feature representation and the positive sample and the negative sample, and the contrast loss is used to measure the effect of the model in distinguishing between positive and negative samples.

[0011] Preferably, in the process of using contrast loss to measure the effect of the model in distinguishing positive and negative samples, the loss function InfoNCE loss of contrastive learning is used to train the model to distinguish positive and negative samples. The formula is as follows:

[0012]

[0013] Among them, z i represents sample i, represents the positive sample of sample i, represents the negative sample of sample i, including the entity that needs to be forgotten, sim(.) represents the similarity function, and τ represents the temperature hyperparameter.

[0014] Preferably, in the process of constructing a large language model, a forgetting weighted loss function is constructed for the negative samples of entities that need to be forgotten, which is used to increase the weight of the negative samples of entities that need to be forgotten in the loss function, thereby prompting the model to reduce the similarity to these entities.

[0015] Preferably, during the model training process, the model parameters are updated through back propagation based on the loss value.

[0016] The present invention also discloses a large language model-based entity disambiguation and forgetting system, which is used to implement the above-mentioned large language model-based entity disambiguation and forgetting method, including:

[0017] The data processing module is used to construct comparative learning samples based on the entity disambiguation dataset and the forgetting dataset, perform data preprocessing to remove irrelevant information, and use the vocabulary of the LLaMA3 model to segment and encode the text to generate a dataset;

[0018] The large language model construction module is used to improve the LLaMA3 model by adding a projection layer and a contrastive learning module. Based on the improved LLaMA3 model, the model is trained according to the dataset to build a large language model. The constructed large language model has the ability to disambiguate entities and forget.

[0019] The present invention discloses the following technical effects:

[0020] (1) The present invention uses the LLaMA3 model as its basic architecture, which fundamentally solves the many defects exposed by previous classic machine learning models (SVM, decision tree), making the model more efficient in entity disambiguation and forgetting tasks, and improving the accuracy and robustness of the model.

[0021] (2) The present invention adds a projection layer after the output layer of the LLaMA3 model to map the high-dimensional text representation to the contrastive learning space. After that, it is further trained through the contrastive learning module to learn to distinguish different entity representations, enhance the entity disambiguation capability, and improve the model's ability to distinguish different entities.

[0022] (3) The LLaMA3 model in the present invention has been pre-trained on a large-scale corpus and possesses rich language knowledge and contextual understanding capabilities. During the training process, negative samples are constructed using contrastive learning to reduce the model's sensitivity to entities that need to be forgotten, thereby achieving the purpose of forgetting. By combining the LLaMA3 large language model with contrastive learning, its powerful language understanding and representation capabilities can be fully utilized, improving the accuracy of entity disambiguation and achieving effective forgetting of specific entity information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 Schematic diagram of the overall framework of the entity disambiguation and forgetting method based on a large language model according to the present invention;

[0025] Figure 2 A schematic diagram of a framework for data preparation according to the present invention;

[0026] Figure 3 Schematic diagram of the framework of the LLaMA3 model of the present invention;

[0027] Figure 4 Schematic diagram of the framework of the comparison module of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0029] like Figure 1-4 As shown, the present invention provides an entity disambiguation and forgetting method based on a large language model, which includes the following steps:

[0030] S1: Data preparation: Figure 2As shown, the given text data is preprocessed, including determining the entity set, constructing sample data, and annotating the data to finally obtain standardized text;

[0031] Step S1: Data preparation, specifically including the following sub-steps:

[0032] S11: Determine the entity set

[0033] The target entity set is the set of entities that the model needs to correctly identify and disambiguate; the forgotten entity set is the set of entities that the model needs to forget, such as outdated, sensitive, or private entities;

[0034] S12: Constructing sample data

[0035] It includes positive samples and negative samples. Positive samples are texts containing target entities, which are encoded using the LLaMA3 model to obtain their representation. There are two types of negative samples. Type 1 is text containing entities that need to be forgotten, which are objects whose similarity should be reduced by the model. Type 2 is text containing other entities that are easily confused with the target entity, which enhances the model's ability to distinguish.

[0036] S13: Data Annotation

[0037] Use named entity recognition (NER) tools (such as spaCy, BioBERT, ScispaCy) to identify and annotate entities in the text to ensure that the entity information of each sample is accurate. Finally, remove redundant characters, unify the case, and form standardized text.

[0038] S2: Model design: Figure 1 As shown in the figure, the standardized text is input into the LLaMA3 model, and after the projection layer and contrastive learning module, the total loss value is output to guide the optimization of model parameters and minimize the contrast loss and forgetting penalty;

[0039] First, the LLaMA model's tokenizer is used to decompose the text into words or subword units and convert the text into integer representations. Then, the text passes through the attention layer and encoding layer in sequence, and then passes through the LLaMA model's multi-layer Transformer encoder to generate context-related feature representations.

[0040] Step S2: Model design, specifically including the following sub-steps:

[0041] S21: LLaMA3 model feature extraction

[0042] Load the LLaMA3 pre-trained model and use its powerful text encoding capabilities to obtain the vector representation of the text from the middle layer or the last layer of the model. The specific steps are as follows:

[0043] Use the word segmenter of the LLaMA3 model to decompose the text into words or subword units, encode the word segmentation results as input IDs, and finally generate an attention mask and convert it into an integer representation. Prepare the input dictionary, which contains the input ID and the attention mask, where the input ID marks the corresponding ID sequence, and the attention mask marks the position of the actual content. Map the input ID to an embedding vector to represent the semantic information of the word. The input ID passes through the embedding layer to obtain the initial word vector representation, that is, the embedding representation. Then, the position information is added to the embedding representation, and then passes through the attention layer and the encoding layer in sequence, and through the multi-layer Transformer encoder of the LLaMA model to generate context-related feature representations;

[0044] S22: Add contrastive learning module

[0045] Add a projection layer after the output layer of the LLaMA3 model to map the high-dimensional text representation to the contrastive learning space and integrate the modules required for contrastive learning, including contrastive loss calculation and positive and negative sample matching. The specific steps are as follows:

[0046] Taking the sequence-level feature representation generated in S21 as input, a linear layer (projection layer) maps the high-dimensional vector to a low-dimensional representation space, and adds a nonlinear activation function (such as ReLU) to enhance the representation ability, generating a low-dimensional feature representation, that is, a numerical vector suitable for contrastive learning;

[0047] like Figure 4 As shown in the figure, low-dimensional feature representation is used as input, and sample pairs are constructed through the contrastive learning module, including positive sample pairs and negative sample pairs. Then, cosine similarity is used to calculate similarity. For positive sample pairs, the similarity of positive sample pairs is shortened to enhance the model's ability to recognize the same entity. For negative sample pairs, the similarity of negative sample pairs is extended to enhance the model's ability to distinguish different entities, especially entities that need to be forgotten. Finally, the loss value and forgetting loss are calculated, and the total loss value is finally obtained to guide the optimization of model parameters and minimize the contrast loss and forgetting penalty.

[0048] S3: Construct loss function: Multiple functions are used for calculation in S22, including: contrast loss function, forgetting loss function and loss adjustment of forgetting mechanism.

[0049] Step S3 constructs the loss function. The details are as follows:

[0050] S31: Contrastive loss function

[0051] The InfoNCE loss function of contrastive learning is used to train the model to distinguish positive and negative samples. The formula is as follows:

[0052]

[0053] Among them, z i represents sample i, represents the positive sample of sample i, represents the negative sample of sample i, including the entity to be forgotten, sim(.) represents the similarity function, and τ represents the temperature hyperparameter;

[0054] S32: Forgetting loss function

[0055] Specifically targeting negative samples of entities to be forgotten, the model is forced to reduce its sensitivity to these entities. The formula is as follows:

[0056]

[0057] Among them, w k Indicates the weight of the negative sample of the entity to be forgotten, Negative sample feature vector representing the entity to be forgotten;

[0058] S33: Loss adjustment of forgetting mechanism

[0059] For negative samples of entities that need to be forgotten, increase their weight in the loss function to encourage the model to reduce the similarity to these entities. The weighted loss formula is as follows:

[0060]

[0061]

[0062] Among them, λ is the forgetting weight coefficient, W k is the weight of the negative sample of the entity to be forgotten, It represents the contrast loss for the entity that needs to be forgotten, and λ represents the forgetting weight coefficient, which controls the strength of forgetting.

[0063] The output of the character-level network, that is, the vector representation of each sentence in the text, is used as input. The output and context vector (that is, the final hidden state) of each time step are obtained through a bidirectional recurrent neural network. The hidden layer dimension is a model hyperparameter and needs to be adjusted according to the specific dataset and training process.

[0064] S4: Model training: Based on the loss value, the model parameters are updated through backpropagation, so that the model can better pull positive samples closer and push negative samples further away. This process is repeated until the model converges. After training, the model has stronger entity disambiguation capabilities and can forget specific entities.

[0065] Step S4: Model training, specifically including the following sub-steps:

[0066] S41: Training Strategy

[0067] Use appropriate batch size to improve training efficiency and stability, adopt learning rate decay or adaptive adjustment strategy to avoid oscillation during training, and use methods such as dropout and weight decay to prevent overfitting;

[0068] S42: Backpropagation and Optimization

[0069] Use the optimization algorithm (AdamW) to update the model parameters and minimize the loss function;

[0070] S43: Model Evaluation

[0071] Including entity disambiguation performance evaluation, forgetting effect evaluation and model generalization ability evaluation.

[0072] The present invention applies the LLaMA3 model to the tasks of entity disambiguation and forgetting, and combines a large-scale language model with an entity disambiguation and forgetting method based on contrastive learning. This can give full play to the powerful representation ability of LLMs, improve the accuracy of entity disambiguation, and achieve effective forgetting of specific entity information. The model: 1. First, the initial text is processed and features are extracted using the LLaMA3 model; 2. A projection layer is added after the output layer of the LLaMA3 model to map the high-dimensional text representation to the contrastive learning space and generate a low-dimensional feature representation, which is a numerical vector suitable for contrastive learning; 3. Using the low-dimensional feature representation as input, the contrastive learning module constructs sample pairs, including positive sample pairs and negative sample pairs, and then uses cosine similarity to calculate similarity. For positive sample pairs, the similarity of the positive sample pairs is shortened to enhance the model's ability to recognize the same entity. For negative sample pairs, the similarity of the negative sample pairs is furthered to improve the model's ability to distinguish different entities, especially the entities that need to be forgotten. Finally, the loss value and the forgetting loss are calculated to obtain the total loss value; 4. Finally, based on the loss value, the model parameters are updated through backpropagation, so that the model can better shorten the positive samples and push the negative samples away. This process is repeated until the model converges. After training, the model has a stronger entity disambiguation ability and achieves forgetting of specified entities.

[0073] The present invention adopts self-supervised learning, does not rely on manually designed features, and can automatically learn the distinguishing features of entities from data; applies contrastive learning to construct positive and negative sample pairs, can dynamically learn new entity features, adapt to new contexts, and utilize the powerful representation ability of large-scale language models to flexibly respond to long-tail entities and new knowledge updating needs. In particular, it greatly improves efficiency in fine-tuning and precise control of large models, expanding the limitations of traditional methods.

[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0076] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for entity disambiguation and forgetting based on a large language model, characterized in that: The following steps are involved: Based on the entity disambiguation dataset and the forgetting dataset, after constructing the comparative learning samples, data preprocessing was performed to remove irrelevant information. The vocabulary of the LLaMA3 model was used to segment and encode the text to generate the dataset. Based on the LLaMA3 model, the model is improved by adding a projection layer and a contrastive learning module, and the improved LLaMA3 model is trained according to the dataset to construct a large language model, so that the constructed large language model has the capabilities of entity disambiguation and forgetting; In the process of improving the LLaMA3 model, a projection layer is added after the output layer of the LLaMA3 model to map the high-dimensional text representation to the contrastive learning space and integrate the modules required for contrastive learning, including contrastive loss calculation and positive and negative sample matching; During model training, for each sample, cosine similarity is used to calculate the similarity between its feature representation and the positive and negative samples, and contrast loss is used to measure the effectiveness of the model in distinguishing positive and negative samples. In the process of using contrast loss to measure the effect of the model in distinguishing positive and negative samples, the loss function of contrastive learning, InfoNCE loss, is used to train the model to distinguish positive and negative samples. The formula is as follows: ,in, represents sample i, represents the positive sample of sample i, represents the negative sample of sample i, including the entities that need to be forgotten, represents the similarity function, represents the temperature hyperparameter.

2. The entity disambiguation and forgetting method based on a large language model according to claim 1, characterized in that: In the process of constructing contrastive learning samples, the contrastive learning samples include positive samples and negative samples, wherein the positive samples include the text of the target entity, which is encoded using the LLaMA3 model to obtain its representation; the negative samples include two types, type one is the text containing the entity that needs to be forgotten, as the object whose similarity the model should reduce, and type two is the text containing other entities that are easily confused with the target entity.

3. The entity disambiguation and forgetting method based on a large language model according to claim 1, characterized in that: In the process of building a large language model, a forgetfulness-weighted loss function is constructed for the negative samples of entities that need to be forgotten. This function increases the weight of the negative samples of entities that need to be forgotten in the loss function, prompting the model to reduce the similarity of these entities.

4. The entity disambiguation and forgetting method based on a large language model according to claim 3, characterized in that: During model training, the model parameters are updated through back propagation based on the loss value.

5. A large language model-based entity disambiguation and forgetting system, used to implement the large language model-based entity disambiguation and forgetting method according to any one of claims 1 to 4, characterized in that: include: The data processing module is used to construct comparative learning samples based on the entity disambiguation dataset and the forgetting dataset, perform data preprocessing to remove irrelevant information, and use the vocabulary of the LLaMA3 model to segment and encode the text to generate a dataset; A large language model construction module is used to improve the LLaMA3 model by adding a projection layer and a contrastive learning module, and to perform model training based on the improved LLaMA3 model and the data set to construct a large language model, so that the constructed large language model has the ability of entity disambiguation and forgetting.

Images