A prompt learning-based named entity recognition method

Abstract

The application discloses a named entity recognition method based on prompt learning, comprising the following steps: inputting query content in a preset data set for retrieval to obtain a storage document with the highest similarity as a discrete prompt; inputting the storage document of the discrete prompt and a continuous prompt into a large language model; the large language model obtains k prompts most similar to the input storage document and the continuous prompt through information retrieval; the method of the application uses prompt learning, finds several sentences most similar to a target in a small amount of data set pre-set through a retrieval enhancement method as discrete prompts, then combines the continuous prompts and uses an LLM as a generation model to generate an extracted result.

Description

Technical Field

[0001] The present invention relates to the technical field of information recognition, and in particular to a named entity recognition method based on prompt learning. Background Art

[0002] Named Entity Recognition (NER) is a common information extraction task that involves identifying and classifying key information in text. Driven by the Transformer architecture, this pre-trained language model has gradually improved the state-of-the-art NER performance over the past decade. However, both training a model from scratch and fine-tuning an existing model requires a large amount of data tailored to the specific domain, which can be highly cost-intensive in real-world applications. In recent years, basic LLMs, such as the GPT model, have demonstrated remarkable capabilities in knowledge reasoning and information extraction, providing us with an alternative solution for NER.

[0003] However, the existing LLM-NER methods have the following problems: it is difficult to collect a large amount of high-quality data for training; the recognition ability of rare entities in specific fields is weak; and the model is not very general. Summary of the Invention

[0004] The purpose of the present invention is to provide a named entity recognition method based on prompt learning to solve the technical problems in the prior art, such as the difficulty in collecting a large amount of high-quality data for training; the weak recognition ability of rare entities in specific fields; and the low versatility of the model.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] A method for named entity recognition based on prompt learning includes the following steps:

[0007] Step 100: Enter the query content in the preset data set to search and obtain the stored document with the highest similarity as a discrete prompt;

[0008] Step 200: The stored document of discrete prompts is combined with the continuous prompts and inputted into the large language model;

[0009] Step 300: The large language model obtains k prompts that are most similar to the input stored documents and continuous prompts through information retrieval.

[0010] As a preferred solution of the present invention, in step 100, the specific implementation method of obtaining the stored document with the highest similarity is:

[0011] The correlation between the query content and the stored documents is estimated by combining the term frequency and inverse document frequency of the query content and the stored documents in the preset data set, and the stored documents are ranked according to the correlation between the query content and the stored documents to obtain the stored documents with the highest similarity.

[0012] As a preferred solution of the present invention, when estimating the correlation between the query content and the stored document by combining the word frequencies of the query content and the stored document, the specific calculation formula is:

[0013]

[0014] Where, f=(q i ,D) is the query content q i Frequency of the term in document D.

[0015] k1 and b are commonly used adjustment parameters. k1 controls the influence of word frequency, and b controls the influence of document length.

[0016] |D| is the length of the document, avgdl is the average length of all stored documents;

[0017] IDF(q i ) is the query word q i The inverse document frequency of

[0018] IDF(q i ) is the query word q i The inverse document frequency is calculated as:

[0019]

[0020] Where N is the total number of documents, n(q i ) is a string containing the query word q i The number of documents.

[0021] As a preferred solution of the present invention, in step 200, obtaining continuous prompts is specifically as follows:

[0022] Learning a set of additional hint embedding vectors for the input query content, and combining the hint embedding vectors with the text embedding vector formed by the actual input query content;

[0023] Combining the text embedding vector with the word frequency and inverse document frequency of the stored documents in the preset data set to estimate the correlation between the query content and the stored documents, and obtaining the stored documents with the highest similarity;

[0024] When the stored documents with the highest similarity obtained from the text embedding combination vector and the query content are the same, the corresponding text embedding combination vector in this case is determined to obtain continuous prompts.

[0025] As a preferred solution of the present invention, in step 300, the large language model processes the input content formed by combining the discrete prompts with the continuous prompts in the following manner:

[0026] The input content formed by combining the discrete prompts with the continuous prompts is input into the encoder, and the obtained encoding vector is input into the decoder, and then the embedded high-dimensional vector is classified into a label for output.

[0027] As a preferred solution of the present invention, the specific implementation of the large language model output data is as follows:

[0028] For a given input sentence X, denoted as Prompt(X);

[0029] Feed the prompt construct Prompt(X) into the large language model to obtain the generated text sequence W = {w1, ..., wn};

[0030] Transform the text sequence W into an entity label sequence to obtain the final result.

[0031] As a preferred solution of the present invention, in step 100, the preset data set is classified according to the field, and the specific classification implementation method is:

[0032] Pre-train the large language model and use the large prediction model to process massive amounts of general text data, including encyclopedia articles, news, scientific papers, and technical documents, to obtain entities in specific fields and their contextual descriptions, thereby classifying all encyclopedia articles, news, scientific papers, and technical documents to form preset data sets classified in different fields.

[0033] As a preferred solution of the present invention, the large language model learns the contextual relationships of entities in a specific field through a self-attention mechanism, so that the large language model can quickly locate and identify related entities.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This paper uses a prompt learning approach, employing a retrieval-enhanced method to find the most similar sentences to the target in a small, pre-set dataset as discrete prompts. This approach then combines continuous prompts with an LLM as a generative model to generate the extracted results. The user simply provides a clear task description, and the model leverages its general knowledge to reason. For example, even with no fine-tuning, the prompt "Extract the names of chemical elements from the following text" produces reasonable results. Given a small number of labeled examples, the model can quickly adapt to the domain using its existing knowledge, resulting in strong zero- and few-shot learning capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0037] Figure 1 Schematic diagram of the process of the named entity recognition method according to an embodiment of the present invention; DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, the present invention provides a method for named entity recognition based on prompt learning, comprising the following steps:

[0040] Step 100: Enter query content in the preset data set to search and obtain the stored documents with the highest similarity as discrete prompts.

[0041] In step 100, the preset data set is classified according to fields. The specific classification implementation method is as follows:

[0042] Pre-train the large language model and use the large prediction model to process massive amounts of general text data, including encyclopedia articles, news, scientific papers, and technical documents, to obtain entities in specific fields and their contextual descriptions, thereby classifying all encyclopedia articles, news, scientific papers, and technical documents to form preset data sets classified in different fields.

[0043] The large language model learns the contextual relationships of entities in a specific domain through a self-attention mechanism, so that the large language model can quickly locate and identify related entities.

[0044] Specifically, the large language model learns contextual relationships through the self-attention mechanism (Transformer architecture) and has a deeper understanding of entities in specific fields. In specific fields, language patterns (such as term collocation and grammatical structure) usually have strong regularities, and the large language model can quickly locate and identify related entities based on these regularities.

[0045] For a specific query term, it appears more frequently in relevant documents than in non-relevant documents. The algorithm calculates the document score by combining the term frequency (TF) and document frequency (DF). In step 100, the specific implementation method for obtaining the stored document with the highest similarity is as follows:

[0046] The correlation between the query content and the stored documents is estimated by combining the term frequency and inverse document frequency of the query content and the stored documents in the preset data set, and the stored documents are ranked according to the correlation between the query content and the stored documents to obtain the stored documents with the highest similarity.

[0047] When estimating the relevance between the query content and the stored document by combining the word frequencies of the query content and the stored document, the specific calculation formula is:

[0048]

[0049] Where, f=(q i ,D) is the query content q i Frequency of the term in document D.

[0050] k1 and b are commonly used adjustment parameters. k1 controls the influence of word frequency, and b controls the influence of document length.

[0051] |D| is the length of the document, avgdl is the average length of all stored documents;

[0052] IDF(q i ) is the query word q i The inverse document frequency of

[0053] IDF(q i ) is the query word q i The inverse document frequency is calculated as:

[0054]

[0055] Where N is the total number of documents, n(q i) is a string containing the query word q i The number of documents.

[0056] Term frequency (TF) refers to the number of times a term appears in a document. BM25 adjusts the traditional TF calculation method, introducing saturation and length normalization to prevent long documents from receiving unfairly high scores due to containing more terms.

[0057] IDF (Inverse Document Frequency) is a metric that measures the rarity of a term. It is calculated based on the entire document collection and is used to reduce the weight of common terms and increase the weight of rare terms.

[0058] Step 200: The stored documents of discrete prompts are combined with continuous prompts and input into the large language model.

[0059] In step 200, obtaining continuous prompts is specifically as follows:

[0060] Learning a set of additional hint embedding vectors for the input query content, and combining the hint embedding vectors with the text embedding vector formed by the actual input query content;

[0061] Combining the text embedding vector with the word frequency and inverse document frequency of the stored documents in the preset data set to estimate the correlation between the query content and the stored documents, and obtaining the stored documents with the highest similarity;

[0062] When the stored documents with the highest similarity obtained from the text embedding combination vector and the query content are the same, the corresponding text embedding combination vector in this case is determined to obtain continuous prompts.

[0063] Discrete hints are derived from information retrieval, while continuous hints, also known as soft hints, serve as model training parameters. During training, soft hints significantly improve model performance by adding additional context to the input text. They are learned and optimized during model training.

[0064] Continuous hinting is a technique for optimizing input hints based on a large language model (LLM), specifically designed to improve model performance in few-shot learning or on specific tasks. The core principle of soft hinting is to guide the model to produce the correct output for the target task by optimizing input hints (usually embedding vectors) rather than modifying the entire model parameters.

[0065] Continuous hinting involves learning a set of additional hint embedding vectors for a specific task, combining them with the actual input text embedding vector, and feeding them into the model for processing. The goal of optimizing these hint vectors is to guide the model's understanding of the specific task. In named entity recognition (NER) tasks, the original input is a sentence containing text, and continuous hints are inserted as additional vectors at the beginning of the input. The model then processes this information to identify relevant entities in the text or perform other tasks.

[0066] Continuous hint optimization is performed using optimization algorithms such as gradient descent. By training a model on a specific task, the optimization goal is to enable the model to better handle the task and accurately predict the outcome. The optimization process only updates the soft hint vector and does not adjust other model parameters.

[0067] Step 300: The large language model obtains k prompts that are most similar to the input stored documents and continuous prompts through information retrieval.

[0068] In step 300, the large language model processes the input content formed by combining the discrete prompts with the continuous prompts in the following manner:

[0069] The input content formed by combining the discrete prompts with the continuous prompts is input into the encoder, and the obtained encoding vector is input into the decoder, and then the embedded high-dimensional vector is classified into a label for output.

[0070] The specific implementation method of the large language model output data is:

[0071] For a given input sentence X, denoted as Prompt(X);

[0072] Feed the prompt construct Prompt(X) into the large language model to obtain the generated text sequence W = {w1, ..., wn};

[0073] Transform the text sequence W into an entity label sequence to obtain the final result.

[0074] For NER tasks, users can design domain-specific discrete or continuous prompts. For example, the prompt "Extract all medical terms from the following text" will significantly improve recognition accuracy.

[0075] The user simply provides a clear task description, and the model leverages its general knowledge to reason. For example, given the prompt "Extract chemical element names from the following text," the model can produce reasonable results even without fine-tuning. Given a small number of labeled examples, the model can quickly adapt to the domain using its existing knowledge, resulting in strong zero-shot and few-shot learning capabilities.

[0076] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A method for named entity recognition based on prompt learning, characterized in that: The following steps are involved: Step 100: Enter the query content in the preset data set to search and obtain the stored document with the highest similarity as a discrete prompt; Step 200: The stored document of discrete prompts is combined with the continuous prompts and inputted into the large language model; Step 300: The large language model obtains k prompts that are most similar to the input stored documents and continuous prompts through information retrieval; In step 200, obtaining continuous prompts is specifically as follows: Learning a set of additional hint embedding vectors for the input query content, and combining the hint embedding vectors with the text embedding vector formed by the actual input query content; Combining the text embedding vector with the word frequency and inverse document frequency of the stored documents in the preset data set to estimate the correlation between the query content and the stored documents, and obtaining the stored documents with the highest similarity; When the stored documents with the highest similarity obtained from the text embedding combination vector and the query content are the same, the corresponding text embedding combination vector is determined in this case to obtain continuous prompts; In step 300, the large language model processes the input content formed by combining the discrete prompts with the continuous prompts in the following manner: The input content formed by combining the discrete prompts with the continuous prompts is input into the encoder, and the obtained encoding vector is input into the decoder, and then the embedded high-dimensional vector is classified into a label for output; The specific implementation method of the large language model output data is: For a given input sentence X, denoted as Prompt(X); Feed the prompt construct Prompt(X) into the large language model to obtain the generated text sequence W={w1,...,wn}; Transform the text sequence W into an entity label sequence to obtain the final result.

2. A method for named entity recognition based on prompt learning according to claim 1, characterized in that In step 100, the specific implementation method of obtaining the stored document with the highest similarity is: The correlation between the query content and the stored documents is estimated by combining the term frequency and inverse document frequency of the query content and the stored documents in the preset data set, and the stored documents are ranked according to the correlation between the query content and the stored documents to obtain the stored documents with the highest similarity.

3. A method for named entity recognition based on prompt learning according to claim 2, characterized in that: When estimating the relevance between the query content and the stored document by combining the word frequencies of the query content and the stored document, the specific calculation formula is: ； in, Is the query content In the documentation word frequency in ; and b are commonly used adjustment parameters, Control the influence of word frequency, and b controls the influence of document length; is the length of the document, is the average length of all stored documents; Is the query word The inverse document frequency of Is the query word The inverse document frequency is calculated as: ； Where N is the total number of documents, n( ) is the query word The number of documents.

4. The method for named entity recognition based on prompt learning according to claim 1, characterized in that: In step 100, the preset data set is classified according to fields. The specific classification implementation method is as follows: Pre-train the large language model and use the large prediction model to process massive amounts of general text data, including encyclopedia articles, news, scientific papers, and technical documents, to obtain entities in specific fields and their contextual descriptions, thereby classifying all encyclopedia articles, news, scientific papers, and technical documents to form preset data sets classified in different fields.

5. A method for named entity recognition based on prompt learning according to claim 4, characterized in that: The large language model learns the contextual relationships of entities in a specific domain through a self-attention mechanism, so that the large language model can quickly locate and identify related entities.

Images