Named entity recognition method based on prompt learning

Abstract

The invention discloses a named entity recognition method based on prompt learning, which comprises the following steps: inputting query content in a preset data set for retrieval to obtain a storage document with the highest similarity as a discrete prompt; the storage document of the discrete prompt is combined with the continuous prompt to be input into a large language model; the large language model obtains k prompts which are most similar to the input storage documents and continuous prompts through information retrieval; according to the method, a prompt learning method is used, several sentences most similar to a target are searched in a small amount of preset data sets through a retrieval enhancement method to serve as discrete prompts, and then an extracted result is generated by combining continuous prompts and using LLM as a generative model.

Description

Technical Field

[0001] The present invention relates to the technical field of information recognition, and in particular to a named entity recognition method based on prompt learning. Background Art

[0002] Named Entity Recognition (NER) is a common information extraction task, which is to identify and classify key information in text. Driven by the Transformer architecture, this pre-trained language model has gradually improved the state-of-the-art NER performance in the past decade. However, both training models from scratch and fine-tuning existing models require a large amount of data customized for specific domains, which can be highly cost-intensive in practical applications. In recent years, basic LLMs, such as the GPT model, have shown excellent capabilities in knowledge reasoning and information extraction, providing us with an alternative solution for NER.

[0003] However, the existing LLM-NER methods have the following problems: it is difficult to collect a large amount of high-quality data for training; the recognition ability of rare entities in specific fields is weak; and the model is not very general. Summary of the invention

[0004] The purpose of the present invention is to provide a named entity recognition method based on prompt learning to solve the technical problems in the prior art that it is difficult to collect a large amount of high-quality data for training; the recognition ability of rare entities in specific fields is weak; and the model is not very general.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions:

[0006] A method for named entity recognition based on prompt learning comprises the following steps:

[0007] Step 100: Enter the query content in the preset data set for retrieval, and obtain the stored document with the highest similarity as a discrete prompt;

[0008] Step 200: The stored documents of discrete prompts are combined with continuous prompts and input into a large language model;

[0009] Step 300: The large language model obtains k prompts that are most similar to the input stored documents and continuous prompts through information retrieval.

[0010] As a preferred solution of the present invention, in step 100, the specific implementation method of obtaining the stored document with the highest similarity is:

[0011] The correlation between the query content and the stored documents is estimated by combining the term frequency and inverse document frequency of the query content and the stored documents in the preset data set, and the stored documents are ranked according to the correlation between the query content and the stored documents to obtain the stored documents with the highest similarity.

[0012] As a preferred solution of the present invention, when estimating the correlation between the query content and the stored document by combining the word frequency of the query content and the stored document, the specific calculation formula is:

[0013]

[0014] Where f = (q i ,D) is the query content q i The frequency of the term in document D.

[0015] k1 and b are commonly used adjustment parameters. k1 controls the influence of word frequency, and b controls the influence of document length.

[0016] |D| is the length of the document, avgdl is the average length of all stored documents;

[0017] IDF(q i ) is the query word q i The inverse document frequency of

[0018] IDF(q i ) is the query word q i The inverse document frequency is calculated as:

[0019]

[0020] Where N is the total number of documents, n(q i ) is a string containing the query word q i The number of documents.

[0021] As a preferred solution of the present invention, in step 200, obtaining continuous prompts is specifically as follows:

[0022] Learning a set of additional hint embedding vectors for the input query content, and combining the hint embedding vectors with the text embedding vector formed by the actual input query content;

[0023] Combining the text embedding vector with the word frequency and inverse document frequency of the stored documents in the preset data set to estimate the correlation between the query content and the stored documents, and obtain the stored documents with the highest similarity;

[0024] When the stored documents with the highest similarity obtained from the two types of input contents, the text embedding combination vector and the query content, are the same, the corresponding text embedding combination vector in this case is determined to obtain continuous prompts.

[0025] As a preferred solution of the present invention, in step 300, the large language model processes the input content formed by combining the discrete prompts with the continuous prompts in the following manner:

[0026] The input content formed by combining the discrete prompts with the continuous prompts is input into the encoder, and the obtained encoding vector is input into the decoder, and then the embedded high-dimensional vector is classified into a label for output.

[0027] As a preferred solution of the present invention, the specific implementation method of the large language model output data is:

[0028] For a given input sentence X, denoted as Prompt(X);

[0029] The prompt construct Prompt(X) is fed into the large language model to obtain the generated text sequence W = {w1, ..., wn;

[0030] Transform the text sequence W into an entity label sequence to obtain the final result.

[0031] As a preferred solution of the present invention, in step 100, the preset data set is classified according to the field, and the specific classification implementation method is:

[0032] Pre-train the large language model and use the large prediction model to process massive general text data, including encyclopedia articles, news, scientific papers, and technical documents, to obtain entities in specific fields and their contextual descriptions, so as to classify all encyclopedia articles, news, scientific papers, and technical documents to form preset data sets classified in different fields.

[0033] As a preferred solution of the present invention, the large language model learns the contextual relationship of entities in a specific field through a self-attention mechanism, so that the large language model can quickly locate and identify related entities.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention uses a prompt learning method to find several sentences that are most similar to the target in a small number of pre-set data sets as discrete prompts through a retrieval enhancement method, and then combines continuous prompts and uses LLM as a generative model to generate the extracted results. The user only needs to provide a clear task description, and the model can use its general knowledge to reason. For example, the prompt "Extract the name of the chemical element from the following text" can give reasonable results even without fine-tuning. If a small number of annotated samples are provided, the model can quickly adapt to the field through its existing knowledge, so it has strong zero-sample and few-sample learning capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0037] Figure 1 A flowchart of a method for named entity recognition according to an embodiment of the present invention is shown; DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, the present invention provides a method for named entity recognition based on prompt learning, comprising the following steps:

[0040] Step 100: Enter query content in a preset data set for retrieval, and obtain the stored document with the highest similarity as a discrete prompt.

[0041] In step 100, the preset data set is classified according to the field, and the specific classification implementation method is as follows:

[0042] Pre-train the large language model and use the large prediction model to process massive general text data, including encyclopedia articles, news, scientific papers, and technical documents, to obtain entities in specific fields and their contextual descriptions, so as to classify all encyclopedia articles, news, scientific papers, and technical documents to form preset data sets classified in different fields.

[0043] The large language model learns the contextual relationships of entities in a specific domain through a self-attention mechanism, so that the large language model can quickly locate and identify related entities.

[0044] Specifically, the large language model learns contextual relationships through the self-attention mechanism (Transformer architecture) and has a deeper understanding of entities in specific fields. In specific fields, language patterns (such as term collocation and grammatical structure) usually have strong regularities, and the large language model can quickly locate and identify related entities through these regularities.

[0045] For a specific query term, its frequency in relevant documents is higher than that in non-relevant documents. The algorithm calculates the score of the document by combining the term frequency (TF) and the document frequency (DF). In step 100, the specific implementation method of obtaining the stored document with the highest similarity is:

[0046] The correlation between the query content and the stored documents is estimated by combining the term frequency and inverse document frequency of the query content and the stored documents in the preset data set, and the stored documents are ranked according to the correlation between the query content and the stored documents to obtain the stored documents with the highest similarity.

[0047] When estimating the correlation between the query content and the stored document by combining the word frequency of the query content and the stored document, the specific calculation formula is:

[0048]

[0049] Where f = (q i ,D) is the query content q i The frequency of the term in document D.

[0050] k1 and b are commonly used adjustment parameters. k1 controls the influence of word frequency, and b controls the influence of document length.

[0051] |D| is the length of the document, avgdl is the average length of all stored documents;

[0052] IDF(q i ) is the query word q i The inverse document frequency of

[0053] IDF(q i ) is the query word q i The inverse document frequency is calculated as:

[0054]

[0055] Where N is the total number of documents, n(q i) is a string containing the query word q i The number of documents.

[0056] TF (Term Frequency) refers to the number of times a term appears in a document. BM25 adjusts the traditional TF calculation method and introduces saturation and length normalization to prevent long documents from getting unfairly high scores due to containing more terms.

[0057] IDF (Inverse Document Frequency) is a measure of the rarity of a term. It is calculated based on the entire document collection and is used to reduce the weight of common terms and increase the weight of rare terms.

[0058] Step 200: The stored documents of discrete prompts are combined with continuous prompts and input into a large language model.

[0059] In step 200, obtaining continuous prompts is specifically as follows:

[0060] Learning a set of additional hint embedding vectors for the input query content, and combining the hint embedding vectors with the text embedding vector formed by the actual input query content;

[0061] Combining the text embedding vector with the word frequency and inverse document frequency of the stored documents in the preset data set to estimate the correlation between the query content and the stored documents, and obtain the stored documents with the highest similarity;

[0062] When the stored documents with the highest similarity obtained from the two types of input contents, the text embedding combination vector and the query content, are the same, the corresponding text embedding combination vector in this case is determined to obtain continuous prompts.

[0063] Discrete prompts are obtained based on information retrieval, and continuous prompts are also called soft prompts. They are used as training parameters of the model. During the training process, soft prompts significantly improve the performance of the model by adding additional contextual information to the input text. They are learned and optimized during the training process of the model.

[0064] Continuous prompts are a technique for optimizing input prompts based on a large language model (LLM), especially for improving the performance of models in few-shot learning or specific tasks. The core principle of soft prompts is to guide the model to produce the correct output of the target task by optimizing the input prompts (usually embedding vectors) instead of modifying the parameters of the entire model.

[0065] The approach of continuous prompting is to learn a set of additional prompt embedding vectors for a specific task, combine them with the actual input text embedding vector, and feed them into the model for processing. The purpose of optimizing these prompt vectors is to enable them to guide the model to understand a specific task. In the named entity recognition (NER) task, the original input is a sentence containing text, and continuous prompts are inserted at the front of the input as additional vectors. The model will process this information to identify relevant entities in the text or perform other tasks.

[0066] The optimization of continuous prompts is performed through optimization algorithms such as gradient descent. By training the model on a specific task, the optimization goal is to enable the model to better handle the task and accurately predict the results. The optimization process only updates the soft prompt vector without adjusting other parameters of the model.

[0067] Step 300: The large language model obtains k prompts that are most similar to the input stored documents and continuous prompts through information retrieval.

[0068] In step 300, the large language model processes the input content formed by combining the discrete prompts with the continuous prompts in the following manner:

[0069] The input content formed by combining the discrete prompts with the continuous prompts is input into the encoder, and the obtained encoding vector is input into the decoder, and then the embedded high-dimensional vector is classified into a label for output.

[0070] The specific implementation method of the large language model output data is:

[0071] For a given input sentence X, denoted as Prompt(X);

[0072] The prompt construct Prompt(X) is fed into the large language model to obtain the generated text sequence W = {w1, ..., wn;

[0073] Transform the text sequence W into an entity label sequence to obtain the final result.

[0074] For NER tasks, users can design domain-specific discrete or continuous prompts. For example, the prompt "Extract all medical terms from the following text" will significantly improve the recognition accuracy.

[0075] The user only needs to provide a clear task description, and the model can use its general knowledge to reason. For example, the prompt "Extract the names of chemical elements from the following text" can give reasonable results even without fine-tuning. If a small number of annotated samples are provided, the model can quickly adapt to the field through its existing knowledge, so it has strong zero-shot and few-shot learning capabilities.

[0076] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A method for named entity recognition based on prompt learning, characterized in that: The following steps are involved: Step 100: Enter the query content in the preset data set for retrieval, and obtain the stored document with the highest similarity as a discrete prompt; Step 200: The stored documents of discrete prompts are combined with continuous prompts and input into the large language model; Step 300: The large language model obtains k prompts that are most similar to the input stored documents and continuous prompts through information retrieval.

2. A method for named entity recognition based on prompt learning according to claim 1, characterized in that: In step 100, the specific implementation method of obtaining the stored document with the highest similarity is: The correlation between the query content and the stored documents is estimated by combining the term frequency and inverse document frequency of the query content and the stored documents in the preset data set, and the stored documents are ranked according to the correlation between the query content and the stored documents to obtain the stored documents with the highest similarity.

3. A method for named entity recognition based on prompt learning according to claim 2, characterized in that: When estimating the correlation between the query content and the stored document by combining the word frequency of the query content and the stored document, the specific calculation formula is: Where f = (q i ,D) is the query content q i The frequency of the term in document D. k1 and b are commonly used adjustment parameters. k1 controls the influence of word frequency, and b controls the influence of document length. |D| is the length of the document, avgdl is the average length of all stored documents; IDF(q i ) is the query word q i The inverse document frequency of IDF(q i ) is the query word q i The inverse document frequency is calculated as: Where N is the total number of documents, n(q i ) is a string containing the query word q i The number of documents.

4. The method for named entity recognition based on prompt learning according to claim 2, characterized in that: In step 200, obtaining continuous prompts is specifically as follows: Learning a set of additional hint embedding vectors for the input query content, and combining the hint embedding vectors with the text embedding vector formed by the actual input query content; Combining the text embedding vector with the word frequency and inverse document frequency of the stored documents in the preset data set to estimate the correlation between the query content and the stored documents, and obtain the stored documents with the highest similarity; When the stored documents with the highest similarity obtained from the two types of input contents, the text embedding combination vector and the query content, are the same, the corresponding text embedding combination vector in this case is determined to obtain continuous prompts.

5. A method for named entity recognition based on prompt learning according to claim 4, characterized in that: In step 300, the large language model processes the input content formed by combining the discrete prompts with the continuous prompts in the following manner: The input content formed by combining the discrete prompts with the continuous prompts is input into the encoder, and the obtained encoding vector is input into the decoder, and then the embedded high-dimensional vector is classified into a label for output.

6. A method for named entity recognition based on prompt learning according to claim 4, characterized in that: The specific implementation method of the large language model output data is: For a given input sentence X, denoted as Prompt(X); The prompt construct Prompt(X) is fed into the large language model to obtain the generated text sequence W = {w1, ..., wn; Transform the text sequence W into an entity label sequence to obtain the final result.

7. The method for named entity recognition based on prompt learning according to claim 1, characterized in that: In step 100, the preset data set is classified according to the field, and the specific classification implementation method is as follows: Pre-train the large language model and use the large prediction model to process massive general text data, including encyclopedia articles, news, scientific papers, and technical documents, to obtain entities in specific fields and their contextual descriptions, so as to classify all encyclopedia articles, news, scientific papers, and technical documents to form preset data sets classified in different fields.

8. A method for named entity recognition based on prompt learning according to claim 7, characterized in that: The large language model learns the contextual relationships of entities in a specific domain through a self-attention mechanism, so that the large language model can quickly locate and identify related entities.

Images