Small sample general named entity identification method

Abstract

The invention discloses a small sample general named entity recognition method, which comprises the following steps of: 1) dividing each sample in a training set into a plurality of sentences, and filling a template for each sentence; wherein the sample comprises a plurality of sentences, and each sentence comprises a plurality of entities; setting a filling frame for each type of entity in the template; filling the entities in the sentences into corresponding filling frames according to the types of the entities; 2) splitting the loss function of the selected large model into a sub-loss function L1 of an input masking strategy and a sub-loss function L2 based on a non-masking strategy; using a template filled according to the sample in the training set as the input of the selected large model to train the large model; and 3) for a to-be-recognized text, dividing the to-be-recognized text into a plurality of sentences, inputting the template filled according to the sentences in the to-be-recognized text into the trained large model, and predicting and outputting entity names corresponding to various types in the to-be-recognized text.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology and relates to a small-sample general named entity recognition method based on prompting learning and loss function optimization of large models. Background Technology

[0002] In recent years, with the rapid development of information technology, products based on natural language processing (NLP) technologies, such as dialogue generation, sentiment analysis, and machine translation, have provided people with speed and convenience. Faced with massive amounts of data, these products need to first transform unstructured text data into structured text data before they can further utilize the data information to provide stable and effective services. The core technology for transforming unstructured text data into structured text data is named entity recognition (NAME). By extracting specific categories of entities such as people, times, and places from the text, downstream tasks can obtain more accurate processing objects, thereby effectively improving the efficiency and accuracy of the entire system.

[0003] Specifically, named entity recognition (NAME) tasks can be categorized into three types based on the structural features of entity objects: ordinary NAME recognition, nested entity recognition, and discontinuous NAME recognition. A general NAME recognition technique is one that can recognize not only ordinary entities but also effectively handle NAME recognition composed of nested and discontinuous entities. Due to its wide range of applications and strong adaptability, general NAME recognition has recently attracted the interest of scholars. Existing general NAME recognition techniques mainly employ methods based on pre-trained language models. The core of these methods is to treat NAME recognition as a sequence labeling or sequence generation task, thereby utilizing the pre-trained knowledge of the language model to achieve entity recognition. Most of these methods utilize an encoder to learn the features of each word in the text, and then use a decoder or additional classifier to select the words with the strongest feature associations and categories to form the recognition result. For example, foreign scholars have used the BERT pre-trained language model to learn the features of each word, used conditional random fields to determine whether there are contextual relationships between words to find entities, and finally designed the BIOHD general representation method to label entities with three structural features, achieving general NAME recognition. The success of these methods has provided significant support and assistance for information mining and acquisition, especially in information extraction in fields such as medicine, biology, and chemistry.

[0004] A major drawback of current mainstream general-purpose named entity recognition (NAME) technologies is their severely limited performance under conditions of insufficient training data and small sample sizes. In experimental settings for traditional general-purpose NAME methods, the amount of training data typically exceeds 1000 entries per category. However, in real-world applications, the amount of training data is extremely limited, often less than a hundred or even just a few dozen entries per category. This is because in many fields, such as military and medical, labeled data requires specialized knowledge or is subject to privacy and confidentiality restrictions, making the acquisition of large amounts of training data extremely costly. Existing general-purpose entity recognition methods based on pre-trained language models suffer from a significant decline in learning ability for NAME recognition tasks under insufficient data conditions due to the small parameter size of the pre-trained language models and the limited and unbalanced types and quantities of pre-trained tasks. This can lead to a drop in recognition accuracy and recall to only half or even one-third of the level achieved with sufficient training data. Therefore, this performance deficiency can greatly hinder the real-world application of traditional general-purpose entity recognition technologies. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide a small-sample general named entity recognition method based on cue learning and loss function optimization of a large model. This invention solves the problem of poor recognition ability in small-sample scenarios caused by the limited performance of pre-trained language models by applying a large model. This method utilizes a general cue template to guide the large model to fully leverage its rich pre-trained knowledge and improves the original loss function of the large model to better suit named entity recognition tasks. First, the general cue template contains labeled instructions in the input and directly outputs the entity name or "not found". Benefiting from this, the large model can not only be guided to recognize and accurately represent common, nested, and discontinuous entities, but also reduces the time cost of the method, eliminating the need for multiple inputs or outputs for each entity category. Second, to better apply the large model to entity recognition tasks, we propose a large model loss function optimization based on a masking strategy, ensuring that the loss function aligns with the task objective. By applying input masking and unmasking strategies in the construction of the loss function, the optimized loss function of this invention can simultaneously focus on the correctness of generated content and the completeness of recognition, thereby improving the accuracy and recall of entity recognition.

[0006] A method for small-sample general named entity recognition based on cue learning and loss function optimization of large models is as follows: Figure 1 As shown, the specific steps include:

[0007] Step 1. Data preprocessing: Divide the sample set into training and testing sets. For each sentence, fill the corresponding positions in the input template with entities of each target type from the text. If no entity of that type exists, fill in "Does not exist" in the corresponding position in the template. This process should ensure that the number of entities of each target type is consistent in the training set.

[0008] Each sample in the sample set includes n sentences, and each sentence includes m entities. A template is created for each sentence in the sample, with a fill box for each entity type. Based on the entity type, the m entities in the sentence are filled into the corresponding fill boxes in the template (for example, if a sentence has 5 entities, 3 of type 1, 2 of type 2, and 0 of type 3, then the 3 type 1 entities are filled into the type 1 boxes in the template, the 2 type 2 entities are filled into the type 2 boxes, and the type 3 boxes are filled with "not found". The advantage of this design is that all recognition results can be obtained with a single output for each sentence. Compared to existing methods that require outputting once for each entity type or each entity result, this template design has a lower time cost during application. During training, existing methods require constructing negative samples separately for the model, otherwise, non-entity results are easily output. This invention cleverly introduces the "not found" content in the template, avoiding extra work in the training preprocessing stage.

[0009] Step 2. Construct a large model with optimized loss function (using GPT or a similar structure). Use the input templates containing the completed statements to be recognized as input to the large model, maintaining its other structure, and improve its loss function. The original loss function of the large model is split into two parts. Input masking and unmasked strategies are applied to the calculation range of each part of the loss function. The two parts are then assigned different weights and added together, resulting in a total weight of 1.

[0010] Step 3. Train the candidate entity generation module. First, the input content from the training data is used as input to the large model. The shallow attention mechanism in the large model represents each word segment as a weighted average of all word segments, thus including correlation features between word segments and between word segments and categories in the segment features. Then, the deep attention mechanism obtains the probability weights of each word segment under the current output, and selects the word segment with the highest weight as the current output by maximizing the current output probability. During this process, the output of the large model is compared with the output content in the training data according to the loss function. The magnitude of the loss function value reflects the difference between the large model's output and the expected output. Next, the loss function is used through backpropagation to iteratively update the model weight parameters of the attention mechanism and other components, continuously improving the representation of entity features, and ultimately optimizing the output probability of the large model for recognizing relevant content. This iterative update process continues until the output of the large model matches the content of the output template in the training data, thus obtaining the final optimal model.

[0011] Step 4. Input data from the test set, already populated with the input template, is fed into the trained large model. Guided by the template prompts in the input data, the large model extracts the relationships between word segments and between word segments and categories, and constructs features. Then, based on the output template features learned during training, it maximizes the output probability of content relevant to the recognition result, selects words for segmentation, and generates the recognition result. Finally, the recognition result is separated into entity names corresponding to each target category according to the output template, thus achieving named entity recognition.

[0012] Step 5. For a text to be identified, divide it into multiple sentences, and input the template filled in according to the sentences in the text to be identified into the trained large model to predict and output the entity names corresponding to each type in the text to be identified.

[0013] The universal prompt template proposed in this invention adopts a question-and-answer format, using the format "Messages: + Content to be identified + Q: + Task description + A:" as input. This universal template introduces tag information into the task description, uses "Messages:" to guide the content to be identified, uses "Q:" to separate the task objective from the content to be identified, and uses "A:" to separate the task objective from the identification result. In terms of output, the template represents the result in the form of "Category iis entity name (or does not exist)," improving the readability of the result and the accuracy of representing complex entities with nested or discontinuous structures. The core of this universal prompt template lies in effectively transforming the entity recognition task into a question-and-answer task, thus more closely aligning with the pre-training task of large models. This allows for better utilization of the ample pre-training knowledge of large models, guiding the attention mechanism within the large model to capture and effectively learn the relationship between each word segment and the target entity category in the text to be processed, constructing a more complete and accurate feature representation, and ultimately achieving the identification of various types of named entities in the form of answering questions. The attention mechanism can be represented as:

[0014]

[0015] This method proposes a large model loss function optimization approach based on a masking strategy. It simultaneously employs input masking and unmasked strategies to guide the large model loss function to focus on the accuracy of the generated recognition results and to fully utilize contextual knowledge to improve the completeness of entity recognition. Specifically, both masking strategies are based on the cross-entropy loss function, maintaining consistency with the original large model loss function in terms of computational framework, thus fully preserving the knowledge learned during the large model's pre-training process. Since the original large model loss function tends to output fluent and context-appropriate results rather than the most accurate ones, this invention uses a sub-loss function based on the input masking strategy, causing the large model to calculate only the conditional probability values ​​of each word at each time step in the generated part during training, thereby prioritizing the accuracy of the results. Furthermore, named entity recognition tasks also emphasize the completeness of entity recognition; therefore, this invention uses a sub-loss function based on the unmasked strategy, causing the large model to calculate the conditional probability values ​​of all input information and existing output content at each time step during training, thus fully considering and utilizing various information in the input template. Finally, by assigning weights to the two sub-loss functions, the optimized large model loss function can simultaneously achieve a balance between entity recognition accuracy and recall.

[0016] The sub-loss function based on the input masking strategy is:

[0017]

[0018] The sub-loss function based on the unmasked strategy is:

[0019]

[0020] The optimized loss function for the large model is:

[0021] L optimized = w×L1+(1-w)×L2.

[0022] Where w is the weight coefficient; x is the input sample (including label information), and y is the weight coefficient. i Let y be the predicted output corresponding to the template filled based on the i-th sentence in sample x, θ be the parameter set of the large model, and y be the predicted output. <i The predicted output of the large model before filling the template for the i-th sentence in sample x, where n is the total number of sentences in sample x.

[0023] Assuming the model needs to output a total of m words for recognizing a given text, where y i This refers to outputting the i-th word, so the i-th word could be either a predicted entity name or a predicted label. For example, if the output recognition result is "China is a country"; m=4, when i=1, the output "China" is an entity name, and when i=4, the output "country" is a predicted label. When calculating the loss, since the correct expected output when i=4 is "country", its theoretical probability is 1 (omitted in the formula), and the actual probability is P(y). i |y <i ), is a probability less than or equal to 1. The loss function reflects the difference between these two values, and the model is trained to minimize this difference, thereby enabling the model to generate P(y). i |y <i To maximize the probability of (i) and generate the desired result at time i.

[0024] The advantages of this invention are as follows:

[0025] 1. This invention addresses the lack of templates for named entity recognition in existing large-scale models by proposing a question-and-answer-based general named entity recognition template. This template incorporates label information into the input in a question-and-answer format, directly outputting the entity name and category in the output, and indicating non-existent targets within that category. This template effectively guides large-scale models in named entity recognition tasks, improving their performance while significantly reducing the complexity of model data preparation and input / output.

[0026] 2. This invention addresses the problem that the pre-trained knowledge of large models is inconsistent with the named entity recognition task, making them unsuitable for direct application. It proposes a loss function optimization method for large models based on a masking strategy. This method improves the loss function by employing masking and unmasking strategies, guiding the large model to focus on the accuracy of generated content and the completeness of entity localization during training, thereby improving the accuracy and recall of the large model on general named entity recognition tasks.

[0027] 3. As shown in Tables 1 and 2 below, comparisons with the latest general-purpose named entity recognition methods from both domestic and international sources on the MIT-Restaurant, MIT-Movie, GENIA, and CADEC datasets with small sample sizes reveal that the method of this invention improves the F1 score by an average of approximately 11 points when each class of training data has only 10 examples. The F1 score is the harmonic mean of precision and recall, comprehensively reflecting these two metrics. The methods compared are all the latest and most widely used general-purpose named entity recognition technologies from 2021 to 2023.

[0028] Table 1 shows the comparison of recognition performance on a common named entity recognition dataset.

[0029]

[0030] Table 2 compares the recognition performance on nested and discontinuous named entity recognition datasets.

[0031] Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0034] To better illustrate the complex named entity recognition method based on prompting guidance and entity representation enhancement proposed in this invention, an example is provided below. Taking the identification of people, locations, times, and organization types in a sentence as an example, the sentence to be identified is "Peter lives in New York, and he works in Washington Bank." Here, the person entity is Peter, the location entity is New York, the organization entity is Washington Bank, and there is no time type entity.

[0035] Step 1. Process data for large-scale model training. The sentence to be recognized, "Peter lives in New York, and he works in Washington Bank.", is entered into the input section of the general prompt template. This yields the input content for the large-scale model: "Messages: Peter lives in New York, and he works in Washington Bank. Q: What is the Person, Location, Time, and Organization entity in the message? A:", where Person, Location, Time, and Organization correspond to person, location, time, and organization. The expected output of the large-scale model for this sentence is "Person is Peter; Location is New York; Time does not exist; Organization is Washington Bank." "Not exists" indicates that the entity in this category does not exist.

[0036] Step 2. Large-scale model training: Next, an attention mechanism is used to represent each word segment as a weighted sum of all word segments, thus obtaining the correlation strength between word segments. The output of the attention mechanism will be used as the feature representation of words for selection during the output process. The large-scale model selects words with strong relevance to the existing output as the next output. The loss function measures the generation quality by comparing the consistency between the generated content and the expected output template. The optimized loss function uses an input masking strategy to ensure the large-scale model pays attention to the correctness of entity names in the generated content, and an unmasked strategy to ensure the large-scale model does not miss entities during recognition. Finally, the parameters in the large-scale model are adjusted using the backpropagation algorithm, so that at the end of training, the large-scale model tends to output "Person: Peter;Location: New York;Time: not exists;Organization: Washington Bank" to the input content "Messages: Peter lives in New York, and he works in Washington Bank. Q: What is the Person, Location, Time and Organization entity in the message? A:".

[0037] Step 3. Large Model Testing. Following Step 1, process the sentences to be recognized to meet the input format of the trained large model. The large model will then output the corresponding recognition results. The recognition results can be segmented by dividing the colons (:) to extract specific recognition information for each category.

[0038] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A general named entity recognition method for small samples, comprising the following steps: 1) For each sample in the training set, divide it into several sentences and fill a template for each sentence; wherein, the sample includes several sentences, and each sentence includes several entities; the template sets a fill box for each type of entity; according to the type of entity, fill the entities in the sentence into the corresponding fill boxes respectively. 2) Decompose the loss function of the selected large model into a sub-loss function L1 based on the input masking strategy and a sub-loss function L2 based on the no-mask strategy to obtain the improved loss function L. optimized =w×L1+(1-w)×L2, where w is the weight coefficient; then, the template filled with samples in the training set is used as the input of the selected large model to train the large model; wherein, the sub-loss function L1 of the input masking strategy is used by the large model to calculate the conditional probability value of each word at each time step of the generated part during the training process, and the sub-loss function L2 of the no-masking strategy is used by the large model to calculate the conditional probability value of all input information and existing output content at each time step during the training process; 3) For a text to be identified, divide it into multiple sentences, and input the template filled according to the sentences in the text to be identified into the large model trained in step 2) to predict and output the entity names corresponding to each type in the text to be identified.

2. The method according to claim 1, characterized in that, The template has the structure "Messages: + content to be identified + Q: + task description + A:"; where "Messages:" guides the content to be identified, "Q:" separates the task purpose from the content to be identified, and "A:" separates the task purpose from the identification result.

3. The method according to claim 2, characterized in that, The large model represents the predicted output in the form of "category iis entity name (or does not exist)".

4. The method according to claim 1, 2, or 3, characterized in that, The sub-loss function of the input masking strategy The sub-loss function based on the maskless strategy is: Where x is the input sample, y i y is the predicted output corresponding to the template filled based on the i-th sentence in sample x. <i θ is the predicted output of the large model before filling the template for the i-th sentence in sample x, θ is the parameter set of the large model, and n is the total number of sentences in sample x.

5. The method according to claim 1, 2, or 3, characterized in that, The loss value calculated based on the loss function is used to adjust the parameters in the large model using the backpropagation algorithm.

6. The method according to claim 1, 2, or 3, characterized in that, The large model is GPT or a model with a similar structure.

Images