Named entity recognition method, device, electronic device and computer storage medium

By pre-training and correcting false negative samples in named entity recognition models, the method addresses the issue of reduced accuracy caused by incomplete labeling, resulting in improved model performance and recognition accuracy.

CN113919348BActive Publication Date: 2024-12-03ALIBABA GROUP HOLDING LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010646870.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-07
Publication Date
2024-12-03
Estimated Expiration
2040-07-07

AI Technical Summary

Technical Problem

Existing named entity recognition models suffer from reduced accuracy due to the presence of false negative examples in incomplete labeled training data, which are incorrectly labeled as negative samples, leading to significant drops in model performance.

Method used

A method and device for named entity recognition that involves pre-training a model to distinguish positive and negative samples, identifying false negative samples through overlap with positive samples, and updating these samples to correct the training data, followed by retraining to improve model accuracy.

Benefits of technology

The approach enhances the model's ability to distinguish false negative samples, improving the accuracy of named entity recognition by reducing the impact of incomplete labeling and enhancing the quality of the training data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113919348B_ABST
    Figure CN113919348B_ABST
Patent Text Reader

Abstract

The disclosed embodiments disclose a method, device, electronic device and computer storage medium for named entity recognition, the method comprising: pre-training a named entity recognition model according to training data, so that the named entity recognition model distinguishes positive samples and negative samples in the training data; obtaining a first negative sample that overlaps with the positive sample in the negative sample; and training the named entity recognition model using the positive sample and the first negative sample, so that the named entity recognition model distinguishes false negative samples in the training data. This technical solution can correct false negative examples in incompletely labeled named entity recognition data to a certain extent, reduce the impact of false negative examples on model training and model quality, thereby improving the accuracy of training data, improving the quality of the named entity recognition model, and ultimately improving the accuracy of named entity classification or recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular to a method, device, electronic device and computer storage medium for named entity recognition. Background Art

[0002] With the development of network science and technology and data processing technology, many fields need to be trained based on labeled data to obtain classification or recognition models for classification or recognition of desired targets. For example, in the field of natural language processing, a very important task is to perform named entity recognition. The goal of named entity recognition is to extract named entities mentioned in unstructured text, wherein the named entities include names of people, organizations, places, medical terms, regulatory terms, time, quantity, monetary value, etc., which have certain meanings. However, in the actual application of named entity recognition, high-quality data annotation usually consumes a lot of time and manpower. Therefore, in order to save time and manpower, the prior art usually obtains some incomplete labeled data through dictionary matching, distance supervision, etc., and then directly trains the model based on these incomplete labeled data. In fact, in the above-mentioned incomplete labeled data, some entities are not labeled, which leads to the presence of more false negative examples in the labeled data, that is, data that should be positive sample data but is labeled as negative sample data, which can also be called false negative sample data. Obviously, these incorrectly labeled samples will cause the accuracy of the trained model to drop significantly. Therefore, how to reduce the impact of false negative examples in partially labeled named entity recognition training data on model training and model quality is of great significance for the application of large-scale automatic annotation data and the improvement of model quality. Summary of the invention

[0003] Embodiments of the present disclosure provide a method, device, electronic device, and computer storage medium for named entity recognition.

[0004] In a first aspect, a method for named entity recognition is provided in an embodiment of the present disclosure.

[0005] Specifically, the named entity recognition method includes:

[0006] Pre-training a named entity recognition model according to the training data so that the named entity recognition model distinguishes positive samples from negative samples in the training data;

[0007] Obtaining, from the negative samples, a first negative sample that overlaps with the positive sample;

[0008] The named entity recognition model is trained using the positive sample and the first negative sample so that the named entity recognition model can distinguish false negative samples in the training data.

[0009] In combination with the first aspect, in a first implementation of the first aspect of the embodiment of the present disclosure, the training data is obtained in the following manner:

[0010] Determine a text segment of the text to be processed and a corresponding segment representation;

[0011] Obtaining the category corresponding to the text fragment;

[0012] The segment representation and the category of the text segment are combined to obtain the training data.

[0013] In combination with the first aspect and the first implementation method of the first aspect, in the second implementation method of the first aspect of the embodiment of the present disclosure, the positive sample is a text fragment classified as an entity, the negative sample is a text fragment classified as a non-entity, and the false negative sample is an entity text fragment classified as a non-entity.

[0014] In combination with the first aspect, the first implementation of the first aspect, and the second implementation of the first aspect, in a third implementation of the first aspect of the present disclosure, the method further includes:

[0015] The named entity recognition model is used to predict the training data, and the false negative samples in the negative samples are updated to positive samples according to the prediction results.

[0016] In combination with the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, and the third implementation manner of the first aspect, in a fourth implementation manner of the first aspect of the present disclosure, the method further includes:

[0017] After the false negative samples are updated to positive samples, they are used as new training data to train the named entity recognition model.

[0018] In combination with the first aspect, the first implementation manner of the first aspect, the second implementation manner of the first aspect, the third implementation manner of the first aspect, and the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect of the present disclosure, the method further includes:

[0019] Named entity recognition is performed based on the named entity recognition model.

[0020] In a second aspect, an embodiment of the present disclosure provides a device for identifying named entities.

[0021] Specifically, the named entity recognition device includes:

[0022] A pre-training module is configured to pre-train a named entity recognition model according to training data so that the named entity recognition model can distinguish between positive samples and negative samples in the training data;

[0023] An obtaining module is configured to obtain, from the negative samples, a first negative sample that overlaps with the positive sample;

[0024] The first training module is configured to train the named entity recognition model using the positive sample and the first negative sample, so that the named entity recognition model can distinguish the false negative samples in the training data.

[0025] In combination with the second aspect, in a first implementation of the second aspect of the embodiment of the present disclosure, the training data is obtained in the following manner:

[0026] Determine a text segment of the text to be processed and a corresponding segment representation;

[0027] Obtaining the category corresponding to the text fragment;

[0028] The segment representation and the category of the text segment are combined to obtain the training data.

[0029] In combination with the second aspect and the first implementation method of the second aspect, in the second implementation method of the second aspect of the embodiment of the present disclosure, the positive sample is a text fragment classified as an entity, the negative sample is a text fragment classified as a non-entity, and the false negative sample is an entity text fragment classified as a non-entity.

[0030] In combination with the second aspect, the first implementation of the second aspect, and the second implementation of the second aspect, in a third implementation of the second aspect of the present disclosure, the device further includes:

[0031] The prediction module is configured to use the named entity recognition model to predict the training data, and update the false negative samples in the negative samples to positive samples according to the prediction results.

[0032] In combination with the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, and the third implementation manner of the second aspect, in a fourth implementation manner of the second aspect of the present disclosure, the device further includes:

[0033] The second training module is configured to update the false negative samples to positive samples and use them as new training data to train the named entity recognition model.

[0034] In combination with the second aspect, the first implementation manner of the second aspect, the second implementation manner of the second aspect, the third implementation manner of the second aspect, and the fourth implementation manner of the second aspect, in a fifth implementation manner of the second aspect of the present disclosure, the device further includes:

[0035] The recognition module is configured to perform named entity recognition based on the named entity recognition model.

[0036] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer instructions that support a named entity recognition device to perform the above-mentioned named entity recognition method, and the processor is configured to execute the computer instructions stored in the memory. The named entity recognition device may also include a communication interface for the named entity recognition device to communicate with other devices or a communication network.

[0037] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by a named entity recognition device, which includes computer instructions involved in the named entity recognition device for executing the above-mentioned named entity recognition method.

[0038] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0039] The above technical solution performs dual training on the named entity recognition model based on the training data and the first negative sample in the negative sample of the training data that overlaps with the positive sample, so that the named entity recognition model can distinguish the false negative samples in the training data. The technical solution can correct the false negative examples in the incompletely labeled named entity recognition data to a certain extent, reduce the impact of the false negative examples on the model training and model quality, thereby improving the accuracy of the training data, improving the quality of the named entity recognition model, and ultimately improving the accuracy of the named entity classification or recognition.

[0040] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Other features, purposes and advantages of the embodiments of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0042] Figure 1 A flowchart of a method for named entity recognition according to an embodiment of the present disclosure is shown;

[0043] Figure 2 A schematic diagram of a named entity recognition model training process according to an embodiment of the present disclosure is shown;

[0044] Figure 3 An overall flow chart of a method for named entity recognition according to an embodiment of the present disclosure is shown;

[0045] Figure 4 A structural block diagram of a named entity recognition device according to an embodiment of the present disclosure is shown;

[0046] Figure 5 It is a structural diagram of a computer system suitable for implementing a method for named entity recognition according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0048] In the embodiments of the present disclosure, it should be understood that terms such as "including" or "having" are intended to indicate the existence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in this specification, and are not intended to exclude the possibility of one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof existing or being added.

[0049] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] The technical solution provided by the embodiment of the present disclosure performs multiple training on the named entity recognition model based on the training data and the first negative sample in the negative sample of the training data that overlaps with the positive sample, so that the named entity recognition model can distinguish the false negative samples in the training data. The technical solution can correct the false negative examples in the incompletely labeled named entity recognition data to a certain extent, reduce the impact of the false negative examples on the model training and model quality, thereby improving the accuracy of the training data, improving the quality of the named entity recognition model, and ultimately improving the accuracy of the named entity classification or recognition.

[0051] Figure 1 A flowchart of a method for named entity recognition according to an embodiment of the present disclosure is shown as follows: Figure 1 As shown, the named entity recognition method includes the following steps S101-S103:

[0052] In step S101, a named entity recognition model is pre-trained according to training data, so that the named entity recognition model can distinguish positive samples and negative samples in the training data;

[0053] In step S102, a first negative sample overlapping with the positive sample is obtained from the negative samples;

[0054] In step S103, the named entity recognition model is trained using the positive sample and the first negative sample, so that the named entity recognition model can distinguish false negative samples in the training data.

[0055] As mentioned above, with the development of network science and technology and data processing technology, many fields need to train classification or recognition models based on labeled data to classify or identify the desired target. For example, in the field of natural language processing, a very important task is to perform named entity recognition. The goal of named entity recognition is to extract the named entities mentioned in unstructured text, wherein the named entities include names of people, organizations, places, medical terms, regulatory terms, time, quantity, monetary value, etc., which have certain meanings. Entity naming, but in the actual application of named entity recognition, high-quality data annotation usually consumes a lot of time and manpower. Therefore, in order to save time and manpower, the prior art usually obtains some incomplete labeled data through dictionary matching, distance supervision, etc., and then directly trains the model based on these incomplete labeled data. In fact, in the above-mentioned incomplete labeled data, some entities are not labeled, which leads to the presence of more false negative examples in the labeled data, that is, data that should be positive sample data but is labeled as negative sample data, which can also be called false negative sample data. Obviously, these incorrectly labeled samples will cause the accuracy of the trained model to drop significantly. Therefore, how to reduce the impact of false negative examples in partially labeled named entity recognition training data on model training and model quality is of great significance for the application of large-scale automatic annotation data and the improvement of model quality.

[0056] In view of the above problems, in this embodiment, a method for named entity recognition is proposed, which performs multiple training on a named entity recognition model based on training data and a first negative sample in the negative sample of the training data that overlaps with a positive sample, so that the named entity recognition model can distinguish false negative samples in the training data. This technical solution can correct the false negative examples in the incompletely labeled named entity recognition data to a certain extent, reduce the impact of false negative examples on model training and model quality, thereby improving the accuracy of the training data, improving the quality of the named entity recognition model, and ultimately improving the accuracy of named entity classification or recognition.

[0057] In one embodiment of the present disclosure, the named entity recognition method may be applicable to computers, computing devices, electronic devices, servers, service clusters, etc. that recognize named entities.

[0058] In one embodiment of the present disclosure, the named entity refers to an entity name with a certain meaning in the field of natural language processing, such as a person's name, organization name, place name, medical terminology, regulatory terminology, time, quantity, monetary value, etc.

[0059] In one embodiment of the present disclosure, the named entity recognition model refers to a model used to identify or classify the named entity, that is, by inputting the information to be identified into the named entity recognition model, a recognition result of whether the information to be identified is a named entity can be obtained.

[0060] In one embodiment of the present disclosure, the training data refers to data used to train a named entity recognition model, wherein the training data includes at least correctly classified positive sample data and negative sample data, and in particular, also includes some incorrectly classified false negative data, such as the incomplete labeled data obtained by dictionary matching, distance supervision, etc. mentioned above, wherein the false negative refers to data that should be positive sample data but is labeled as negative sample data, which can also be called false negative sample data or false negative samples.

[0061] In the prior art, the training data including positive samples, negative samples and false negative samples is usually used directly for training to obtain a named entity recognition model that is actually used to identify named entities. However, due to the influence of false negative samples in the training data, the accuracy of the named entity recognition model trained at this time is insufficient and the recognition effect needs to be improved.

[0062] In one embodiment of the present disclosure, the named entity recognition model is trained multiple times with the help of corrected training data to achieve the purpose of improving the accuracy of the named entity recognition model and improving the recognition effect of the named entity recognition model.

[0063] In this embodiment, the training data including positive samples, negative samples and false negative samples is first used to pre-train the named entity recognition model, so that the named entity recognition model can distinguish between positive samples and negative samples in the training data, that is, a preliminary named entity recognition model can be obtained based on the above pre-training steps. Although there are some false negative samples in the training data based on which the named entity recognition model obtained at this time is based, which will affect the accuracy of the named entity recognition model, the named entity recognition model obtained at this time is at least capable of identifying true samples and true negative samples in the training data, wherein the true sample refers to a positive sample that is correctly marked as a positive sample, and the true negative sample refers to a negative sample that is correctly marked as a negative sample. At the same time, since the named entity recognition model uses the full amount of training data for training, the bias of the model can also be effectively avoided. The named entity recognition model can be retrained later by correcting the training data to correct the named entity recognition model and reduce the impact of false negative samples on it.

[0064] Then, among the negative samples in the training data, a first negative sample that overlaps with the positive sample is determined, and the named entity recognition model is trained again using the positive sample and the first negative sample, so that the named entity recognition model can distinguish the false negative samples in the training data, thereby reducing the negative impact of the false negative samples on the training of the named entity recognition model.

[0065] Among them, the first negative sample refers to a negative sample in the training data that has content overlap with the positive sample in the training data. As mentioned above, for named entities, most of the time, the real named entities will be separated and labeled, and then considered as non-named entities, that is, there are many false negative examples. Therefore, in this embodiment, all negative samples (false negative samples) that have content overlap with entity fragments, that is, positive samples (real samples) can be screened out from all non-entity fragments, that is, negative samples (including true negative samples and false negative samples), and the remaining non-entity fragments (true negative samples) and entity fragments (real samples) are or are more likely to be true samples with higher accuracy, of course, except for overlapping and nested entities. Using the remaining true samples, that is, true samples and false negative samples, to continue training the named entity recognition model, the named entity recognition model can be able to distinguish the false negative samples in the training data, so as to achieve the purpose of correcting the named entity recognition model and improving its recognition accuracy.

[0066] In one embodiment of the present disclosure, the training data can be obtained in the following manner:

[0067] Determine a text segment of the text to be processed and a corresponding segment representation;

[0068] Obtaining the category corresponding to the text fragment;

[0069] The segment representation and the category of the text segment are combined to obtain the training data.

[0070] In order to improve the comprehensiveness of the training data, in this embodiment, the segments of the text to be processed and their categories are obtained to form the training data. Specifically, first determine the text segments of the text to be processed and the corresponding segment representation, wherein the text segments of the text to be processed include all the text segments in the text to be processed, assuming that the length of the processed text is L, then the text segments include all text segments of length 1 to L, which are composed of unit characters that appear continuously, wherein the unit character refers to a character unit that cannot be further divided, such as a Chinese character or an English word; then obtain the category corresponding to the text segment, wherein the category corresponding to the text segment can be either a manually marked category or a category obtained by annotating tools, and the category of the text segment obtained at this time may be wrong, and the wrong category annotation will be corrected by obtaining the first negative sample and retraining the named entity recognition model; finally, combine the segment representation and category of the text segment to obtain the training data, wherein the positive sample in the training data refers to the text segment with the category of entity, the negative sample in the training data refers to the text segment with the category of non-entity, and the false negative sample in the training data refers to the entity text segment with the category of non-entity.

[0071] Among them, the segment representation of the text segment can be obtained through segment encoders such as CNN (Convolutional Neural Networks), GRU (Gated Recurrent Unit), Transformer, etc.

[0072] In one embodiment of the present disclosure, a segment representation of a text segment of a certain text to be processed can be represented as:

[0073] S ij =[h i ;h j ;x ij ; l ij ]

[0074] Among them, S ij represents the segment representation corresponding to the segment from the i-th unit character to the j-th unit character extracted from the text to be processed; h i and h j It is the character representation corresponding to the i-th unit character and the j-th unit character obtained by the bidirectional LSTM (Long Short-Term Memory). t=i,j, character representation calculated using LSTM from left to right Character representation calculated using LSTM from right to left x t For t The corresponding word representation, x t =[w t ; char t ;elmo t ],w t Based on h t The Glove output of the corresponding words, char t Based on h t The CNN output of the corresponding words, elmo t Based on h t The corresponding words are obtained from the pre-trained language model ELMo output; x ij is the fragment representation obtained by the attention mechanism, α=softmax(Wx i:j )=[α i …α t …α j ], W is a learnable matrix parameter, x i:j =[x i …x j ] is a vector; l ij A vector representation of the fragment width.

[0075] In one embodiment of the present disclosure, when the named entity recognition model is pre-trained according to the training data, the convergence of the model pre-training can be achieved with the help of a loss function. In this embodiment, the named entity recognition model obtained by pre-training can be expressed as:

[0076] P ij =softmax(W1(ReLU(W2S ij )))

[0077] Among them, P ij Represents the recognition probability of each category of named entity, softmax(.) represents the softmax function, W1 and W2 are learnable parameters, and ReLU(.) represents the linear rectification function or the rectified linear unit (Rectified Linear Unit).

[0078] The loss function can be taken as the cross entropy function L ij , the cross entropy function L ij It can be expressed as:

[0079]

[0080] Among them, k represents the number of named entity categories, y ijRepresents the category of the segment. For example, if there are two named entity categories, A and B, then k = 2, t = 1 represents category A, t = 2 represents category B. If a segment S ij If the category is A, then For the first category A segment S ij The category feature is true, represented by 1, for the second category B fragment S ij The category feature is false and is represented by 0.

[0081] The training process of the above named entity recognition model can be as follows Figure 2 As shown in the example of three fragment combinations, Figure 2 In fragment 1, i=1, j=2, in fragment 2, i=3, j=6, in fragment 3, i=7, j=8, and from x1~x8 and the neighboring values ​​of h1~h8, h1~h8 can be obtained. By α=softmax(Wx i:j )=[α i …α t …α j ],as well as You can get x 12 、x 36 、x 78 , based on l 12 , l 36 , l 78 and S ij =[h i ;h j ;x ij ; l ij ], we can get S 12 , S 36 , S 78 , assuming that the number of categories is 4: category 1, category 2, category 3 and category 4, then based on P ij =softmax(W1(ReLU(W2S ij ))), we can get the probability sizes used to distinguish different fragments as category 1, category 2, category 3 and category 4 respectively.

[0082] After the named entity recognition model is trained again using the positive sample and the first negative sample so that the named entity recognition model can distinguish the false negative samples in the training data, in order to further improve the accuracy of named entity recognition, the named entity recognition model can be used to return to predict the training data, i.e., class recognition, to correct the training data and correct the sample class that is incorrectly labeled in the training data. That is, in this embodiment, the method further includes the following steps:

[0083] The named entity recognition model is used to predict the training data, and the false negative samples in the negative samples are updated to positive samples according to the prediction results.

[0084] For example, after using the named entity recognition model to predict the training data, if a segment previously labeled as a non-entity category is found to be an entity category with a high confidence level, the non-entity category label of the segment can be corrected or updated to an entity category label, that is, the false negative sample can be corrected to a positive sample. In this way, corrected training data with higher accuracy can be obtained. Obviously, the corrected training data will include more true samples and fewer false negative samples.

[0085] After the training data is updated, training can be performed again based on the updated training data, that is, the updated training data is used as new training data to train the named entity recognition model again. Since the proportion of false negative samples in the updated training data has been greatly reduced, the accuracy of the named entity recognition model obtained based on the updated training data can be greatly improved, and the named entity recognition can be performed subsequently according to the named entity recognition model. That is, in this embodiment, the method also includes the following steps:

[0086] After the false negative samples are updated to positive samples, they are used as new training data to train the named entity recognition model.

[0087] After training to obtain a named entity recognition model with a high accuracy, the named entity recognition operation can be performed on the information to be processed based on the obtained named entity recognition model. That is, in this embodiment, the method further includes the following steps:

[0088] Named entity recognition is performed based on the named entity recognition model.

[0089] Figure 3 The overall flow chart of the named entity recognition method according to an embodiment of the present disclosure is shown as follows: Figure 3As shown, in this embodiment, firstly, the text segment of the text to be processed and the corresponding segment representation and category are determined based on the segment encoder to obtain the segment training data; then, pre-training is performed based on the segment training data to obtain an initial named entity recognition model that can distinguish between positive samples and negative samples in the training data; a first negative sample that overlaps with the positive sample in the negative sample is obtained, and the initial named entity recognition model is retrained using the positive sample and the first negative sample to obtain a revised named entity recognition model that can distinguish between false negative samples in the training data; the segment training data is predicted using the revised named entity recognition model, and the false negative samples in the negative samples are updated to positive samples according to the prediction results; and the updated segment training data is used to train to obtain a final named entity recognition model that can perform named entity recognition on the information to be processed.

[0090] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0091] Figure 4 FIG. 1 is a block diagram of a named entity recognition device according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 4 As shown, the named entity recognition device includes:

[0092] A pre-training module 401 is configured to pre-train a named entity recognition model according to training data, so that the named entity recognition model can distinguish between positive samples and negative samples in the training data;

[0093] An obtaining module 402 is configured to obtain, from the negative samples, a first negative sample that overlaps with the positive sample;

[0094] The first training module 403 is configured to train the named entity recognition model using the positive samples and the first negative samples, so that the named entity recognition model can distinguish false negative samples in the training data.

[0095] As mentioned above, with the development of network science and technology and data processing technology, many fields need to train classification or recognition models based on labeled data to classify or identify the desired target. For example, in the field of natural language processing, a very important task is to perform named entity recognition. The goal of named entity recognition is to extract the named entities mentioned in unstructured text, wherein the named entities include names of people, organizations, places, medical terms, regulatory terms, time, quantity, monetary value, etc., which have certain meanings. Entity naming, but in the actual application of named entity recognition, high-quality data annotation usually consumes a lot of time and manpower. Therefore, in order to save time and manpower, the prior art usually obtains some incomplete labeled data through dictionary matching, distance supervision, etc., and then directly trains the model based on these incomplete labeled data. In fact, in the above-mentioned incomplete labeled data, some entities are not labeled, which leads to the presence of more false negative examples in the labeled data, that is, data that should be positive sample data but is labeled as negative sample data, which can also be called false negative sample data. Obviously, these incorrectly labeled samples will cause the accuracy of the trained model to drop significantly. Therefore, how to reduce the impact of false negative examples in partially labeled named entity recognition training data on model training and model quality is of great significance for the application of large-scale automatic annotation data and the improvement of model quality.

[0096] In view of the above problems, in this embodiment, a named entity recognition device is proposed, which performs multiple training on a named entity recognition model based on training data and a first negative sample in the negative sample of the training data that overlaps with a positive sample, so that the named entity recognition model can distinguish false negative samples in the training data. This technical solution can correct the false negative examples in the incompletely labeled named entity recognition data to a certain extent, reduce the impact of false negative examples on model training and model quality, thereby improving the accuracy of the training data, improving the quality of the named entity recognition model, and ultimately improving the accuracy of named entity classification or recognition.

[0097] In one embodiment of the present disclosure, the named entity recognition apparatus may be implemented as a computer, a computing device, an electronic device, a server, a service cluster, etc. for recognizing named entities.

[0098] In one embodiment of the present disclosure, the named entity refers to an entity name with a certain meaning in the field of natural language processing, such as a person's name, organization name, place name, medical terminology, regulatory terminology, time, quantity, monetary value, etc.

[0099] In one embodiment of the present disclosure, the named entity recognition model refers to a model used to identify or classify the named entity, that is, by inputting the information to be identified into the named entity recognition model, a recognition result of whether the information to be identified is a named entity can be obtained.

[0100] In one embodiment of the present disclosure, the training data refers to data used to train a named entity recognition model, wherein the training data includes at least correctly classified positive sample data and negative sample data, and in particular, also includes some incorrectly classified false negative data, such as the incomplete labeled data obtained by dictionary matching, distance supervision, etc. mentioned above, wherein the false negative refers to data that should be positive sample data but is labeled as negative sample data, which can also be called false negative sample data or false negative samples.

[0101] In the prior art, the training data including positive samples, negative samples and false negative samples is usually used directly for training to obtain a named entity recognition model that is actually used to identify named entities. However, due to the influence of false negative samples in the training data, the accuracy of the named entity recognition model trained at this time is insufficient and the recognition effect needs to be improved.

[0102] In one embodiment of the present disclosure, the named entity recognition model is trained multiple times with the help of corrected training data to achieve the purpose of improving the accuracy of the named entity recognition model and improving the recognition effect of the named entity recognition model.

[0103] In this embodiment, the training data including positive samples, negative samples and false negative samples is first used to pre-train the named entity recognition model, so that the named entity recognition model can distinguish between positive samples and negative samples in the training data, that is, a preliminary named entity recognition model can be obtained based on the above pre-training steps. Although there are some false negative samples in the training data based on which the named entity recognition model obtained at this time is based, which will affect the accuracy of the named entity recognition model, the named entity recognition model obtained at this time is at least capable of identifying true samples and true negative samples in the training data, wherein the true sample refers to a positive sample that is correctly marked as a positive sample, and the true negative sample refers to a negative sample that is correctly marked as a negative sample. At the same time, since the named entity recognition model uses the full amount of training data for training, the bias of the model can also be effectively avoided. The named entity recognition model can be retrained later by correcting the training data to correct the named entity recognition model and reduce the impact of false negative samples on it.

[0104] Then, among the negative samples in the training data, a first negative sample that overlaps with the positive sample is determined, and the named entity recognition model is trained again using the positive sample and the first negative sample, so that the named entity recognition model can distinguish the false negative samples in the training data, thereby reducing the negative impact of the false negative samples on the training of the named entity recognition model.

[0105] Among them, the first negative sample refers to a negative sample in the training data that has content overlap with the positive sample in the training data. As mentioned above, for named entities, most of the time, the real named entities will be separated and labeled, and then considered as non-named entities, that is, there are many false negative examples. Therefore, in this embodiment, all negative samples (false negative samples) that have content overlap with entity fragments, that is, positive samples (real samples) can be screened out from all non-entity fragments, that is, negative samples (including true negative samples and false negative samples), and the remaining non-entity fragments (true negative samples) and entity fragments (real samples) are or are more likely to be true samples with higher accuracy, of course, except for overlapping and nested entities. Using the remaining true samples, that is, true samples and false negative samples, to continue training the named entity recognition model, the named entity recognition model can be able to distinguish the false negative samples in the training data, so as to achieve the purpose of correcting the named entity recognition model and improving its recognition accuracy.

[0106] In one embodiment of the present disclosure, the training data can be obtained in the following manner:

[0107] Determine a text segment of the text to be processed and a corresponding segment representation;

[0108] Obtaining the category corresponding to the text fragment;

[0109] The segment representation and the category of the text segment are combined to obtain the training data.

[0110] In order to improve the comprehensiveness of the training data, in this embodiment, the segments of the text to be processed and their categories are obtained to form the training data. Specifically, first determine the text segments of the text to be processed and the corresponding segment representation, wherein the text segments of the text to be processed include all the text segments in the text to be processed, assuming that the length of the processed text is L, then the text segments include all text segments of length 1 to L, which are composed of unit characters that appear continuously, wherein the unit character refers to a character unit that cannot be further divided, such as a Chinese character or an English word; then obtain the category corresponding to the text segment, wherein the category corresponding to the text segment can be either a manually marked category or a category obtained by annotating tools, and the category of the text segment obtained at this time may be wrong, and the wrong category annotation will be corrected by obtaining the first negative sample and retraining the named entity recognition model; finally, combine the segment representation and category of the text segment to obtain the training data, wherein the positive sample in the training data refers to the text segment with the category of entity, the negative sample in the training data refers to the text segment with the category of non-entity, and the false negative sample in the training data refers to the entity text segment with the category of non-entity.

[0111] Among them, the segment representation of the text segment can be obtained through segment encoders such as CNN (Convolutional Neural Networks), GRU (Gated Recurrent Unit), Transformer, etc.

[0112] In one embodiment of the present disclosure, a segment representation of a text segment of a certain text to be processed can be represented as:

[0113] S ij =[h i ;h j ;x ij ; l ij ]

[0114] Among them, S ij represents the segment representation corresponding to the segment from the i-th unit character to the j-th unit character extracted from the text to be processed; h i and h j It is the character representation corresponding to the i-th unit character and the j-th unit character obtained by the bidirectional LSTM (Long Short-Term Memory). t=i,j, character representation calculated using LSTM from left to right Character representation calculated using LSTM from right to left x t For t The corresponding word representation, x t =[w t ; char t ;elmo t ],w t Based on h t The Glove output of the corresponding words, char t Based on h t The CNN output of the corresponding words, elmo t Based on h t The corresponding words are obtained from the pre-trained language model ELMo output; x ij is the fragment representation obtained by the attention mechanism, α=softmax(Wx i:j )=[α i …α t …α j ], W is a learnable matrix parameter, x i:j =[x i …x j ] is a vector; l ij A vector representation of the fragment width.

[0115] In one embodiment of the present disclosure, when the named entity recognition model is pre-trained according to the training data, the convergence of the model pre-training can be achieved with the help of a loss function. In this embodiment, the named entity recognition model obtained by pre-training can be expressed as:

[0116] P ij =softmax(W1(ReLU(W2S ij )))

[0117] Among them, P ij Represents the recognition probability of each category of named entity, softmax(.) represents the softmax function, W1 and W2 are learnable parameters, and ReLU(.) represents the linear rectification function or the rectified linear unit (Rectified Linear Unit).

[0118] The loss function can be taken as the cross entropy function L ij , the cross entropy function L ij It can be expressed as:

[0119]

[0120] Among them, k represents the number of named entity categories, y ijRepresents the category of the segment. For example, if there are two named entity categories, A and B, then k = 2, t = 1 represents category A, t = 2 represents category B. If a segment S ij If the category is A, then For the first category A segment S ij The category feature is true, represented by 1, for the second category B fragment S ij The category feature is false and is represented by 0.

[0121] The training process of the above named entity recognition model can be as follows Figure 2 As shown in the example of three fragment combinations, Figure 2 In fragment 1, i=1, j=2, in fragment 2, i=3, j=6, in fragment 3, i=7, j=8, and from x1~x8 and the neighboring values ​​of h1~h8, h1~h8 can be obtained. By α=softmax(Wx i:j )=[α i …α t …α j ],as well as You can get x 12 、x 36 、x 78 , based on l 12 , l 36 , l 78 and S ij =[h i ;h j ;x ij ; l ij ], we can get S 12 , S 36 , S 78 , assuming that the number of categories is 4: category 1, category 2, category 3 and category 4, then based on P ij =softmax(W1(ReLU(W2S ij ))), we can get the probability sizes used to distinguish different fragments as category 1, category 2, category 3 and category 4 respectively.

[0122] After the named entity recognition model is trained again using the positive sample and the first negative sample so that the named entity recognition model can distinguish the false negative samples in the training data, in order to further improve the accuracy of named entity recognition, the named entity recognition model can be used to return to predict the training data, that is, category recognition, to correct the training data and correct the sample category that is incorrectly labeled in the training data. That is, in this embodiment, the device also includes:

[0123] The prediction module is configured to use the named entity recognition model to predict the training data, and update the false negative samples in the negative samples to positive samples according to the prediction results.

[0124] For example, after using the named entity recognition model to predict the training data, if a segment previously labeled as a non-entity category is found to be an entity category with a high confidence level, the non-entity category label of the segment can be corrected or updated to an entity category label, that is, the false negative sample can be corrected to a positive sample. In this way, corrected training data with higher accuracy can be obtained. Obviously, the corrected training data will include more true samples and fewer false negative samples.

[0125] After the training data is updated, training can be performed again based on the updated training data, that is, the updated training data is used as new training data to train the named entity recognition model again. Since the proportion of false negative samples in the updated training data has been greatly reduced, the accuracy of the named entity recognition model obtained based on the updated training data can be greatly improved, and the named entity recognition can be performed according to the named entity recognition model later. That is, in this embodiment, the device also includes:

[0126] The second training module is configured to update the false negative samples to positive samples and use them as new training data to train the named entity recognition model.

[0127] After training to obtain a named entity recognition model with a high accuracy, a named entity recognition operation can be performed on the information to be processed based on the obtained named entity recognition model. That is, in this embodiment, the device also includes:

[0128] The recognition module is configured to perform named entity recognition based on the named entity recognition model.

[0129] The present disclosure also discloses an electronic device, which includes a memory and a processor; wherein:

[0130] The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement any of the above method steps.

[0131] Figure 5 It is a structural diagram of a computer system suitable for implementing a method for named entity recognition according to an embodiment of the present disclosure.

[0132] like Figure 5As shown, the computer system 500 includes a processing unit 501, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The processing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0133] The following components are connected to the I / O interface 505: an input part 506 including a keyboard, a mouse, etc.; an output part 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part 508 including a hard disk, etc.; and a communication part 509 including a network interface card such as a LAN card, a modem, etc. The communication part 509 performs communication processing via a network such as the Internet. The drive 510 is also connected to the I / O interface 505 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage part 508 as needed. Among them, the processing unit 501 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0134] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a readable medium thereof, and the computer program includes a program code for executing the named entity recognition method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 509, and / or installed from a removable medium 511.

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the road map or block diagram can represent a part of a module, program segment or code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a different order from the order marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0137] As another aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the methods described in the embodiments of the present disclosure.

[0138] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for named entity recognition, comprising: Acquire training data based on segment representations and categories corresponding to text segments of the text to be processed; Pre-training a named entity recognition model according to the training data so that the named entity recognition model distinguishes positive samples from negative samples in the training data; Obtaining, from the negative samples, a first negative sample that overlaps with the positive sample; The named entity recognition model is trained using the positive sample and the first negative sample so that the named entity recognition model can distinguish false negative samples in the training data.

2. According to the method of claim 1, the training data is obtained by: Determine a text segment of the text to be processed and a corresponding segment representation; Obtaining the category corresponding to the text fragment; The segment representation and the category of the text segment are combined to obtain the training data.

3. According to the method of claim 2, the positive sample is a text segment classified as an entity, the negative sample is a text segment classified as a non-entity, and the false negative sample is an entity text segment classified as a non-entity.

4. The method according to any one of claims 1, further comprising: The named entity recognition model is used to predict the training data, and the false negative samples in the negative samples are updated to positive samples according to the prediction results.

5. The method according to claim 4, further comprising: After the false negative samples are updated to positive samples, they are used as new training data to train the named entity recognition model.

6. The method according to any one of claims 1 to 5, further comprising: Named entity recognition is performed based on the named entity recognition model.

7. A named entity recognition device, comprising: A pre-training module is configured to obtain training data based on segment representations and categories corresponding to text segments of the text to be processed; Pre-training a named entity recognition model according to the training data so that the named entity recognition model distinguishes positive samples from negative samples in the training data; An obtaining module is configured to obtain, from the negative samples, a first negative sample that overlaps with the positive sample; The first training module is configured to train the named entity recognition model using the positive sample and the first negative sample, so that the named entity recognition model can distinguish the false negative samples in the training data.

8. The apparatus according to claim 7, wherein the training data is obtained by: Determine a text segment of the text to be processed and a corresponding segment representation; Obtaining the category corresponding to the text fragment; The segment representation and the category of the text segment are combined to obtain the training data.

9. According to the device of claim 8, the positive sample is a text segment classified as an entity, the negative sample is a text segment classified as a non-entity, and the false negative sample is an entity text segment classified as a non-entity.

10. The device according to any one of claims 7, further comprising: The prediction module is configured to use the named entity recognition model to predict the training data, and update the false negative samples in the negative samples to positive samples according to the prediction results.

11. The device according to claim 10, further comprising: The second training module is configured to update the false negative samples to positive samples and use them as new training data to train the named entity recognition model.

12. The device according to any one of claims 7 to 11, further comprising: The recognition module is configured to perform named entity recognition based on the named entity recognition model.

13. An electronic device, characterized in that: comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps described in any one of claims 1-6.

14. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Named entity identification method and device based on improved loss function

    CN110362814A

  • Model optimization method and device, computer equipment and storage medium

    CN110457683A