Named entity recognition method and recognition device

Abstract

This application discloses a method and apparatus for identifying named entities, belonging to the field of keyword extraction technology. A method for identifying named entities includes the following steps: Step 100: Dividing the data text into a labeled dataset and an unlabeled dataset, and labeling the labeled dataset; Step 200: Calculating the uncertainty score of samples in the unlabeled dataset; Step 300: Calculating the diversity score of samples in the unlabeled dataset; Step 400: Manually labeling unlabeled samples with high information content; Step 500: Calculating the predicted score of unlabeled samples using a data classification module; Step 600: Using unlabeled samples with predicted scores higher than a threshold as labeled samples, and repeating steps 200 to 600. The beneficial effect of this application is that it provides a method and apparatus for identifying named entities that can reduce the amount of manual labeling while ensuring recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a named entity recognition method and device. BACKGROUND

[0002] Named entity recognition is beneficial to the construction of large-scale knowledge base in the big data era. It can label text information according to pre-set label items from a large amount of data text, so as to identify the required text from a large amount of data text according to the demand.

[0003] In the invention patent "named entity recognition model training method, recognition method and device" (patent application number: CN202110797174.5), a named entity recognition method based on deep learning is proposed. The initial neural network model of the method is obtained by combining keyword level coding and word level coding to represent the vector of scientific paper data, then introducing the word level vector and the word level vector into the bidirectional long short-term memory network, and then introducing the word level vector into the self-attention mechanism model, so as to realize the named entity recognition of scientific papers.

[0004] In the prior art, as mentioned in the above-mentioned invention patent, the method based on deep learning is used to recognize and label data text. The recognition method based on deep learning has high accuracy in practice. However, in practical application, a large amount of labeled data is required, so the existing named entity recognition method is restricted by a large amount of manual labeling data. If the accuracy of the recognized entity needs to be ensured, a large amount of manpower and material resources will be spent on manual labeling.

[0005] In summary, there is currently a lack of a named entity recognition method and device that can reduce the amount of manual labeling and ensure recognition accuracy. SUMMARY

[0006] The summary part of the present application is used to introduce the concept in a simple form, which will be described in detail in the specific embodiment part. The summary part of the present application is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0007] As a first aspect of the present application, in order to solve the technical problems mentioned in the background part, some embodiments of the present application provide a named entity recognition method, comprising the following steps:

[0008] In step 100, the data acquisition module acquires the text data published on the network, divides the text data into a labeled data set and an unlabeled data set, and manually labels the data in the labeled data set using a plurality of preset label items.

[0009] In step 200, the labeled samples in the labeled data set are input into the uncertainty calculation module, the uncertainty calculation module is trained, and then the unlabeled samples in the unlabeled data set are input into the uncertainty calculation module, and the uncertainty scores of the unlabeled samples are calculated.

[0010] In step 300, the labeled data set and the unlabeled data set in step 100 are input into the diversity calculation module, the similarity of the labeled data set and the unlabeled samples is calculated by the diversity calculation module, and the diversity scores of the unlabeled samples are obtained.

[0011] In step 400, the diversity score and the uncertainty score of each unlabeled sample are input into the data information degree calculation module, the data information degree calculation module calculates the comprehensive score of the uncertainty score and the diversity score of each unlabeled sample according to the pre-set compromise parameter, and a plurality of unlabeled samples are selected according to the comprehensive score to obtain a new labeled data set.

[0012] In step 500, the labeled data set with the new labeled data set added and the unlabeled data set with the new labeled data set reduced are input into the data classification module; wherein the labeled data set is used to train the data classification module, so that the data classification module calculates the prediction score of the unlabeled sample.

[0013] In step 600, the data selection module manually labels the sample data in the unlabeled sample whose prediction score exceeds the preset threshold value and adds it to the labeled data set, and the sample data in the unlabeled sample whose prediction score is lower than the preset threshold value is taken as the unlabeled data set, and steps 200-600 are executed again until all the unlabeled samples in the unlabeled data set are processed as labeled samples.

[0014] The technical scheme provided in the application trains the data classification module by using the labeled samples, so that the data classification samples have high accuracy when identifying unlabeled samples and corresponding label items. For complicated manual labeling, on one hand, the application regards the unlabeled samples and a label item exceeding a prediction threshold as labeled samples, so that the number of labeled samples can be increased through repeated operations. On the other hand, the application repeatedly calculates the diversity and uncertainty between the unlabeled samples and the labeled samples, so that the unlabeled samples with low similarity and large uncertainty in the unlabeled data set are manually labeled compared with the labeled data set. Therefore, when the labeled samples are used to train the data classification module, the sample types have sufficient breadth. Meanwhile, for manual labeling, a large number of complicated sample labeling work is provided by the unlabeled samples with a prediction score exceeding the threshold, so the amount of data for manual labeling can be greatly reduced. Manual labeling is only needed for samples with sufficient information. In this way, the technical scheme provided in the application can reduce the amount of manual labeling and ensure the recognition accuracy.

[0015] Further, step 200 specifically includes the following steps:

[0016] Step 201: input the labeled samples in the labeled data set L into the initial classifier of the uncertainty calculation module to train the initial classifier.

[0017] Step 202: input the unlabeled samples in the unlabeled data set U into the initial classifier of the uncertainty calculation module to calculate the uncertainty score of the unlabeled samples and the corresponding label items.

[0018] The initial classifier is first trained by using the labeled samples, and then the unlabeled samples are input into the initial classifier. The uncertainty score obtained is related to the similarity between the labeled samples and the unlabeled samples. It can be predicted that the higher the similarity between the labeled samples and the unlabeled samples, the lower the uncertainty score of the unlabeled samples should be. Correspondingly, the probability of inputting the unlabeled samples into the data classification module and obtaining a prediction score higher than the prediction score is greater. The reason why the application does not use the prediction score to reflect the similarity between the labeled samples and the unlabeled samples is that in the data classification module, a large number of complicated operations are needed to accurately calculate the prediction probability of the unlabeled samples and the corresponding label items, while in the initial classifier, only an uncertainty score is roughly determined. Therefore, the calculation amount can be reduced and the data processing efficiency can be increased.

[0019] Further,

[0020] In step 201, for the labeled data set L={L1, L2, …L i …L n}, wherein the i-th labeled sample is L i ; for each labeled sample L i , is a sentence composed of a words;

[0021] For the unlabeled data set U = {U1, U2, … U x …U m}, wherein each is a sentence composed of a words; for the unlabeled sample X has a words, and each word is labeled with a label, for which the unlabeled sample X = tk1, tk2, … tk i …tk α ; wherein the i-th word is tk i ; the label of the unlabeled sample X is The label of the i-th word tk i is

[0022] In step 202, the formula for calculating the uncertainty score of the unlabeled sample is:

[0023] wherein, is the label of the word tk i ,

[0024] is the posterior probability of the label corresponding to the unlabeled sample X.

[0025] The probability of the unlabeled instance is calculated, and when the probability is closer to 1, it indicates that the word is the corresponding label; on the contrary, when the probability is close to 0, it indicates that the word is not the corresponding label; when the probability is close to 0.5, it indicates that the model cannot determine whether the word is the corresponding label, then we filter out such instances for manual annotation. We can improve the accuracy of the training data set by this method, thereby improving the effect of the model.

[0026] Further, in step 300, for each unlabeled sample in the unlabeled data set U and each labeled sample in the labeled data set, the same word window is selected, and the unlabeled sample X and the sample L i are compared to obtain the diversity score of the unlabeled sample X.

[0027] The concept of window comes from the dynamic context window of word2vec. By sliding the window, the similarity of the context is compared to calculate the similarity between instances. In this way, the similarity between the unlabeled sample and the labeled sample under the same word window can facilitate the comparison between the labeled sample and the unlabeled sample, and can compare the similarity between the unlabeled sample and the labeled sample under the same word selection range.

[0028] Further, the calculation formula of the comprehensive score is: S BScor LC B Scor sim 1-B ; wherein B = 0.1 ~ 0.9;

[0029] Step 400 includes the following steps:

[0030] Step 401: determine the number of selected values m of the compromise parameter B and the size of each corresponding selected value, wherein B = 0.1 ~ 0.9;

[0031] Step 402: divide all unlabeled samples in the unlabeled data set U into several subsets S1, S2, S3, … S i ……S n ; wherein the i-th subset is S i ; wherein the unlabeled samples in each subset are k;

[0032] Step 403: calculate the comprehensive score of all unlabeled samples in the unlabeled data set U under the selected value of the compromise parameter B, and extract the unlabeled sample X with the lowest comprehensive score in each subset under each compromise parameter to form the newly added sample labeled data set CS, k > m.

[0033] Here, the way to extract the unlabeled sample is to extract the unlabeled sample with the lowest comprehensive score within a subset, rather than setting a comprehensive score threshold in advance. In this way, the number of samples in the newly added sample labeled data set obtained when step 400 is executed can be set by adjusting the number of subsets. Compared with the scheme of setting a fixed threshold to make the unlabeled samples with a comprehensive score lower than the preset threshold be manually labeled, the present application has higher accuracy, avoids the need for a large amount of manual labeling when the set threshold is too large, or the number of manual labeling is not enough when the set threshold is too small; and as steps 200-600 are continuously executed, the comprehensive score between the labeled samples and the unlabeled samples will become more and more ideal, so the comprehensive score as a whole is in a state of change, and setting a too accurate threshold will result in the inability to adapt to the continuously changing comprehensive score. The end screening mechanism set by the present application can obtain the expected number of labeled samples each time it is executed, and in the case of reducing the labeling workload, the samples that most need to be manually labeled are obtained each time, so the accuracy of the data classification module prediction can be continuously increased.

[0034] Further, the data classification module includes a BERT model, an LSTM model, and a CRF model.

[0035] BERT obtains the attention weight of the input sequence through the self-attention mechanism and the multi-head attention mechanism, and obtains the feature of the key word. Then, the BERT output is input into the BiLSTM, the BiLSTM propagates forward and backward, each character contains context information, and then each character and each named entity label form a transition matrix. Finally, the CRF selects the sequence with the highest prediction score as the optimal path according to the path and the corresponding prediction score of different labels.

[0036] Further,

[0037] In step 500, the steps of obtaining the unlabeled sample and the corresponding label prediction score are as follows:

[0038] Step 501: input the unlabeled data set U' into the BERT model, and the BERT model obtains the attention weight of each unlabeled sample input, and then obtains the feature of the key word in each unlabeled sample;

[0039] Step 502: input the obtained attention weight and key word feature of each unlabeled sample into the BILSTM model to obtain the probability matrix of each word in each unlabeled sample compared with each label item; for each unlabeled sample input into the BILSTM, the forward propagation hidden state of each word in the unlabeled sample X is calculated first The backward propagation hidden state And the merged forward and backward propagation hidden state h t ;

[0040] Step 503: for each unlabeled sample compared with each label item, the CRF selects the sequence with the highest prediction score as the optimal path according to the path and the corresponding prediction score of different labels, and obtains the final labeled sequence

[0041] Further,

[0042] Step 503 specifically includes the following steps:

[0043] Step 5031: obtain the probability matrix of each word and the corresponding label of the input unlabeled sample X from step 502;

[0044] Step 5032: obtain the transition score matrix of each word and the corresponding label of each input unlabeled sample X through the probability matrix;

[0045] Step 5032: calculate the label score of the input unlabeled sample X compared with the corresponding label item, and obtain the optimal entity label sequence through gradient descent, and then calculate the label score of the unlabeled sample X.

[0046] BERT is a pre-training model that uses a deep bidirectional transformer to build the entire model, which can obtain global context information of language representation. The probability matrix of the corresponding label obtained by the character and BiLSTM, and the score of the label propagation to the label, calculate the score for the input sequence and the arbitrary path of the named entity label. This method avoids getting the wrong label sequence. CRF can obtain the optimal prediction sequence through the relationship between adjacent labels, making up for the shortcomings of BiLSTM.

[0047] Further, step 100 specifically comprises the following steps:

[0048] Step 101: The data acquisition module acquires the target text data published on the network platform, and uses the Stanford natural language processing tool to perform word segmentation on the target text data.

[0049] Step 102: Part of the data in the text data set is manually annotated to obtain a labeled data set L={L1, L2, …L i …L n}, and the remaining data that has not been manually annotated is used as an unlabeled data set U={U1, U2, …U x …U m}.

[0050] After the text data is segmented by the Stanford natural language processing tool, all sentences in the data text will be divided into individual words, so that the speed of manual annotation can be improved when manual annotation is performed, and the influence of hesitation during word segmentation on the annotation efficiency can be reduced.

[0051] As a second aspect of the present application, in order to solve the technical problems mentioned in the background art, a named entity recognition device is provided, which comprises a data acquisition module, an uncertainty calculation module, a diversity calculation module, a data information degree calculation module, a data classification module and a data selection module; The data acquisition module is signal connected with the uncertainty calculation module and the diversity calculation module; The data information degree calculation module is signal connected with the diversity calculation module and the uncertainty calculation module; The data classification module is signal connected with the data information degree calculation module and the data selection module; The data selection module is signal connected with the uncertainty calculation module and the diversity calculation module;

[0052] The data acquisition module acquires the text data published on the network, divides the text data into a labeled data set and an unlabeled data set, and manually annotates the data in the labeled data set using a plurality of preset label items;

[0053] The uncertainty calculation module is trained by using the labeled data set, and then calculates the uncertainty scores of the unlabeled samples in the unlabeled data set;

[0054] The diversity calculation module calculates the similarity of the samples in the labeled data set and the unlabeled samples to obtain the diversity scores of each unlabeled sample;

[0055] The data information degree calculation module calculates the comprehensive scores of the uncertainty scores and the diversity scores of each unlabeled sample according to the pre-set compromise parameter, and manually labels several unlabeled samples with high comprehensive scores to obtain a newly added labeled data set;

[0056] The data classification module calculates the prediction scores of the remaining unlabeled samples in the unlabeled data set according to the newly added labeled data set and the labeled data set;

[0057] The data selection module manually labels the sample data with a prediction score higher than a preset threshold in the unlabeled samples as the labeled data set, and adds the sample data with a prediction score lower than the preset threshold in the unlabeled samples as the unlabeled data set, and repeats the above operation until all the samples in the unlabeled data set are processed as labeled samples.

[0058] In summary, the present application calculates the uncertainty and diversity of the unlabeled samples, manually labels the unlabeled data rich in information degree, trains the BERT-BiLSTM-CRF model using these labeled data, and uses the trained model to predict the remaining unlabeled samples. Through repeated retraining of the model, the accuracy of the model is improved. This method greatly reduces the demand for high-quality labeled data for deep models and is suitable for use in new fields. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and are incorporated herein for a purpose of

[0060] Fig. 1 is a flowchart of the method for recognizing named entities;

[0061] Fig. 2 is a flowchart of the uncertainty score calculation;

[0062] Fig. 3 is a flowchart of obtaining a newly added sample labeled data set;

[0063] Fig. 4 is a structural schematic diagram of the named entity recognition device;

[0064] Fig. 5Schematic diagram of a named entity recognition device.

[0065] In addition, throughout the drawings, same or similar reference numerals designate same or similar elements. It should be understood that the drawings are schematic, and elements and elements are not necessarily drawn to scale. DETAILED DESCRIPTION

[0066] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0067] It should also be noted that, for ease of description, only parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0068] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0069] Reference Figs. 1-5 The named entity recognition method comprises the following steps:

[0070] In step 100, the data acquisition module acquires the text data published on the network, divides the text data into a labeled data set and an unlabeled data set, and manually labels the data in the labeled data set using a plurality of preset label items.

[0071] Further, the data acquisition module acquires the data published on the network through a web crawler.

[0072] Further, the labeling format adopts the "BIO" labeling method. "B" represents the starting word of the entity, "I" represents the remaining words of the entity except the starting word, and "O" represents the words that are not entities.

[0073] Further, the Stanford Natural Language Processing Tool is used to segment the text data, and the sentences in the text data are divided into individual words.

[0074] After the text data is segmented by the Stanford Natural Language Processing Tool, all the sentences in the data text will be divided into individual words, so that the speed of manual labeling can be improved and the influence of manual labeling on labeling efficiency due to hesitation in word segmentation can be reduced.

[0075] Further, the label items include location, organization, person, miscellaneous and event; wherein the location refers to a country or a city; the organization refers to a university, a scientific research institution or a company, etc.; the person refers to a historical celebrity or a state leader; the miscellaneous refers to an artistic work, a book title, a technical product, an agreement, etc.; and the event refers to a political meeting or an academic conference, etc.

[0076] In the embodiments provided in the present application, the relevant information in the field of education is mainly concerned, so the label items are set as the above-mentioned "location, organization, person, miscellaneous and event"; if the information in the field of weather is concerned, the label items can be set as time, location and weather type. In addition to the above-mentioned label item setting mode, the required label items can also be set according to actual needs.

[0077] The step 100 specifically includes the following steps:

[0078] Step 101: The data acquisition module acquires the target text data published on the network platform, and performs word segmentation on the target text data by using the Stanford natural language processing tool.

[0079] Step 102: Part of the data in the text data set is manually labeled to obtain a labeled data set L={L1, L2, …L i …L n}, and the remaining data which is not manually labeled is used as an unlabeled data set U={U1, U2, …U x …U m}.

[0080] In step 102, for the labeled data set L={L1, L2, …L i …L n}, the i-th sample is L i . Each sample is a complete sentence, and a corresponding label is set for each word in the sentence.

[0081] For the unlabeled data set U={U1, U2, …U x …U m}, each unlabeled sample in the unlabeled data set U is defined as unlabeled sample 1, unlabeled sample 2, …, unlabeled sample X, …, unlabeled sample M, and the X-th unlabeled sample is unlabeled sample X; each unlabeled sample is a complete sentence.

[0082] For example, in step 100, the collected data text is: "Ningbo Education Bureau held the 2023 annual education work conference. The reporter learned from the meeting that in 2022, Ningbo made new breakthroughs in education comprehensive reform and took new steps in accelerating the process of education modernization, and was approved as the second batch of Ministry of Education basic education comprehensive reform experimental zone. In the national public service quality monitoring report, Ningbo's "public education" satisfaction rate ranked fifth in the country."

[0083] In this data text, one has three sentences respectively

[0084] "Ningbo Education Bureau held the 2023 annual education work conference."

[0085] "The reporter learned from the meeting that in 2022, Ningbo made new breakthroughs in education comprehensive reform and took new steps in accelerating the process of education modernization, and was approved as the second batch of Ministry of Education basic education comprehensive reform experimental zone."

[0086] "In the national public service quality monitoring report, Ningbo's 'public education' satisfaction rate ranked fifth in the country."

[0087] In this way, the first sentence is labeled by artificial labeling, and the first sentence forms a labeled data set L. For the labeled data set L, it has only one 1 sample.

[0088] The second and third sentences become unlabeled data sets. For the unlabeled data set, it has two unlabeled samples.

[0089] Step 200: input the labeled sample in the labeled data set into the uncertainty calculation module, train the uncertainty calculation module, and then input the unlabeled sample in the unlabeled data set into the uncertainty calculation module to calculate the uncertainty score of the unlabeled sample.

[0090] Step 200 specifically includes the following steps:

[0091] Step 201: input the labeled sample in the labeled data set L into the initial classifier of the uncertainty calculation module to train the initial classifier.

[0092] In step 201, for the labeled data set L={L1, L2, …L i …L n}, where the i-th labeled sample is L i ; for the unlabeled data set U={U1, U2, …U x …U m}, where the Xth unlabeled sample is unlabeled sample X, which is a sentence composed of α words; for each unlabeled sample X, there are α words, and each word is labeled. Therefore, unlabeled sample X = tk1, tk2, ... tk i …tk α ; where the i-th word is tk i The label for sample X is not labeled. The i-th word tk i The tag is

[0093] Step 202: Input the unlabeled samples in the unlabeled dataset U into the initial classifier of the uncertainty calculation module to calculate the uncertainty score of the unlabeled sample and the corresponding label item; where, the greater the uncertainty, the lower the ability of the initial classifier to identify the sample.

[0094] In step 202, the formula for calculating the uncertainty fraction of unlabeled samples is:

[0095] in, It is the word tk i The tag, y is the posterior probability of the label corresponding to the unlabeled sample X, and Θ is the set of all labels. i argmin predicts labels for the model. tki It is the sample X corresponding to tki The position of the minimum value of the fraction. For Take the absolute value of all posterior probabilities after subtracting 0.5, and select the one with the highest probability. As tk i Maximum determinism; then, select all tk. i The one with the lowest probability among ∈X is used as the criterion for judging whether an unlabeled sample X is uncertain. Thus, the above formula can be used to calculate the uncertainty score of each unlabeled sample in each unlabeled dataset U relative to each labeled item.

[0096] Step 300: Input the labeled dataset and unlabeled dataset from step 100 into the diversity calculation module. The diversity calculation module calculates the similarity between the labeled dataset and the unlabeled samples to obtain the diversity score of the unlabeled samples.

[0097] Furthermore, for each unlabeled sample in the unlabeled dataset U, the unlabeled sample X is a sentence composed of multiple words, where unlabeled sample X = word 1, word 2, ..., word i, ..., word n. For the words in unlabeled sample X, word i is the word at the i-th position in unlabeled sample X.

[0098] In step 300, the calculation method of the diversity of each unlabeled sample in the unlabeled data set U and the whole labeled data set L is as follows: for each unlabeled sample in the unlabeled data set U and each labeled sample in the labeled data set, the same word window is selected, and the word window of the unlabeled sample X and the word window of the labeled sample L i The diversity score of the unlabeled sample X is obtained by comparison.

[0099] The specific calculation formula is as follows:

[0100] Wherein, n is the word quantity size of the preset attention window, and n is the part-of-speech tagging number of the unlabeled sample or the labeled sample; is the part-of-speech tagging vector corresponding to the position i in the attention window of the unlabeled sample X, is the context vector of the unlabeled sample X. is the part-of-speech tagging vector corresponding to the position i in the attention window of the labeled sample L i is the part-of-speech tagging vector corresponding to the position i in the attention window of the labeled sample L is the context vector of the labeled sample L i The context vector of the unlabeled sample. Sim() is a function of calculating the number of same elements of two vectors. If the similarity of an unlabeled sample is smaller, the unlabeled sample is more diverse. In this way, the unlabeled data set U and the labeled data set L are input into the uncertainty calculation module, and for each unlabeled sample in the unlabeled data set U, the diversity score compared with the labeled data set L can be obtained.

[0101] Step 400: input the diversity score and the uncertainty score of each unlabeled sample into the data information degree calculation module, the data information degree calculation module calculates the comprehensive score of the uncertainty score and the diversity score of each unlabeled sample according to the pre-set compromise parameter, and screens out a plurality of unlabeled samples for manual labeling to obtain a new labeled data set according to the high and low of the comprehensive score.

[0102] Through the calculation of steps 200 and 300, the uncertainty score and the diversity score of each unlabeled sample in the unlabeled sample U relative to all samples in the labeled sample L can be obtained, so that the comprehensive score of each unlabeled sample relative to all samples in the labeled sample L can be calculated through the compromise parameter.

[0103] The calculation formula of the comprehensive score is as follows: B Scor LC B Scor sim 1-B ; wherein B=0.1-0.9.

[0104] Specifically, step 400 includes the following steps:

[0105] Step 401: determine the number of selected values m of the compromise parameter B and the size of each corresponding selected value, wherein B = 0.1-0.9.

[0106] The compromise parameter B can be any value within the value range, and is generally 0.1, 0.2, …, 0.9. The number of selected values of the compromise parameter can be set according to requirements.

[0107] Step 402: divide all unlabeled samples in the unlabeled data set U into several subsets S1, S2, S3, …, S i ……S n ; wherein the i-th subset is S i ; wherein the unlabeled samples in each subset are k.

[0108] Step 403: calculate the comprehensive score of all unlabeled samples in the unlabeled data set U under the selected value of the compromise parameter B, and extract the unlabeled sample X with the lowest comprehensive score in each subset under each compromise parameter to form a new sample labeled data set CS, k > m.

[0109] For example, the selected values of the compromise parameter B are set to 0.1, 0.2, …, 0.9, all the unlabeled samples in the unlabeled data set U are divided into 20 subsets each containing 20 unlabeled samples, and then the comprehensive score of all the unlabeled samples X in each subset under the 9 compromise parameters 0.1, 0.2, …, 0.9 is calculated. The unlabeled sample X with the lowest comprehensive score in each subset is extracted to form a new sample labeled data set. Therefore, for the 20 unlabeled samples in each subset S i , 9 unlabeled samples will be extracted to form a new sample labeled data set.

[0110] Step 500: input the labeled data set increased by the new labeled data set and the unlabeled data set reduced by the new labeled data set into the data classification module; wherein the labeled data set is used to train the data classification module, so that the data classification module calculates the predicted score of the unlabeled sample;

[0111] In step 500, the labeled data set increased by the new labeled data set is the post-labeled data set L', and the unlabeled data set reduced by the new labeled data set is the post-unlabeled data set U', L' = L-CS; U' = U-CS.

[0112] Further, the data classification module includes a BERT model, an LSTM model, and a CRF model.

[0113] Further, for the unlabeled data set and the labeled data set input into the data classification module, the data classification module has the same processing manner, the sequence input into the data classification module will obtain the probability of the sequence and each preset label item; however, when the labeled data set is input into the data classification module, the data classification module will compare the highest probability of the output preset label item with the manually labeled label, so as to continuously train the data classification module, so that the output result of the data classification module and the manually labeled result are consistent with each other.

[0114] For example, inputting the sequence of "Ningbo City" into the data classification module, the highest probability label input into the data classification module for the first time is "miscellaneous", and the manually labeled result is "location", so it is indicated that the data classification module has not been trained completely, so the sequence of "Ningbo City" needs to be input into the data classification module again for repeated training until the highest probability output by the data classification module is "location". Then, it is indicated that the data classification module has been trained completely, and at this time, the remaining sequences which are not manually labeled are input into the data classification module.

[0115] Further, because the data classification module outputs the prediction score of the input sequence and each label item for the labeled data set and the unlabeled data set, the data classification module has the same processing manner when processing the input labeled data and unlabeled data, only the labeled data needs to be used for training to correct the data classification module, therefore, how the data classification module obtains the prediction score of the unlabeled sample in the post-labeled data is introduced below.

[0116] Specifically, in step 500, the steps of obtaining the unlabeled sample and the corresponding label prediction score are as follows:

[0117] Step 501: inputting the post-unlabeled data set U' into the BERT model, the BERT model obtaining the attention weight of each unlabeled sample input, and then obtaining the feature of the keyword in each unlabeled sample.

[0118] Step 502: inputting the obtained attention weight of each unlabeled sample and the feature of the keyword into the BILSTM model, and obtaining the probability matrix of each word in each unlabeled sample compared with each label item.

[0119] In step 502, for each unlabeled sample input into the BILSTM, the forward propagation hidden state of each word in the unlabeled sample X is calculated first the backward propagation hidden state and the merged forward and backward propagation hidden state h t .

[0120] Step 503: For each unlabeled sample, compared with the probability matrix of each label item, the path of CRF different label sorting and the corresponding prediction score, the sequence with the highest prediction score is selected as the optimal path, and the final labeled sequence is obtained

[0121] Step 503 specifically comprises the following steps:

[0122] Step 5031: From step 502, the probability matrix of each word and the corresponding label of the input unlabeled sample X is obtained;

[0123] Step 5032: And through the probability matrix, the transition score matrix of each word and the corresponding label of each input unlabeled sample X is obtained;

[0124] Step 5032: Calculate the labeling score of the input unlabeled sample X compared with the corresponding label item, and obtain the optimal entity label sequence through gradient descent, and then calculate the labeling score of the unlabeled sample X.

[0125] Step 600: The data selection module manually annotates the sample data in the unlabeled sample whose prediction score exceeds the preset threshold value, and adds it to the labeled data set, and takes the sample in the unlabeled sample whose prediction score is lower than the preset threshold value as the unlabeled data set, and re-executes steps 200-600 until all samples in the unlabeled data set are processed as labeled data.

[0126] In step 600, by Selecting high confidence data.

[0127] Wherein, η = 0.5, X is an unlabeled sample X, tk i Is the i-th word in the unlabeled sample X, y i Is the predicted label item of word tk i , Θ is the set of all labels, Is the posterior probability of the unlabeled sample X corresponding to the label. Is the minimum posterior probability of tk i Corresponding .

[0128] The application also provides a named entity recognition device for recognizing named entities using the named entity recognition method described above.

[0129] The named entity recognition device comprises a data acquisition module, an uncertainty calculation module, a diversity calculation module, a data information degree calculation module, a data classification module, and a data selection module; the data acquisition module is signal connected with the uncertainty calculation module and the diversity calculation module; the data information degree calculation module is signal connected with the diversity calculation module and the uncertainty calculation module; the data classification module is signal connected with the data information degree calculation module and the data selection module; and the data selection module is signal connected with the uncertainty calculation module and the diversity calculation module.

[0130] The data acquisition module acquires the text data published on the network, divides the text data into a labeled data set and an unlabeled data set, and manually labels the data in the labeled data set by using multiple preset label items.

[0131] The data acquisition module processes the data text by using the method described in the foregoing step 100 to obtain the unlabeled data set and the labeled data set.

[0132] The uncertainty calculation module is trained by using the labeled data set, and then calculates the uncertainty scores of the unlabeled samples in the unlabeled data set.

[0133] The uncertainty calculation module processes the unlabeled data set and the labeled data set by using the steps described in the foregoing step 200 to obtain the uncertainty scores of the unlabeled samples.

[0134] The diversity calculation module calculates the similarity of the samples in the labeled data set and the unlabeled samples to obtain the diversity scores of the unlabeled samples.

[0135] The diversity calculation module processes the unlabeled data set and the labeled data set by using the steps described in the foregoing step 300 to obtain the diversity scores of the unlabeled samples.

[0136] The data information degree calculation module calculates the comprehensive scores of the uncertainty scores and the diversity scores of each unlabeled sample according to a pre-set compromise parameter, and manually labels a plurality of unlabeled samples with high comprehensive scores to obtain an added labeled data set.

[0137] The data information degree calculation module processes the unlabeled samples by using the steps described in the foregoing step 400 to obtain the added labeled data set.

[0138] The data classification module calculates the prediction scores of the remaining unlabeled samples from the added labeled data set and the labeled data set.

[0139] The data classification module calculates the prediction scores of the remaining unlabeled samples by using the steps described in the foregoing step 500.

[0140] The data selection module manually annotates sample data in the unlabeled sample whose prediction score exceeds the preset threshold value, adds the sample data to the annotated data set, takes the sample in the unlabeled sample whose prediction score is lower than the preset threshold value as the unlabeled data set, and repeatedly executes the above operation until all samples in the unlabeled data set are processed as annotated samples.

[0141] The data selection module adopts the steps described in the foregoing step 600 to process all samples in the unlabeled data set as annotated samples.

[0142] The above description is merely some preferred embodiments of the present disclosure and a description of the principles of the technology used. Those skilled in the art should understand that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above inventive concept. For example, the above features are replaced with each other to form a technical solution with similar functions disclosed in the embodiments of the present disclosure (but not limited to).

Claims

1. A method of named entity recognition, the method comprising: The method comprises the following steps: Step 100, the data acquisition module acquires the text data published on the network, divides the text data into a labeled data set and an unlabeled data set, and manually labels the data in the labeled data set using multiple preset label items; Step 200: input the labeled samples in the labeled data set into the uncertainty calculation module, train the uncertainty calculation module, then input the unlabeled samples in the unlabeled data set into the uncertainty calculation module, and calculate the uncertainty scores of the unlabeled samples; Step 300: input the labeled data set and the unlabeled data set in step 100 into the diversity calculation module, the diversity calculation module calculates the similarity of the labeled data set and the unlabeled samples to obtain the diversity scores of the unlabeled samples; Step 400: input the diversity scores and the uncertainty scores of each unlabeled sample into the data information degree calculation module, the data information degree calculation module calculates the comprehensive scores of the uncertainty scores and the diversity scores of each unlabeled sample according to a pre-set compromise parameter, and selects a plurality of unlabeled samples for manual labeling to obtain a newly added labeled data set according to the high and low of the comprehensive scores; Step 500: input the labeled data set with the newly added labeled data set and the unlabeled data set with the newly added labeled data set into the data classification module; wherein the labeled data set is used to train the data classification module, so that the data classification module calculates the prediction scores of the unlabeled samples; The data classification module comprises a BERT model, an LSTM model and a CRF model; In step 500, the steps of obtaining the prediction scores of the unlabeled samples and the corresponding labels are as follows: Step 501: input the unlabeled data set U' into the BERT model, the BERT model obtains the attention weight of each input unlabeled sample, and then obtains the features of the keywords in each unlabeled sample; Step 502: input the obtained attention weight of each unlabeled sample and the feature of the keyword into the BILSTM model to obtain a probability matrix of each word in each unlabeled sample compared with each label item; for each unlabeled sample input into the BILSTM, the forward propagation hidden state of each word in the unlabeled sample X is calculated first , the backward propagation hidden state , and the merged hidden state h t of the forward and backward propagation ; Step 503: For each unlabeled sample, the CRF calculates the path of different label orders and the corresponding prediction scores compared with the probability matrix of each label item, selects the sequence with the highest prediction score as the optimal path, and obtains the final labeling sequence ; Step 600: the data selection module manually labels the sample data with a prediction score higher than a preset threshold in the unlabeled samples and adds it to the labeled data set, and takes the sample data with a prediction score lower than the preset threshold in the unlabeled samples as an unlabeled data set, and re-executes steps 200-600 until all the unlabeled samples in the unlabeled data set are processed as labeled samples.

2. The named entity recognition method according to claim 1, wherein: Step 201: input the labeled samples in the labeled data set L into the initial classifier of the uncertainty calculation module to train the initial classifier; Step 202: input the unlabeled samples in the unlabeled data set U into the initial classifier of the uncertainty calculation module to calculate the uncertainty scores of the unlabeled samples and the corresponding labels.

3. The named entity recognition method according to claim 2, wherein: In step 201, for the labeled data set L = {L1, L2, … L i …L n}, the i-th labeled sample is L i ; for the unlabeled data set U = {U1, U2, … U x …U m}, the X-th unlabeled sample is unlabeled sample X, and both are sentences composed of α words; for the unlabeled sample X, there are α words, and each word is labeled with a label, so the unlabeled sample X = tk1, tk2, … tk i …tk α ; where the i-th word is tk i ; the label of the unlabeled sample X is , , … , and the label of the i-th word tk i is ; In step 202, the formula for calculating the uncertainty score of the unlabeled sample is: ; wherein, is the label of the word tk i , is an unlabeled sample corresponding label.

4. The method of claim 3, wherein: In step 300, for each unlabeled sample in the unlabeled data set U and each labeled sample L in the labeled data set i , the same word window is selected, and the unlabeled sample X and the labeled sample L i are compared, and then the diversity score of the unlabeled sample X is obtained.

5. The method of claim 4, wherein: The calculation formula of the comprehensive score is: ; wherein B is a compromise parameter, B = 0.1-0.9; is an uncertainty score, is a diversity score; Step 400 comprises the following steps: Step 401: determine the number m of selected values of the compromise parameter B and the size of each corresponding selected value; Step 402: divide all unlabeled samples in the unlabeled data set U into several subsets S1, S2, S3, … S i … n ; wherein the ith subset is S i the unlabeled samples in each subset are k; Step 403: calculate the comprehensive score of all unlabeled samples in the unlabeled data set U under the selected value of the compromise parameter B, and extract the unlabeled sample X with the lowest comprehensive score in each subset under each compromise parameter to form the newly added sample labeled data set CS, k > m.

6. The named entity recognition method of claim 4, characterized in that: Step 503 specifically comprises the following steps: Step 5031: from step 502, obtain the probability matrix of each word and corresponding label of the input unlabeled sample X; Step 5032: obtain the transition score matrix of each word and corresponding label of each input unlabeled sample X through the probability matrix; Step 5032: calculate the labeling score of the input unlabeled sample X compared to the corresponding label item, and obtain the optimal entity label sequence through gradient descent to calculate the labeling score of the unlabeled sample X.

7. The method of claim 1, wherein: Step 100 specifically comprises the following steps: Step 101: the data acquisition module acquires the target text data published on the network platform, and performs word segmentation on the target text data using the Stanford natural language processing tool; Step 102: manually annotate part of the data in the text dataset to obtain a labeled dataset L = {L1, L2, …L i … n}, and the remaining data which is not manually annotated as an unlabeled dataset U = {U1, U2, …U x … m}.

8. An apparatus for recognizing a named entity, applied to the method for recognizing a named entity according to any one of claims 1-7, characterized in that: The data acquisition module, the uncertainty calculation module, the diversity calculation module, the data information degree calculation module, the data classification module, and the data selection module are connected in signal; the data acquisition module is connected with the uncertainty calculation module and the diversity calculation module in signal; the data information degree calculation module is connected with the diversity calculation module and the uncertainty calculation module in signal; the data classification module is connected with the data information degree calculation module and the data selection module in signal; the data selection module is connected with the uncertainty calculation module and the diversity calculation module in signal; The data acquisition module acquires the text data published on the network, divides the text data into labeled data set and unlabeled data set, and manually labels the data in the labeled data set using multiple preset label items; The uncertainty calculation module is trained using the labeled data set, and then calculates the uncertainty score of the unlabeled samples in the unlabeled data set; The diversity calculation module calculates the similarity of the samples and the unlabeled samples in the labeled data set to obtain the diversity score of each unlabeled sample; The data information degree calculation module calculates the comprehensive score of the uncertainty score and the diversity score of each unlabeled sample according to the pre-set compromise parameter, and manually labels several unlabeled samples with high comprehensive score to obtain a newly added labeled data set; The data classification module calculates the prediction score of the remaining unlabeled samples in the unlabeled data set according to the newly added labeled data set and the labeled data set; The data selection module manually labels the sample data with a prediction score exceeding the preset threshold in the unlabeled sample data and adds it to the labeled data set, and labels the sample data with a prediction score lower than the preset threshold in the unlabeled sample data as an unlabeled data set, and repeats the above operation until all samples in the unlabeled data set are processed as labeled samples.

Images