Method for chemical production named entity recognition by pre-training BERT

Abstract

The present invention belongs to the technical field of entity recognition, and particularly relates to a method for chemical production named entity recognition by pre-training BERT. The method comprises S1, delivering the text to be recognized to the pre-trained LM embedding model ChemBERT, thereby obtaining a semantic representation vector and a semantic retrieval library; S2, applying a decoder to the semantic representation vector of step S1; S3, voting using a decoded label and a KNN clustering label, and the voting result is used as the final output. The method solves the problem of long-tail data and small sample data in the data set of the existing entity recognition method.

Description

Technical Field

[0001] The present invention belongs to the technical field of entity recognition, and in particular to a method for chemical production named entity recognition by pre-training BERT. Background Art

[0002] Named entity recognition is a subtask of information extraction, which aims to locate and classify named entities in text into predefined categories such as people, organizations, locations, time expressions, quantities, monetary values, percentages, etc. At present, most of the named entity recognition work using machine learning methods is based on different combinations of BERT, LSTM and CRF. Sequence models such as long short-term memory (LSTM) are usually used as baseline models, and multi-task learning and multi-step training are combined to improve the performance of clinical NER datasets. However, it takes a long time for LSTM to process long texts. Considering the context, bidirectional long short-term memory (BiLSTM) is adopted. The most representative model is BiLSTM-CRF, which uses BiLSTM network to encode biomedical text and CRF to decode named entity tags. There are also methods that incorporate BERT, using attention mechanisms, which can parallelly calculate the weight of each position of the text relative to another position. BERT directly learns WordPiece embedding before training and fine-tuning, and uses a single output layer based on its last layer representation to only calculate the token-level BIO probability. Compared with other methods, this greatly improves its performance on named entity recognition tasks.

[0003] Chemical named entity recognition (NER) is the basis of information extraction tasks in the field of chemistry. However, the extraction of key information of chemicals is not effective at present, especially when the literature contains complex information such as chemical names, response plans, and storage environments. However, these information categories are clear and related to the chemicals themselves, and these rules have not been fully utilized. In the field of hazardous chemicals, most entities can be classified into "chemicals", "management plans", and "mediums", resulting in a small number of entities in other categories such as "emergency methods" and "scenarios", resulting in the problem of long-tail data and small sample data in the dataset. Summary of the invention

[0004] The present invention proposes a method for chemical production named entity recognition by pre-training BERT, which solves the problem of long-tail data and small sample data in the data set of the existing entity recognition method.

[0005] The technical solution of the present invention is achieved in this way:

[0006] A pre-trained BERT LM embedding model, ChemBERT, includes an embedding layer, a KNN retrieval layer, a CRF decoding layer, and an output layer.

[0007] The method for chemical production named entity recognition by pre-training BERT specifically includes the following steps:

[0008] S1. The text to be recognized is fed into the pre-trained LM embedding model ChemBERT to obtain the semantic representation vector and semantic retrieval library.

[0009] S2, applying the decoder to the semantic representation vector of step S1;

[0010] S3. Use the decoded labels and KNN clustering labels to vote, and the voting results are used as the final output.

[0011] Optionally, in step S1, the specific steps of obtaining the semantic representation vector are as follows:

[0012] A1. Obtain the contextual semantics at the word and sentence levels by encoding the segmented sentences in the text;

[0013] A2. Pre-train the input text to generate dynamic tag vectors and automatically extract a large number of word-level and sentence semantic features from the text;

[0014] A3. Use CRF layer constraints to predict the dependencies between labels and obtain the global optimal label sequence.

[0015] A4. After the sentences in the text are segmented, individual words are used as the minimum processing unit of the model input, i.e., tokens. The model is fine-tuned by using the chemical entry text and labels in the structured corpus and adding some manually annotated text and labels from the safety hazard emergency plan.

[0016] A5. Apply BERT to sentences, adding special tokens [CLS] and inter-sentence tokens [SEP] at the beginning of the sequence to construct the input sequence:

[0017] ;

[0018] A6. Input the sequence into Chinese BERT for feature extraction to obtain a semantic representation vector;

[0019] ;

[0020] in, Indicates the number of tokens in the sequence. .

[0021] Optionally, in step S1, the specific steps of obtaining the semantic search library are as follows:

[0022] Match the labeled embedding vector S of the training set with the corresponding label Y to form a key-value pair, and then form a retrieval library as follows:

[0023] ;

[0024] in, represents the token embedding vector, Indicates the label corresponding to the token, Indicates Tokens, Indicates The label corresponding to the token;

[0025] Optionally, in step S2, the specific steps of applying the semantic representation vector by the decoder are as follows:

[0026] S21. Use conditional random field CRF to perform label prediction task on sentence embedding. Given an input sentence , label sequence The probability is:

[0027] ;

[0028] in, represents all possible label sequences of the input, express for The emission weight, represents the transfer matrix weight of two adjacent labels, Indicates the true category label of the token;

[0029] S22. Given a text sequence , the tokens obtained by BERT are embedded into Then use The Euclidean distance is used as the criterion for the Perform k-nearest neighbor search on the tags in the table to obtain the nearest neighbor set , and convert it into an output distribution based on the RBF kernel method as follows:

[0030] ;

[0031] in, Represents the token tag set retrieved by k nearest neighbors. express Search key-value pairs in; Represents the jth entity in the label category table, which is predicted to be an entity The probability of all N equal to The value of is proportional to the sum of the nuclear outputs; is the indicator function, The value is 1 when it is, otherwise it is 0; Indicates Token semantic representation vector and search key-value pairs Middle Nearby Point L2 distance; is the scaling parameter used to smooth the distribution, Indicates the first tokens, and correspond.

[0032] Optionally, in step S3, the specific steps of voting are as follows:

[0033] Combine the two prediction distributions into the final output;

[0034] ;

[0035] in, The value range is (0, 1), which changes with the training process and represents the weight of the two prediction distributions;

[0036] The result of the vote is the label with the highest decoding score:

[0037] ;

[0038] in, Indicates Tokens, Indicates The label of a token.

[0039] After adopting the above technical solution, the beneficial effects of the present invention are:

[0040] The present invention uses the BERT pre-trained model to represent entities in context. The built-in MASK mode of BERT enables the model to take into account the specific usage context of vocabulary in sentences, so that the present invention can have more accurate prediction capabilities in the chemical industry. The clustering algorithm KNN is combined to learn a large amount of unlabeled text and entities, so that the present invention can quickly adapt to unknown vocabulary and expressions in the chemical industry and has stronger generalization.

[0041] In real industrial scenarios, the long-tail problem is usually encountered (some categories have a lot of data, while other categories have very little data). For types with very small amounts of data, the problem of small sample data arises when training the model. The present invention uses two different sampling methods, namely, using KNN to sample the neighbors of the entity to form the nearest neighbor set of the entity and using BERT to directly sample the text where the entity is located to form a representation of the long-distance dependency of the entity, to avoid the small sample representation vector error caused by the small amount of data, and finally weighting the two sampling methods to infer the category of the entity, thereby solving the long-tail problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 Schematic diagram of the structure of the entity recognition method in the embodiment. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The embodiment of the present application discloses a method for chemical production named entity recognition by pre-training BERT.

[0046] Example: According to Figure 1 As shown, the method for chemical production named entity recognition by pre-training BERT includes the following steps:

[0047] S1. The text to be recognized is fed into the pre-trained LM embedding model ChemBERT to obtain the semantic representation vector and semantic retrieval library.

[0048] The specific steps to obtain the semantic representation vector are as follows:

[0049] A1. Obtain the contextual semantics at the word and sentence levels by encoding the segmented sentences in the text;

[0050] A2. Pre-train the input text to generate dynamic tag vectors and automatically extract a large number of word-level and sentence semantic features from the text;

[0051] A3. Use CRF layer constraints to predict the dependencies between category labels and obtain the global optimal label sequence.

[0052] A4. After the sentences in the text are segmented, individual words are used as the minimum processing unit of the model input, i.e., tokens. The model is fine-tuned by using the chemical entry texts and labels in the structured corpus and adding some manually annotated texts and labels from the safety hazard emergency plan.

[0053] A5. To obtain the contextual representation of each token in the input sentence, BERT is first applied to the sentence, adding a special token [CLS] and an inter-sentence token [SEP] at the beginning of the sequence to construct the input sequence:

[0054] ;

[0055] A6. Input the sequence into Chinese BERT for feature extraction to obtain a semantic representation vector;

[0056] ;

[0057] in, Indicates the number of tokens in the sequence. .

[0058] The specific steps to obtain the semantic retrieval library are as follows:

[0059] Since a word is used as the tag input of the BERT model after sentence segmentation, the tag embedding vector can be directly used to build the retrieval library of the KNN model. First, the tag embedding vector S of the tag in the training set is matched with the corresponding label Y to form a key-value pair, and then a retrieval library is formed, as shown below:

[0060] ;

[0061] in, represents the token embedding vector, Indicates the label corresponding to the token, Indicates Tokens, Indicates The label corresponding to the token.

[0062] Afterwards, given a text sequence , the tokens obtained by BERT are embedded into Then use The Euclidean distance is used as the criterion for the Perform k-nearest neighbor search on the tags in the table. Get the nearest neighbor set , and convert it into an output distribution based on the RBF kernel method as follows:

[0063] ;

[0064] in, Represents the token tag set retrieved by k nearest neighbors. express Search key-value pairs in; Represents the jth entity in the label category table, which is predicted to be an entity The probability of all N equal to The value of is proportional to the sum of the nuclear outputs; is the indicator function, The value is 1 when it is, otherwise it is 0; Indicates Token semantic representation vector and search key-value pairs Middle Nearby Point L2 distance; is the scaling parameter used to smooth the distribution, Indicates the first tokens, and correspond;

[0065] S2, apply the decoder to the semantic representation vector of step S1; use the conditional random field CRF to perform label prediction on the semantic representation vector of the sentence. Given an input sentence , label sequence The probability is:

[0066] ;

[0067] in, represents all possible label sequences of the input, express for The emission weight, represents the transfer matrix weight of two adjacent labels, Indicates the real label of the token;

[0068] S3, use the decoded labels and KNN clustering labels to vote, and the voting results are used as the final output;

[0069] The specific steps for voting are as follows:

[0070] S31. Combine the two predicted distributions into the final output as follows:

[0071] ;

[0072] in, The value range is (0, 1), which changes with the training process and represents the weight of the two prediction distributions;

[0073] The result of the vote is the label with the highest decoding score:

[0074] ;

[0075] in, Indicates Tokens, Indicates The label of a token.

[0076] The experiment of the entity recognition method in the present invention is as follows:

[0077] 1. Experimental data

[0078] Two public corpora and one professional dataset were used: the traffic hazard corpus, the vehicle maintenance corpus, and the chemical production dataset. Overall statistics for each dataset are provided in the supplementary material BioCreative Corpus. Like many teams in the challenge, the original training set and the development set were used as the training set. Then, 10% of the training set was randomly selected as the validation set to adjust the hyperparameters. Chemical NER performance is measured by the F score, which gives equal importance to the precision and recall of the test set.

[0079] Boundary definition: With reference to the MSDS chemical corpus and safety hazard emergency plan text of the "Shanghai Institute of Organic Chemistry Database", it is stipulated that the identification label categories are divided into 4 categories, a total of 15 types, as shown in the following table:

[0080] Table 1 Classification of token identification labels

[0081]

[0082] 2. Experimental Setup

[0083] Adam optimization algorithm is used as optimizer with a learning rate of 1e -5. The maximum sentence length is set to 175 for the two public corpora and 300 for the chemical production dataset, covering more than 99% of the sentences. The hidden size of BERT is set to 512. The random dropout rate of neurons set in the BERT layer is set to 0.2. The Adam optimization algorithm is used as the optimizer with a learning rate of 1e-5. The number of model training iterations (epochs) is set to 10, and the batch size of each epoch is set to 35. The BERT pre-trained model selects "bert-base-chinese-accidentreason-classifier" pre-trained from scratch on the same corpus as the vocabulary for embedding sentences. Entity embeddings are trained on the chemical production dataset using this BERT. All hyperparameters for entity embedding training in this BERT are set to default values, except for the following hyperparameters: epochs is set to 10 and batch size is set to 35. All experiments are run on RTX 3090.

[0084] 3. Evaluation indicators

[0085] Multiple metrics are used to evaluate model performance, including precision, recall, and F1 score. Precision is the ratio of true positives among identified positive samples, reflecting the model’s classification accuracy for BioNEs tokens:

[0086] ;

[0087] Among them, TP represents the number of positive classes predicted as positive classes, and FP represents the number of negative classes predicted as positive classes. The recall rate represents the proportion of all positive samples in the test set that are correctly identified as positive samples, reflecting the ability of the model to distinguish BioNEs:

[0088] ;

[0089] Among them, FN represents the number of positive classes predicted as negative classes. F1 score is one of the most commonly used indicators in classification and information retrieval, reflecting the average performance of model precision and recall:

[0090] .

[0091] 4. Experimental results

[0092] The model is compared with existing methods on two datasets.

[0093] First, we use NB-CRF without considering rule information for testing. Then we compare our method with the SOTA method using BERT as the embedding layer, including BERT-base-chinese and BERT-base-chinese-accidentreason-classifer that have been fine-tuned on the traffic hazard corpus.

[0094] NB-CRF is a machine learning model that uses a Bayesian approach to NER.

[0095] Hierarchical BiLSTM-CRF is a neural hierarchical model for nested NER.

[0096] MLP+softmax treats the NER sequence labeling task as a multi-classification problem. The label of each word is predicted independently based on the context-dependent representation without considering the results of its neighboring labels.

[0097] MLP+CRF treats the NER sequence tagging task as a multi-classification problem. The label of each word is predicted independently based on the context-dependent representation, without considering the results of its neighboring labels.

[0098] BERT-base is a contextualized word representation model based on a masked language model, pre-trained with a bidirectional transformer. We use its embeddings with a BiLSTM-CRF architecture as a baseline.

[0099] BioBERT v1.0 is a pre-trained model based on BERT, using 200,000 PubMed abstracts and 270,000 PMC full-text articles. A NER fine-tuned version of the BiLSTM-CRF architecture is used as the baseline.

[0100] BioBERT v1.1 is the same as BioBERT v1.0, but is pre-trained using 1 million PubMed abstracts.

[0101] MKRGCN utilizes Chinese vocabulary and domain knowledge graph concepts to consider the boundaries of non-monotonic elements.

[0102] EnRel-G combines entity mention relationships based on global reference relationships and local dependency relationships through graph neural networks.

[0103] The results are shown in the following table:

[0104] Table 2 Comparison of test results

[0105]

[0106] On the chemical production dataset and vehicle maintenance corpus, RiBERT outperforms NB-CRF, hierarchical BiLSTM-CRF, MLP+softmax, MLP-CRF, RNN-CRF, Transformer-CRF, BERT-base-chinese, and BERT-accidentreason. RiBERT achieves an F1 score of 69.77% on the chemical production dataset and 88.12% on the vehicle maintenance corpus. Compared with BERT-accidentreason, RiBERT improves the F1 score by 0.7% and 0.42% on the chemical production dataset and vehicle maintenance corpus, respectively. The improvements in the rule information encoding and fusion layers are significant. Among the methods that can exploit both rule information and local dependencies, RiBERT performs better on NCBI-disease, but sometimes performs worse on BC2GM (MKRGCN-max is higher than RiBERT-min). This result shows that it is not simple to exploit local dependencies of external knowledge for NER. The reason may be that the representation space of article words is different from the representation space of KG concepts.

[0107] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for chemical production named entity recognition by pre-training BERT, characterized in that: A pre-trained BERT LM embedding model ChemBERT is used for entity recognition, wherein the embedding model ChemBERT includes an embedding layer, a KNN retrieval layer, a CRF decoding layer and an output layer; The method specifically comprises the following steps: S1. The text to be recognized is fed into the pre-trained LM embedding model ChemBERT to obtain the semantic representation vector and semantic retrieval library. S2, applying the decoder to the semantic representation vector of step S1; S3, use the decoded labels and KNN clustering labels to vote, and the voting results are used as the final output; In step S1, the specific steps of obtaining the semantic representation vector are as follows: A1. Obtain the contextual semantics at the word and sentence levels by encoding the segmented sentences in the text; A2. Pre-train the input text to generate dynamic tag vectors and automatically extract a large number of word-level and sentence semantic features from the text; A3. Use CRF layer constraints to predict the dependencies between labels and obtain the global optimal label sequence; A4. After the sentences in the text are segmented, individual words are used as the minimum processing unit of the model input, i.e. tokens. The model is fine-tuned by using the chemical entry texts and labels in the structured corpus and adding some manually annotated texts and labels from the safety hazard emergency plan. A5. Apply BERT to sentences, adding tokens [CLS] and inter-sentence tokens [SEP] at the beginning of the sequence to construct the input sequence: ； A6. Input the sequence into Chinese BERT for feature extraction to obtain a semantic representation vector; ； in, Indicates the number of tokens in the sequence. ; In step S1, the specific steps of obtaining the semantic retrieval library are as follows: Match the labeled embedding vector S of the training set with the corresponding label Y to form a key-value pair, and then form a retrieval library as follows: ； in, represents the token embedding vector, Indicates the label corresponding to the token, Indicates Tokens, Indicates The label corresponding to the token.

2. The method for chemical production named entity recognition by pre-training BERT according to claim 1, characterized in that: In step S2, the specific steps of applying the semantic representation vector by the decoder are as follows: S21. Use conditional random field CRF to perform label prediction task on sentence embedding. Given an input sentence , label sequence The probability is: ； in, represents all possible label sequences of the input, express for The emission weight, represents the transfer matrix weight of two adjacent labels, Indicates the true category label of the token; S22. Given a text sequence , the tokens obtained by BERT are embedded into Then use The Euclidean distance is used as the criterion for the Perform k-nearest neighbor search on the tags in the table to obtain the nearest neighbor set , and convert it into an output distribution based on the RBF kernel method as follows: ； in, Represents the token tag set retrieved by k nearest neighbors. express Search key-value pairs in ; Represents the jth entity in the label category table, which is predicted to be an entity The probability of all N equal to is proportional to the sum of the nuclear outputs of the value of ; is the indicator function, The value is 1 when it is, otherwise it is 0; Indicates Token semantic representation vector and search key-value pairs Middle Nearby Point L2 distance; is the scaling parameter used to smooth the distribution, Indicates the first tokens, and correspond.

3. The method for chemical production named entity recognition by pre-training BERT according to claim 2, characterized in that: In step S3, the specific steps of voting are as follows: The two predicted distributions are merged into the final output as shown below: ； in, The value range is (0, 1), which changes with the training process and represents the weight of the two prediction distributions; The result of the vote is the label with the highest decoding score: ； in, Indicates Tokens, Indicates The label of a token.

Images