Multi-level wide-category named entity identification method in CO2 catalysis field
By applying named entity recognition methods of deep learning and machine learning in the field of electrocatalytics, a knowledge map in the field of CO2 electrocatalytic reduction was constructed, which solved the problem of extracting useful information from scientific and technological literature, and improved the efficiency and repeatability of catalyst design and screening.
Patent Information
- Application Number
- CN202411645805.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art is difficult to effectively extract useful information from the scientific and technological literature in the field of electrocatalysis, resulting in a large number of manual labor and repeatability problems in the catalyst design and screening process.
Using a multi-level, wide-category named entity recognition method based on deep learning and machine learning, we identify and classify entities in papers, including methods, materials, products and Faraday efficiency by constructing a knowledge graph in the field of CO2 electrocatalytic reduction.
The construction of the electrocatalytic field database has been achieved, the innovation efficiency in the material field has been improved, manual labor has been reduced, and the repeatability of the catalyst design and screening process has been improved.
Smart Images
Figure CN119886139A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of materials and computers, and realizes a multi-level and wide-category named entity recognition method in the field of material CO2 catalysis based on deep learning and machine learning. Background Art
[0002] Electrocatalysis offers a low-carbon technology for producing sustainable fuels and chemicals while reducing air pollutant emissions from fossil fuels. Although a large amount of experimental work has been devoted to the development of efficient electrocatalysts, these trial-and-error processes require a lot of manual labor and face reproducibility issues. Theoretical calculations and simulations can accelerate catalyst screening through activity descriptors that link catalyst structure to catalyst activity, and have long been regarded as principles for catalyst design. However, in addition to simulating the detailed properties of catalysts at the atomic or molecular level, the characteristics of catalysts are also closely related to their electronic structure, adsorption free energy, surface atomic coordination, etc., so there is an urgent need to grasp more influencing factors at the height of the entire field to guide the design of catalysts. In fact, this knowledge is already mature in the catalytic science literature, so the problem becomes how to construct a domain-specific knowledge graph to extract useful information from the scientific literature in the field of electrocatalysis.
[0003] The combination of artificial intelligence and catalysis science has demonstrated great potential for extracting and analyzing key information from a large amount of academic literature. However, knowledge graphs have not yet been applied to the field of electrocatalysts. As a semantic network with a graph structure, it provides an appropriate technology to visualize multiple relationships in a knowledge network. In a knowledge graph, discrete information consisting of entities and relationships is represented in a graph-structured knowledge database, which converts textual information into scientific knowledge. Natural language processing (NLP) technology can achieve the extraction and analysis of textual information in scientific literature while retaining its syntactic and semantic features, and has recently shown potential in biological and thermoelectric research. In contrast, catalysis science is a complex system in which the activity of catalysts is determined by multiple properties, and NLP technology cannot directly extract and analyze key terms from unstructured and heterogeneous scientific documents in its prototype mode. Summary of the invention
[0004] Therefore, in order to solve the above problems, the present invention proposes a named entity recognition method in the field of materials based on a deep learning model. The present invention first constructs a set of entity standards to be extracted for the literature collection in the field of CO2 electrocatalytic reduction. The entity types include 4 major categories and 39 minor categories. Based on deep learning and machine learning methods, the expert-annotated data is trained to obtain a multi-level and wide-category entity recognition model, completing the construction of the electrocatalytic field database, which is conducive to innovation in the field of materials.
[0005] The technical solution adopted by the present invention is as follows:
[0006] A multi-level, broad-category named entity recognition method in the field of CO2 catalysis, comprising the following steps:
[0007] 1) Obtaining papers on CO2 electrocatalytic reduction to obtain a collection of papers; annotating entities in the title and abstract of each paper in the collection of papers to obtain a training sample;
[0008] 2) generating a semantic representation of the training sample;
[0009] 3) inputting the semantic representation of the training sample into the entity recognition model to predict the type of each entity in the training sample; optimizing the entity recognition model according to the prediction results and labeling results of the training sample;
[0010] 4) Perform fine-grained annotation on entities of various types according to the set named entity classification hierarchy, and save the annotation results in the data dictionary;
[0011] 5) For a paper to be processed, the semantic representation of the paper to be processed is input into the optimized entity recognition model to predict the type of each entity in the paper to be processed; then, according to the type of the entity, a matching fine-grained type is searched in the data dictionary to obtain a fine-grained type recognition result for each entity in the paper to be processed.
[0012] Furthermore, the types of entities include methods, materials, products and Faraday efficiency.
[0013] Furthermore, for entity A identified as a Faraday efficiency type in step 2), a method based on regular expression rules is used to predict the entity A. If the predicted type of the entity A is not a Faraday efficiency type, a voting method is used to determine the final type of the entity A.
[0014] Furthermore, the training samples are input into the Sci-BERT model to obtain semantic representations of the training samples.
[0015] Furthermore, step 5) also includes a classification model. When a matching fine-grained type is not found in the data dictionary according to the type of entity i in the paper to be processed, the entity i is input into the classification model to predict the fine-grained type recognition result of the entity i.
[0016] Furthermore, the classification model is a pre-trained NLP classification model based on the maximum entropy principle.
[0017] Furthermore, the entity recognition model is a bidirectional long short-term memory neural network BiLSTM-CRF with a conditional random field top layer.
[0018] A server, characterized in that it comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the above method.
[0019] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the above method when executed by a processor.
[0020] In order to address the challenges of a large number of entity categories, a wide range of levels, and a small number of labeled samples in the materials field, we transformed the traditional named entity recognition model into a multi-task model consisting of two stages. In the first stage, an entity recognition model is built for the coarse-grained entity categories of the first layer; in the second stage, a classification model is built for the fine-grained entity categories of the second layer, and the coarse-grained entity category labels of the first stage are added to the features of the classification model to improve the accuracy of the overall model.
[0021] A coarse-grained named entity recognition method based on deep learning. In the first stage, the coarse-grained entity recognition model includes a bidirectional long short-term memory network (Bi-directional Long Short-Term Memory, Bi-LSTM) and a conditional random field layer (Conditional Random Field, CRF), which can identify the semantic feature information of words based on the semantics of the word itself and its context. The Sci-BERT model is built on the basis of the Transformer encoder and is a scientific variant of BERT. It retains the original architecture of BERT and is pre-trained on a scientific corpus to generate vector representations of labeled data. In this way, domain knowledge will be integrated into Sci-BERT, thereby improving its performance on downstream tasks. The overall training framework consists of two stages: Pre-training and Fine-tuning. Its steps include:
[0022] 1) Collect and download papers on CO2 electrocatalytic reduction in the past three years from Web of Science Core Collection. A series of entity standards are predefined by experts in the field of materials. The entity categories include four categories. The definitions and descriptions of the categories are shown in Table 1. Then the experts in the field annotate the downloaded paper titles and abstracts, annotating each entity in the paper titles and abstracts to belong to the corresponding type in Table 1. A total of 767 documents were annotated, 1783 corpus sentences and 55158 entity tags were generated. The NER model was trained using the annotated training set, and then the trained NER model was used to automatically annotate the paper titles and abstracts.
[0023] Table 1 Description of entity types in the sample
[0024]
[0025] 2) Based on the labeled data, process it into sequence annotation BIO format and divide it into training set, validation set and test set in a ratio of 6:2:2.
[0026] 3) Pre-trained model. Since the Transformer-based Sci-BERT model has a better effect on word segmentation prediction of scientific paper metadata, Sci-BERT is selected as a large pre-trained model to generate word embeddings. The Sci-BERT model is used to segment the annotated data, and then the word embedding vector representation of the annotated data is generated; then fine-tuning is performed on the divided data set.
[0027] 4) BiLSTM-CRF prediction. In the named entity recognition task, there is a dependency between the labels before and after. For example, the I label can only exist after the B label and the I label, and is associated with different entity categories. In order to capture this dependency, a bidirectional long short-term memory neural network (BiLSTM-CRF) layer with a conditional random field top layer is added to the pre-trained Sci-BERT word embedding model to predict the coarse-grained category of each entity, making the coarse-grained entity recognition result of this step more accurate.
[0028] Considering that some entity types such as material and product entities are usually described by chemical formulas, the entity previously identified as Faraday efficiency type is predicted again using a regular expression rule-based method. If the prediction result is different from the previous prediction, a voting method is used to determine the final type; Faraday efficiency entities are usually expressed in the format of "number + unit", so a regular expression rule-based method is proposed to assist model identification. In view of the different results produced by the two methods, an accuracy voting method and various reference boundary coordination strategies are adopted to make the final decision.
[0029] A fine-grained entity classification method based on machine learning, the steps of which include:
[0030] 1) The entity categories in the material field are very complex, so based on the coarse-grained entity categories obtained by the above method, a finer-grained division is required. The hierarchical structure of named entity classification is shown in Table 2. Here, domain experts perform more detailed annotations according to this structure. Fine-grained entity labels provide more detailed label subcategories. For example, when the annotator annotates a material described as an electrocatalyst, he / she needs to specify what type of material it is, such as Cu, Cu / C, CuOx, etc.
[0031] Table 2 Named entity classification hierarchy
[0032]
[0033]
[0034] 2) Based on the labeled data, the training set and test set are divided into two sets in a ratio of 7:3. The entities labeled by the fine-grained classification in the previous step are saved using a data dictionary, and the entities are classified according to the type of coarse-grained entities.
[0035] 3) For entity categories that cannot be matched in the data dictionary, an NLP classification model based on the maximum entropy principle can be trained based on the labeled data, and the classification model can be used for prediction. Considering that finer granularity means that entities under the same coarse-grained type will present a closer distance in semantics, the coarse-grained category features obtained in the first stage will also improve the efficiency of the classification model. Finally, the features of each entity are combined with the word embedding vector, context text vector, word clustering information and coarse-grained entity category information through a simple mapping function. The word embedding vector is obtained by using the ChemWordTokenizer of the ChemDataExtractor tool to tag sentences; the context text vector is obtained by training the MLM task of the Sci-BERT model; the word clustering information is obtained using the existing Brown clustering method, which is a hierarchical clustering form based on the context of word appearance, which can improve the performance of part-of-speech tagging and named entity recognition methods in various fields; the coarse-grained entity category information is obtained in the coarse-grained entity recognition task in the previous stage.
[0036] 4) Integrate the results of the data dictionary and classification model to obtain fine-grained entity recognition results.
[0037] Furthermore, deep learning methods include the Transformer-based Sci-BERT model and the BiLSTM-CRF model, and machine learning methods include data dictionary and maximum entropy classification model methods.
[0038] A storage medium stores a computer program, wherein the computer program is configured to execute the method described above when running.
[0039] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer to execute the method described above.
[0040] Compared with the prior art, the present invention has the following advantages:
[0041] 1. The method adopted by the present invention is a sequence annotation model based on the pre-trained model of Sci-BERT and the Viterbi decoding of BiLSTM-CRF, which is also one of the mainstream models of NER at present. The Sci-BERT model is a variant of the BERT model for the scientific field, which is more suitable for NLP tasks in the direction of scientific papers. At the same time, the CRF layer has two types of feature functions, which can further learn the constraints of the sentence compared to the traditional linear classifier to ensure the effectiveness of the final prediction result.
[0042] 2. The current NER model only extracts entities of different categories based on coarse-grained classification, while the entity categories provided in this competition include 4 major categories and 39 minor categories. Therefore, we divide this task into two subtasks, a coarse-grained entity recognition module and a fine-grained entity classification module, to improve the efficiency and accuracy of entity extraction in a multi-task form.
[0043] 3. In the fine-grained classification module, we choose to use the dictionary-first-model-later approach hierarchically. Compared with extracting fine-grained entities from the entire sentence, this amplifies the subtle differences between each fine-grained entity, thereby amplifying the space for further distinguishing chemical formula entities with similar language features. Finally, the features of each entity are combined with the word embedding vector, context text vector, word clustering information, and coarse-grained entity category information through a simple mapping function. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is the overall architecture diagram of the model of the present invention.
[0045] Figure 2 The data flow diagram of the model.
[0046] Figure 3 Module diagram of the coarse-grained named entity model based on deep learning.
[0047] Figure 4 Module diagram of the fine-grained entity classification model based on machine learning. DETAILED DESCRIPTION
[0048] The present invention is further described in detail below through specific embodiments and drawings.
[0049] Please refer to Figure 1 The multi-level and wide-category named entity recognition method in the field of CO2 catalysis based on deep learning and machine learning proposed in the present invention comprises the following steps:
[0050] Step S1: Data preprocessing. Domain experts perform entity annotation on the downloaded documents on the annotation platform doccano, and the export format is json, including three fields: "id": a single id for each sentence; "data": sentence content; "label": entity label, each entity label consists of three fields. Taking [386,408,"CuSx"] as an example, "386" is the number of characters at the beginning of the entity, "408" is the number of characters at the end of the entity, and "CuSx" is the fine-grained category to which the entity belongs. The preprocessing of the original data is divided into two parts, the BIO annotation part and the fine-grained classification part, which are used to model the NER model and the fine-grained classification model respectively.
[0051] like Figure 2 As shown in the figure, for the NER model, the data is annotated into BIO format according to the major categories to which the entities belong, where each word corresponds to a label. BM means that the word is the first word of the M-type entity, IM means that the word is the middle word of the M-type entity, and O means that the word does not belong to any entity type.
[0052] For the entity classification model, the data is divided into sentences, each sentence contains the original sentence itself, the main category to which the entity belongs, and the subcategory to which the entity belongs.
[0053] Step S2: NER model. Since the semantic information between entities of different categories in the data will affect each other, in order to improve the accuracy, a NER model is trained for each category of entities. For example, taking the method category as an example, first filter out all method category labels from the annotated BIO format data, and other categories are marked as O (other labels), so the labels output by the model are only B-method, I-method and O categories. Although this approach increases the number of models that need to be trained, the training time of each model is shortened, and the interference of other entities is eliminated, and the accuracy will be improved compared to training together.
[0054] like Figure 3As shown in the figure, the NER model first calls the Transformer-based Sci-BERT model for pre-training, tokenizes and tags the input sentences, and then relies on self-attention to calculate its input and output. The encoder and decoder are constructed using stacked self-attention and point-by-point fully connected layers to link different positions of a single sequence to obtain the semantic representation of the text. The semantic embedding is then passed to the BiLSTM-CRF layer, and the Viterbi matrix of emission and transfer is established using the constraint relationship between the words in the sentence to obtain the predicted label string, and the corresponding extracted entity can be obtained. For other entities including chemical formulas and numbers, a method based on regular expression rules is used to assist model recognition. We apply the recognition model to the test data for entity extraction. If there is a span overlap, the voting classification method is used to select the final category to which it belongs. The voting results are weighted according to the accuracy of each model.
[0055] Step S3: Follow Figure 2 The data flow diagram of the NER output label is processed into the format of [left_index, right_index, span, entity_label], where left_index and right_index represent the offset of the extracted entity in the sentence, span is the span of the extracted entity, and entity_label is the category to which the extracted entity belongs. The data in this format is passed to the fine-grained classification model to obtain the fine-grained category corresponding to the entity.
[0056] The training of fine-grained classification models is divided into two steps. Figure 4As shown in the figure, first, by analyzing the annotated data, the entities in the field of materials have very obvious semantic features, so we consider using the data dictionary method to capture the features. The dictionary layer constructs a dictionary for the entity names and labels in the preprocessed data to handle most of the coarse-grained entity classification tasks. For entities that cannot be matched by the dictionary, the sentence classification model is trained with the entity name, label and original sentence in the feature classification layer, and the classification layer classifies the sentences to be predicted separately. The features of each entity combine the word embedding vector, context text vector, word clustering information and coarse-grained entity category information through a simple mapping function. We first use the ChemWordTokenizer of the ChemDataExtractor tool to tokenize the sentences and obtain the word embedding vector. The context vector of each word is obtained by training the MLM task of the Sci-BERT model. The word clustering information uses the Brown clustering method, which is a hierarchical clustering form based on the context in which the word appears, which can improve the performance of part-of-speech tagging and named entity recognition methods in various fields. We clustered the full text and titles of 2,123 material articles published in ACS, RSC, and Springer journals, which contained about 20 million words in about 700,000 sentences. The clustered information was used as entity classification features for maximum entropy model training. Finally, the fine-grained entities obtained by the dictionary layer and feature classification layer were integrated as the final output.
[0057] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. A multi-level, broad-category named entity recognition method in the field of CO2 catalysis, comprising the following steps: 1) Obtain papers on CO2 electrocatalytic reduction and obtain a collection of papers; Annotating entities in the title and abstract of each paper in the collection of papers to obtain a training sample; 2) generating a semantic representation of the training sample; 3) inputting the semantic representation of the training sample into the entity recognition model to predict the type of each entity in the training sample; optimizing the entity recognition model according to the prediction results and labeling results of the training sample; 4) Perform fine-grained annotation on entities of various types according to the set named entity classification hierarchy, and save the annotation results in the data dictionary; 5) For a paper to be processed, the semantic representation of the paper to be processed is input into the optimized entity recognition model to predict the type of each entity in the paper to be processed; then, according to the type of the entity, a matching fine-grained type is searched in the data dictionary to obtain a fine-grained type recognition result for each entity in the paper to be processed.
2. The method according to claim 1, characterized in that The types of entities include methods, materials, products, and faradaic efficiencies.
3. The method according to claim 2, characterized in that For entity A identified as the Faraday efficiency type in step 2), a method based on regular expression rules is used to predict the entity A. If the predicted type of the entity A is not the Faraday efficiency type, a voting method is used to determine the final type of the entity A.
4. The method according to claim 1, characterized in that: The training sample is input into the Sci-BERT model to obtain a semantic representation of the training sample.
5. The method according to claim 1, 2 or 3, characterized in that: Step 5) also includes a classification model. When no matching fine-grained type is found in the data dictionary according to the type of entity i in the paper to be processed, the entity i is input into the classification model to predict the fine-grained type recognition result of the entity i.
6. The method according to claim 5, characterized in that The classification model is a pre-trained NLP classification model based on the maximum entropy principle.
7. The method according to claim 1, 2 or 3, characterized in that: The entity recognition model is a bidirectional long short-term memory neural network BiLSTM-CRF with a conditional random field top layer.
8. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method according to any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.