Method, device and medium for identifying descriptors of text data

Abstract

The present invention discloses a method, device and medium for identifying descriptors of text data. The method comprises: separating text data into at least one sentence sequence, and separating each sentence sequence into separate tags, and labeling each tag based on a preset entity tag, wherein the preset entity tag is used to define a descriptor; randomly masking some words in the sentence sequence, and predicting the masked words through the learned contextual semantic relationship to enhance the text data; training a recognition model based on the enhanced text data, and using the trained recognition model to identify the descriptors of the text data, thereby screening out performance-driven high-quality descriptors. The present invention can not only automatically extract descriptors from materials science literature from both coarse-grained and fine-grained aspects, but also embed domain knowledge into the descriptor recognition method, so that the present invention can screen high-quality descriptors according to user needs.

Description

Technical Field

[0001] The present invention relates to the technical field of text data processing, and more specifically, to a method, device and medium for identifying a descriptor of text data. Background Art

[0002] With the development of materials science, the accumulation of material data is becoming increasingly large. Extracting useful information from hundreds of millions of complex data, analyzing and sorting out the structure-activity relationship of material composition-structure-process-performance has become the core and key of material research. Machine learning can establish a mapping relationship between material influencing factors (such as composition, process, external environment and other descriptors) and target quantities (such as performance, etc.), thereby realizing the prediction of material composition, structure, process, performance and the discovery of new materials. However, the performance of machine learning models is constrained by data quality. Therefore, scientifically and rationally constructing descriptor features related to target quantities is still the basis for establishing high-precision machine learning prediction models and assisting in understanding material mechanisms. Therefore, the rapid and effective selection of appropriate descriptors is of great significance for the study of material structure-activity relationships.

[0003] In the study of structure-activity relationship in the field of materials, the selection of descriptors mainly relies on the experience and knowledge of experts or manual reading of literature. For example, Jalem et al. (2014) selected chemical composition descriptors such as charge and coordination number of elements and crystal structure descriptors such as lattice constant, unit cell volume, bond length and bond angle of polyhedron and interatomic distance to construct learning samples to predict the activation energy of olivine compound LiMXO_4 (M, X as main group elements) by summarizing the existing material knowledge. Sendek et al. (2016) manually screened 12831 lithium-containing crystalline solids from the Material Project database and screened out lithium with high structural and chemical stability, low electronic conductivity and low cost. Then, 21 descriptors related to crystal structure and chemical composition were screened by conductivity to explore the structure-composition-conductivity relationship of solid electrolytes for lithium-ion batteries. Xu et al. (2020) collected ionic conductivity data of 70 NASICON-type compounds of R3 c space groups based on a large amount of literature and extrapolated the Arrhenius law, and constructed a prediction model for ionic conductivity using logistic regression. They empirically selected 16 chemical composition descriptors such as element radius, element electronegativity, number of ions, and 12 structural descriptors such as lattice constant, unit cell volume, and atomic volume to explore the structure-activity relationship with conductivity. However, experts spend a lot of time summarizing words or phrases in published literature to obtain suitable descriptors, which will lead to an increase in the number of materials science literature publications. This approach will seriously affect the selection of descriptors, thereby affecting the study of material structure-activity relationships.

[0004] Named Entity Recognition (NER) can automatically extract information from unstructured text. Usually, this type of task is regarded as a supervised machine learning problem, in which the model learns to recognize keywords or phrases in sentences. Currently, NER has been applied to information extraction of organic and inorganic materials. For example, Kim et al. (2017) used NER to parse 76,000 articles related to oxide material synthesis, extracted key information, and encoded this information into a database; Mysore et al. (2017) extracted action graph structures from material science synthesis programs through NER, and was able to accurately extract action graph synthesis text when the data set was small; In addition, some chemical-based NER systems can extract inorganic materials. For example, Krallinger et al. (2017) used NER to achieve effective acquisition of chemical information contained in scientific literature, patents, technical reports or networks; Leaman et al. (2015) used NER technology to accurately identify chemical terms, attributes, and relationships in literature. Recently, some researchers have achieved large-scale extraction of inorganic material information from literature by building deep learning NER models. For example, He et al. (2020) built a Bi-LSTM-based model to achieve accurate extraction of precursors and targets in inorganic solid-state synthesis reactions reported in the literature; Weston et al. (2019) used NER to identify inorganic material references, sample descriptors, phase labels, material properties and applications, as well as any synthesis and characterization methods used from the literature. The research results of NER in inorganic material information extraction have attracted the attention of organic material researchers. For example, Zhao et al. (2021) used named entity recognition (NER) and BiLSTM-CNN-CRF deep learning models to automatically extract organic material information from literature. However, there is no research on materials literature science using NER. In addition, materials informatics research often predicts hundreds or thousands of materials, and extracting descriptor features from materials literature is very useful for studying structure-activity relationships.

[0005] At present, most methods for selecting descriptors are to select words or phrases suitable as descriptors from material literature through experts' domain knowledge. At present, most NER methods in the field of materials use unsupervised entity recognition, supervised entity recognition, and entity recognition based on deep learning. Among them, unsupervised entity recognition is mainly based on rules, and the design of rules requires domain-related knowledge bases, dictionaries, and even requires careful design by experts; supervised entity recognition is mostly implemented using conditional random fields (CRF); deep learning-based entity recognition is mostly implemented by combining Bi-LSTM with CRF, where Bi-LSTM is a bidirectional long short-term memory network (LSTM), and LSTM is an improved RNN. The following introduces rule-based entity recognition, CRF, RNN, LSTM, and Bi-LSTM respectively.

[0006] (1) Rule-based

[0007] The more famous rule-based entity recognition systems include LaSIE-II, NetOwl, Facile, SAR, FASTUS and LTG. These systems mainly identify entities based on manually designed semantic and syntactic rules, such as tagging sentences with parts of speech and treating noun phrases that meet certain restrictions as entities. When the dictionary resources are very rich, good performance can usually be achieved. The KnowItAll system is unsupervised and can automatically extract a large number of entities (and relations) from web pages using domain-independent rule templates. The advantage of unsupervised learning entity recognition is that it does not require any labeled data, and a large number of entities can be obtained with the help of dictionaries and manually designed rules. However, because the rules are domain-specific and the dictionary is incomplete, these systems often have high precision and low recall, and it is difficult to apply this system to other fields.

[0008] (2) Based on traditional machine learning

[0009] CRF is a classic sequence labeling model. It extracts the features at each position l and the features between adjacent output labels. Assuming there are N types of features, it is expressed as The conditional probability of the model is:

[0010]

[0011] Among them, Z(x) is the normalization function.

[0012]

[0013] (3) Based on deep learning

[0014] RNN is a neural network that can process sequence data. It takes sequence data as input, recursively in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain-like manner.

[0015] In traditional neural networks, the input layer and hidden layer, as well as the hidden layer and output layer are fully connected. The nodes within each layer are not connected, and can only process inputs one by one. The previous input and the next input are completely unrelated. However, when processing sequence data such as sentences, it is definitely inappropriate to understand each word in the sentence separately. For example, when performing part-of-speech tagging, the part-of-speech of the previous word has a great influence on the part-of-speech prediction of the current word. If the previous word is a verb, the probability that the current word is a noun is much greater than the probability that the current word is a verb. Therefore, in order to better handle similar problems, it is necessary to use the information of other words before the current word, or to use some historical information to help the current prediction, and traditional neural networks cannot provide such historical information.

[0016] RNN adds recursive connections to the hidden layer units, so that the historical information of the network can be propagated through the recursive connections, so that the hidden layer of the network has the function of saving and using historical information. Specifically, the input of the hidden layer contains not only the information of the input layer, but also the output of the hidden layer at the previous moment. In RNN, the hidden layer input at time t is in addition to x t In addition, it also includes the output h of the hidden layer at the previous moment t-1 , so that when processing the tth word in the sentence, the information of other words before the word in the sentence can be used (h t-1 ), which helps to deal with sequence prediction problems.

[0017] h t =H(W[h t-1 ,x t ]+b) Formula (3)

[0018] In the formula, h t is the hidden layer output, H is a nonlinear function (such as tanh function), and b is a bias.

[0019] However, the standard RNN also has two problems: first, if the gradient transfer is too long, it will make it difficult to capture the long-distance dependencies in the sequence; second, when processing long sequences, the gradient will disappear or explode.

[0020] (4) LSTM model

[0021] Compared with RNN, LSTM adds three gate structures in the hidden layer unit for information memory, update and utilization. These three gates are input gate (i), forget gate (f) and output gate (o), and a memory unit (c) is added. Among them, the function of input gate i is to determine which new information can be stored in the memory unit, forget gate f controls how much historical information should be forgotten, and output gate o determines which information can be output. Its calculation method is shown in formulas (4)-(9):

[0022] i t =σ(W i [h t-1 ,x t ]+b i ) Formula (4)

[0023] f t =σ(W f [h t-1 ,x t ]+b f ) Formula (5)

[0024]

[0025]

[0026] o t =σ(W o [h t-1 ,x t ]+b o ) Formula (8)

[0027] h t =o t tan(c t ) Formula (9)

[0028] Among them, the symbols σ and tanh represent different activation functions, and W represents dot product. i , W f , W c , W o is the weight matrix, b i , b f , b c , b o is the bias value. t is the input vector at time t, h t is the hidden layer state and the output vector, which contains all valid information before time t. t 、f t , o t They represent the control of the input gate, forget gate, and output gate at time t.

[0029] Then, both RNN and LSTM can only capture the historical information of the sequence. Due to the complexity of the natural language sentence structure, future information is sometimes needed when performing sequence labeling. Corresponding to the sentence, when processing the current word, the information of the word after the current word (to the right) may be needed.

[0030] (5) Bi-LSTM model

[0031] The Bi-LSTM network is composed of a forward LSTM unit and a backward LSTM unit. The basic idea is to use two LSTMs in the hidden layer to model the sequence from front to back (forward) and from back to front (reverse), and then connect their outputs. The hidden layer of the forward unit is represented as The hidden layer of the backward unit is represented as Through formulas (4)-(9), the output of the unidirectional hidden layer at time t is obtained, as shown in formulas (10)-(11). The hidden layer output of Bi-LSTM is obtained by concatenating the hidden layer outputs of the forward LSTM unit and the backward LSTM unit, as shown in formula (12).

[0032]

[0033]

[0034]

[0035] Therefore, the most widely used method for selecting descriptors is to select words or phrases from material documents manually through expert experience. This method is time-consuming, and the selection results depend on the expert knowledge of researchers. There are certain limitations and subjectivity, that is, different researchers may select different descriptors for a certain material, so the generalization of the descriptors selected by this method will also be limited. At the same time, when dealing with the problem of named entity recognition of materials, the traditional model: 1) lacks the ability to encode words in long sentences in material documents (that is, for long sentences, it is difficult to capture the dependencies between each word, so that each word cannot be represented in a good context, which affects the classification performance of the entity); 2) there is a problem of being unable to represent the polysemy of words (there are many words with the same meaning but different representations in the sentences of material documents, such as chemical formulas, their abbreviations, and their English names in material documents). The traditional Word2Vec method extracts different embeddings for entities with polysemy, so it is difficult to distinguish them, or train a classifier to judge synonyms after entity recognition. Traditional NER methods have weak ability to learn long-distance dependencies and require the combination of external knowledge and a lot of manual participation to extract and process features. Summary of the invention

[0036] The present invention is provided to solve the above-mentioned problems existing in the prior art. The present invention is a descriptor recognition method, device and medium for text data. By obtaining semantic vectors of words and sentences with sufficient semantic information, the embedding of words, the embedding of sentences and the embedding semantic vector of the word position are considered simultaneously when encoding the word vector. Therefore, it can effectively alleviate the ambiguity problem in the material field to a certain extent, thereby obtaining the semantic features of words with sufficient semantic information. Next, for long sentences in the material text, the entire sentence is modeled to effectively capture the long-distance dependency of words to extract the local contextual semantic features of words. Finally, the CRF model is used to learn the dependency relationship between labels, and finally the optimal label sequence is obtained, and the corresponding descriptors are identified, so that the entities can be accurately classified.

[0037] The present invention specifically adopts the following technical solutions:

[0038] According to a first embodiment of the present invention, there is provided a method for identifying a descriptor of text data, the method comprising:

[0039] Use the trained recognition model to identify the descriptors of text data using the following method:

[0040] Based on the text data, determine the input sequence w=(w 1 ,w 2 ,...,w n ) and the label sequence y corresponding to the feature vector y = (y 1 ,y 2 ,...,y n ), where w n is the feature vector of the nth word;

[0041] A set of label sequences with the largest total probability score is calculated using the following formulas (14)-(17):

[0042]

[0043]

[0044]

[0045]

[0046] Among them, score(W,y) is the evaluation score of all input sequences, T is the migration matrix, for y i Migrate to y i+1 The probability score of The i-th word is labeled as y iThe probability score, p(y|S) represents the probability that the sentence S is labeled with the label sequence y, is the true label, formula (16) represents the likelihood function of the label sequence during training, Y W represents the set of all possible annotations, y * Represents a set of label sequences with the largest total probability score;

[0047] Determine the coarse-grained descriptor based on the tag sequence with the largest total probability score;

[0048] Dynamically add coarse-grained descriptors and their corresponding sentence sequences to build a knowledge base;

[0049] Based on the coarse-grained descriptors in the knowledge base, performance-driven high-quality descriptors are screened out through the principle that descriptors co-appear in the same sentence and the importance of each coarse-grained descriptor in the corresponding sentence sequence.

[0050] According to a second embodiment of the present invention, a device for identifying descriptors of text data is provided, the device comprising a processor, wherein the processor is configured to: identify descriptors of text data using a trained recognition model by the following method:

[0051] Based on the enhanced text data, determine the input sequence w=(w 1 ,w 2 ,...,w n ) and the label sequence y corresponding to the feature vector y = (y 1 ,y 2 ,...,y n ), where w n is the feature vector of the nth word;

[0052] A set of label sequences with the largest total probability score is calculated using the following formulas (14)-(17):

[0053]

[0054]

[0055]

[0056]

[0057] Among them, score(W,y) is the evaluation score of all input sequences, T is the migration matrix, for y i Migrate to y i+1 The probability score of The i-th word is labeled as y iThe probability score, p(y|S) represents the probability that the sentence S is labeled with the label sequence y, is the true label, formula (16) represents the likelihood function of the label sequence during training, Y W represents the set of all possible annotations, y * Represents a set of label sequences with the largest total probability score;

[0058] Determine the coarse-grained descriptor based on the tag sequence with the largest total probability score;

[0059] Dynamically add coarse-grained descriptors and their corresponding sentence sequences to build a knowledge base;

[0060] Based on the coarse-grained descriptors in the knowledge base, performance-driven high-quality descriptors are screened out through the principle that descriptors co-appear in the same sentence and the importance of each coarse-grained descriptor in the corresponding sentence sequence.

[0061] According to a third aspect of the present invention, a non-transitory computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor, the method according to any embodiment of the present invention is executed.

[0062] According to the descriptor recognition method, device and medium for text data described in each embodiment of the present invention, not only can descriptors be automatically extracted from materials science literature from both coarse-grained and fine-grained aspects, but domain knowledge can also be embedded in the descriptor recognition method, so that the present invention can screen high-quality descriptors according to user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. The same reference numerals with letter suffixes or different letter suffixes may represent different instances of similar parts. The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the specification and claims, are used to illustrate the embodiments of the invention. When appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.

[0064] Figure 1 A flow chart of a method for identifying a descriptor of text data according to an embodiment of the present invention is shown.

[0065] Figure 2 A flow chart of a method for identifying a descriptor of text data according to an embodiment of the present invention is shown.

[0066] Figure 3 A flow chart of a method for identifying a descriptor of text data according to an embodiment of the present invention is shown.

[0067] Figure 4 A specific flow chart of a descriptor recognition method for text data according to an embodiment of the present invention is shown.

[0068] Figure 5 A schematic diagram of a descriptor identifier according to an embodiment of the present invention is shown.

[0069] Figure 6 A conditional data enhancement flow chart for material domain knowledge fusion according to an embodiment of the present invention is shown.

[0070] Figure 7 A structural diagram of a coarse-grained descriptor identifier according to an embodiment of the present invention is shown.

[0071] Figure 8 A schematic diagram of word vector representation input based on the MatBERT model according to an embodiment of the present invention is shown.

[0072] Fig. 9 A knowledge base structure diagram according to an embodiment of the present invention is shown.

[0073] Fig.10 A schematic diagram of example prediction of a NER model according to an embodiment of the present invention is shown.

[0074] Fig.11 A flow chart of a method for calculating descriptor importance according to an embodiment of the present invention is shown.

[0075] Fig.12 An example of FGDR screening according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings and specific embodiments, but are not intended to limit the present invention. For the various steps described herein, if there is no necessity for a causal relationship between each other, the order in which they are described as examples herein should not be regarded as a limitation, and those skilled in the art should know that they can be adjusted in order, as long as the logic between them is not destroyed, resulting in the inability to implement the entire process.

[0077] The embodiment of the present invention provides a method for identifying a descriptor of text data, such as Figure 1 As shown, the method includes using the trained recognition model to recognize the descriptors of the text data through the following steps:

[0078] Step S100, based on the text data, determine the input sequence w=(w 1 ,w 2,…,w n ) and the label sequence y corresponding to the feature vector y = (y 1 ,y 2 ,…,y n ), where w n is the feature vector of the nth word.

[0079] Step S200, calculate a set of tag sequences with the largest total probability score using formulas (14)-(17):

[0080]

[0081]

[0082]

[0083]

[0084] Among them, score(W,y) is the evaluation score of all input sequences, T is the migration matrix, for y i Migrate to y i+1 The probability score of The i-th word is labeled as y i The probability score, p(y|S) represents the probability that the sentence S is labeled with the label sequence y, is the true label, formula (16) represents the likelihood function of the label sequence during training, Y W represents the set of all possible annotations, y * Represents a set of label sequences with the largest total probability score;

[0085] In step S300, a coarse-grained descriptor is determined based on the tag sequence with the largest total probability score. In this step, a coarse-grained descriptor set can be obtained, and the corresponding sentence sequence can be determined according to the corresponding coarse-grained descriptor. Therefore, the present invention can not only identify the corresponding descriptor, but also be applied to text classification.

[0086] Step S400, dynamically add coarse-grained descriptors and their corresponding sentence sequences to construct a knowledge base. As an example, the constructed knowledge base is as follows: Fig. 9 shown.

[0087] Step S500 , based on the coarse-grained descriptors in the knowledge base, the performance-driven high-quality descriptors are screened out by the principle that the descriptors co-appear in the same sentence and by calculating the importance of each coarse-grained descriptor in the corresponding sentence sequence.

[0088] In some embodiments, Figure 2As shown, the step S500, based on the coarse-grained descriptors in the knowledge base, screens out performance-driven descriptors by the principle that the descriptors appear together in the same sentence and calculating the importance of each coarse-grained descriptor in the corresponding sentence sequence, including:

[0089] Step S501, list the coarse-grained descriptor list in the knowledge base: D = [D 1 ,D 2 ,...D n ], and lists the statements corresponding to the descriptor: S = [S 1 ,S 2 ,...S n ];

[0090] Step S502, select a descriptor, then create a temporary queue, put the descriptor into the temporary queue, take the coarse-grained descriptor and the statement from the corresponding coarse-grained descriptor list and statement list, add the descriptor in which the coarse-grained descriptor and the statement appear together to the temporary queue, and if the temporary queue is not an empty queue, remove the head element in the temporary queue from the temporary queue and assign it to the performance-driven descriptor set to obtain a performance-driven high-quality descriptor set;

[0091] Step S503, calculating the importance of the descriptors in the performance-driven high-quality descriptor set in the corresponding sentence sequence by formula (18):

[0092]

[0093] Among them, I i Indicates the importance of the i-th word, E i is the embedding vector of the i-th word, S [CLS] Embed the corresponding sentence vector;

[0094] Step S504: Screen out performance-driven high-quality descriptors based on a threshold of descriptor importance.

[0095] In some embodiments, based on a threshold of descriptor importance, performance-driven high-quality descriptors are screened out by the following formula (19);

[0096]

[0097] Among them, D i represents a performance-driven high-quality descriptor set, T is the threshold of descriptor importance, true represents the descriptor retained in the performance-driven high-quality descriptor set, and false represents the descriptor deleted in the performance-driven high-quality descriptor set.

[0098] In some embodiments, before using the trained recognition model to recognize the descriptor of the text data by the following method, Figure 3 As shown, the method also includes:

[0099] Step S1001, dividing text data into at least one sentence sequence, and dividing each sentence sequence into separate tokens, and annotating each token based on a preset entity tag, wherein the preset entity tag is used to define a descriptor;

[0100] Step S1002, randomly masking some words in the sentence sequence, and predicting the masked words through the learned contextual semantic relationship to enhance the text data;

[0101] Step S1003, training the recognition model using the enhanced text data. As an example only, the training method may be to train the recognition model using a method of recognizing descriptors. The specific steps have been described above and will not be repeated here.

[0102] In some embodiments, before separating the text data into at least one sentence sequence, the method also includes cleaning the text information to obtain text data; the cleaning of the text information to obtain text data includes: removing invalid data in the text information through regular expression matching, the invalid data including garbled characters and pictures; in the event of garbled characters, converting the characters that cause the garbled characters into special symbol marks.

[0103] The embodiment of the present invention further provides a device for identifying a descriptor of text data, the device comprising a processor, and the processor is configured to:

[0104] Use the trained recognition model to identify the descriptors of text data using the following method:

[0105] Based on the enhanced text data, determine the input sequence w=(w 1 ,w 2 ,...,w n ) and the label sequence y corresponding to the feature vector y = (y 1 ,y 2 ,...,y n ), where w n is the feature vector of the nth word;

[0106] A set of label sequences with the largest total probability score is calculated using the following formulas (14)-(17):

[0107]

[0108]

[0109]

[0110]

[0111] Among them, score(W,y) is the evaluation score of all input sequences, T is the migration matrix, for y i Migrate to y i+1 The probability score of The i-th word is labeled as y i The probability score, p(y|S) represents the probability that the sentence S is labeled with the label sequence y, is the true label, formula (16) represents the likelihood function of the label sequence during training, Y W represents the set of all possible annotations, and y* represents a set of label sequences with the largest total probability score;

[0112] Determine the coarse-grained descriptor based on the tag sequence with the largest total probability score;

[0113] Dynamically add coarse-grained descriptors and their corresponding sentence sequences to build a knowledge base;

[0114] Based on the coarse-grained descriptors in the knowledge base, performance-driven high-quality descriptors are screened out through the principle that descriptors co-appear in the same sentence and the importance of each coarse-grained descriptor in the corresponding sentence sequence.

[0115] It should be noted that the processor may be a processing device including one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), etc. More specifically, the processor may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor running other instruction sets, or a processor running a combination of instruction sets. The processor may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc.

[0116] The processor may be communicatively coupled to the memory and configured to execute computer executable instructions stored thereon to perform the descriptor recognition method for text data according to various embodiments of the present invention.

[0117] In some embodiments, the processor is further configured to: separate the text data into at least one sentence sequence, and separate each sentence sequence into separate tags, and label each tag based on a preset entity tag, wherein the preset entity tag is used to define a descriptor; randomly mask some words in the sentence sequence, and predict the masked words through the learned contextual semantic relationship to enhance the text data; and train the recognition model through the enhanced text data.

[0118] In some embodiments, the processor is further configured to: list a coarse-grained descriptor list in the knowledge base: D = [D 1 ,D 2 ,...D n ], and lists the statements corresponding to the descriptor: s = [s 1 ,s 2 ,...s n ]; select a descriptor, then create a temporary queue, and put the descriptor into the temporary queue, take the coarse-grained descriptor and the sentence from the corresponding coarse-grained descriptor list and sentence list, add the descriptor in which the coarse-grained descriptor and the sentence appear together to the temporary queue, and if the temporary queue is not an empty queue, remove the head element in the temporary queue from the temporary queue and assign it to the performance-driven high-quality descriptor set to obtain a performance-driven high-quality descriptor set; calculate the importance of the descriptors in the performance-driven high-quality descriptor set in the corresponding sentence sequence by the following formula (18):

[0119]

[0120] Among them, I i Indicates the importance of the i-th word, E i is the embedding vector of the i-th word, S [CLS] Embed the corresponding sentence vector;

[0121] Performance-driven high-quality descriptors are screened out based on a threshold of descriptor importance.

[0122] In some embodiments, the processor is further configured to: based on a threshold of descriptor importance, filter out performance-driven high-quality descriptors by the following formula (19);

[0123]

[0124] Among them, D irepresents a performance-driven high-quality descriptor set, T is the threshold of descriptor importance, true represents the descriptor retained in the performance-driven high-quality descriptor set, and false represents the descriptor deleted in the performance-driven high-quality descriptor set.

[0125] In some embodiments, the processor is further configured to: remove invalid data in text information through regular expression matching, the invalid data including garbled characters and pictures; in the event of garbled characters, convert the characters that cause the garbled characters into special symbol marks.

[0126] An embodiment of the present invention provides a non-transitory computer-readable storage medium storing instructions. When the instructions are executed by a processor, a descriptor recognition method for text data according to various embodiments of the present invention is performed.

[0127] The following embodiments of the present invention will be combined with specific application examples to further illustrate the feasibility and advancement of the present invention.

[0128] The embodiment of the present invention intends to process text data using the following steps: 1) preprocessing the text data using a data processor; 2) screening entities using a coarse-grained descriptor identifier; 3) further screening entities using a fine-grained descriptor identifier as needed. The process of this method is as follows Figure 4 The detailed schematic diagram is shown in Figure 5 shown.

[0129] 1) Use a data processor to preprocess text data

[0130] Through the Crystallographic Information File (CIF), 55 NASICON materials science documents suitable for corpus mining are collected. The full text information of these documents (including title, author, abstract, keywords, institution, publisher and publication year) is then stored as a separate document and extracted by PDF parsing (Python toolkit). The resulting NASICONNER dataset contains 65690 data, 2434 sentences, and 6036 words. Afterwards, preprocessing will be performed on these documents.

[0131] ①Text cleaning

[0132] Because the text information of the document converted from PDF contains a lot of invalid data, such as garbled characters, other information besides text, etc., we delete them through regular expression matching. These special symbols may also appear as garbled characters; however, we cannot delete them directly because some may be useful information, such as chemical units; therefore, in the next step, we convert all these symbols into special tokens<sym>In this way, we can get relatively clean documents from PDF documents.

[0133] ② Segment and mark text data

[0134] For this work, we first perform word segmentation on the cleaned documents through ChemDataExtractor, which involves segmenting the raw text data into sentences and then segmenting each sentence into individual tokens. Then, in order to annotate these tokens, 8 entity tags of descriptors are defined, including: Composition, Structure, Property, Processing, Characterization, Application, Feature and Condition, which can summarize most of the information of material descriptors. Table 1 gives the definition and examples of each tag.

[0135] Table 1. Definitions of 8 descriptor entity types in the materials domain

[0136]

[0137]

[0138] Using the labeling scheme described above, 55 materials science papers were manually annotated. In addition, when annotating, an inside-outside-beginning (IBO) format was used, which can account for multi-word entities such as "activation energy". In this approach, tokens can be at beginning (B), inside (I), or outside (O) to represent special labels. For example, a sentence in the NASICON paper "The ionic conductivity decreases with increasing activation energy" would be annotated with (token; IOB-label) pairs in the following way: (The; O), (ionic; B-Property), (conductivity; I-Property), (decreases; O), (with; O), (increasing; O), (activation; B-Property), (energy; I-Property).

[0139] ③Data enhancement

[0140] Training supervised NER models requires a large amount of labeled data, which is time-consuming and laborious. In this method, a conditional data augmentation method (cDA-DK) that integrates material domain knowledge is proposed to address the problem of insufficient NER data. Figure 6 shown.

[0141] Since data analysis methods are often affected by noise, in order to reduce the impact of noise while generating high-quality data as much as possible, we introduce material domain knowledge such as material text and label constraints as input to the pre-trained DistilRoBERTa model. Figure 2 As shown in Figure 2, we fine-tune the DistilRoBERTa model for large-scale augmented data. In fact, the augmented data is generated by the mask language model (MLM) of the DistilRoBERTa model, which can randomly mask some words in the sentence and then predict the masked words through the learned contextual semantic relations.

[0142] For example, given an input sentence "The ionic conductivity decreased with increased activation energy", two words in the sentence are masked and become "The <mask>conductivitydecreased with increasing <mask>energy". Then, the fine-tuned DistilRoBERTa model predicts and fills <mask>Finally, the sentence "The electrode conductivity decreases withincreasing electric energy" is generated, as shown in Table 2. In Table 2, italic bold is used to highlight the word changes.

[0143] Table 2. Initial training and augmentation examples

[0144]

[0145] 2) Filter entities through coarse-grained descriptor identifiers

[0146] For this work, the goal is to train a NER model in such a way that materials science knowledge is encoded; for example, we want to let the computer know that the words "activation energy" and "ionic conductivity" represent descriptors of material properties, while "tetrahedra" and "polyhedra" represent descriptors of material structure. Therefore, we designed CGDR, which builds a NER model (MatBERT-BiLSTM-CRF) to identify different categories of coarse-grained descriptors from materials science literature. There are three main types of information that can be used to let the model recognize which words or phrases correspond to a specific descriptor type: ① MatBERT-based word representation; ② BiLSTM-based sentence context feature extraction; ③ CRF-based descriptor classification (such as Figure 7 shown).

[0147] ①Word representation based on MatBERT

[0148] Based on MatBERT's word representation, we design and use the MatBERT model to obtain vector representations of words and sentences. Figure 7 As shown in the figure, the MatBERT model is derived from the pre-trained bidirectional encoder representation in the Transformer (BERT) model, which needs to be fine-tuned with the material literature text data. Through the analysis of the material discourse, we found that words with the same or similar meanings may express different meanings in different contexts. For example, the English word "Bottleneck" not only means "limited", but also indicates the structural information of the material crystal, which shows that the context is very important. Therefore, it is necessary to consider contextual information when encoding the complex text of the material. However, the word embedding generated by the Word2vec method of Mikolov et al. (2013) is context-independent (static embedding) and does not have complex features (such as grammar, semantics). In short, the disadvantage of the Word2vec method is that it is not conducive to the computer to fully understand the material vocabulary, thereby affecting the accuracy of descriptor extraction. This paper adopts the MatBERT method to encode the material text because this method can fully capture the contextual information of the word (i.e., word embedding, segment embedding, and position embedding), thereby obtaining a vector representation with richer semantic information.

[0149] Specifically, this paper gives a sentence sequence. Mattbert uses a fine-tuning parameter mechanism. The input sequence is set to w = ([CLS], w 1 ,w 2 ,...,w n ,[SEP]), where [CLS] represents the beginning of the sample sentence sequence, and [SEP] represents the interval symbol between sentences. They are all used for sentence-level training tasks. The vector representation of each word consists of three parts: word embedding vector (Word Embedding Vector), sentence embedding vector (Sentence Embedding Vector) and position embedding vector (Position Embedding Vector), which are defined as Here, the word embedding vector is determined by the vocabulary provided by MatBERT. Since the training example is a sentence, the sentence embedding vector is set to 0. The three embedding vectors are added together to obtain the word features as the input of MatBERT, as shown in Figure 5 By training the input word vector, the final word vector representation is shown in formula (13), which is used as the input of BiLSTM.

[0150] x=[x 1 ,x 2 ,...,x n ] Formula (13)

[0151] ② Sentence context feature extraction based on BiLSTM

[0152] The BiLSTM model is used to capture the contextual features of the material text. NER is a sequence labeling problem and a tag-level classification task, that is, each word in the sentence needs to be classified. Therefore, it is necessary to consider the local context of each word in the sentence. For example, in the sentence "The overall_____are near to 10-5S / cm-1at200℃", it is obvious that the missing word is conductivity (a coarse-grained descriptor of an attribute class). Although the position information introduced by MatBERT completes the local context information, the self-attention mechanism of MatBERT weakens the position information when fine-tuning. Therefore, RNN is used to solve the above problems. RNN has the ability to capture temporal information for sequence-to-sequence classification. However, RNN often has the problem of gradient disappearance and gradient explosion in the process of temporal information propagation. We used a variant of RNN-long short-term memory (LSTM). LSTM introduces three gate units, namely input gate, forget gate and output gate, such as Figure 8 As shown in Figure 2, the gate structure can selectively save context information and solve the above problems of RNN. Therefore, LSTM is superior to RNN in capturing long-range dependencies.

[0153] Its parameter settings are shown in Table 3:

[0154] Table 3 Bi-LSTM parameter settings

[0155] Parameter name Parameter Value Word vector dimension 768 LSTM cell dimensions 128 Dropout rate 0.1 Learning rate 0.00003 Optimizer AdamW Batch size 32 Early Morning Patience 3 Max sentence length 75 Tag schema BIO

[0156] ③CRF-based descriptor classification

[0157] CRF can predict the optimal label sequence by learning the dependencies between labels and achieve more accurate entity classification. At the same time, as a classifier for sequence labeling problems, CRF can also capture the strong dependencies between output labels and obtain the optimal label sequence. Since the entity label of each word in a sentence needs to be predicted by a classifier, and there is often a certain migration relationship between adjacent entity labels. Therefore, it is useful to consider the correlation between labels in the neighborhood and decode the best label chain for a given input sentence. Therefore, the classifier layer of NER uses CRF instead of the traditional Softmax layer.

[0158] Specifically, w=(w 1 ,w 2 ,...,w n ) is represented as a general input sequence. i is the input vector of the i-th word, y=(y 1 ,y 2 ,...,y n ) represents the label sequence corresponding to the input vector. The evaluation score of the CRF model calculation formula is shown in formula (14), where T is the migration matrix, for y i Migrate to y i+1 The probability score of The i-th word is labeled as y i p(y|S) represents the probability that the sentence S is labeled as the label sequence y, which is calculated by formula (15), where is the true label. In addition, the likelihood function of the label sequence during training is shown in formula (16), where Y W represents the set of all possible annotations. It should be noted that a valid output sequence can be obtained through the likelihood function. Finally, a set of sequences with the largest total probability score can be calculated through formula (17).

[0159]

[0160]

[0161]

[0162]

[0163] In summary, we can use the NER model of CGDR to identify coarse-grained descriptors in material texts, and then build a knowledge base to store them, such as Fig. 9 In addition, the dynamic addition of descriptors and their corresponding sentences in the knowledge base is also implemented. Fig. 9 In the example, "activation energy", "occupancy", and "safety" are property-type descriptors, and "conduction channels", "bottleneck", and "rhombohedral symmetry" are structure-type descriptors. In addition, each descriptor is followed by the corresponding sentence in which it appears.

[0164] Using the NER model trained with CGDR, we can accurately extract descriptor information from materials science literature. The performance of the NER model is shown in Figure 2. Fig.10 As shown in the figure, we can see that the sentence "For those NASICON materials which show a phase transition, the activation energy differed at low temperature (LT) and high temperature (HT)." is input into the trained NER model and can classify each word in the sentence. The results show that "NASICON materials" is a descriptor of the feature class, "phase transition" and "activation energy" belong to the attribute class, and "low temperature" and "high temperature" belong to the condition class. It can be seen that the model can accurately identify the descriptor information in the text.

[0165] 3) Use fine-grained descriptor identifiers to further filter entities as needed

[0166] There are a large number of different categories of coarse-grained descriptors in the knowledge base for material property prediction or new material discovery. However, if we completely manually screen the relevant descriptors, the workload is no less than selecting from the literature. In addition, the quality of the coarse-grained descriptors in the knowledge base is also an important factor in the screening process. Therefore, FGDR is designed to quickly screen high-quality descriptors that are relevant to the target material properties of the study. Note that FGDR combines performance-driven and importance calculations, which helps researchers build a sample dataset of descriptors. Then, using this dataset, ML models can be used to study structure-activity relationships.

[0167] The specific process of performance driving is as follows. First, input a descriptor D seed , which is the target material property to be studied; then a temporary queue Q is created and D seed Put it into the queue. As long as Q is not empty, the head element of Q will be removed from the queue and assigned to D current In addition, d i and i Also taken from the corresponding lists of D and S. Note that d i and D current Here is the same descriptor. Then, s i Middle and d i Co-occurring descriptors w j Added to Q and D associate Finally, the loop terminates until there are no elements in Q and the performance-driven descriptor set D is obtained. associate .

[0168] High-quality descriptors are screened by calculating importance, such as Fig.11 To calculate the importance of each descriptor in the corresponding sentence, we perform inner product on the word vector and sentence vector of the last layer of MatBERT, and then use the Softmax function to normalize the result of the previous step to obtain the final importance. The normalized calculation formula is shown in formula (18), where I i Indicates the importance of the i-th word, E i is the embedding vector of the i-th word output by MatBERT, Embed the corresponding sentence vector. After that, we will set a threshold for descriptor screening. As shown in formula (19), true represents the retained descriptor, false represents the deleted descriptor, and T is the threshold of descriptor importance. The MatBERT model here is the same as the MatBERT in CGDR, except that the latter does not need to output the vectors of words and corresponding sentences, but provides them to the downstream model for further feature extraction.

[0169]

[0170]

[0171] FGDR can accurately screen high-quality descriptors driven by performance in the corresponding context to a certain extent. Taking "activation energies" as an example to screen high-quality relevant descriptors, its effectiveness is shown in Fig.12 As shown in the figure. As can be seen from the figure, the sentence "The calculated potential barriers are in good agreement with the activation energies obtained from ac measurements of polycrystalline samples." in the knowledge base is obtained by retrieving descriptors that co-occur with "activation energies". The MatBERT-BiLSTM-CRF model is used to identify other descriptors of the sentence ("potential barriers" and "polycrystalline samples"). Through FGDR, the importance of each word in this sentence is calculated. However, we focus on the descriptors (i.e., "potential barriers", "activation energies", and "polycrystalline samples"). Finally, it can be found that the importance of the first two descriptors in this context is greater than the threshold, while the importance of the latter descriptor is less than the threshold. The results show that FGDR can effectively screen out high-quality performance-driven descriptors.

[0172] In view of the shortcomings of the prior art, the present invention aims to mine available high-quality descriptors from materials science literature from both coarse-grained and fine-grained dimensions. The present invention also comprehensively considers the problems existing in the field of materials science in NER, and solves the problem that traditional methods have weak long-distance dependency capabilities and require the combination of external knowledge and a large amount of manual participation to extract and process features. By pre-training the BERT model with materials science literature, it can achieve the best results with a shorter number of training times. In addition, we also use pre-trained BERT to perform data enhancement on entities, solving the problem of insufficient number of named entities in materials science. At the same time, this method introduces domain knowledge to achieve accurate screening of descriptors, thereby obtaining high-quality descriptors that meet user needs.

[0173] Table 4 shows the performance of CGDR for the classification of eight named entities, where F1-Score is the harmonic mean of precision P (precision) and recall R (recall). From the table, we can see that the overall F1-score of our model is 0.87, and its performance is close to that of the latest NER model (2018) (0.92). The model training and evaluation are both performed on manually annotated news articles with only three entity labels. However, due to different datasets, it is not possible to directly compare the performance between models by the numerical values ​​of performance indicators. It is worth noting that our model is trained and evaluated on more entity labels and more complex texts. The CGCR model achieves the highest F1-Score (0.94) in the Composition category, while the F1-Score in the Application category is only 0.58, which may be due to the small amount of training data in this category and the failure of the model to fully capture the dependency between this type of entity and its label. The F1-score of other entity categories is above 0.80, indicating that the model performs well in identifying descriptors of different categories.

[0174] Table 4 Comprehensive performance of NER for 8 entity classifications

[0175]

[0176]

[0177] Compared with the performance of the baseline model (BiLSTM-CNNs-CRF), the F1-score of this method is 0.87, and its performance is improved by 16%. The results are shown in Table 5, which further verifies the effectiveness of the CGDR model and is suitable for the automatic recognition of descriptors in the material field.

[0178] Table 5 Comparison of model results

[0179] Model Precision Recall F1-score BiLSTM-CNNs-CRF 0.74 0.69 0.71 DA+MatBERT-BiLSTM-CRF 0.86 0.87 0.87

[0180] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present invention with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. The elements in the claims will be interpreted broadly based on the language adopted in the claims, and are not limited to the examples described in this specification or during the implementation of this application, and the examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.

[0181] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, those of ordinary skill in the art can use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the present invention. This should not be interpreted as a feature of an invention that is not claimed for protection being necessary for any claim. On the contrary, the subject matter of the present invention may be less than all the features of the embodiments of a specific invention. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently used as a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the full scope of the equivalent forms of the attached claims and these claims.< / mask> < / mask> < / mask> < / sym>

Claims

1. A method for identifying descriptors of text data, characterized in that: The method comprises: Use the trained recognition model to identify the descriptors of text data using the following method: Based on the text data, determine the input sequence w=(w1,w2,...,w n ) and the label sequence y corresponding to the feature vector y=(y1,y2,...,y n ), where w n is the feature vector of the nth word; A set of label sequences with the largest total probability score is calculated using the following formulas (14)-(17): Among them, score(W,y) is the evaluation score of all input sequences, T is the migration matrix, for y i Migrate to y i+1 The probability score of The i-th word is labeled as y i The probability score, p(y|S) represents the probability that the sentence S is labeled with the label sequence y, is the true label, formula (16) represents the likelihood function of the label sequence during training, Y W represents the set of all possible annotations, y * Represents a set of label sequences with the largest total probability score; Determine the coarse-grained descriptor based on the tag sequence with the largest total probability score; Dynamically add coarse-grained descriptors and their corresponding sentence sequences to build a knowledge base; List the coarse-grained descriptors in the knowledge base: D = [D1, D2, ... D n ], and list the statements corresponding to the descriptors: s=[s1, s2, ...s n ]; Select a descriptor, then create a temporary queue, and put the descriptor into the temporary queue, take the coarse-grained descriptor and the statement from the corresponding coarse-grained descriptor list and statement list, add the coarse-grained descriptor appearing in the statement to the temporary queue, and if the temporary queue is not an empty queue, remove the head element in the temporary queue from the temporary queue and assign it to the performance-driven high-quality descriptor set to obtain the performance-driven high-quality descriptor set; The importance of the descriptors in the performance-driven high-quality descriptor set in the corresponding sentence sequence is calculated by the following formula (18): Among them, I i Indicates the importance of the i-th word, E i is the embedding vector of the i-th word, S [CLS] Embed the corresponding sentence vector; Performance-driven descriptors are filtered out based on a threshold of descriptor importance.

2. The method according to claim 1, characterized in that Before using the trained recognition model to recognize the descriptor of the text data by the following method, the method further includes: Separating the text data into at least one sentence sequence, and separating each sentence sequence into separate tokens, and labeling each token based on a preset entity tag, wherein the preset entity tag is used to define a descriptor; Randomly masking some words in the sentence sequence, and predicting the masked words through the learned contextual semantic relationship to enhance the text data; The recognition model is trained using the enhanced text data.

3. The method according to claim 1, characterized in that Based on the threshold of descriptor importance, high-quality descriptors are screened out using the following formula (19); Among them, D i represents a performance-driven high-quality descriptor set, T is the threshold of descriptor importance, true represents the descriptor retained in the performance-driven high-quality descriptor set, and false represents the descriptor deleted in the performance-driven high-quality descriptor set.

4. The method according to claim 2, characterized in that: Before separating the text data into at least one sentence sequence, the method further includes cleaning the text information to obtain the text data; The cleaning of text information to obtain text data comprises: Remove invalid data in text information by regular expression matching, the invalid data includes garbled characters and pictures; In the case where garbled characters occur, the characters causing the garbled characters are converted into special symbol marks.

5. A descriptor recognition device for text data, characterized in that: The apparatus comprises a processor configured to: Use the trained recognition model to identify the descriptors of text data using the following method: Based on the text data, determine the input sequence w=(w1,w2,...,w n ) and the label sequence y corresponding to the feature vector y=(y1,y2,...,y n ), where w n is the feature vector of the nth word; A set of label sequences with the largest total probability score is calculated using the following formulas (14)-(17): Among them, score(W,y) is the evaluation score of all input sequences, T is the migration matrix, for y i Migrate to y i+1 The probability score of The i-th word is labeled as y i The probability score, p(y|S) represents the probability that the sentence S is labeled with the label sequence y, is the true label, formula (16) represents the likelihood function of the label sequence during training, Y W represents the set of all possible annotations, y * Represents a set of label sequences with the largest total probability score; Determine the coarse-grained descriptor based on the tag sequence with the largest total probability score; Dynamically add coarse-grained descriptors and their corresponding sentence sequences to build a knowledge base; List the coarse-grained descriptors in the knowledge base: D = [D1, D2, ... D n ], and lists the statements corresponding to the descriptors: S = [S1, S2, ... S n ]; Select a descriptor, then create a temporary queue, and put the descriptor into the temporary queue, take the coarse-grained descriptor and the statement from the corresponding coarse-grained descriptor list and statement list, add the coarse-grained descriptor appearing in the statement to the temporary queue, and if the temporary queue is not an empty queue, remove the head element in the temporary queue from the temporary queue and assign it to the performance-driven descriptor set to obtain a performance-driven high-quality descriptor set; The importance of the descriptors in the performance-driven high-quality descriptor set in the corresponding sentence sequence is calculated by the following formula (18): Among them, I i Indicates the importance of the i-th word, E i is the embedding vector of the i-th word, S [CLS] Embed the corresponding sentence vector; Performance-driven high-quality descriptors are screened out based on a threshold of descriptor importance.

6. The device according to claim 5, characterized in that The processor is further configured to: Separating the text data into at least one sentence sequence, and separating each sentence sequence into separate tokens, and labeling each token based on a preset entity tag, wherein the preset entity tag is used to define a descriptor; Randomly masking some words in the sentence sequence, and predicting the masked words through the learned contextual semantic relationship to enhance the text data; The recognition model is trained using the enhanced text data.

7. The device according to claim 6, characterized in that The processor is further configured to: Based on the threshold of descriptor importance, performance-driven high-quality descriptors are screened out using the following formula (19): Where T is the threshold of descriptor importance, true indicates the descriptor is retained in the performance-driven high-quality descriptor set, and false indicates the descriptor is deleted in the performance-driven high-quality descriptor set. 8 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, executes the method according to claim 1 .

Images