Multi-domain knowledge extraction method

Abstract

The application discloses a multi-field knowledge extraction method, which comprises the following steps: collecting text information in the field of process knowledge and generating triplets represented by word vectors; acquiring the dependency relationship between each word and the grammatical role of the word in a sentence by adopting dependency syntax analysis; modeling the features of the sentence by using a long short-term memory network as a feature extractor; modeling the constraint relationship between labels by using a CRF model; extracting the semantic relationship between entities by using a support vector machine classifier for the labeled entities and entity relationships; and linking the semantic relationship between the entities to known entities in a knowledge graph by using an entity linking open source tool Dexter2. The application automatically establishes a lexical relationship resource library from a large-scale, cross-field text sample set, captures as many deep lexical implicit relationships and text topics as possible while ensuring the speed of the model, and is used for answering related QA questions.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data, and in particular to a multi-domain knowledge extraction algorithm. BACKGROUND

[0002] For the current popular large-scale DeepQA system, how to obtain a large amount of knowledge so as to answer more open-domain questions is crucial. In the language and text database, many knowledge is expressed in an unstructured form. Effective information such as date, name, and relationship between nouns may be embedded in the text information. It is necessary to correctly parse sentences, identify key entities, embed information and relationship information, and merge information about the same entity through co-reference resolution. A standard knowledge representation method can make knowledge have stronger interpretability and stronger logic. However, it is difficult and expensive to extract structured knowledge from unstructured data. In order to solve this problem, it is assumed that the implicit semantic rules can be mined from a large number of sentences, and the rules can be directly applied to the question and answer system. Therefore, a two-stage method is needed to extract the knowledge and the implicit semantics in the sentence. The shallow knowledge is automatically extracted from a large number of documents, and the additional semantics is inferred from the shallow knowledge. Based on the rapid development of deep learning, it has been successfully applied to knowledge extraction. Therefore, the deep learning technology is combined with the multi-domain knowledge extraction problem to promote the rapid development of the multi-domain knowledge extraction text mining direction. SUMMARY

[0003] The present application provides a knowledge extraction algorithm for digitizing knowledge by aggregating all text information.

[0004] In order to achieve the above purpose, the present application provides a multi-domain knowledge extraction method, which comprises:

[0005] S1: collecting text information in the process knowledge field and converting the text information into triples by word segmentation, stop word removal, and part-of-speech tagging, and then encoding each triple to obtain a triple represented by a word vector e i ;

[0006] S2: adopting dependency syntax analysis to split the components of the sentences in the text information, obtaining the dependency relationship between each word and the grammatical role thereof in the sentence, and the dependency relationship is represented by a data structure of a dependency tree;

[0007] S3: using a long short-term memory network (LSTM) as a feature extractor to model the features of the sentences, and the word vector e i of each word in the sentence is the input quantity of the LSTM, and the hidden state of the forward LSTM and the hidden state of the backward LSTM are output ​ The calculation formula is as follows:

[0008]

[0009]

[0010] In the formula, LSTM f Indicates forward LSTM, LSTM b Indicates backward LSTM, and They represent e respectively i The LSTM hidden states of the preceding and following words;

[0011] S4: Use a CRF model to annotate entities and entity relationships. The specific steps are as follows:

[0012] S401: Each word e i Hidden states of a forward LSTM The hidden state h of the backward LSTM i b Given the input quantity, predict the labeling probability of each word. The formula for calculating the predicted labeling probability is as follows:

[0013]

[0014] In the formula, n represents the number of words in the sentence, and W∈R 2d*T represents the weight matrix, b is the bias matrix, and the softmax function performs normalization.

[0015] S402: Use a CRF model to model the constraint relationships between labels. Given an input sequence x and a label sequence y, calculate p(y|x), where p(y|x) represents the probability that the output label sequence is y given the input sequence x.

[0016]

[0017] In the formula, the given input sequence x = x1, x2, ..., x... n Calculate the label sequence y = y1, y2, ..., y n The probability of f; function f θ (x,y) is the feature function, which represents the probability given x and y; k is the total number of labels;

[0018] Z(x) is the partition function used for normalization, and its calculation formula is as follows:

[0019]

[0020] S5: Decode the output label sequence of the CRF model using the Viterbi dynamic programming algorithm. Based on the dependency tree data structure obtained in the previous step, find the optimal path on this directed acyclic graph and predict the label sequence with the highest probability. The specific steps are as follows:

[0021] S501: Define δ i,j Let y be the last label in the first i positions. j The maximum probability, that is:

[0022]

[0023] S502: Define ψ i,j Let y be the last label in the first i positions. j The previous tag y j-1 The possible values ​​of are:

[0024]

[0025] S503: Calculate δ and ψ using the following recursive formulas, and output the annotated entities and entity relationships;

[0026]

[0027] Where, δ i (t) represents the probability value of the sequence with the highest probability among all observed sequences at time t and state i; ψ i (t) represents the previous state at time t and state i in the sequence with the highest probability;

[0028] S6: For the labeled entities and entity relationships, a support vector machine classifier is used to extract the semantic relationships between entities. The specific steps are as follows:

[0029] S601: Extract features from the sentences containing entities and entity relationships;

[0030] S602: Use the Word2vec word embedding model to obtain the feature vector of the feature, and use it as the input of the support vector machine classifier;

[0031] S603: The aforementioned feature vectors are used as a training set to train the support vector machine classifier. The trained support vector machine classifier is used for prediction to determine whether there is a certain semantic relationship between entities.

[0032] S7: Use the open-source entity linking tool Dexter2 to link the semantic relationships between the entities to known entities in the knowledge graph.

[0033] In a preferred mode, the features include at least one of the following: distance between the two entities, length of the sentence in which the two entities are located, text similarity of the two entities, and co-occurring words in the sentence in which the two entities are located.

[0034] In a preferred mode, the unstructured data is segmented using the jieba package provided with Python to obtain a word list l i of the i-th sentence s i , and the calculation formula is as follows:

[0035] l i = jieba.cut(s i )

[0036] In a preferred mode, the stopword types include at least one of the following: symbols, mood auxiliaries, adverbs, prepositions, conjunctions, and determiners.

[0037] In a preferred mode, the part-of-speech tagging of the jieba library is used to obtain the text triple relationship: each triple is composed of two entities and a relationship, and the triple can be represented as (h i , r i , t i ), where h i is the head entity, r i is the relationship, and t i is the tail entity, and the set of all entities is represented as E, and the set of all relationships is represented as R.

[0038] In a preferred mode, after extracting all entities E and relationships R, the word2vec word embedding model is used to encode each entity and relationship to generate a word vector e i .

[0039] e i = emb(w i )

[0040] In the formula, w i represents the i-th entity or relationship, and emb(w i ) represents the word vector of w i .

[0041] The present application has the beneficial effects that: the present application aims to automatically infer fine-grained lexical relation information from a large amount of data, and we also do not limit ourselves to verbs, binary semantic relations or specific types of hierarchies, because doing so will limit the content of the knowledge base finally derived, and the method mainly has three differences compared with previous work; one is that the stored framework is more complete, most automatic parsers tend to use dependency parsers to define the relationship between professional terms, compared with them, the method adopted by our method can greatly improve the extraction range of sparse information, and the label requirement of training data is not large, and the knowledge extraction model training can be carried out in a semi-supervised manner; two is that the category and relationship of the method are not limited, previous methods use given instances to limit the type of knowledge extracted in order to quickly extract useful information from large-scale corpus, resulting in the loss of part of the edge knowledge, while the fine-grained resolution used in the present method can well solve this problem; three is that it is not necessary to limit the knowledge extraction in a certain specific corpus, previous models usually use a specific domain corpus for learning, so as to better aggregate statistical data, and improve the system performance of the knowledge learning model, while our method can use an open domain corpus to obtain multi-domain knowledge information, and finally use the overall aggregated statistical data to comprehensively learn the knowledge learned for subsequent identification work. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The output structure example diagram of the dependency syntax analysis of the present application. DETAILED DESCRIPTION Specific embodiments:

[0044] The multi-domain knowledge extraction algorithm of the present application comprises the following steps:

[0045] S1: text data preprocessing, in this stage, the task is to collect relevant text information in the process knowledge field and convert them into the format of model input, collect the text information in the process knowledge field, and convert the text information into triplets through word segmentation, stop word removal and part-of-speech tagging, then encode each triplet and obtain the triplet represented by the word vector e i ;

[0046] S2: Analyze the structure of the whole sentence. In this step, dependency parsing is used to split the components of the sentence, obtain the dependency relationship between each word and their grammatical role in the sentence; these dependency relationships are represented by the data structure of dependency tree. Dependency parsing can help us better understand the language structure and semantic relationship in the domain text, and provide more rich context information for knowledge extraction; the present invention uses the open source NLP tool-fastHan, reference paper Zhichao Geng, Hang Yan, Xipeng Qiu and Xuanjing Huang, fastHan: A BERT-based Multi-Task Toolkit for Chinese NLP, ACL, 2021. to complete the dependency parsing, the input is a piece of text, and the output is a dependency tree containing all the words and their dependency relationships, as follows Figure 1 is a simple example showing the output structure of dependency parsing; the dependency tree after dependency parsing will be used in the third step of frame extraction, to provide basic structure and context information for entity and entity relationship labeling;

[0047] S3: Use long short-term memory network LSTM, Long Short-Term Memory, as a feature extractor to model the features of the sentence, the word vector e i is the input quantity of LSTM, the hidden state of forward LSTM and the hidden state of backward LSTM The calculation formula is as follows:

[0048]

[0049]

[0050] In the formula, LSTM f represents forward LSTM, LSTM b represents backward LSTM, and respectively represent the LSTM hidden state of the previous and next word of e i ;

[0051] S4: Use CRF model to label entities and entity relationships, CRF model can consider the bidirectional constraint relationship between entities and entity relationships when labeling, the specific steps are as follows:

[0052] S401: The hidden state of forward LSTM and the hidden state of backward LSTM h i of each word e ib For the input quantity, the label probability of each word is predicted, and the calculation formula of the predicted label probability is:

[0053]

[0054] In the formula, n represents the number of words in the sentence, W ∈ R 2d*T represents a weight matrix, b is a bias matrix, and the softmax function realizes normalization processing; we define a label set as T, and the label of each word e i y i ∈ T. It is assumed that the data set has only two entity categories: Person and Organization, and for each category, it is divided into start words and intermediate words, so there are five categories: B-Person, which is the first word of Person, I-Person, which is the intermediate word of Person, B-Organization, which is the first word of Organization, I-Organization, which is the intermediate word of Organization, and O, which is other entities. A matrix P is defined to represent the label probability of each word.

[0055] S402: Use the CRF model to model the constraint relationship between labels, given the input sequence x and the label sequence y, calculate p(y|x), which represents the probability that the output label sequence is y given the input sequence x, that is:

[0056]

[0057] In the formula, given the input sequence x = x1, x2,..., x n , the probability of the label sequence y = y1, y2,..., y n is calculated; the function f θ (x, y) is a feature function, which represents the probability given x and y; k is the total number of labels;

[0058] Z(x) is the partition function for normalization, and its calculation formula is:

[0059]

[0060] S5: The output label sequence of the CRF model is decoded by the viterbi dynamic programming algorithm, and the optimal path is found on the directed acyclic graph according to the dependency tree data structure obtained in the previous step to predict the label sequence with the maximum probability. The viterbi algorithm uses the idea of dynamic programming to recursively calculate the maximum probability of each label at each position, and finally obtains the label sequence with the maximum probability. The specific steps are as follows:

[0061] S501: define δ i,j the maximum probability that the last label in the first i positions is y j , that is:

[0062]

[0063] S502: define ψ i,j the value of the last label y j when the last label in the first i positions is y j-1 , that is:

[0064]

[0065] S503: Calculate δ and ψ by the following recursive formula, and output the annotated entity and entity relationship;

[0066]

[0067] where δ i (t) represents the probability value of the sequence with the maximum probability in all observed sequences at time t and state i; ψ i (t) represents the previous state of state i at time t in the sequence with the maximum probability; In this process, we use the dependency tree obtained in the previous step; The dependency tree here provides the structural information of the sentence, so that we can better understand the context and semantic roles of each word, and thus help us to annotate the entity and entity relationship more accurately; The output annotated entity and entity relationship is used as the input of the relationship extraction step;

[0068] S6: Relationship extraction, extract the semantic relationship between entities from the annotated entity and entity relationship; In the process knowledge graph, these relationships can be the order of process flow, the dependency relationship between materials or processes, and in this invention, the Support Vector Machine, SVM, algorithm is used, reference paper Tang Y. Deep Learning using Linear Support Vector Machines [J]. Computer Science, 2013. DOI: 10.48550 / arXiv.1306.0239. The sentence between the entity pairs is used as a feature vector to input the classifier for training to predict whether there is a certain relationship between the entity pairs, and the specific steps are as follows:

[0069] S601: Extract features from the sentence where the entity and entity relationship are located;

[0070] S602: obtaining a feature vector of the feature using a Word2vec word embedding model as an input of a support vector machine classifier;

[0071] S603: training the support vector machine classifier using the part of the feature vector as a training set, and using the trained support vector machine classifier for prediction to determine whether there is a certain semantic relationship between entities;

[0072] S7: entity linking is linking the entities and relationships extracted in the above steps to the known entities in the knowledge graph; this process is automatically completed using the open source tool Dexter2 for entity linking, which uses context information, entity attributes and semantic information to make judgments and disambiguate. When there are multiple entities with the same name or similar entities in the knowledge graph, Dexter2 can correctly link the entities in the text to the corresponding entities. Through entity linking, we can associate the entities in the text with the entities in the knowledge graph, further enrich and expand the content of the knowledge graph by adding entity links in the knowledge graph, and improve the application value of the knowledge graph.

[0073] The features include at least one of the following: the distance between the two entities, the length of the sentence in which the two entities are located, the text similarity of the two entities, and the co-occurring words in the sentence in which the two entities are located.

[0074] Use the jieba package of python to perform word segmentation on unstructured data to obtain the word list l of the i-th sentence s i i The calculation formula is:

[0075] l i = jieba.cut(s i )

[0076] Remove stop words. Stop words are common words in natural language text that appear frequently but are usually ignored, such as "a", "an", "the", "and", "is", "of" in English. These words appear frequently in text, but usually contribute little to the semantic information of the text. This invention uses the HIT stop word table, which contains 746 Chinese stop words, including symbol, mood auxiliary word, adverb, preposition, conjunction, and qualifier. After removing stop words from the word list l i

[0077] Use the part-of-speech tagging of the jieba library to obtain the text triple relationship: each triple is composed of two entities and a relationship, and the triple can be represented as (h i , r i , t i ​​), where h i Head entity, r i Relation, t i Tail entity, the set of all entities is denoted as E, and the set of all relations is denoted as R.

[0078] After extracting all the entities E and relations R, a unique code is assigned to each entity and relation. The word embedding model aims to convert words in natural language into dense vectors that can be understood by computers. Each word is represented as a fixed-length vector. In this invention, the word2vec word embedding model is used, as described in the paper Rong X. word2vec Parameter Learning Explained. 2014 [2023-08-17]. DOI: 10.48550 / arXiv.1411.2738. The word2vec word embedding model is used to encode each entity and relation, generating a word vector e i :

[0079] e i = emb(w i )

[0080] where w i represents the i-th entity or relation, and emb(w i ) represents the word vector of w i . After generating the word vector, these vectors can be used to represent the original triple. Each triple can be converted into a vector group consisting of two entity vectors and one relation vector, i.e., from (h i , r i , t i ) to

[0081] Current text repositories are mainly of two types: manually built and automatically created. These repositories have their own characteristics. Manually built repositories are usually constructed by humans and include WordNet, FrameNet, and VerbNet. WordNet is a lexical repository that contains information about words, including definitions, synonyms, and antonyms. However, it has limited knowledge of predicate verbs. FrameNet is a lexical database that contains frame structures for words. It represents a verb through a frame, such as "eat" in the "eliminate" frame, which contains a series of semantic parameters that make up the predicate verb. However, different words may be mapped to different frames, so the same word may represent different meanings in different frames, i.e., polysemy. VerbNet classifies things through the levin classification method, containing syntactic and semantic information about verbs. These methods represent knowledge in a descriptive way.

[0082] Our approach, on the other hand, digitizes knowledge by aggregating all textual information, making it easier to perform statistical text analysis and adding new features such as type constraints. Additionally, our approach defines frames differently. In our approach, frames are not abstract concepts applied to a set of words, but are defined by the words in a sentence and the relationships between them. This definition avoids the problem of limiting the meaning of words within a frame when processing text automatically, and allows for more accurate recognition of different variants when the corpus is large enough. Furthermore, our approach avoids the problem of mapping parser relations to frame elements, resulting in smaller definitions of slots for vocabulary. This design not only reduces the complexity of the model, but also better adapts to real-world application scenarios. In addition to manually built repositories, automatically created repositories are also gaining attention. Automatically created repositories are typically generated by computer algorithms and include KNEX, DART, TextRunner, DIRT, and NELL. These methods vary in their implementation, but all can automatically extract implicit knowledge from text information.

[0083] KNEX is one of the earliest methods for automatically extracting implicit knowledge from text. It uses a set of predefined syntactic patterns to parse the syntax tree and extract the topic of the sentence. The advantage of this method is that it does not require any pre-trained models or manually labeled data, so it can be applied to large-scale text data. However, its accuracy may be limited by the choice of syntactic patterns.

[0084] DART is another method for automatically building repositories. It analyzes text information to parse implicit rules and applies propositions to various tasks. Unlike KNEX, DART uses semantic and lexical features to identify and extract relationships and generates a proposition graph to represent semantic information in text. Since it does not require manually labeled data, it can also be applied to large-scale text data.

[0085] TextRunner is another method for automatically identifying and extracting relationships. It uses conditional random fields to automatically identify and extract relationships. Compared to KNEX and DART, TextRunner has the advantage of being able to handle large-scale Internet data and using a complete dependency parser to identify and extract relationships. It can automatically identify and extract various relationships, such as noun-noun, noun-verb, noun-adjective, etc.

[0086] DIRT is another method for automatically constructing repositories that automatically identifies inference rules that tend to link the same parameter dependency paths. This method performs text pattern recognition by analyzing the similarity of the paths between words. The advantage of this method is that it can discover hidden rules and apply them to various tasks. However, it can be limited by the choice of the relationship paths.

[0087] NELL is a recently developed method for automatically constructing repositories that treats textual information as seed classes and relationships to automatically acquire the relationships between text examples and extract new relationship classes and instance information from millions of web pages. This method can handle large amounts of textual data and automatically construct and expand repositories, so it can be applied to large-scale knowledge acquisition tasks.

[0088] Compared with KNEX and DART, this method requires a syntax parse tree as input, and compared with TextRunner, the definition of relationships in this method is more flexible, while compared with DIRT and NELL, this method has higher accuracy and scalability. In general, this method has the following innovations: (1) By aggregating all textual information, knowledge is digitized and new features such as type constraints are added, which can more accurately represent knowledge. (2) The framework in this method is not an abstract concept applied to a set of words, but is defined by the words in a sentence and the relationships between them, which helps to avoid limiting the meaning of words to a specific framework when automatically processing text and improves the ability to recognize different variants. (3) The definition slot of the vocabulary in this method is relatively small, avoiding the problem of parser relationship mapping to framework elements. (4) Compared with manually created repositories, automatically created repositories have greater scale and wider application range.

[0089] Compared with KNEX and DART, this method requires a syntax parse tree as input, and compared with TextRunner, the definition of relationships in this method is more flexible, while compared with DIRT and NELL, this method has higher accuracy and scalability. In general, this method has the following innovations: (1) By aggregating all textual information, knowledge is digitized and new features such as type constraints are added, which can more accurately represent knowledge. (2) The framework in this method is not an abstract concept applied to a set of words, but is defined by the words in a sentence and the relationships between them, which helps to avoid limiting the meaning of words to a specific framework when automatically processing text and improves the ability to recognize different variants. (3) The definition slot of the vocabulary in this method is relatively small, avoiding the problem of parser relationship mapping to framework elements. (4) Compared with manually created repositories, automatically created repositories have greater scale and wider application range.

[0090] The key point of the present application is how to automatically establish a lexical relationship resource library from a large-scale, cross-domain text sample set, while ensuring the speed of the model and capturing as many deep lexical implicit relationships and text main treatments as possible, so as to be used for answering related QA questions. The present application provides users with all-round knowledge about predicate verbs and related vocabularies, which is very useful for various artificial intelligence application programs.

[0091] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can make equivalent replacement or change within the technical range disclosed by the present application according to the technical solution and the inventive concept of the present application, which should be covered in the protection scope of the present application.

Claims

1. A method for extracting knowledge from multiple domains, characterized in that, It includes: S1: Collect textual information in the field of process knowledge, and transform the textual information into triples through word segmentation, stop word removal, and part-of-speech tagging. Then, encode each triple and obtain word vectors. The triplet is represented; S2: Dependency parsing is used to break down the components of the sentences in the text information, obtain the dependency relationships between each word and its grammatical role in the sentence, and the dependency relationships are represented by a dependency tree data structure; S3: Use Long Short-Term Memory Network LSTM As a feature extractor, it models the features of sentences, specifically the word vectors of each word in the sentence. for LSTM Input quantity, output forward quantity LSTM Hidden state and backward LSTM Hidden state The calculation formula is as follows: In the formula, Indicates forward LSTM , Indicates backward LSTM , and They represent The first and second words LSTM Hidden state; S4: Use a CRF model to annotate entities and entity relationships. The specific steps are as follows: S401: Each word Forward LSTM Hidden state and backward LSTM Hidden state Given the input, predict the labeling probability of each word. The formula for calculating the predicted labeling probability is: In the formula, n Indicates the number of words in a sentence. Represents the weight matrix. b The bias matrix, softmax The function implements normalization processing; S402: Model the constraints between labels using a CRF model, given an input sequence. x and label sequence y ,calculate Represents a given input sequence x Under the given conditions, the output label sequence is y The probability of that, that is: In the formula, given the input sequence Calculate the label sequence The probability; function Let be the characteristic function, which represents the state of a given condition. x and y Probability under given conditions; k The total number of tags; The partition function used for normalization is calculated as follows: S5: Decode the output label sequence of the CRF model using the Viterbi dynamic programming algorithm. Based on the dependency tree data structure obtained in the previous step, find the optimal path on this directed acyclic graph and predict the label sequence with the highest probability. The specific steps are as follows: S501: Definition For the front i The last label in each position is The maximum probability, that is: S502: Definition For the front i The last label in each position is Previous tag The possible values ​​of are: S503: Calculated using the following recursive formula and It outputs the annotated entities and entity relationships; in, Indicates time t Status is i And the probability value of the sequence with the highest probability among all observed sequences; This indicates that in the sequence with the highest probability, time... t Status is i The previous state; S6: For the labeled entities and entity relationships, a support vector machine classifier is used to extract the semantic relationships between entities. The specific steps are as follows: S601: Extract features from the sentences containing entities and entity relationships; S602: Use the Word2vec word embedding model to obtain the feature vector of the feature, and use it as the input of the support vector machine classifier; S603: A portion of the feature vectors is used as a training set to train the support vector machine classifier. The trained support vector machine classifier is used for prediction to determine whether there is a certain semantic relationship between entities. S7: Use the open-source entity linking tool Dexter2 to link the semantic relationships between the entities to known entities in the knowledge graph.

2. The multi-domain knowledge extraction method according to claim 1, characterized in that, The features include at least one of the following: the distance between the two entities, the length of the sentence containing the two entities, the text similarity between the two entities, and the co-occurring words in the sentence containing the two entities.

3. The multi-domain knowledge extraction method according to claim 1, characterized in that, Use Python's built-in jieba package to perform word segmentation on unstructured data and obtain the first word. i a sentence word list The calculation formula is as follows: 。 4. The multi-domain knowledge extraction method according to claim 1, characterized in that, The types of stop words include at least one of the following: symbols, modal particles, adverbs, prepositions, conjunctions, and determiners.

5. The multi-domain knowledge extraction method according to claim 1, characterized in that, Using the jieba library for part-of-speech tagging to obtain text triple relations: Each triple consists of two entities and one relation, and a triple can be represented as follows: , of which Head entity, For the relationship, For the tail entity, the set of all entities is represented as: E The set of all relations is represented as R .

6. The multi-domain knowledge extraction method according to claim 1, characterized in that, Extract all entities E and relationships R Then, the word2vec word embedding model is used to encode each entity and relation and generate word vectors. : In the formula, Indicates the first i An entity or relationship, express Word vectors.

Images