A method and device for constructing a knowledge graph

Abstract

The application discloses a kind of construction method and device of knowledge graph, the method includes: original text is converted and Chinese word segmentation, obtains to be processed open field text data;Then, the first knowledge graph is constructed, the first knowledge graph refers to the knowledge graph that is constructed by the knowledge extracted after knowledge extraction to the to-be-processed open field text data;Finally, the second knowledge graph is constructed, the second knowledge graph refers to the knowledge graph that is obtained by knowledge reasoning to the knowledge included in the first knowledge graph.The knowledge graph constructed by the method, the knowledge in it has higher accuracy, is more higher than traditional knowledge graph knowledge abundance, and has faster new knowledge completion ability.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and artificial intelligence application technology, and specifically relates to a method and apparatus for constructing a knowledge graph. Background Technology

[0002] Knowledge graphs, as the core of language intelligence, have been successfully applied in various fields such as intelligent question-answering robots and intelligent search. The rapid development of language intelligence technology has also actively promoted the research and development of knowledge graph construction technology and knowledge reasoning technology. Currently, both knowledge graphs and knowledge reasoning mechanisms are built based on natural language processing. The construction of knowledge graphs relies on knowledge extraction algorithms, while the goal of knowledge reasoning is to solve the problem of incomplete knowledge links in knowledge graphs.

[0003] Currently, knowledge extraction is the process of discovering hidden information in text from the results of natural language processing. Entity extraction and relation extraction are two important subtasks of knowledge extraction. Through entity extraction and relation extraction tasks, the results of knowledge extraction should be presented in the form of (entity 1, relation type, entity 2) triples. Attribute extraction is another subtask of knowledge extraction. Its core task is to automatically identify and extract language fragments describing specific attributes from structured and semi-structured text. Currently, there is relatively little research on applying attribute extraction technology to the construction of open-domain knowledge bases. The mainstream approach is to transform the attribute recognition task into a sequence labeling task, use neural networks to obtain high-dimensional representations of text, and then use machine learning methods to extract attribute words.

[0004] The purpose of knowledge reasoning is to analyze the text information and the inherent connections between texts based on a given pair of natural language texts, and to deduce the logical relationship between the text pairs, so that new fact triples are generated in the existing knowledge graph, making the knowledge graph more complete.

[0005] Because of the different types of relations involved in knowledge extraction, research on knowledge extraction is divided into open-ended knowledge extraction and knowledge extraction based on predefined relation sets. Currently, knowledge extraction methods based on predefined relation sets include: entity-relation joint extraction methods and pipelined methods. Pipeline methods divide the extraction of entities and relations into two stages: the first stage finds all possible entity pairs in the text, and the second stage attempts to classify the relations between these entity pairs.

[0006] Pipeline methods suffer from error accumulation due to mistakes at each stage. Existing joint knowledge extraction models perform poorly in extracting triples with overlapping entities. Furthermore, existing link prediction models are slow to train and lack comprehensive relation representation. Summary of the Invention

[0007] The application aims to provide a knowledge graph construction method and device to solve the problems of error accumulation, poor entity overlapping triple extraction effect in the existing knowledge graph construction method, and slow training speed and insufficient comprehensive relation expression of the existing link prediction model.

[0008] In a first aspect, the application provides a knowledge graph construction method, which comprises:

[0009] acquiring open field text data to be processed;

[0010] constructing a first knowledge graph, which is a knowledge graph constructed by knowledge extracted from the open field text data to be processed after knowledge extraction.

[0011] constructing a second knowledge graph, which is a knowledge graph obtained by knowledge reasoning on the knowledge contained in the first knowledge graph.

[0012] Further, the method of acquiring open field text data to be processed comprises:

[0013] performing format conversion on the original text to obtain text with unified format;

[0014] performing open field Chinese word segmentation on the text with unified format using a word segmentation method to obtain open field text data to be processed.

[0015] Further, the method of constructing a first knowledge graph comprises:

[0016] training a knowledge extraction model using a pre-trained knowledge extraction model to extract knowledge from the open field text data and construct a first knowledge graph from the extracted knowledge;

[0017] combining a pre-trained relation extraction model, a pre-trained first entity extraction model, a pre-trained second entity extraction model, and a pre-trained attribute extraction model into the pre-trained knowledge extraction model;

[0018] combining a relation extraction model, a first entity extraction model, a second entity extraction model, and an attribute extraction model into the knowledge extraction model;

[0019] combining the first type of knowledge constructed, the second type of knowledge constructed, and the third type of knowledge constructed into the first knowledge graph constructed.

[0020] Further, the method of constructing a first knowledge graph further comprises:

[0021] using the relation extraction model to extract text relation information from the open field text data to be processed after relation extraction.

[0022] In the case that there is only one text relation type in the text relation information, performing entity extraction on the open field text data by using the first entity extraction model to obtain text first-type entity information, and constructing the first-type knowledge by using the text relation information and the text first-type entity information;

[0023] In the case that there are at least two text relation types in the text relation information, performing entity extraction on the open field text data by using the second entity extraction model to obtain the text second-type entity information, and constructing the second-type knowledge by using the text relation information and the text second-type entity information;

[0024] performing attribute extraction on the open field text data by using the attribute extraction model to obtain text attribute information, and constructing the third-type knowledge by using the text first-type entity information or the text second-type entity information and the text attribute information;

[0025] Further, the method for constructing the second knowledge graph comprises:

[0026] training a knowledge graph embedding model by using a pre-trained knowledge graph embedding model to obtain a knowledge graph embedding model, and obtaining new knowledge embedding representation of new knowledge generated by the knowledge extraction model by knowledge embedding;

[0027] generating embedding representation of the first knowledge graph by using the knowledge graph embedding model;

[0028] measuring similarity between the new knowledge embedding representation and all knowledge embedding representations in the embedding representation of the first knowledge graph by using a scoring function, and completing new knowledge by using the most similar existing knowledge to obtain completed knowledge;

[0029] adding the completed knowledge to the first knowledge graph to construct the second knowledge graph.

[0030] Further, the method for constructing the first knowledge graph and the method for constructing the second knowledge graph further comprise:

[0031] obtaining training data, wherein the training data is collected from a public data set SKE;

[0032] generating a knowledge extraction model, wherein the knowledge extraction model is obtained by training the pre-trained knowledge extraction model based on the training data.

[0033] generating a knowledge reasoning model, wherein the knowledge graph embedding model is obtained by training the pre-trained knowledge graph embedding model based on the training data.

[0034] In a second aspect, the present application provides a knowledge graph construction device, which comprises:

[0035] a data acquisition unit configured to acquire open field text data to be processed;

[0036] a knowledge extraction unit configured to extract knowledge from the open field text data to be processed by using a knowledge extraction model to obtain a first knowledge graph;

[0037] a knowledge reasoning unit configured to complete the first knowledge graph by using a knowledge reasoning model to obtain a second knowledge graph.

[0038] a second training unit configured to train a pre-trained knowledge graph embedding model by using training data collected in a public data set SKE to obtain a knowledge graph embedding model.

[0039] In a third aspect, the present application provides a computer storage medium storing a computer program, which can realize the method of the first aspect when executed.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] (1) The error propagation problem in the knowledge extraction process is reduced, and the overall accuracy of knowledge extraction is improved.

[0042] (2) The attribute extraction model and the open field text data are fully utilized, and the first knowledge graph constructed is more knowledge-rich than the traditional knowledge graph which only uses the entity extraction model and the relation extraction model.

[0043] (3) The present application uses a knowledge graph embedding model different from the traditional one, which can learn the features of the first knowledge graph more quickly and comprehensively, thereby improving the knowledge graph completion speed. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the knowledge graph construction method provided by the embodiment of the present application is shown.

[0045] Figure 2 The detailed flowchart of the knowledge graph construction method provided by the embodiment of the present application is shown.

[0046] Figure 3 The pre-trained relation extraction model provided by the embodiment of the present application is shown.

[0047] Figure 4 The pre-trained first entity extraction model provided by the embodiment of the present application is shown.

[0048] Figure 5 A pre-training second entity extraction model diagram provided for an embodiment of the present application.

[0049] Figure 6 A pre-training attribute extraction model diagram provided for an embodiment of the present application.

[0050] Figure 7 A pre-training knowledge extraction model training process diagram provided for an embodiment of the present application.

[0051] Figure 8 A pre-training knowledge graph embedding model training process diagram provided for an embodiment of the present application.

[0052] Figure 9 A pre-training knowledge graph embedding model modeling relationship in a triple diagram provided for an embodiment of the present application.

[0053] Figure 10 A pre-training knowledge graph embedding model inverse modeling relationship in a triple diagram provided for an embodiment of the present application.

[0054] Figure 11 A knowledge graph construction device diagram provided for an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the drawings and technical scheme.

[0056] In order to construct a knowledge graph, it is necessary to extract entities, corresponding relationships and attribute information from a large amount of heterogeneous data. The knowledge triplets in the knowledge graph include relationship triplets and attribute triplets, and the structure of the relationship triplets is represented as (entity, relationship, entity), and the structure of the attribute triplets is represented as (entity, attribute, attribute value). Among them, the relationship triplets represent the relationship between different entities, for example: (Taytum, director, The Imitation Game); the attribute triplets describe the inherent properties of the entity and the specific data or state of the property, for example: (Turing, related movies, The Imitation Game).

[0057] In order to construct a complete and rich knowledge graph, it is also necessary to reason the knowledge in the knowledge graph, that is, to infer the knowledge triplets with missing entities or relationships according to the entities and relationships of the existing knowledge triplets in the knowledge graph.

[0058] The existing pipeline knowledge extraction method extracts entities, relationships and attributes in the knowledge triplets from open field Chinese data in a synchronous manner, and the shortcomings are that the correlation of each extraction process is low, and when any one of the entity, relationship and attribute extraction tasks is wrong, the extracted knowledge triplets will be wrong.

[0059] The current reasoning process in the knowledge graph is divided into two stages. In the first stage, the structural features of all entities and relationships in the existing training set are learned by supervised machine learning to form an embedding vector representation model of entities and relationships, i.e., a knowledge graph embedding model. In the second stage, the triplets with missing entity information or relationship information are input into the trained reasoning model to generate entity embedding vectors and relationship embedding vectors. Since the model has converted all entity vectors and relationship vectors in the training set into a candidate set during training, the entity vectors and relationship vectors generated by the newly input triplets are matched with the candidate entity pairs in the candidate set one by one to calculate the scores, and the triplet with the highest score is completed in the knowledge graph. Among them, the existing first stage method has the problem of slow model training speed.

[0060] Based on this, the process of constructing a knowledge graph by using a pipeline knowledge extraction method and a knowledge reasoning method using machine learning is described below.

[0061] Referring to Figure 1 and Figure 2 , Figure 1 the flowchart of the knowledge graph construction method provided by the embodiment of the present application, Figure 2 the detailed flowchart of the knowledge graph construction method provided by the embodiment of the present application.

[0062] It should be noted that the technical solution of the present application can be applied to construct the knowledge graph of an open field intelligent question and answer robot. By using the method of combining pre-training models and deep neural networks, knowledge in a wide field can be automatically extracted from open field text data, and structured knowledge for constructing the first knowledge graph can be formed.

[0063] Step S1: obtaining open field text data to be processed.

[0064] It should be noted that in order to construct an open field knowledge graph, open field text data to be processed needs to be obtained. It should be understood that the open field text data to be processed is stored in a certain encoding form in the computer system, and the open field text data to be processed may come from different platforms, different countries. Since the character encoding forms used by different countries, regions and network platforms are inconsistent, illegal characters and extra spaces may occur, which may lead to the situation that all characters cannot be normally read and processed.

[0065] It should also be noted that in order to facilitate subsequent calculation, the open field text data to be processed needs to be processed into a vector form. Before generating the vector, the open field text data to be processed needs to be segmented and composed into a string in the form of one character followed by one space, so as to facilitate the pre-training model to convert the string into a word vector, wherein each character in the string corresponds to a number in the word vector.

[0066] Therefore, in the technical solution of the present invention, the step of obtaining the open-domain text data to be processed includes:

[0067] Step S11: Convert the format of the original text to obtain text with a uniform format;

[0068] Step S12: Perform open-domain Chinese word segmentation on the uniformly formatted text using a word segmentation method to obtain the open-domain text data to be processed.

[0069] It should be noted that when using word segmentation methods for open-domain Chinese word segmentation, not just any word segmentation method can achieve a word segmentation form where a character is followed by a space. The technical solution of this invention adopts the WordPiece word segmentation method.

[0070] Step S2: Construct a first knowledge graph. The first knowledge graph refers to the knowledge graph constructed from the extracted knowledge after extracting knowledge from the open domain text data to be processed. Specifically, it includes the following steps.

[0071] Step S21: Train the knowledge extraction model using the pre-trained knowledge extraction model, which specifically includes the following steps;

[0072] Step S211: ... Figure 3 Pre-trained relation extraction model, Figure 4 Pre-trained first entity extraction model Figure 5 Pre-trained second entity extraction model and Figure 6 The pre-trained attribute extraction models are combined to form the pre-trained knowledge extraction model;

[0073] It should be noted that, Figure 3 The pre-trained relation extraction model consists of a BERT (Bidirectional Encoder Representation from Transformers) pre-trained model, an untrained Bidirectional Long Short Term Memory (BiLSTM) network, an untrained attention mechanism, and a Softmax normalization function. Figure 4 The pre-trained first entity extraction model consists of a BERT pre-trained model, an untrained bidirectional long short-term memory network, and a conditional random field (CRF). Figure 5 The pre-trained second entity extraction model consists of a BERT pre-trained model, an untrained bidirectional long short-term memory network, a relation-based attention mechanism, a relation gating unit, and an entity decoder. Figure 6The pre-trained attribute extraction model is composed of a BERT pre-training model, an untrained residual network, an untrained bidirectional long short-term memory network and a conditional random field.

[0074] Step S212: combining the relation extraction model, the first entity extraction model, the second entity extraction model and the attribute extraction model into the knowledge extraction model;

[0075] It should be noted that the relation extraction model is composed of a BERT (Bidirectional Encoder Representation from Transformers) fine-tuning model, a trained bidirectional long short-term memory network (BiLSTM), a trained attention mechanism and a Softmax normalization function; the first entity extraction model is composed of a BERT fine-tuning model, a trained bidirectional long short-term memory network and a conditional random field (CRF); the second entity extraction model is composed of a BERT fine-tuning model, a trained bidirectional long short-term memory network, a relation-based attention mechanism, a relation gating unit and an entity decoder; and the attribute extraction model is composed of a BERT fine-tuning model, a trained residual network, a trained bidirectional long short-term memory network and a conditional random field.

[0076] Step S22: performing knowledge extraction on the open domain text data, specifically including the following steps:

[0077] It should be noted that, since the knowledge extraction model has an upper limit on the number of characters in the input text that can be accepted at one time, only one piece of data is input at a time during the knowledge extraction process.

[0078] Step S221: using the relation extraction model to perform relation extraction on a piece of data in the open domain text data to obtain relation information of the piece of data; if there is only one relation type in the relation information of the piece of data, performing step S222; if there are at least two relation types in the relation information of the piece of data, performing step S223;

[0079] Step S222: using the first entity extraction model to perform entity extraction on a piece of data in the open domain text data to obtain first-type entity information, and using the relation information of the piece of data and the first-type entity information to construct first-type knowledge, and then performing step S224;

[0080] Step S223: performing entity extraction on one piece of data in the open field text data by using the second entity extraction model to obtain the second type of entity information, and constructing the second type of knowledge by using the relationship information and the text second type of entity information;

[0081] Step S224: performing attribute extraction on one piece of data in the open field text data by using the attribute extraction model to obtain text attribute information, and constructing the third type of knowledge by using the text first type of entity information or the text second type of entity information and the text attribute information;

[0082] Step S225: combining the first type of knowledge, the second type of knowledge and the third type of knowledge constructed after the knowledge extraction model performs knowledge extraction on the open field text data to construct the first knowledge graph, and one piece of data in the first knowledge graph is shown in Table 1.

[0083] Table 1: One piece of data in the first knowledge graph

[0084]

[0085] Step S3: constructing a second knowledge graph, the second knowledge graph refers to a knowledge graph obtained by performing knowledge reasoning on the knowledge contained in the first knowledge graph, and the following steps are included.

[0086] Step S31: training a knowledge graph embedding model by using the pre-trained knowledge graph embedding model, inputting new knowledge generated by the knowledge extraction model into the knowledge graph embedding model to obtain new knowledge embedding representation;

[0087] Step S32: generating embedding representation of the first knowledge graph by using the knowledge graph embedding model;

[0088] Step S33: measuring the similarity between the new knowledge embedding representation and all knowledge embedding representations in the embedding representation of the first knowledge graph by using a scoring function, and completing the new knowledge with the highest similarity to obtain completed knowledge;

[0089] Step S34: adding the completed knowledge to the first knowledge graph to construct the second knowledge graph.

[0090] It should be noted that the knowledge extraction model used for constructing the first knowledge graph in step S2 needs to be trained based on a pre-trained knowledge extraction model to obtain a knowledge extraction model; the knowledge graph embedding model used for constructing the second knowledge graph in step S3 needs to be trained based on a pre-trained knowledge graph embedding model to obtain a knowledge graph embedding model. The training process of the pre-trained knowledge extraction model is described in detail in the following.Figure 7 The training process of the pre-training knowledge graph embedding model is described in Figure 8 and specifically includes the following steps:

[0091] Step S4: obtaining training data and using the training data for training to generate a pre-training knowledge extraction model and a pre-training knowledge graph embedding model to generate a knowledge extraction model and a knowledge graph embedding model, specifically including the following steps:

[0092] Step S41: obtaining training data, which is training data collected from a public data set SKE;

[0093] SKE refers to a schema based Chinese information extraction data set (Schema based Knowledge Extraction, where the sentences come from Baidu Encyclopedia and Baidu information flow text. The SKE data set contains 24 entity categories, 40 predefined relationship types, 10 predefined attribute types, 170,000 training data, 20,000 validation data and 430,000 triple data.

[0094] Step S42: generating a knowledge extraction model, which is a knowledge extraction model obtained by training the pre-training knowledge extraction model based on the training data;

[0095] It should be noted that the pre-training knowledge extraction model is composed of a pre-training relationship extraction model, a pre-training first entity extraction model, a pre-training second entity extraction model and a pre-training attribute extraction model.

[0096] The pre-training relationship extraction model uses the BERT pre-training model to obtain the semantic information of the context in the input training data and represents it in the form of a vector. BiLSTM is used to encode the vector generated by BERT to obtain the semantic relationship between multiple inputs of the user, and the attention mechanism is used to merge the information at each time step in BiLSTM to obtain information that has a greater impact on the extraction result. Finally, the Softmax normalization function is used to calculate the prediction value of each word in the information for different label types, and the label type corresponding to the maximum value is taken as the label type of each word. According to the comparison of the predicted label type and the annotated type, the network parameters are adjusted by the gradient descent method (Gradient Descent) according to the comparison result, and the model training is completed after the network parameter adjustment is completed.

[0097] The pre-trained first entity extraction model uses BERT to obtain the semantic information of the context in the input training data and represents it as a vector. BiLSTM is used to encode the vectors generated by BERT to obtain the semantic relationships between multiple user inputs. Finally, CRF is used to add inter-label constraints to the label prediction results for each word to ensure the reasonableness of the output labels. The predicted label types and the labeled types are compared, and gradient descent is used to adjust the network parameters based on the comparison results. Once the network parameters are adjusted, the model training is complete.

[0098] The pre-trained second entity extraction model consists of the following steps;

[0099] Step S421: First, BERT generates a vector representation for each word in the sentence to obtain the contextual features of the sentence;

[0100] For the input text sequence S = {w1, w2, ..., w...}, n}, w i Given the i-th word in the sentence, BERT transforms S into a feature vector X = {x1, x2, ..., xn} with word embeddings, sentence embeddings, and position embeddings. n}, where n is the number of vectors being transformed. Let be the vector representation of the i-th word. For word-based word feature embedding, It is the embedding of the word in the sentence. It refers to the embedding of words in the sentence, d w The dimension representing the word feature embedding. It is of dimension d w The real number field, d s This represents the dimension of the sentence feature embedding. It is of dimension d s The real number field, d p The embedding dimension representing the position of a word in a sentence. It is of dimension d p The real number field.

[0101] Step S422: Then, a Bi-LSTM network is used to capture word relevance of word feature sequence X to obtain the encoded representation of sentence context features;

[0102] Let x be the vector representation of the i-th word. i The forward and backward LSTM hidden states are concatenated and used as context word representations, as shown in Equation (1).

[0103]

[0104] in, dhe denotes the dimension of the Bi-LSTM hidden state, denotes the forward LSTM hidden state of the i-th word vector x i denotes the backward LSTM hidden state of the i-th word vector x i , wherein the forward LSTM hidden state and the backward LSTM hidden state are generated by different LSTMs of the i-th word vector x i . Then the encoded sentence context feature is represented as S c = {h1, h2,..., h n}.

[0105] Step S422: obtaining the relations extracted from the input text sequence S by the relation extraction model, and using a context attention mechanism to assign different weights to the context words of each relation;

[0106] An average vector s g is extracted from the encoded sentence context feature, as shown in equation (2):

[0107] s g = avg {h1, h2,..., h n} (2)

[0108] Then the weight of the context word of each relation is calculated using the context attention mechanism, as shown in equations (3) and (4):

[0109] e ik = v T tanh (W r r k + W g s g + W h h i ) (3)

[0110]

[0111] wherein, is the vector representation of the k-th relation, W g , is a model parameter, d r denotes the dimension of the vector representation of the relation, is a real field with dimension d r , d att denotes the dimension of the context attention mechanism, is a real field with dimension d att , e ik is the relation vector r k ​The attention score for the i-th word. Finally, the sentence representation s k h can be represented by the context words of the i-th word in the input text sequence S. i With the corresponding attention weight α ik The product is obtained by multiplying and then summing, as shown in equation (5).

[0112]

[0113] Step S423: Use a relational gating mechanism to preserve the sentence representation s k Information that is helpful for entity annotation is discarded, while irrelevant information is removed.

[0114] Relation-based attention mechanisms acquire sentence representations with relational information by focusing on words in the input sentence that are related to specific relations. However, it is only necessary to perform subsequent entity extraction when the relation and the sentence are positively related. When the relation and the sentence description are negative or interrogative, the sentence representation with relational information will only interfere with the subsequent decoding process.

[0115] For the k-th relation, the gating unit compares the existing sentence representations s. g and relation-based representations k The role of entity extraction and retention of the most influential relation features u k As shown in equations (6) and (7):

[0116]

[0117] u k =9 k ⊙tan(W3s k +b3) (7)

[0118] Among them, W1, W2, b1, b2, For training parameters, d g This indicates the dimension of the gate unit. It is of dimension d g The real number field, is a vector concatenation operation, and ⊙ is a vector dot product operation. σ represents the Sigmoid activation function, which returns a value between 0 and 1. Finally, the context word representation h of the i-th word of the input text sequence S is used. i Relationship features with preservation u k Concatenate them to obtain the final representation of the i-th word, as shown in equation (8):

[0119]

[0120] in Therefore, the representation of the input text S is

[0121] Step S423: using an entity decoder to sequence mark the representation S of the input text S k to extract entities in the input text S.

[0122] using a BiLSTM network to receive the representation S of the input text S k to obtain the context encoding of the input text S wherein, j∈[1, n]. Then, each word in the sequence is mapped into a label space using a Sofimax normalization function, as shown in equation (9),

[0123]

[0124] wherein, is a training parameter, j∈[1, n], dhd is the dimension of the hidden state in the BiLSTM network, n l is the total number of labels, is the entity prediction label of the jth word under the relationship r k Compare the predicted label type with the annotated type, and adjust the network parameters according to the comparison result using the gradient descent method. The network parameter adjustment is completed, and the model training is completed.

[0125] The pre-trained attribute extraction model is composed of BERT, residual network (Residual Network), BiLSTM and CRF. The model uses the residual network to obtain the local features of the sentence from the BERT character embedding, the BiLSTM is used to obtain the context information of the sentence, and the CRF is used to predict the label of each character in the sentence. The pre-trained attribute extraction model specifically consists of the following steps:

[0126] Step S424: first, BERT performs vector representation on each word in the sentence to obtain the context features of the sentence;

[0127] Step S425: then, the residual network is used to obtain the local features of the sentence from the context features of the sentence;

[0128] ​Deep residual networks typically employ identity mapping based on shortcut connections to train very deep neural networks. In a pre-trained attribute extraction model, a 9-layer residual network (ResNet9) is used to extract sentence-local features from character embeddings. This network consists of convolutional layers and residual blocks, but does not include maxpooling layers or fully connected layers.

[0129] For convolutional layers, when BERT outputs the context features of the sentence [X1, X2, ... X...], n After (n is the number of input characters), calculate using equation (10):

[0130] c i =f(w·X) i +b) (10)

[0131] Where, w∈R h×d (d equals the i-th eigenvector X) i The length of is the filter matrix in the convolution operation, the activation function f is the ReLU function, b∈R is the bias term, and the feature c = [c1, c2, ..., c] is obtained. n ], where c∈R s (s=n).

[0132] For a residual block, given the convolutional filter matrices w1, w2 ∈ R of two residual blocks in a 9-layer residual network. h×1 The output features of the convolutional layer in the first residual block can be calculated using equation (11), and the output features of the convolutional layer in the second residual block can be calculated using equation (4.18):

[0133] C1 i =f(w1·X) i +b1) (11)

[0134] C2 i =f(w2·C1) i +b2) (12)

[0135] Where b1 and b2 are bias terms, the local text features generated by the residual network can be given by equation (13).

[0136] C = C + C² (13)

[0137] Step S426: After obtaining the text local features, the features are input into the BiLSTM to extract the context feature information of the sentence; then, according to the context feature information of the sentence, the CRF is used to predict the label of each word in the context feature; finally, the predicted label type and the labeled type are compared, and the network parameters are adjusted according to the comparison result by using the gradient descent method, and the model training is completed after the network parameter adjustment is completed.

[0138] It should be noted that the difference between the use method and the training method of the knowledge extraction model generated in step S42 is that, when the knowledge extraction model is used, since there is no given label in the input sentence, the predicted label output is not used for model parameter adjustment, but as the final prediction result.

[0139] Step S43: generating a knowledge reasoning model, the knowledge graph embedding model refers to training the pre-trained knowledge graph embedding model based on the training data to obtain the knowledge graph embedding model, which specifically consists of the following steps:

[0140] Step S431: the pre-trained knowledge graph embedding model calculates the scores of all relationship triples in the first knowledge graph through the scoring function.

[0141] For the relationship triple (h, r, t) in the first knowledge graph, where h is the head entity, t is the tail entity, and r is the relationship, the pre-trained knowledge graph embedding model maps the head entity h and the tail entity t to the complex vector space through the Euler identity, so that h, (d is the dimension of the vector space), the relationship r is modeled as a rotation of the head entity complex vector h to the tail entity complex vector t on the complex plane, establishing a relationship as shown in Figure 9 The inverse relationship r -1 is modeled as a rotation of the tail entity complex vector t to the head entity complex vector h on the complex plane, establishing an inverse relationship as shown in Figure 10 That is, for any relationship triple in the knowledge base, the formulas (14) and (15) are established.

[0142]

[0143]

[0144] where, represents element-wise multiplication or Hadamard product, and according to the Euler identity property, |r i | = 1. Then the score of the relationship triple is given by the scoring function formula (16).

[0145]

[0146] where ||·|| applies the L1 norm calculation rule.

[0147] Finally, all the relation triples in the first knowledge graph are input into the pre-trained knowledge graph embedding model to calculate the scores of all the relation triples.

[0148] Step S432: all the relation triples in the first knowledge graph are taken as positive samples, and negative samples are constructed;

[0149] The negative sample construction method is that, first, the head entity, tail entity and relation of each positive example triple are masked to generate negative samples, and the positive-to-negative sample ratio is 1:3; then, the negative sampling distribution shown in formulas (17), (18) and (19) is used, and the scoring function of the negative samples is calculated using formula (16).

[0150]

[0151]

[0152]

[0153] where L(·) is the scoring function of the negative samples, σ(·) represents the Sigmoid function, d r (·) is the scoring function, p(·) is the sampling probability distribution of the negative samples, is an interval hyperparameter, (h′ i , r, t′ i ) and (h′ j , r, t′ j ) represent the i-th negative sample triple and the j-th negative sample triple respectively, γ is an interval hyperparameter, α is used to adjust the negative sampling strategy, μ is a dynamic influence factor, and k is the number of iterations of the model algorithm.

[0154] When α>0, the lower the score of the constructed negative sample relation triple, the closer the negative sample to the positive sample relation triple, and the greater the contribution of such negative sample to the model training, so the weight of the negative sample is higher, and the probability of sampling such negative sample next time is higher.

[0155] After all the positive samples and negative samples are used to adjust the parameters of the pre-trained knowledge graph embedding model, the pre-trained knowledge graph embedding model is trained.

[0156] The above, in combination Figures 1 to 10 The construction method of the knowledge graph in the embodiment of the application is described in detail. The following, in combination Figure 11The embodiment of the application discloses a knowledge graph construction device. It should be noted that the method for constructing a knowledge graph in the embodiment of the application corresponds to the description of the device for constructing a knowledge graph in the embodiment of the application, and therefore, the content not described in detail is described in the method of the above embodiment, and for brevity, will not be repeated here.

[0157] Reference is made to Figure 11 , Figure 11 is a structural block diagram of an embodiment of the device for constructing a knowledge graph in the application. As shown in Figure 11 , the device 700 can include a data acquisition unit 701, a knowledge extraction unit 702, a knowledge reasoning unit 703, a first training unit 704, and a second training unit 705. The device 700 can perform the operation of acquiring open field text data, can perform the operation of knowledge extraction in the method embodiment shown in the above Figures 3 to 6 , and can perform the operation of model training in the method embodiment shown in the above Figures 7 to 8 .

[0158] For example, the data acquisition unit 701 can be used to acquire the open field text data to be processed.

[0159] The knowledge extraction unit 702 can be used to extract the first knowledge graph from the open field text data to be processed.

[0160] The knowledge reasoning unit 703 can be used to complete the first knowledge graph to obtain the second knowledge graph.

[0161] The first training unit 704 can be used to train the pre-trained knowledge extraction model by using the training data collected from the public data set SKE to obtain the knowledge extraction model.

[0162] The second training unit 705 can be used to train the pre-trained knowledge graph embedding model by using the training data collected from the public data set SKE to obtain the knowledge graph embedding model.

[0163] The above embodiments are only used to illustrate the technical solutions of the application, and not to limit the same; although the application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for constructing a knowledge graph, characterized in that, The method comprises: acquiring to-be-processed open field text data; constructing a first knowledge graph, the first knowledge graph being a knowledge graph constructed by knowledge extracted after knowledge extraction on the to-be-processed open field text data; the method for constructing the first knowledge graph comprises: training a knowledge extraction model by using a pre-trained knowledge extraction model to obtain the knowledge extraction model, performing knowledge extraction on the open field text data, and constructing a first knowledge graph by the extracted knowledge; combining a pre-trained relation extraction model, a pre-trained first entity extraction model, a pre-trained second entity extraction model and a pre-trained attribute extraction model into the pre-trained knowledge extraction model; combining a relation extraction model, a first entity extraction model, a second entity extraction model and an attribute extraction model into the knowledge extraction model; combining the constructed first type of knowledge, the constructed second type of knowledge and the constructed third type of knowledge into the constructed first knowledge graph; performing relation extraction on the open field text data by using the relation extraction model to obtain text relation information extracted; in the case that there is only one text relation type in the text relation information, performing entity extraction on the open field text data by using the first entity extraction model to obtain text first type of entity information, and constructing the first type of knowledge by using the text relation information and the text first type of entity information; in the case that there are at least two text relation types in the text relation information, performing entity extraction on the open field text data by using the second entity extraction model to obtain text second type of entity information, and constructing the second type of knowledge by using the text relation information and the text second type of entity information; performing attribute extraction on the open field text data by using the attribute extraction model to obtain text attribute information, and constructing the third type of knowledge by using the text first type of entity information or the text second type of entity information and the text attribute information; constructing a second knowledge graph, the second knowledge graph being a knowledge graph obtained by knowledge reasoning on the knowledge contained in the first knowledge graph.

2. The method of claim 1, wherein, The method for acquiring to-be-processed open field text data comprises: performing format conversion on original text to obtain text with unified format; performing open field Chinese word segmentation on the text with unified format by using a word segmentation method to obtain to-be-processed open field text data.

3. The method of claim 1, wherein, The method for constructing a second knowledge graph comprises: training a knowledge graph embedding model by using a pre-trained knowledge graph embedding model to obtain the knowledge graph embedding model, generating new knowledge embedding representation by knowledge embedding on new knowledge generated by the knowledge extraction model; generating embedding representation of the first knowledge graph by using the knowledge graph embedding model; measuring similarity between the new knowledge embedding representation and all knowledge embedding representations in the embedding representation of the first knowledge graph by using a scoring function, and completing new knowledge by using the most similar existing knowledge to obtain completed knowledge; adding the completed knowledge to the first knowledge graph to construct the second knowledge graph.

4. The method of claim 3, wherein, The method further comprises: The training data is collected from a public data set; The knowledge extraction model is obtained by training the pre-trained knowledge extraction model based on the training data; The knowledge reasoning model is obtained by training the pre-trained knowledge graph embedding model based on the training data. 5.A device for constructing a knowledge graph, characterized in that, The device for implementing the method of claim 1 comprises: A knowledge acquisition unit for acquiring open field text data to be processed; A knowledge extraction unit for extracting knowledge from the open field text data to be processed by using a knowledge extraction model to obtain a first knowledge graph; A knowledge reasoning unit for completing the knowledge set by using a knowledge reasoning model to obtain a second knowledge graph; A first training unit for training a pre-trained knowledge extraction model by using training data collected from a public data set to obtain a knowledge extraction model; A second training unit for training a pre-trained knowledge graph embedding model by using training data collected from a public data set to obtain a knowledge graph embedding model. 6.The knowledge graph construction apparatus of claim 5, wherein, The knowledge extraction model is used to extract knowledge from the open field text data to be processed to obtain a first knowledge graph, and the specific process is as follows: The pre-trained knowledge extraction model is trained to obtain a knowledge extraction model, and the open field text data is extracted by using the knowledge extraction model to obtain a first knowledge graph; The pre-trained relationship extraction model, the pre-trained first entity extraction model, the pre-trained second entity extraction model, and the pre-trained attribute extraction model are combined to obtain the pre-trained knowledge extraction model; The relationship extraction model, the first entity extraction model, the second entity extraction model, and the attribute extraction model are combined to obtain the knowledge extraction model; The first type of knowledge obtained by construction, the second type of knowledge obtained by construction, and the third type of knowledge obtained by construction are combined to obtain the first knowledge graph obtained by construction. 7.The knowledge graph construction apparatus of claim 6, wherein, The knowledge reasoning model is used to complete the knowledge set to obtain a second knowledge graph, and the specific process is as follows: The pre-trained knowledge graph embedding model is trained to obtain a knowledge graph embedding model, and the new knowledge generated by the knowledge extraction model is embedded to obtain a new knowledge embedding representation; The knowledge graph embedding model is used to generate an embedding representation of the first knowledge graph; A scoring function is used to measure the similarity between the new knowledge embedding representation and all knowledge embedding representations in the embedding representation of the first knowledge graph, and the highest similarity is used to complete the new knowledge to obtain completed knowledge; The completed knowledge is added to the first knowledge graph to obtain the second knowledge graph.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-4.

Images