Knowledge graph construction method and device based on CIEM and deep learning

Abstract

The invention relates to the technical field of graph construction, and discloses a knowledge graph construction method based on CIEM and deep learning, and the method comprises the steps: constructing an input sequence, recognizing entity information and attribute information through employing a constructed multi-task transfer learning model, and extracting the relation between entities through employing a regularization-based method; performing learning training on knowledge graph representation and text relationship representation by utilizing a knowledge graph and text joint representation model based on an interactive attention mechanism according to the identified entity information, attribute information and relationships among entities; and completing knowledge graph construction based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism. By applying the method, a full-process-oriented heterogeneous knowledge expression organization and storage retrieval method is constructed for massive heterogeneous information, and a large-scale knowledge graph is constructed through entity alignment, disambiguation and verification on the basis of information which is mapped to cross-domain sharing by a CI EM dictionary, so that the coverage and accuracy of the knowledge graph are improved.

Description

Technical Field

[0001] The present application relates to the technical field of graph construction, and in particular to a method and device for constructing a knowledge graph based on CIEM and deep learning. Background Art

[0002] In order to predict the missing elements of triples, knowledge graph representation learning techniques based on translation have been widely used. Representation learning embeds entities and relations into vector space and has produced many successful translation models.

[0003] In the multi-task transfer learning layer, the language model and sequence annotation layer are intended to share the same character-level input layer, which meets the requirements of the multi-task learning and transfer learning framework. However, the language model and sequence annotation tasks are not very related. In order to alleviate the negative impact of this low correlation problem in multi-task learning, the low correlation problem often affects the effect of multi-task learning.

[0004] The traditional JET (Joint Entity and Triple Extraction) model is a model for fine-grained sentiment analysis, which aims to extract sentiment triplets from text, namely entity, aspect, and sentiment. This model usually involves in-depth analysis of the text to identify attributes related to specific entities and determine the sentiment tendency of these attributes. The JET model has limitations in the completeness of feature extraction, which may result in the model being unable to fully capture all relevant information of the text.

[0005] The knowledge graph constructed by the traditional joint learning model of knowledge graph and text may be incomplete and have the problem of missing information. Low-quality knowledge graphs will limit the effectiveness of the model. The computational cost is high when processing large-scale heterogeneous graphs and text data. Over time, the knowledge graph needs to be continuously updated to maintain the accuracy and timeliness of information, and the performance of the model depends largely on the coverage and accuracy of the knowledge graph. The model lacks generalization ability and overfits to specific fields or tasks.

[0006] In view of this, how to provide a knowledge graph construction technology and build a large-scale knowledge graph has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The embodiments of the present application provide a knowledge graph construction method based on CIEM and deep learning, a knowledge graph construction device based on CIEM and deep learning, a computing device and a computer storage medium, so as to solve the problems of small knowledge graph coverage and low accuracy existing in the prior art.

[0008] In a first aspect, an embodiment of the present application provides a method for constructing a knowledge graph based on CIEM and deep learning, including:

[0009] Constructing an input sequence to identify entity information and attribute information using the constructed multi-task transfer learning model, and extracting the relationship between entities using a regularization-based method, wherein the multi-task transfer learning model includes a character-level bidirectional LSTM model, a word-level bidirectional LSTM model, and a CRF layer, the character-level bidirectional LSTM model is used as a language model, the word-level bidirectional LSTM model is used for sequence labeling feature extraction, and the CRF layer is used for sequence decoding;

[0010] The identified entity information, attribute information, and relationships between entities are used to learn and train the knowledge graph representation and text relationship representation using a knowledge graph and text joint representation model based on an interactive attention mechanism.

[0011] The knowledge graph is constructed based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism.

[0012] In a second aspect, an embodiment of the present application provides a knowledge graph construction device based on CIEM and deep learning, including:

[0013] An extraction module is configured to construct an input sequence to identify entity information and attribute information using the constructed multi-task transfer learning model, and extract the relationship between entities using a regularization-based method, wherein the multi-task transfer learning model includes a character-level bidirectional LSTM model, a word-level bidirectional LSTM model and a CRF layer, the character-level bidirectional LSTM model is used as a language model, the word-level bidirectional LSTM model is used for sequence labeling feature extraction, and the CRF layer is used for sequence decoding;

[0014] A training module is configured to use the knowledge graph and text joint representation model based on the interactive attention mechanism to learn and train the knowledge graph representation and text relationship representation based on the identified entity information, attribute information and the relationship between entities;

[0015] The building module is configured to complete the knowledge graph construction based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism.

[0016] In a third aspect, an embodiment of the present application provides a computing device, including:

[0017] Memory and processor;

[0018] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned knowledge graph construction method based on CIEM and deep learning are implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned method for constructing a knowledge graph based on CIEM and deep learning.

[0020] The embodiment of the present application proposes a method for constructing a knowledge graph based on CIEM and deep learning, including: constructing an input sequence to use the constructed multi-task transfer learning model to identify entity information and attribute information, and extracting the relationship between entities using a regularization-based method, wherein the multi-task transfer learning model includes a character-level bidirectional LSTM model, a word-level bidirectional LSTM model and a CRF layer, the character-level bidirectional LSTM model is used as a language model, the word-level bidirectional LSTM model is used for sequence labeling feature extraction, and the CRF layer is used for sequence decoding; the identified entity information, attribute information and the relationship between entities are used to learn and train the knowledge graph representation and text relationship representation using a knowledge graph and text joint representation model based on an interactive attention mechanism; and the knowledge graph construction is completed based on the trained knowledge graph and text joint representation model based on an interactive attention mechanism.

[0021] The method provided in the embodiment of the present application is intended to build a full-process heterogeneous knowledge expression, organization, storage and retrieval method of "knowledge extraction-knowledge processing-knowledge fusion-knowledge evaluation" for "cross-system, cross-region, cross-department, cross-level, and cross-business" massive heterogeneous information in all fields and categories, including venues, professional talents, and cultural teams. It also maps the CIEM dictionary to information that can be shared across domains, and builds a large-scale knowledge graph through entity alignment, disambiguation and verification, thereby achieving the accuracy and timeliness of knowledge graph update information and improving the coverage and accuracy of the knowledge graph.

[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0024] Figure 1 A flowchart of a method for constructing a knowledge graph based on CIEM and deep learning provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of an information extraction structure based on a semi-supervised mode in a knowledge graph construction method based on CIEM and deep learning provided in an embodiment of the application;

[0026] Figure 3 A schematic diagram showing the structural comparison between a traditional extraction model and a balanced model in a knowledge graph construction method based on CIEM and deep learning provided in an embodiment of the application;

[0027] Figure 4 A schematic diagram of an entity linking structure based on a tensor neural network in a knowledge graph construction method based on CIEM and deep learning provided in an embodiment of the application;

[0028] Figure 5 A schematic diagram of a knowledge graph and text joint representation learning model based on an interactive attention mechanism in a knowledge graph construction method based on CIEM and deep learning provided in an embodiment of the application;

[0029] Figure 6 A schematic diagram of the structure of a knowledge graph construction device based on CIEM and deep learning provided in an embodiment of the present application;

[0030] Figure 7 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] Representation learning embeds entities and relations into vector space, resulting in many successful translation models, such as the TransE model, TransH model, and TransR model.

[0033] The TransE model is a translation distance model for knowledge graph embedding. It maps entities and relations in the knowledge graph into a low-dimensional vector space, and represents and infers the relations between entities through vector operations. The training process of the TransE model is usually optimized using a stochastic gradient descent algorithm, which obtains vector representations of entities and relations by minimizing the loss function on the training data. The TransE model is simple to understand, highly scalable and computationally efficient, and can solve the problem of data sparsity. The TransE model also has some limitations. It assumes that the relationship between entities and relations is linear, and may not be able to fully capture the complex structure in the graph; it is difficult to handle complex relationships: TransE may have difficulties in handling one-to-many, many-to-one, and many-to-many relationships.

[0034] The TransH model is an improvement on the TransE model. It introduces the concept of hyperplanes and maps entities to the hyperplane where relationships are located, rather than translating directly in the entity space like TransE. The TransH model also improves the problem of TransE that may introduce erroneous labels during negative sampling by setting a certain probability to replace the head entity or the tail entity, thereby reducing the introduction of erroneous labels. The TransH model has significant performance improvements over the TransE model in tasks such as link prediction, triple classification, and relationship extraction, but it still assumes that entities and relationships are in the same semantic space, which to some extent limits the representation capabilities of TransH.

[0035] The core idea of ​​the TransR model is to define an independent vector space for each type of relationship, and also use the stochastic gradient descent algorithm to optimize the vector representation of entities and relationships by minimizing the loss function. The loss function usually adopts a negative sampling method and uses the L1 or L2 norm to measure the distance between correct triples and incorrect triples. The TransR model has shown good performance in tasks such as knowledge graph completion, entity linking, and relationship extraction, and has significant performance improvements over the TransE and TransH models when dealing with relationships with complex semantic characteristics. However, a potential limitation of the TransR model is that it needs to maintain a mapping matrix for each relationship, which may cause the number of model parameters to increase rapidly, thereby increasing the cost of computation and storage.

[0036] Deep neural network models and representation learning strategies have been widely used in previous studies for text relation extraction, where the relations between entities in the text are projected into a low-dimensional semantic space. Compared with traditional algorithms, deep learning-based algorithms can accurately model the semantic relations between entities from context fragments without the need for additional, explicit syntactic features.

[0037] The first embodiment of the present application provides a method for constructing a knowledge graph based on CIEM and deep learning, such as Figure 1 As shown, the specific steps include:

[0038] Step S101: construct an input sequence to use the constructed multi-task transfer learning model to identify entity information and attribute information, and use a regularization-based method to extract the relationship between entities, wherein the multi-task transfer learning model includes a character-level bidirectional LSTM model, a word-level bidirectional LSTM model and a CRF layer, the character-level bidirectional LSTM model is used as a language model, the word-level bidirectional LSTM model is used for sequence labeling feature extraction, and the CRF layer is used for sequence decoding; it should be noted that step S101 is used to realize knowledge graph entity extraction based on a semi-supervised mode and textual relationship extraction combining templates and features.

[0039] In an embodiment of the present application, the input sequence is constructed to identify entity information and attribute information by using the constructed multi-task transfer learning model, including: constructing an input sequence, using a character-level bidirectional LSTM model as a language model for the observation values ​​corresponding to each sequence member in the input sequence, and combining the word-level bidirectional LSTM model to process the obtained sequence annotation feature extraction layer to generate a multi-task transfer learning layer, wherein the character-level bidirectional LSTM model and the word-level bidirectional LSTM model share the same character-level input layer; using a CRF layer to sequence decode the multi-task transfer learning layer, and identifying entities and attribute information based on the label sequence, wherein the label sequence carries the entity starting state, entity internal state, and entity external state.

[0040] Specifically, the observation values ​​corresponding to each sequence member in the input sequence adopt a character-level bidirectional LSTM model as a language model, and are combined with a word-level bidirectional LSTM model to obtain a sequence annotation feature extraction layer to generate a multi-task transfer learning layer, and also includes: in the word vector layer, by expanding the word vector, splicing the existing domain knowledge on the word vector in the form of a vector, fusing the information of the word vector and the existing domain knowledge information, and generating a bidirectional LSTM vector; using the bidirectional LSTM vector as the updated domain knowledge information to generate a multi-task transfer learning layer.

[0041] First, the knowledge graph entity extraction based on the semi-supervised model is introduced.

[0042] Deep learning technology is used to combine sequence labeling and recurrent neural networks to explore the characteristics of multi-channel corpora and multi-source heterogeneous features obtained through cross-domain sharing and exchange, conduct semantic-based entity mining, and finally generate entity mining tools that can be used for multi-channel corpora.

[0043] The proposed method is based on the semi-supervised information extraction structure as follows Figure 2 For an input sentence s, the corresponding label sequence is y_s=(y_1,y_2,…,y_n); the word level input form is x_s=(x_1,x_2,…,x_n); the character level input form is c_s=(c_(0,_),c_1,1,c_1,2,c_(1,b),…,c_2,1,…,c_(n,_)), where c_(i,_) is the space after the i-th word.

[0044] The sentence after each word segmentation is taken as a word sequence. Each word (i.e., symbol string) in the sequence can be regarded as an observation value of the sequence member. For each type of entity, three labels are defined: entity start B, entity inside I, and entity outside O, represented by B-TYPE, I-TYPE, and O respectively. These labels constitute the state value of the sequence members. Given the observation value, the character-level bidirectional LSTM model is used to implement the language model, and the multi-task transfer learning layer is formed with the sequence annotation feature extraction layer implemented by the word-level bidirectional LSTM model. Finally, the CRF layer is used for sequence decoding. Finally, the entity and attribute information are identified based on the label sequence.

[0045] In the multi-task transfer learning layer, the language model and the sequence annotation layer are intended to share the same character-level input layer, which meets the requirements of the multi-task learning and transfer learning framework. However, the two tasks of language model and sequence annotation are not very related. In order to alleviate the negative impact of this low correlation problem in multi-task learning, the embodiment of the present application adjusts the output of the language model, hoping to map the output of the character-level language model to the semantic space at the word level. Therefore, the character-level language model is used to predict at the end of a word, and the probability distribution of the next word is predicted, rather than the probability distribution of the next character. In this way, not only can the output of the character-level language model be mapped to the semantic space at the word level, but the language model can also be more focused on learning the word formation knowledge of words, rather than just remembering the spelling of words.

[0046] In the word vector layer, it is planned to expand the word vector and splice the existing domain knowledge on the word vector in the form of vectors, integrating the information of the word vector and the existing domain knowledge information. It is planned to save and update the bidirectional LSTM vector generated by the splicing of each word as the updated domain knowledge information during the model iteration process, so as to achieve the purpose of reverse expansion of domain knowledge.

[0047] Secondly, the textual relationship extraction combining templates and features is introduced.

[0048] In an embodiment of the present application, the method of extracting the relationship between entities using a regularization-based method includes: determining candidate words carried in an input sequence, and marking a pattern set associated with the candidate words, wherein the candidate words include verbs, nouns, and adjectives; in response to the candidate word successfully matching the pattern set, the pattern set will determine that the entity corresponding to the candidate word is a target entity, wherein the target entity is a potential relationship entity; determining entities other than the target entity through an entity classifier, and in response to the existence of entities other than the target entity, using a role classifier to assign roles to different entities, and applying a reporting relationship classifier to extract the relationship between entities.

[0049] Research on relation extraction aims to combine the advantages of templates and features and fully consider the association between entities in a sentence. Usually, the key problem in relation extraction is that templates cannot cover all situations, features are too flexible and inaccurate, and the relationship between entities has not received enough attention.

[0050] Therefore, the embodiment of the present application balances the influence of templates and feature models in the model, and makes full use of the relationship between candidate entities based on a regularization method: nouns, verbs, and adjectives in each sentence can be used as candidate words for the relationship; when a candidate word is found, the pattern set associated with the candidate word will be re-labeled. If this pattern matching process is successful, the best pattern will mention some entities in the sentence as entities of potential relationships. Subsequently, an entity classifier is used to determine the remaining entities that need to be identified. If it is an entity, a role classifier is used to assign a role to it. Finally, a reporting relationship classifier is applied to decide whether this relationship should be recognized.

[0051] See also Figure 3 , which is different from the traditional JET model, such as Figure 3 As shown in (a), the technical features of the balance model proposed in the embodiment of the present application are: (1) the influence of the balance mode and other characteristics and (2) the use of a regularization-based method to fully utilize the relationship between candidate entities. Figure 3 The thick border in (b) represents the improved method. Since JET only uses patterns when predicting relationship types, the embodiment of the present application uses an SVM classifier to determine the relationship type of each candidate word. The classifier uses entity word embedding, sentence-level embedding and pattern features together to balance the effect.

[0052] Secondly, entity linking and disambiguation based on tensor neural networks are introduced.

[0053] After step S101, the embodiment of the present application further provides a method for entity linking and disambiguation based on a tensor neural network, which specifically includes: modeling the referential context through a convolutional neural network to obtain a vectorized representation of the context; capturing the common information in the referential features and candidate entity features through a tensor neural network, wherein the referential features carry a vectorized representation of the referential vector and the context; the candidate entity features carry entity terms and entity types; and scoring the candidate entity by calculating the semantic similarity between the candidate entity features and the referential context modeling.

[0054] In order to improve the representation and reasoning ability of the model, the embodiment of the present application adopts a combination of convolutional neural networks and tensor neural networks to achieve end-to-end entity linking. First, the referential context is modeled through a convolutional neural network, and the referential context, referent, candidate entity, candidate entity category, etc. are mapped to the same semantic space. Then, the tensor neural network is used to capture the common information in the referential features and candidate entity features respectively, and finally the candidate entity is scored by calculating the semantic similarity between the candidate entity and the referential modeling part.

[0055] The entity linking structure based on tensor neural network is as follows Figure 4 As shown. In the referential context and referential modeling part, based on the sentence representation model, a convolutional neural network is used to capture the topic of the query sentence, and the vectorized representation Vc of the context is obtained. Then, the common information of the referential vectors Vm and Vc is extracted through the tensor matrix. In the candidate entity modeling part, the tensor matrix is ​​also used to extract the common information in the entity words Vew and the entity category Vec. Finally, with the goal of minimizing the semantic distance between the query sentence and the linked entity, the model is solved using stochastic gradient descent as the optimization algorithm. Its objective function is as follows:

[0056]

[0057] Among them, e represents the best linked entity and e' represents the incorrectly linked entity.

[0058] Finally, the knowledge graph and text joint representation learning model based on the interactive attention mechanism is introduced.

[0059] Step S102: The identified entity information, attribute information and the relationship between entities are used to learn and train the knowledge graph representation and text relationship representation using a knowledge graph and text joint representation model based on an interactive attention mechanism; it should be noted that step S102 is used to implement the knowledge graph and text joint representation learning based on an interactive attention mechanism.

[0060] Step S103: Complete the knowledge graph construction based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism.

[0061] In an embodiment of the present application, the knowledge graph and text joint representation model based on the trained interactive attention mechanism completes the knowledge graph construction, including: learning the embedded representation of the entity vector representation and the relationship vector representation from the knowledge graph; learning the embedded representation of the relationship vector representation from the plain text of the corpus; integrating the embedded representation of the entity vector representation, the relationship vector representation and the embedded representation of the relationship vector representation to complete the knowledge graph construction.

[0062] Specifically, the method of learning the embedded representation of entity vector representation and the relationship vector representation from the knowledge graph includes: learning the embedded representation of entity vector representation and the relationship vector representation from the knowledge graph by minimizing the margin-based pairwise sorting loss.

[0063] Specifically, the method of learning the embedded representation and relationship vector representation of entity vector representation from the knowledge graph includes: using the link between the word vector and the corresponding position vector in the knowledge graph as the input of the convolutional neural network to learn the embedded representation and relationship vector representation of entity vector representation.

[0064] Specifically, the embedded representation of the entity vector representation, the relationship vector representation and the embedded representation of the relationship vector representation are integrated to complete the knowledge graph construction, including: a first attention mechanism based on softmax, learning the embedded representation of the entity vector representation and the relationship vector representation from the knowledge graph, wherein the first attention mechanism is the attention direction from the corpus to the knowledge graph; the relationship vector representation includes an implicit relationship vector and an explicit relationship vector; a second attention mechanism based on softmax, learning the embedded representation of the relationship vector representation from the textual relationship representation, wherein the second attention mechanism is the attention direction from the knowledge graph to the corpus; integrating the embedded representation of the entity vector representation, the relationship vector representation and the embedded representation of the relationship vector representation to complete the knowledge graph construction.

[0065] A knowledge graph is a graph-structured knowledge base that represents factual knowledge in the form of relationships between entities. From the perspective of the embodiments of the present application, a knowledge graph is a heterogeneous information network containing entity nodes and relationship edges, corresponding to an RDF-style triple (h, r, t), where h represents the head entity. The embodiments of the present application propose a text-enhanced knowledge graph representation model with collaborative attention. The embodiments of the present application model each entity and each relationship contained in the knowledge graph from two aspects: the structural signal of the knowledge graph itself and the text signal of the plain text. Unlike the traditional joint learning model of knowledge graphs and texts, this model introduces an interactive attention mechanism to fuse the above-mentioned heterogeneous signals.

[0066] (1) Overall model architecture

[0067] The joint representation of entities, relations, and words is recorded as model parameters θ = {θ_(E_G),θ_(E_C),θ_(R_G),θ_(R_C),θ_V}.

[0068] Among them, θ_(E_G) represents the parameters of learning entity vector representation from knowledge graph G (i.e., the structural representation of entity); θ_(E_C) represents the parameters of learning entity vector representation from plain text C (i.e., the textual representation of entity); θ_(R_G) represents the parameters of learning relation vector representation from knowledge graph G (i.e., the structural representation of relation); θ_(R_D) represents the parameters of learning relation vector representation from plain text in text corpus C (i.e., the textual representation of relation). In addition, θ_V represents the parameters of vector representation of words in vocabulary V. Find the best parameters:

[0069]

[0070] Among them, l Θ (G, C) represents the loss function defined on the knowledge graph G and the text corpus C given the parameter Θ, which can be formed as follows:

[0071]

[0072] Among them, α and λ represent harmonic factors, α is also used to balance the learning rate of the knowledge graph G and the plain text in the corpus C, ||Θ|| l represents the secondary norm. Used to learn vector representations of entities and relations from a given knowledge graph G, The aim is to learn vector representations of relations from plain text in corpus C.

[0073] From the above formula, we can see that under the definition of this loss function, the knowledge graph representation learning process and the text representation learning process are tightly coupled. In addition, the Skip-Gram model and other methods use negative sampling to construct word vectors w∈R k .

[0074] See also Figure 5 , Figure 5 The architecture of the joint learning model of text and knowledge graph based on interactive attention mechanism is outlined, which combines knowledge graph representation learning and text relationship representation learning. Finally, the stochastic gradient descent strategy is applied to the optimization function.

[0075] Knowledge graph representation learning module: This module learns the embedded representation of entity vector representation from the knowledge graph G and learns the relationship vector representation from the knowledge graph G, as shown in the green part of the figure above.

[0076] Textual Relation Representation Learning Module: This module learns the embedding representation of the relation vector representation from the plain text of corpus C, as shown in the blue part of the figure above.

[0077] Interactive attention mechanism module: During the training process, the interactive attention mechanism proposed in the embodiment of the present application integrates the above modules, as shown in the orange part of the above figure (in particular, the green / blue cross-dashed lines in the above figure represent the mutual mechanism between knowledge graph representation learning and text relationship representation learning. Each component will be described in detail below.

[0078] (2) Knowledge Graph Representation Learning

[0079] The knowledge graph and text joint representation model based on the interactive attention mechanism proposed in the embodiment of the present application has strong versatility. Any knowledge graph representation learning model based on representation learning technology can be transplanted into it, including the knowledge graph completion model integrating lexical semantics and the knowledge graph completion model integrating common sense constraints.

[0080] Taking the TransE model as an example, the precise vectors of entities and relations are generated according to the principle of h+r≈t, which means that t is "translated" from h by r. Here, the tail entity t is used as an example to describe the modeling process. It should be noted that any other knowledge graph embedding method can be used here, because these methods usually learn continuous, low-dimensional vector representations of entities and relations by minimizing the margin-based pairwise ranking loss.

[0081] Given each pair of head entity h and tail entity t defined in the knowledge graph G, the present embodiment assumes that there is an implicit relationship vector rh,t, which represents the "translation" from the head entity vector hG (relative to the head entity h) to the tail entity vector tG (relative to the tail entity t), as shown below:

[0082] r h，t =t G -h G

[0083] In addition, the embodiment of the present application can G It is represented as an explicit relation vector for each triple (h, r, t) ∈ T defined in the knowledge graph G, indicating h G to G The following formula shows that for each (h, r, t)∈T, the embodiment of the present application hopes that t G -h G ≈r G , so the score function for each triple (h, r, t) can be defined as follows:

[0084] φ(h,t)=||r h，t -r G || l =||t G -h G -r G ||l

[0085] Usually, the l2 norm is used in the above formula. Based on this score function, the loss function on all triples is It can be defined as follows:

[0086]

[0087] in, μ >0 indicates edge parameter. If x is positive, then [x] + =x, if x is negative, then [x] + =0. T′ is a negative triplet related to T defined above, that is, the embodiment of the present application needs to sample the negative triplet (h′, r, t′) to calculate the loss function, given the positive triplet (h, r, t)∈T. The embodiment of the present application replaces the head entity h or the tail entity t with a random entity uniformly sampled from the knowledge graph G to construct a set of negative triples. Therefore, h′∈E and t'∈E represent the negative head entity and tail entity obtained by random sampling, respectively.

[0088] (3) Text Relation Representation Learning

[0089] The main purpose of text relation extraction is to determine the type of relationship between two entities that appear simultaneously in a text. For each occurrence of the target entity pair h and t in a given sentence d, the embodiments of the present application should specify a relationship type r∈R for h and t, representing the implicit semantics of the textual relationship between the two entities. Recently, deep neural network models and representation learning strategies have been widely used in previous work to capture textual relations, which are projected into low-dimensional semantic spaces in these models. Compared with traditional algorithms, algorithms based on deep learning can accurately model semantic relationships between entities from context fragments without the need for additional, explicit syntactic features. The embodiments of the present application use convolutional neural networks (CNNs) for textual relationship representation learning.

[0090] Therefore, given a sentence d = {w 1 ,…,w |d|}, the embodiment of the present application assumes that this sentence contains semantic signals about the textual relationship rC. The embodiment of the present application uses CNN to reveal the implicit semantics of the textual relationship between entity h and entity t. There is a defined relationship r between h and t in the knowledge graph, and the corresponding relationship vector is represented as rG. The connection between the word vector (wi) and its corresponding position vector (pi) is used as the input of CNN to obtain the final embedding vector rC of the textual relationship with the pooling layer and convolution layer of CNN. The score function of a given sentence d can be defined as:

[0091] ψ r)d)=||r C -r G || l

[0092] On this basis, the loss function on all sentences in corpus C is It can be expressed as:

[0093]

[0094] (4) Joint learning based on interactive attention mechanism

[0095] The mutual attention mechanism for joint learning of text and knowledge graph proposed in the embodiment of the present application can fully integrate signals from multiple directions, namely, signals from knowledge graph representation learning to text relationship representation learning, and vice versa, to obtain the joint learning results of entities, and model the semantics embedded in a given knowledge graph.

[0096] 1) Signal modeling from knowledge graph representation learning to textual relationship representation learning

[0097] As mentioned above, given a relation r defined in the knowledge graph, r has two relation vectors:

[0098] (1) Implicit relationship vector: This embodiment of the application assumes that there are m pairs of entities that conform to the relationship r, {(h 1 ,t 1 ),…,(h m ,t m )}, the corresponding implicit relationship vector is Represents the entity vector from the head (corresponding to the head entity hi) to the tail entity vector (corresponding to the tail entity ti) conversion.

[0099] (2) Explicit relation vector: There is an explicit relation vector rG corresponding to the relation r defined in the knowledge graph. However, not every All contribute the same to rG, not to mention the noise.

[0100] In order to overcome this problem, the embodiment of the present application attempts to introduce a softmax-based attention mechanism (the symbol "C→G" represents the attention direction from the corpus to the knowledge graph, as shown below:

[0101]

[0102] Among them, M C→G and b C→G is part of the parameters that need to be trained. Through the above work, the implicit relationship representation r h,t This can be modeled as follows:

[0103]

[0104] Among them, ATT C→G [i] indicates the corresponding implicit vector The i-th attention of represents the confidence of the implicit vector.

[0105] Therefore, the embodiment of the present application can redefine each triple (h, r, t)∈{(h 1 ,r,t 1 ),…,(h m ,r,t m )} is as follows:

[0106] φ r (h, t) = || h，t -r G || l

[0107] Intuitively, this scoring function is labeled as “bridging r h,t -r G || l and r G the gap between them".

[0108] Based on this scoring function, can be redefined as:

[0109] 2) Signal modeling from textual relationship representation learning to knowledge graph representation learning

[0110] As mentioned above, for each relationship r∈R defined in the knowledge graph G, the present embodiment can derive a set of sentences {d 1 ,…,d n}, which reveals the implicit semantics of the textual relation rC of relation r and the occurrence of the target entity pair h and t in the sentence, while (h, r, t) is defined in the given knowledge graph. In addition, the corresponding output embedding relation vector is On the other hand, there is an explicit relation vector rG corresponding to the relation r. n Modeling to generate Thus bridging the gap between rC and rG. However, facing the same difficulty of learning knowledge graph representation, not every d j The contributions to rC are the same. To overcome this problem, the embodiment of the present application introduces a softmax-based attention mechanism (the symbol "G→C" represents the attention direction from the knowledge graph to the corpus), as shown below:

[0111]

[0112] Among them, M G→C and b G→C is part of the parameters that need to be trained. Through the above work, the final embedding vector rC of the text relationship can be modeled as follows:

[0113]

[0114] Among them, ATT G→C [j] corresponds to the attention of the jth sentence. The target entity pair h and t appear in the sentence, and the triple (h, r, t) is defined in the given knowledge graph G. The corresponding embedding vector is measured Therefore, the embodiment of the present application can redefine {d 1 ,…,d n The score function for each sentence in} is as follows:

[0115] ψ r (d)=||r C -r G || l

[0116] This score function is labeled "Bridge r C and r G Through the above efforts, based on this score function, the embodiment of the present application has 1 ,…,d n}Form a loss function on all sentences As shown below:

[0117]

[0118] The embodiment of the present application designs a knowledge graph construction method based on CIEM and deep learning, relying on the cross-domain multi-dimensional heterogeneous data sharing and exchange capabilities provided by the CIEM model and exchange sharing platform, based on big data collection results and huge amounts of public channel data, facing all fields and all categories of massive heterogeneous information, completing the "knowledge extraction-knowledge processing-knowledge fusion-knowledge evaluation" full-process heterogeneous knowledge expression organization and storage retrieval method for "cross-system, cross-region, cross-department, cross-level, cross-business", mapping all the above elements to cross-domain shareable information based on the CIEM dictionary, completing entity alignment, disambiguation and verification, and constructing a large-scale knowledge graph.

[0119] Corresponding to the above method embodiment, this specification also provides an embodiment of a knowledge graph construction device based on CIEM and deep learning, Figure 6 A schematic diagram of a knowledge graph construction device based on CIEM and deep learning provided in an embodiment of the present application. Figure 6 As shown, the device comprises:

[0120] The extraction module 601 is configured to construct an input sequence to identify entity information and attribute information using the constructed multi-task transfer learning model, and extract the relationship between entities using a regularization-based method, wherein the multi-task transfer learning model includes a character-level bidirectional LSTM model, a word-level bidirectional LSTM model and a CRF layer, the character-level bidirectional LSTM model is used as a language model, the word-level bidirectional LSTM model is used for sequence labeling feature extraction, and the CRF layer is used for sequence decoding;

[0121] The training module 602 is configured to use the knowledge graph and text joint representation model based on the interactive attention mechanism to learn and train the knowledge graph representation and text relationship representation based on the identified entity information, attribute information and the relationship between entities;

[0122] The construction module 603 is configured to complete the knowledge graph construction based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism.

[0123] In an optional embodiment, the device further includes:

[0124] The referential context is modeled through a convolutional neural network to obtain a vectorized representation of the context;

[0125] The common information in the referential features and candidate entity features is captured respectively by means of a tensor neural network, wherein the referential features carry a vectorized representation of the referential vector and the context; and the candidate entity features carry entity terms and entity types;

[0126] The candidate entity is scored by calculating the semantic similarity between the candidate entity features and the referential context modeling.

[0127] In an optional embodiment, the extraction module 601 is further configured to:

[0128] Construct an input sequence, and for the observation values ​​corresponding to each sequence member in the input sequence, use a character-level bidirectional LSTM model as a language model, and combine it with a sequence annotation feature extraction layer obtained by processing the word-level bidirectional LSTM model to generate a multi-task transfer learning layer, wherein the character-level bidirectional LSTM model and the word-level bidirectional LSTM model share the same character-level input layer;

[0129] The CRF layer is used to perform sequence decoding on the multi-task transfer learning layer, and the entity and attribute information are identified according to the label sequence, wherein the label sequence carries the entity starting state, entity internal state and entity external state.

[0130] In an optional embodiment, the extraction module 601 is further configured to:

[0131] In the word vector layer, by expanding the word vector, the existing domain knowledge is spliced ​​on the word vector in the form of vectors, and the information of the word vector and the existing domain knowledge are integrated to generate a bidirectional LSTM vector;

[0132] The bidirectional LSTM vector is used as the updated domain knowledge information to generate a multi-task transfer learning layer.

[0133] In an optional embodiment, the extraction module 601 is further configured to:

[0134] Determining candidate words carried in an input sequence and marking a pattern set associated with the candidate words, wherein the candidate words include verbs, nouns, and adjectives;

[0135] In response to the candidate word successfully matching the pattern set, the pattern set determines that the entity corresponding to the candidate word is a target entity, wherein the target entity is a potential relationship entity;

[0136] Entities other than the target entity are determined by an entity classifier. In response to the existence of entities other than the target entity, roles are assigned to different entities using a role classifier, and a reporting relationship classifier is applied to extract the relationship between entities.

[0137] In an optional embodiment, the construction module 603 is further configured to:

[0138] Learn embedded representations of entity vector representations and relation vector representations from knowledge graphs;

[0139] Learn embeddings of relation vector representations from plain text in a corpus;

[0140] The embedded representation of the entity vector representation, the relationship vector representation and the embedded representation of the relationship vector representation are integrated to complete the knowledge graph construction.

[0141] In an optional embodiment, the construction module 603 is further configured to:

[0142] Embedded representations of entity vector representations and relation vector representations are learned from knowledge graphs by minimizing a margin-based pairwise ranking loss.

[0143] In an optional embodiment, the construction module 603 is further configured to:

[0144] The link between the word vector and the corresponding position vector in the knowledge graph is used as the input of the convolutional neural network to learn the embedded representation of the entity vector representation and the relationship vector representation.

[0145] In an optional embodiment, the construction module 603 is further configured to:

[0146] Based on the first attention mechanism of softmax, the embedded representation and relation vector representation of entity vector representation are learned from the knowledge graph, wherein the first attention mechanism is the attention direction from the corpus to the knowledge graph; the relation vector representation includes implicit relation vector and explicit relation vector;

[0147] A second attention mechanism based on softmax learns the embedding representation of the relation vector representation from the textual relation representation, wherein the second attention mechanism is the attention direction from the knowledge graph to the corpus;

[0148] The embedded representation of the entity vector representation, the relationship vector representation and the embedded representation of the relationship vector representation are integrated to complete the knowledge graph construction.

[0149] The embodiment of the present application designs a knowledge graph construction device based on CIEM and deep learning. Relying on the cross-domain multi-dimensional heterogeneous data sharing and exchange capabilities provided by the CIEM model and exchange sharing platform, based on big data collection results and huge amounts of public channel data, it faces all fields and all categories of massive heterogeneous information, completes the "knowledge extraction-knowledge processing-knowledge fusion-knowledge evaluation" full-process heterogeneous knowledge expression organization and storage retrieval method for "cross-system, cross-region, cross-department, cross-level, cross-business", maps all the above elements to information that can be shared across domains based on the CIEM dictionary, completes entity alignment, disambiguation and verification, and constructs a large-scale knowledge graph.

[0150] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the knowledge graph construction device based on CIEM and deep learning, since it is basically similar to the knowledge graph construction method embodiment based on CIEM and deep learning, the description is relatively simple, and the relevant parts can be referred to the partial description of the knowledge graph construction method embodiment based on CIEM and deep learning.

[0151] Figure 7 This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0152] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0153] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0154] Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device, a mobile phone, a wearable computing device or other type of mobile device, or a stationary computing device such as a desktop computer or a personal computer. Computing device 700 may also be a mobile or stationary server.

[0155] Among them, the processor 720 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned knowledge graph construction method based on CIEM and deep learning.

[0156] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the knowledge graph construction method embodiment based on CIEM and deep learning, the description is relatively simple, and the relevant parts can be referred to the partial description of the knowledge graph construction method embodiment based on CIEM and deep learning.

[0157] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned method for constructing a knowledge graph based on CIEM and deep learning.

[0158] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the knowledge graph construction method embodiment based on CIEM and deep learning, the description is relatively simple, and the relevant parts can be referred to the partial description of the knowledge graph construction method embodiment based on CIEM and deep learning.

[0159] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned method for constructing a knowledge graph based on CIEM and deep learning.

[0160] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the knowledge graph construction method embodiment based on CIEM and deep learning, the description is relatively simple, and the relevant parts can be referred to the partial description of the knowledge graph construction method embodiment based on CIEM and deep learning.

[0161] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0162] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0163] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0164] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0165] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A knowledge graph construction method based on CIEM and deep learning, characterized in that: include: Constructing an input sequence to identify entity information and attribute information using the constructed multi-task transfer learning model, and extracting the relationship between entities using a regularization-based method, wherein the multi-task transfer learning model includes a character-level bidirectional LSTM model, a word-level bidirectional LSTM model, and a CRF layer, the character-level bidirectional LSTM model is used as a language model, the word-level bidirectional LSTM model is used for sequence labeling feature extraction, and the CRF layer is used for sequence decoding; The identified entity information, attribute information, and relationships between entities are used to learn and train the knowledge graph representation and text relationship representation using a knowledge graph and text joint representation model based on an interactive attention mechanism. The knowledge graph is constructed based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism.

2. The method according to claim 1, characterized in that The method further comprises: The referential context is modeled through a convolutional neural network to obtain a vectorized representation of the context; The common information in the referential features and candidate entity features is captured respectively by means of a tensor neural network, wherein the referential features carry a vectorized representation of the referential vector and the context; and the candidate entity features carry entity terms and entity types; The candidate entity is scored by calculating the semantic similarity between the candidate entity features and the referential context modeling.

3. The method according to claim 1, characterized in that The constructing of the input sequence to identify entity information and attribute information using the constructed multi-task transfer learning model includes: Construct an input sequence, and for the observation values ​​corresponding to each sequence member in the input sequence, use a character-level bidirectional LSTM model as a language model, and combine it with a sequence annotation feature extraction layer obtained by processing the word-level bidirectional LSTM model to generate a multi-task transfer learning layer, wherein the character-level bidirectional LSTM model and the word-level bidirectional LSTM model share the same character-level input layer; The CRF layer is used to perform sequence decoding on the multi-task transfer learning layer, and the entity and attribute information are identified according to the label sequence, wherein the label sequence carries the entity starting state, entity internal state and entity external state.

4. The method according to claim 3, characterized in that The observation values ​​corresponding to each sequence member in the input sequence are processed by using a character-level bidirectional LSTM model as a language model, and a sequence annotation feature extraction layer is obtained by combining the word-level bidirectional LSTM model to generate a multi-task transfer learning layer, and further includes: In the word vector layer, by expanding the word vector, the existing domain knowledge is spliced ​​on the word vector in the form of vectors, and the information of the word vector and the existing domain knowledge are integrated to generate a bidirectional LSTM vector; The bidirectional LSTM vector is used as the updated domain knowledge information to generate a multi-task transfer learning layer.

5. The method according to claim 1, characterized in that The method based on regularization is used to extract the relationship between entities, including: Determining candidate words carried in an input sequence and marking a pattern set associated with the candidate words, wherein the candidate words include verbs, nouns, and adjectives; In response to the candidate word successfully matching the pattern set, the pattern set determines that the entity corresponding to the candidate word is a target entity, wherein the target entity is a potential relationship entity; Entities other than the target entity are determined by an entity classifier. In response to the existence of entities other than the target entity, roles are assigned to different entities using a role classifier, and a reporting relationship classifier is applied to extract the relationship between entities.

6. The method according to claim 1, characterized in that The knowledge graph construction is completed based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism, including: Learn embedded representations of entity vector representations and relation vector representations from knowledge graphs; Learn embeddings of relation vector representations from plain text in a corpus; The embedded representation of the entity vector representation, the relationship vector representation and the embedded representation of the relationship vector representation are integrated to complete the knowledge graph construction.

7. The method according to claim 6, characterized in that The embedded representation and relation vector representation of entity vector representation learned from the knowledge graph include: Embedded representations of entity vector representations and relation vector representations are learned from knowledge graphs by minimizing a margin-based pairwise ranking loss.

8. The method according to claim 6, characterized in that The embedded representation and relation vector representation of entity vector representation learned from the knowledge graph include: The link between the word vector and the corresponding position vector in the knowledge graph is used as the input of the convolutional neural network to learn the embedded representation of the entity vector representation and the relationship vector representation.

9. The method according to claim 6, characterized in that The step of integrating the embedded representation of the entity vector representation, the relationship vector representation, and the embedded representation of the relationship vector representation to complete the knowledge graph construction includes: Based on the first attention mechanism of softmax, the embedded representation and relation vector representation of entity vector representation are learned from the knowledge graph, wherein the first attention mechanism is the attention direction from the corpus to the knowledge graph; the relation vector representation includes implicit relation vector and explicit relation vector; A second attention mechanism based on softmax learns the embedding representation of the relation vector representation from the textual relation representation, wherein the second attention mechanism is the attention direction from the knowledge graph to the corpus; The embedded representation of the entity vector representation, the relationship vector representation and the embedded representation of the relationship vector representation are integrated to complete the knowledge graph construction.

10. A knowledge graph construction device based on CIEM and deep learning, characterized in that: include: An extraction module is configured to construct an input sequence to identify entity information and attribute information using the constructed multi-task transfer learning model, and extract the relationship between entities using a regularization-based method, wherein the multi-task transfer learning model includes a character-level bidirectional LSTM model, a word-level bidirectional LSTM model and a CRF layer, the character-level bidirectional LSTM model is used as a language model, the word-level bidirectional LSTM model is used for sequence labeling feature extraction, and the CRF layer is used for sequence decoding; A training module is configured to use the knowledge graph and text joint representation model based on the interactive attention mechanism to learn and train the knowledge graph representation and text relationship representation based on the identified entity information, attribute information and the relationship between entities; The building module is configured to complete the knowledge graph construction based on the trained knowledge graph and text joint representation model based on the interactive attention mechanism.

Images