System and method for building a knowledge graph using capsule neural networks

Abstract

A system for knowledge graph construction is provided. The system includes a computing device. The computing device has a processor and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to: define entities and relations of a knowledge graph; provide a file having sentences; convert the sentences to fixed-length sentence embeddings and treat the sentence embeddings as a primary capsule layer; learn entity capsules and relation capsules from the primary capsule layer using a set transformer; for each triple, project a head entity and a tail entity from an entity space to a particular relation space, determine that a relation exists when a sum of a projected head entity vector and a relation vector substantially equals a projected tail entity vector; and construct the knowledge graph using the head entity, the tail entity, and the determined relation.

Description

[0001] CROSS-REFERENCE

[0002] This disclosure claims priority to U.S. Patent Application No. 17 / 116,906, filed December 9, 2020, the entire contents of which are incorporated herein by reference.

[0003] In the description of the present disclosure, some references are cited and discussed, which can include patents, patent applications, and various publications. The citation and / or discussion of such references in the present disclosure is provided merely for the purpose of illustrating the present disclosure and does not constitute an admission that any such reference is prior art to the present disclosure. All references cited or discussed in this specification are incorporated herein by reference in their entirety and the citation or discussion of a reference is not to be construed as an admission that it is prior art to the present disclosure. To the extent that any meaning or definition of a term in this document conflicts with the meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to that term in this document shall govern. TECHNICAL FIELD

[0004] The present disclosure relates generally to the field of natural language processing, and more specifically, to systems and methods for constructing a knowledge graph from unstructured text using capsule neural networks. BACKGROUND

[0005] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the inventors, to the extent the work is described in this background section, as well as aspects of the description that can not otherwise qualify as prior art to the disclosure, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0006] A knowledge graph (KG) represents a collection of interlinked descriptions of entities (real-world objects, events, situations, or abstract concepts). They enable automated reasoning, for example, the ability to infer unobserved facts from observed evidence, make logical “jumps,” and adapt data to work in graph analytics for decades. Today, knowledge graphs are at the core of many tools we use in our daily lives, such as voice assistants (ALEXA, SIRI, or Google Assistant), intuitive search applications (Google, Baidu), and online store recommendation programs (Amazon, Jingdong). Building and using knowledge graphs have proven effective for many computational tasks related to linguistics and social sciences. For example, structured knowledge is used as seeds to generate interpretable and controllable text. However, due to the system complexity of designing and integrating all the complex components, including entity extraction, relation extraction, and missing link prediction, it is challenging to construct a KG from unstructured text.

[0007] Knowledge graph construction typically considers the following tasks: 1) knowledge recognition, 2) entity mapping, 3) triple extraction, and 4) predicate mapping. Although these systems extract a large amount of information into a knowledge graph, they miss link prediction and inference, and the KGs they build are far from complete.

[0008] Furthermore, these existing systems and methods design and implement algorithms and models for each task separately and loosely connect the tasks in a pipeline. Typically, such a system construction process includes 3 steps: 1) training each model with annotated datasets separately; 2) processing unstructured text datasets and integrating the extracted entities and relations, storing them as triples; and 3) reasoning new knowledge based on the extracted triples. Each component works independently, and errors in each step propagate to downstream steps. Furthermore, the whole system is highly complex.

[0009] For the missing link prediction task, many methods, such as TransE, TransH, TransR, TransD, STransE, TranSparse, DISMULT, ComplEx, and CapsE, propose to learn embedding models that represent entities and relations as vectors. Based on the learned vector representations, valid triples obtain higher plausibility scores compared to invalid triples. However, except for CapsE, these embedding models use direct operators including addition, subtraction, or multiplication to encode linear relationships between entities, which can be insufficient. On the other hand, CapsE uses a dynamic routing algorithm to dynamically route information between two capsule layers. Due to the complex various semantic surface forms of high-level entities, this is insufficient to construct high-level entities from low-level entities with high quality.

[0010] Therefore, there is an unaddressed need in the art to address the above-referenced deficiencies and inadequacies. SUMMARY

[0011] In certain aspects, the present disclosure relates to a system for knowledge graph construction and for using a knowledge graph. In certain embodiments, the system includes a computing device. The computing device includes a processor and a storage device storing computer executable code. The computer executable code, when executed at the processor, is configured to:

[0012] provide a first number of entities, a second number of relations, and a plurality of documents, each document of the plurality of documents including at least one sentence;

[0013] convert each sentence of the at least one sentence into a third number of sentence embeddings, i.e., a third number of sentence embedding layers;

[0014] form a third number of primary capsule layers, each primary capsule layer of the third number of primary capsule layers corresponding to one sentence embedding of the third number of sentence embeddings (in certain embodiments, the third number of sentence embedding layers are directly considered as the third number of primary capsule layers);

[0015] learning, from a third number of primary capsule layers, a first number of entity capsule layers and a second number of relation capsule layers, an i-th entity and a j-th entity from the first number of entity capsule layers and an m-th relation from the second number of relation capsule layers form a head entity-tail entity-relation triple, where i, j and m are positive integers, i and j can be in the range of hundreds to thousands or more, and m can be in the range of 3-50;

[0016] projecting the i-th entity in the entity space into the m-th relation space to form an i-th projection, projecting the j-th entity in the entity space into the m-th relation space to form a j-th projection, and determining that the i-th entity and the j-th entity exist the m-th relation if a sum of the i-th projection (vector) and the m-th relation (vector) substantially equals the j-th projection (vector), where there is one entity space and the second number of relation spaces; and

[0017] using the determined triples to construct a knowledge graph, such as the i-th entity, the j-th entity and the m-th relation triple.

[0018] In certain embodiments, the set transformer includes an encoder and a decoder, the encoder includes a plurality of self-attention blocks (SABs), and the decoder includes a multi-head attention pooling (PMA) block and a plurality of self-attention blocks.

[0019] In certain embodiments, the computer executable code is configured to project the i-th entity and the j-th entity into the m-th relation space using a projection matrix, the projection matrix is learned during training of the entire model defined by the computer executable code.

[0020] In certain embodiments, the computer executable code is configured to convert each sentence in the at least one sentence into a third number of sentence embeddings by:

[0021] encoding tokens in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to a token in the at least one sentence, where the tokens include words and optional punctuation;

[0022] embedding each one-hot vector in the plurality of one-hot vectors into a word embedding;

[0023] performing LSTM on the word embeddings to obtain a plurality of feature vectors, each feature vector corresponding to a token in the at least one sentence; and

[0024] performing a self-structured attention module on the plurality of feature vectors to obtain the third number of sentence embeddings (sentence embedding layer).

[0025] In certain embodiments, the step of embedding each one-hot vector in the plurality of one-hot vectors into a word embedding is performed using word2vec.

[0026] In certain embodiments, the computer-executable code is configured to convert each sentence in the at least one sentence into a third number of sentence embeddings by:

[0027] encoding tokens in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to a token in the at least one sentence, wherein the tokens comprise words and optional punctuation;

[0028] transforming the one-hot vectors by a transformer to obtain a plurality of feature vectors, each feature vector corresponding to a token in the at least one sentence; and

[0029] performing a self-structured attention module on the plurality of feature vectors to obtain the third number of sentence embeddings.

[0030] In certain embodiments, the transformer comprises a Bidirectional Encoder Representations from Transformers (BERT).

[0031] In certain embodiments, the plurality of documents are product descriptions, the entities are a plurality of products, and the relationships comprise matching relationships between the plurality of products, the computer-executable code is further configured to:

[0032] upon receiving a query product, query the knowledge graph using the query product to obtain a query entity corresponding to the query product and matching entities having matching relationships with the query entity; and

[0033] provide the matching products to the query product, wherein the matching products correspond to the matching entities.

[0034] The number of matching entities can be one or more. In certain embodiments, the query product and the matching products form a set of clothing.

[0035] In certain embodiments, the plurality of documents are product questions and answers, the entities are a plurality of product questions and a plurality of product answers, and the relationships comprise ground truth relationships linking one product answer in the plurality of product answers to one product question in the plurality of product questions, the computer-executable code is further configured to:

[0036] upon receiving a product question, query the knowledge graph using the product question to obtain a product question entity and obtain product answer entities having ground truth relationships with the product question entity among the product answer entities; and

[0037] provide an answer to the product question, wherein the answer corresponds to one product answer entity.

[0038] In certain embodiments, the plurality of documents are product service requests and answers, the entities are a plurality of product service request entities and a plurality of product service answer entities, the relationships include true relationships linking one of the plurality of product service request entities to one of the plurality of product service answer entities, the computer executable code is further configured to:

[0039] upon receiving a product service request, querying the knowledge graph using the product service request to obtain one of the product service request entities and to obtain product service answer entities having a true relationship with the product service request entity; and

[0040] providing an answer to the product service request, wherein the answer corresponds to the product service answer entity.

[0041] In certain aspects, the present disclosure relates to a method. In certain embodiments, the method includes:

[0042] providing, by a computing device, a first number of entities, a second number of relationships, and a plurality of documents, each of the plurality of documents including at least one sentence;

[0043] transforming, by the computing device, each of the at least one sentence into a third number of sentence embeddings, i.e., a third number of sentence embedding layers;

[0044] forming, by the computing device, a third number of primary capsule layers, each of the third number of primary capsule layers corresponding to one of the third number of sentence embeddings (in certain embodiments, the third number of sentence embedding layers is directly considered as the third number of primary capsule layers);

[0045] learning, by the computing device, a first number of entity capsule layers and a second number of relationship capsule layers from the third number of primary capsule layers using a set transformer, an ith entity and a jth entity from the first number of entity capsule layers and an mth relationship from the second number of relationship capsule layers forming a head entity-tail entity-relationship triple;

[0046] projecting, by the computing device, the ith entity in an entity space into an mth relationship space to form an ith projection and projecting the jth entity in the entity space into the mth relationship space to form a jth projection, determining that the ith entity and the jth entity have the mth relationship if a sum of the ith projection (vector) and the mth relationship (vector) substantially equals the jth projection (vector); and

[0047] constructing a knowledge graph using the determined triples, wherein the head entity and the tail entity in the triples have a corresponding relationship in the triples.

[0048] In certain embodiments, the set transformer comprises an encoder comprising a plurality of self-attention blocks (SABs) and a decoder comprising a multi-head attention pooling (PMA) block and a plurality of self-attention blocks.

[0049] In certain embodiments, the step of converting each of the at least one sentence into a third number of sentence embeddings comprises:

[0050] encoding tokens in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to a token in the at least one sentence, wherein the tokens comprise words and optionally punctuation;

[0051] embedding each one-hot vector in the plurality of one-hot vectors into a word embedding;

[0052] performing LSTM on the word embeddings to obtain a plurality of feature vectors, each feature vector corresponding to a token in the at least one sentence; and

[0053] performing a self-structured attention module on the plurality of feature vectors to obtain the third number of sentence embeddings, i.e., a third number of sentence embedding layers.

[0054] In certain embodiments, the step of converting each of the at least one sentence into a third number of sentence embeddings comprises:

[0055] encoding tokens in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to a token in the at least one sentence, wherein the tokens comprise words and punctuation;

[0056] transforming the one-hot vectors by a transformer to obtain a plurality of feature vectors, each feature vector corresponding to a word in the at least one sentence; and

[0057] performing a self-structured attention module on the plurality of feature vectors to obtain the third number of sentence embeddings, i.e., a third number of sentence embedding layers.

[0058] In certain embodiments, the plurality of documents are product descriptions such as clothing matches, the entities are a plurality of products, the relationships comprise matching relationships between the plurality of products, and the method further comprises:

[0059] upon receiving a query product, querying the knowledge graph using the query product to obtain a query entity corresponding to the query product and at least one matching entity having a matching relationship with the query entity; and

[0060] providing a matching product corresponding to the at least one matching entity.

[0061] In certain embodiments, the plurality of documents are product questions and answers, the entities are a plurality of product questions and a plurality of product answers, the relationships include true relationships linking one of the plurality of product answers to one of the plurality of product questions, and the method further comprises:

[0062] upon receiving the product question, querying the knowledge graph using the product question to obtain one of the product question entities, and obtaining the product answer entities that have true relationships with the product question entity; and

[0063] providing an answer to the product question, wherein the answer corresponds to the product answer entity.

[0064] In certain embodiments, the plurality of documents are product service requests and answers, the entities are a plurality of product service request entities and a plurality of product service answer entities, the relationships include true relationships linking one of the plurality of product service request entities to one of the plurality of product service answer entities, and the method further comprises:

[0065] upon receiving the product service request, querying the knowledge graph using the product service request to obtain one of the product service request entities corresponding to the product service request, and obtaining the product service answer entities that have true relationships with the product service request entity; and

[0066] providing an answer to the product service request, wherein the answer corresponds to the product service answer entity.

[0067] In certain aspects, the present disclosure relates to a non-transitory computer- readable medium storing computer-executable code. In certain embodiments, the computer- executable code, when executed at a processor of a computing device, is configured to perform the above-described method.

[0068] These and other aspects of the present disclosure will become apparent from the following description of preferred embodiments thereof, taken in conjunction with the following drawings, wherein like elements are numbered alike, and their changes are shown where appropriate. BRIEF DESCRIPTION OF DRAWINGS

[0069] The present disclosure will become more fully understood from the detailed description and the accompanying drawings, in which one or more embodiments of the present disclosure are shown, by way of illustration. The drawings illustrate embodiments of the present disclosure and, together with the written description, serve to explain the principles of the present disclosure. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like components, in which:

[0070] FIG. 1 A system for knowledge graph construction and utilization according to certain embodiments of the present disclosure is schematically illustrated.

[0071] FIG. 2 An architecture of a knowledge learning module according to certain embodiments of the present disclosure is illustratively shown.

[0072] FIG. 3 A relationship inference mechanism according to certain embodiments of the present disclosure is illustratively shown.

[0073] FIG. 4 Training of a knowledge graph learning according to certain embodiments of the present disclosure is illustratively shown.

[0074] FIG. 5 Construction or completion of a knowledge graph according to certain embodiments of the present disclosure and use of the knowledge graph is illustratively shown. DETAILED DESCRIPTION

[0075] The present disclosure is described in more detail in the following examples, which are intended to be illustrative only, as many modifications and variations will be apparent to persons skilled in the art. Various embodiments of the present disclosure are now described in detail. Referring to the drawings, like numbers (if any) indicate like components throughout the set of drawings. The following description is provided in relation to various embodiments of the present disclosure.

[0076] Terms used in this specification generally have their ordinary meanings in the art, in the context of the present disclosure, and in the specific context where each term is used. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. In case of conflict, the present document, including definitions, controls. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide additional guidance to the practitioner in understanding the description of the present disclosure. For convenience, certain terms can be highlighted, such as using italics and / or quotation marks. The use of highlighting has no influence on the scope and meaning of a term. The scope and meaning of a term are the same, whether or not it is highlighted. It can be appreciated that the same thing can be expressed in more than one way. Consequently, alternative language and synonyms can be used for any one or more terms discussed herein, no substitution of a synonym excludes the use of other synonyms. The use of examples anywhere in this specification, including examples of any terms discussed herein, is illustrative only, and in no way limits the scope and meaning of the present disclosure or of any exemplified term. Likewise, the present disclosure is not limited to the various embodiments given in this specification.

[0077] As used in the description herein and throughout the claims that follow, the meaning of “a,” “an,” and “the” includes plural reference unless the context clearly dictates otherwise. As described herein, the terms “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to.

[0078] As described herein, the phrase at least one of A, B, and C should be interpreted to mean the logic (A or B or C) using non-exclusive OR. It will be understood that one or more steps within a method, unless otherwise specified, can be carried out in differing order (or concurrently) without altering the principles of the present disclosure.

[0079] As described herein, the term "module" or "unit" can indicate, be part of or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a Field Programmable Gate Array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module or unit can include a (shared, dedicated, or group) memory that stores code, executed by a processor.

[0080] The term "code," as used herein, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or objects. The term "shared," as used above, means that some or all code from multiple modules can be executed using a single (shared) processor. Also, some or all code from multiple modules can be stored in a single (shared) memory. The term "group," as used above, means that some or all code from a single module can be executed using a group of processors. Also, some or all code from a single module can be stored using a group of memories.

[0081] As described herein, the term "interface" generally refers to a communication means or device used to perform data communication between components at an interaction point between the components. Generally, an interface can be applicable both at a hardware level and at a software level, and an interface can be a unidirectional interface or a bidirectional interface. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O (Input / Output) devices or components. For example, components in communication with the interface can be multiple components or peripheral devices of a computer system.

[0082] The present disclosure relates to a computer system. As shown in the accompanying drawings, computer components can include physical hardware components, which are shown using solid-line boxes, and virtual software components, which are shown using dashed-line boxes. Those of ordinary skill in the art will understand that these computer components can be implemented in the form of software, firmware, or hardware components or a combination thereof, but are not limited to these forms.

[0083] The apparatus, systems, and methods described herein can be implemented by one or more computer programs executed by one or more processors. The computer program includes processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer program may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage, and optical storage.

[0084] The present disclosure is described more fully below with reference to the accompanying drawings, in which embodiments of the disclosure are shown. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art.

[0085] In some aspects, this disclosure provides a unified framework integrating knowledge recognition, mapping, triple extraction, and missing link prediction and inference. This unified framework creatively combines capsule networks, ensemble transformers, entity and relation spaces to construct knowledge graphs (KGs) from unstructured text. This disclosure uses capsule neurons to represent entities and relations in the knowledge graph, ensemble transformers to learn embeddings, and predicts missing links in the unified framework. The ensemble transformer is responsible for constructing high-level entities and relations from low-level entities and relations. The capsule networks and ensemble transformers enable the system to efficiently handle various semantic surface forms and interactions between complex entities and relations. Based on this representation, this disclosure also provides a learning strategy that enables the corresponding neural network to simultaneously and automatically learn representations of entities and relations in the knowledge graph. The use of capsule networks, ensemble transformers, and separate entity and relation spaces outperforms related techniques in KG construction.

[0086] FIG. 1 A system for knowledge graph construction and utilization according to certain embodiments of this disclosure is illustrated schematically. FIG. 1 As shown, system 100 includes computing device 110. In some embodiments, computing device 110 may be a server computer, cluster, cloud computer, general-purpose computer, headless computer, or special-purpose computer for generating content plans. Computing device 110 may include, but is not limited to, processor 112, memory 114, and storage device 116. In some embodiments, computing device 110 may include other hardware and software components (not shown) to perform their respective tasks. Examples of these hardware and software components may include, but are not limited to, other required memory, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0087] The processor 112 can be a central processing unit (CPU) that controls operation of the computing device 110. In some embodiments, the processor 112 can execute an operating system (OS) or other applications of the computing device 110. In some embodiments, the computing device 110 can have more than one CPU as a processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. The memory 114 can be a volatile memory, such as a random-access memory (RAM), used for storing data and information during operation of the computing device 110. In some embodiments, the memory 114 can be a volatile memory array. In some embodiments, the computing device 110 can operate on more than one processor 112 and / or more than one memory 114. The storage device 116 is a non-volatile data storage medium or device. Examples of the storage device 116 can include a flash memory, a memory card, a USB drive, a solid-state drive, or other types of non-volatile storage devices, such as a hard disk drive, a floppy disk, an optical disk drive, or any other type of data storage device. In some embodiments, the computing device 110 can have more than one storage device 116. In some embodiments, the computing device 110 can also include a remote storage device 116.

[0088] The storage device 116 stores computer-executable code. The computer-executable code includes a knowledge graph application 118. The knowledge graph application 118 includes code or instructions that, when executed at the processor 112, generate knowledge for knowledge graph construction, construct a knowledge graph, and perform related functions with the knowledge graph. In some embodiments, the knowledge graph application 118 can not be executable code, but rather in the form of circuitry corresponding to the functions of the executable code. By providing circuitry instead of executable code, the speed of operation of the knowledge graph application 118 is greatly improved. In some embodiments, as shown, the knowledge graph application 118 includes a data preparation module 120, a knowledge learning module 130, a knowledge graph construction module 150, a function module 160, and a user interface 170. FIG. 1

[0089] ​Data preparation module 120 is configured to prepare training data for training knowledge learning module 130, provide inference data to knowledge learning module 130 for knowledge inference, and send training data or inference data to knowledge learning module 130. For each knowledge graph, a large number of entities and a small number of relationships are predefined. Training data includes training documents, each of which may include one or more sentences, each sentence may have labeled entities and relationships between labeled entities (if available). Training sentences, entity labels, and relationship labels can be used as input to train knowledge learning module 130. After training, the trained knowledge learning module 130 can be used for entity and label prediction and inference. Therefore, data preparation module 120 is also configured to prepare inference data. Inference data includes a large number of documents, each of which may have one or more sentences. These sentences are not labeled with entities and relationships. In some embodiments, training data and inference data are of the same type, such as product description data, customer review data, or customer service data.

[0090] The knowledge learning module 130 is configured to perform training when it receives training data from the data preparation module 120, and after training, infer entities and relations when it receives inference data, and send the inferred entities and relations to the relation determination module 150. FIG. 2 The detailed architecture of the knowledge learning module 130 according to certain embodiments of the present disclosure is illustrated schematically.

[0091] like FIG. 2 As shown, the input text is 131. For example, text input 131 is a sentence from the training data or inference data. The sentence consists of multiple consecutive words, and the number of words in each sentence in the training data or inference data can be different. The knowledge learning module 130 is configured to convert the sentence into a one-hot encoded representation {w1, w2, ..., w...}. i , ..., w T}, where T is the number of word segments in the sentence, including words and punctuation.

[0092] In some embodiments, the knowledge learning module 130 is configured to embed one-hot representations into a word embedding sequence: E = {e1, e2, ..., e...} i , ..., e T Each unique hot word representation has a corresponding word embedding. Here, e i It is a vector representing the d-dimensional word embedding of the i-th word in the document. To obtain the word embedding representation E, the knowledge learning module 130 first looks up the embedding matrix. Where V is a fixed-size vocabulary, d emb This is the size of the word embedding. Matrix W wrd These are the parameters that are to be learned along with the model, d embThese are hyperparameters that must be selected by the user. Knowledge learning module 130 utilizes the matrix-vector product e. i =W wrd v i The word w i Convert to its word embedding e i , where v i It is the word w in V i A vector of indices. In some embodiments, the word embedding layer may be word2vec. After word embedding, the knowledge learning module 130 is further configured to embed the words {e1, e2, ..., e...} i , ..., e T The input is fed into a Long Short-Term Memory (LSTM) encoder 132 to capture the dependencies between words in the sequence. The output of the LSTM encoder 132 is a set of feature vectors {u1, u2, ..., u...}. i , ..., u T}, where u i The semantics of the i-th word in a given sentence are encoded. The LSTM encoder 132 can be a unidirectional LSTM or a bidirectional LSTM.

[0093] Alternatively, the knowledge learning module 130 does not use LSTM layers, but is instead configured to use a transformer encoder to capture dependencies between words in a sentence, which can be achieved using an attention mechanism. Specifically, the one-hot encoding represents {w1, w2, ..., w...} i , ..., w T The input is fed into the converter encoder 132, and the converter encoder 132 outputs a feature vector group 133 {u1, u2, ..., u}. i , ..., u T In some embodiments, the transformer encoder 132 is based on the bidirectional encoder representations from transformers (BERT).

[0094] Feature vector group 133{u1, u2, ..., u i , ..., u T The feature vectors are input into the self-structured attention module 134, which converts them into fixed-length sentence embeddings 135. These fixed-length sentence embeddings 135 are directly used as fixed-length primary capsules 142. Specifically, the self-structured attention module 134 processes the entire feature embedding U = {u1, u2, ..., u...}. i , ..., u T} as input, and output a vector of weights A:

[0095] A = softmax(W) s2 tanh(W s1 U T )) (1)

[0096] Here, W s1 W is a weight matrix with the shape t x u (one-way LSTM encoder 132) or t x 2 u (two-way LSTM encoder 32), where t or T is the total number of words, u is the number of hidden state vectors of LSTM encoder 132, and W s2 It is an r-by-t weight matrix, where r is a hyperparameter that can be arbitrarily set. In this setting, this disclosure uses a softmax classifier to predict the discrete class Y of sentence S. Labels. The classifier takes word embeddings E as input, and U is the hidden state vector of the LSTM encoder 132:

[0097] as well as

[0098]

[0099] Here, A (S) U is a fixed-length sentence embedding of 135, b (S) It is a bias parameter.

[0100] The fixed-length sentence embedding 135 is in the form of a 2D matrix and is considered as a primary capsule 136. Each row of the matrix relates to a portion of the sentence. After obtaining the primary capsule 136, the knowledge learning module 130 is also configured to use a set transformer mechanism to learn the relationships between the primary capsule 136 and abstract entity and relation capsules, obtaining entity / relation capsule 142. FIG. 2 As shown, the ensemble transformer includes an encoder 137 and a decoder 139. The encoder 137 includes multiple self-attention blocks (SABs) 138, and the decoder 139 includes a multi-head attention pooling (PMA) block 140 and multiple self-attention blocks (SABs) 141. The encoder 137 encodes primary capsules 136 to obtain encoded primary capsules, and the decoder 139 uses the encoded primary capsules and entity / relation seed embeddings to compute entity / relation capsules. Note that a predefined number of entity / relation seed embeddings can be randomly set to initialize training; the number of entity / relation seeds is equal to the number of entity / relation capsules obtained. Because the entity / relation seed embedding information is stored in the model, it is not necessary to input entity / relation seed embeddings during subsequent training.

[0101] A self-attention block SAB is a special type of multi-head attention block (MAB). A MAB is an adaptation of the encoder block of a Transformer without positional encoding and dropout. The attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. The input consists of the query and keys of dimension d k and values of dimension d kv :

[0102] and

[0103] MultiHead(Q, K, V) = Concat(head1,... head h ) W O (5)

[0104] where the projections are parameter matrices and

[0105] MAB(X, Y) = LayerNorm(H + rFF(H)) (6)

[0106] where H = LayerNorm(X + Multihead(X, Y, Y)). rFF is any row-wise feed forward layer (i.e., it processes each instance independently and identically), and LayerNorm is layer normalization.

[0107] SAB(X): = MAB(X, X) (7)

[0108] A PMA with seed hypothesis vectors is defined as:

[0109] PMA k (Z) = MAB(S, rFF(Z)) (8)

[0110] where k denotes k hypothesis entity or relation vectors, and Z denotes the input set vector.

[0111] After obtaining the entity / relation capsules 149, the knowledge learning module 130 is further configured to tune the model by the relation inference rules 143. In particular, during the learning process, the knowledge learning module 130 attempts to optimize a loss function combining a classification loss and a relation inference loss:

[0112] L = L c + L r (9)

[0113] For classification loss, the present disclosure uses cross-entropy loss. For multi-class classification, the present disclosure uses multi-class cross-entropy loss. For multi-label classification, the present disclosure uses binary cross-entropy loss.

[0114] As a novel feature, the knowledge learning module 130 is configured to model entities and relations in different spaces (i.e., entity space and relation space), and perform transformation in the relation space. For each triple (h, r, t), the entity embedding is set as The relation embedding is set as For each relation r, the present disclosure sets a projection matrix The projection matrix can project entities from the entity space to the relation space, and is learned during training.

[0115] With the mapping matrix, the present disclosure defines the projection vector of an entity as:

[0116] h r = hM r (10), and

[0117] t r = tM r (11)

[0118] The scoring function is accordingly defined as:

[0119]

[0120] The present disclosure defines the following marginal-based scoring function as the training objective:

[0121] L r = ∑ (h，r，t)∈S ∑ (h′，r，t′)∈S′ max(0, f r (h, t) + γ - f r (h', t')) (13)

[0122] where max(x, y) aims to obtain the maximum value between x and y, γ is the margin, S is the set of correct triples, and S' is the set of incorrect triples.

[0123] FIG. 3 The basic idea of the relation inference mechanism according to certain embodiments of the present disclosure is schematically shown. As FIG. 3 shown, for each triple (h, r, t), first, the entities in the entity space are projected into the r relation space as h r and t r by the operation M r , then h r + r = t rThe relation-specific projection can bring the head / tail entities that actually maintain the relation (denoted as solid circles) closer to each other, and away from those that do not maintain the relation (denoted as solid triangles). In certain embodiments, the proximity is predefined, e.g., within 5% of the value of the relation vector r. After obtaining the head-tail-relation triples, the knowledge learning module 130 is further configured to send the triples to the knowledge graph construction module 150.

[0124] In short, the knowledge learning module 130 is configured to train the learning model by inputting the text, encoding the text to the primary capsule layer, performing the set transformer to obtain the entity / relation capsule layer, performing the relation inference to infer the relations, comparing the inferred relations with the labeled relations (computing the loss) in order to adjust the parameters for encoding, the set transformer, and the projection matrix. After the training, the knowledge learning module 130 can use new inputted text to infer the relations, the newly inferred relations are sent to the knowledge graph construction module 150.

[0125] The knowledge graph construction module 150 is configured to, upon receiving the triples, construct a new knowledge graph or complete the available knowledge graph using the obtained triples. The constructed or completed knowledge graph is accessible to the functional module 160.

[0126] The functional module 160 is configured to, when the knowledge graph is constructed or substantially complete, use the knowledge graph to perform certain functions. The functional module 160 can be stored in the computing device 110 or any other computing device in communication with the computing device 110. In certain embodiments, the function is clothing recommendation, the knowledge graph is constructed using the clothing matching documents. The knowledge graph includes clothing entities, edges represent matches, and clothing belonging to the same outfit are linked by edges. When a customer purchases a clothing in the e-commerce platform, the functional module 160 is configured to query the knowledge graph using the purchased clothing, obtain the clothing that matches the queried clothing, and recommend the queried clothing to the customer.

[0127] In certain embodiments, the function is to provide answers to customer questions, the knowledge graph is constructed using the customer questions and answer documents. The knowledge graph includes question entities and answer entities, and edges represent the appropriate answers to the questions. When a customer purchases a product in the e-commerce platform, the functional module 160 is configured to query the knowledge graph using the question, obtain a customized answer to the question, and provide the answer to the customer. In certain embodiments, the knowledge graph includes subgraphs corresponding to different types of products, so that the answer provided to the customer can be more accurate.

[0128] In certain embodiments, the function is to provide a service to a customer request using the customer service document to build a knowledge graph. The knowledge graph includes service request entities and service entities, and optionally links to service providers for specific services. The edges represent suitable services for a service request. When a customer seeks a service for a product, the function module 160 is configured to use the service request to query the knowledge graph, obtain a customized service for the request, and link the service provider to the request. In certain embodiments, the function can also include making a service appointment for the customer with the service provider. In certain embodiments, the function can also include a troubleshooting process. The knowledge graph includes troubleshooting entities related to the service request, and the function module 160 is configured to provide troubleshooting instructions corresponding to the service request. Before seeking help from a service provider, the customer can be able to solve his problem according to the provided instructions.

[0129] The user interface 170 is configured to provide a user interface or graphical user interface on the computing device 110 or a remote terminal. In certain embodiments, a user of the system administrator can configure parameters for the computing device 110, in particular parameters used in the knowledge graph application 118, including for example hyperparameters of the knowledge learning module 130, the number of primary capsule layers, the number of self-attention blocks SAB in the encoder 137, the number of self-attention blocks SAB in the decoder 139, entities and relations, etc.

[0130] FIG. 4 Training of knowledge graph learning according to certain embodiments of the present disclosure is schematically shown. In certain embodiments, the method 400 as shown can be implemented on the computing device 110 as shown. FIG. 4 FIG. 1 It should be noted in particular that the steps of the method can be arranged in a different order unless otherwise specified in the present disclosure, and thus are not limited to the order as shown. FIG. 4

[0131] At process 402, the data preparation module 120 prepares training data for building or completing the knowledge graph, and sends the training data to the knowledge learning module 130. The entities and labels of the knowledge graph are predefined. The training data includes a plurality of documents, each document having one or more sentences. The entities and relations in each sentence are labeled. The data preparation module 120 can provide the training data for training in batches, each batch including for example 10 labeled sentences.

[0132] At process 404, upon receiving the batch of training data, the knowledge learning module 130 converts the T tokens (including words and punctuation) in each sentence into a sequential one-hot encoding representation {w1, w2,..., w i ,..., w T ​​} and sends the one-hot encoded representation to a word embedding module of the knowledge learning module 130. Each sentence in training is processed substantially independently, however, information from a batch of sentences is used collaboratively to adjust the model parameters in each training iteration.

[0133] At process 406, the word embedding module embeds the sequential one-hot encoded representation {w1, w2,..., w i ,..., w T} into a sequence of word embeddings {e1, e2,..., e i ,..., e T} and sends the sequence of word embeddings to the LSTM encoder 132. Word embeddings are extracted from a vocabulary, each word embedding has a predefined dimension. The word embedding parameters will be learned during training. The word embeddings can be word2vec.

[0134] At process 408, upon receiving the word embeddings, the LSTM encoder 132 encodes the word embeddings {e1, e2,..., e i ,..., e T} into feature vectors {u1, u2,..., u i ,..., u T} and sends the feature vectors to the self-structural attention module 134. The feature vectors are the hidden vectors of the LSTM encoder 132 and include features of the words and semantic relationships of the words in the sentence. In certain embodiments, process 406 and process 408 can also be performed by a transformer that learns the feature vectors {u1, u2,..., u i ,..., u T} directly from the one-hot vectors {w1, w2,..., w i ,..., w T}. In certain embodiments, the transformer is BERT.

[0135] At process 410, upon receiving the feature vectors, the self-structural attention module 134 converts the feature vectors {u1, u2,..., u i ,..., u T} into a sentence embedding 135. The sentence embedding 135 is treated as a primary capsule 136 that goes through an encoder 137. The sentence embedding 135 has a fixed length that is independent of the length of the sentence. The sentence embedding 135 is used as a primary capsule 136 that has a fixed number of primary capsules. The importance of a capsule in the primary capsule 136 can be different for different sentences.

[0136] In process 412, encoder 137 encodes primary capsule 136 to obtain encoded primary capsule and sends the encoded primary capsule to decoder 139. Encoder 137 includes multiple self-attention blocks SAB 138, the number of which can vary depending on the knowledge to be learned.

[0137] In process 414, the PMA 140 of decoder 139 processes the encoded primary capsules and seed entity / relation embeddings, and after further applying the self-attention block SAB 141, obtains entity / relation capsules. The seed entity / relation embeddings define all entities and relations to be constructed in the knowledge graph, and the total number of entity / relation capsules is the predefined total number of entities and relations. In some embodiments, the seed entity / relation embeddings may be random value embeddings or empty embeddings. The seed entity / relation embeddings are used as input during training initiation and are not required after initiation.

[0138] In process 416, after obtaining the entity / relation capsule, for each head entity-relation-tail entity triple, knowledge learning module 130 uses operation M. r Project the head entity and tail entity from the entity space to the corresponding relation space to obtain the projected head entity h. r and the projected tail entity t r If h r +r=t r This determines the relationship between the head entity and the tail entity. In some embodiments, operation M... r The value can be random or empty during training startup and will be learned during subsequent training. Note that there is one entity space for all entities, and each relation has its own relation space.

[0139] In process 418, the knowledge learning module 130 calculates the loss based on the obtained head entity-tail entity relationship, adjusts the model parameters based on the loss, and runs another training iteration to minimize the loss.

[0140] Processes 402 to 418 can be executed iteratively on the same batch of training data for a predetermined number of iterations, or until the parameters converge. Then, the knowledge learning module 130 trains using another batch of training data. After training is complete using all the training data, the model is considered trained. In some embodiments, the system 100 can use certain criteria to evaluate the training; if these criteria are met, the model is considered trained.

[0141] FIG. 5 The illustration schematically depicts the construction or completion of a knowledge graph according to certain embodiments of this disclosure and the use of that knowledge graph. In some embodiments, such as FIG. 5 The method 500 shown can be used in, for exampleFIG. 1 implemented on the computing device 110 as shown. It should be specifically noted that the steps of the method can be arranged in different orders unless otherwise specified in this disclosure, and thus are not limited to the order as shown. FIG. 5

[0142] At process 502, the data preparation module 120 prepares learning data for building or completing the knowledge graph, and sends the learning data to the knowledge learning module 130. The learning data includes a large number of documents, each of which has one or more sentences. The learning data can provide the learning data in batches, and each batch can include, for example, 10 sentences.

[0143] At step 504, the knowledge learning module 130 converts the words in each sentence into a one-hot vector, embeds the one-hot vector into a sequential word embedding, encodes the word embedding through LSTM to obtain a feature vector, performs a self-structural attention module on the feature vector to obtain a fixed-length sentence embedding, regards the fixed-length sentence embedding as a primary capsule, performs a set transformer on the primary capsule to obtain an entity / relationship capsule, and extracts head entity-relation-tail entity information from the obtained entity / relationship capsule using relation inference. Process 504 substantially corresponds to processes 404-416. But there is no real entity and relation label for comparison, so there is no need to adjust the model parameters. Through the above processes, the entity and relationship in each sentence can be learned through one operation of the learning process, and there is no need to learn one round for each triple, each entity or each relationship. Therefore, the learning process is efficient.

[0144] At step 506, after learning the head entity-relation-tail entity triple from the learning data, the knowledge graph construction module 150 uses the learned triple to build or complete the knowledge graph. The built or completed knowledge graph is available to the functional module 160.

[0145] At process 508, the functional module 160 uses the knowledge graph to perform functions. In some embodiments, the knowledge graph is about clothing matching, and each entity is a piece of clothing, and the relationship or edge indicates whether the clothing belongs to the same suit. When a customer evaluates or purchases a piece of clothing from an e-commerce platform, the functional module 160 uses the evaluated or purchased clothing as a query to query the knowledge graph, finds the clothing that matches the evaluated or purchased clothing, and recommends the found clothing from the knowledge graph to the customer, for example, by pushing a message of the clothing to the customer, or showing the recommended clothing to the customer when the customer enters the e-commerce platform.

[0146] ​In certain embodiments, the knowledge graph is about product questions and answers, the entities include products and features of the products, and the relations or edges are whether a product has a corresponding feature. The functional module 160 provides a question and answer interface, such as a chat box, to the customer. When the customer is interested in a product and asks a question about a feature of the product, the functional module 160 uses the product and the question as a query to the knowledge graph to obtain the corresponding feature of the product and includes the corresponding feature in the answer to the customer.

[0147] In certain embodiments, the knowledge graph is about product service requests, the entities include products, service requests for the products, and service solutions for the service requests. The relations or edges link the product entities and the corresponding service solutions. The service solutions can include instructions for the service requests or contact information of service providers for the service requests. The functional module 160 provides a service interface to the customer. When the customer has a service request for a product he purchased, he can describe the service request via the interface, the functional module 160 finds the solution for the service request, and includes the query result as an answer to the customer. The answer can instruct the customer to solve the service problem himself or provide the customer with contact information of a customer service provider. In certain embodiments, the functional module 160 can also schedule a service appointment between the customer and the service provider.

[0148] In certain aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable code. The code, when executed at the processor 112 of the computing device 110, can perform a method as described above. In certain embodiments, the non-transitory computer readable medium can include, but is not limited to, any type of physical or virtual storage media. In certain embodiments, the non-transitory computer readable medium can be implemented as the storage device 116 of the computing device 110 as shown in FIG. 1 FIG. 1.

[0149] Certain embodiments of the present disclosure have the following advantages, among others. (1) It is the first time to propose the use of capsule networks to achieve the isometry of viewpoints in image classification, solving the bag-of-features problem caused by the use of pooling operations in CNNs (Convolutional Neural Networks). Through the routing algorithm between low-level features and high-level features, they can be generalized to recognize the same object with different viewpoints in the image. The present disclosure uses this generalization capability of the capsule network to learn the layer relationships and complex patterns in NLP (Natural Language Processing) applications, abstracted from different surface implementations, and applies the capsule network to solve the knowledge graph construction of unstructured text. (2) Set transformer aims to solve the problem of set input that meets two properties: first, when the order of input instances changes, the output of the model remains unchanged; second, the model can accept inputs of any size. Set transformer captures pairwise and high-order interactions between elements in a set through a self-attention mechanism. Therefore, the present disclosure uses the features of set transformer to solve complex problems such as knowledge recognition and missing link prediction. (3) The present disclosure uses the self-attention mechanism in set transformer to aggregate features, which is conducive to effectively capturing the interaction between entities and relationships and learning more accurate representations. It is novel to combine set transformer to process text compared to using set transformer to process images. (4) Considering that entities are usually complex and contain multiple aspects with relationships centered on the corresponding aspects, projecting entities and relationships into the same embedding space cannot effectively distinguish the aspects of the entities and their relationships. To solve this problem, the present disclosure models entity and relationship embeddings into separate entity and relationship spaces based on capsule networks and set transformers. Then, the present disclosure performs entity-to-relationship space conversion during the learning process and uses it as a regularizer. It is also novel to represent entities and relationships by vectors. (5) The system provides a unified and integrated framework. The framework uses an end-to-end capsule neural network to learn all representations at once. The three components of the framework include: a) a text encoder such as LSTM or transformer with a self-structural attention module to encode the original text into primary capsules; b) a set transformer mechanism to learn the relationships between primary capsules, abstract entity, and relationship capsules; c) During the learning process, the present disclosure uses relationship inference as a regularizer.

[0150] The foregoing description of certain examples of the present disclosure has been presented for the purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching.

[0151] The embodiments are chosen and described in order to explain the principles of the disclosure and its practical application to thereby enable others skilled in the art to best utilize the disclosure and various embodiments and with various modifications as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure pertains, without departing from the scope of the present disclosure. Accordingly, the scope of the present disclosure is defined only by the claims appended hereto and equivalents thereof.

Claims

1. A system for knowledge graph construction, wherein, The system includes a computing device including a processor and a storage device storing computer executable code that when executed at the processor is configured to: provide a first number of entities, a second number of relations, and a plurality of documents, each document of the plurality of documents including at least one sentence; transform each sentence of the at least one sentence into a third number of sentence embeddings; form a third number of primary capsule layers, each primary capsule layer of the third number of primary capsule layers corresponding to one of the third number of sentence embeddings; learn, using a set transformer, the first number of entity capsule layers and the second number of relation capsule layers, an ith entity and a jth entity from the first number of entity capsule layers and an mth relation from the second number of relation capsule layers forming a head entity-tail entity-relation triple; project the ith entity in an entity space into an mth relation space to form an ith projection and project the jth entity in the entity space into the mth relation space to form a jth projection, determine that the ith entity and the jth entity exist the mth relation if a sum of the ith projection and the mth relation substantially equals the jth projection; and use the determined mth relation to construct the knowledge graph.

2. The system of claim 1, wherein, The set transformer includes an encoder including a plurality of self-attention blocks (SABs) and a decoder including a multi-head attention pooling (PMA) block and a plurality of SBA.

3. The system of claim 1, wherein, The computer executable code is configured to project the ith entity and the jth entity into the mth relation space using a projection matrix, the projection matrix learned during training.

4. The system of claim 1, wherein, The computer executable code is configured to transform each sentence of the at least one sentence into the third number of sentence embeddings by: encoding tokens in the at least one sentence into a plurality of one-hot vectors, each one-hot vector of the plurality of one-hot vectors corresponding to one token in the at least one sentence, wherein the tokens include words and punctuation; embedding each one-hot vector of the plurality of one-hot vectors into a word embedding; performing LSTM on the word embedding to obtain a plurality of feature vectors, each feature vector corresponding to one token in the at least one sentence; and performing self-structured attention on the plurality of feature vectors to obtain the third number of sentence embeddings.

5. The system of claim 4, wherein, The step of embedding each one-hot vector of the plurality of one-hot vectors into the word embedding is performed using word2vec.

6. The system of claim 1, wherein, The computer executable code is configured to transform each sentence of the at least one sentence into the third number of sentence embeddings by: encoding words in the at least one sentence into a plurality of one-hot vectors, each one-hot vector of the plurality of one-hot vectors corresponding to one token in the at least one sentence, wherein the tokens include words and punctuation; transforming, by a transformer, a one-hot vector to obtain a plurality of feature vectors, each feature vector corresponding to a token in the at least one sentence; and performing, by the computing device, self-structured attention on the plurality of feature vectors to obtain a third number of sentence embeddings.

7. The system of claim 6, wherein, The transformer comprises a transformer-based bidirectional encoder representation (BERT).

8. The system of claim 1, wherein, The plurality of documents are product descriptions, the entities are a plurality of products, and the relationships comprise matching relationships between the plurality of products, and the computer executable code is further configured to: upon receiving a query product, query the knowledge graph using the query product to obtain a query entity corresponding to the query product and a matching entity having the matching relationship with the query entity; and provide a matching product corresponding to the matching entity.

9. The system of claim 1, wherein, The plurality of documents are product questions and answers, the entities are a plurality of product questions and a plurality of product answers, and the relationships comprise a true relationship linking one product answer in the plurality of product answers to one product question in the plurality of product questions, and the computer executable code is further configured to: upon receiving a product question, query the knowledge graph using the product question to obtain one product question entity in a plurality of product question entities and obtain a product answer entity in a plurality of product answer entities having the true relationship with the product question entity; and provide an answer corresponding to the product answer entity.

10. The system of claim 1, wherein, The plurality of documents are product service requests and answers, the entities are a plurality of product service request entities and a plurality of product service answer entities, and the relationships comprise a true relationship linking one product service request entity in the plurality of product service request entities to one product service answer entity in the plurality of product service answer entities, and the computer executable code is further configured to: upon receiving a product service request, query the knowledge graph using the product service request to obtain one product service request entity in the product service request entities and obtain a product service answer entity in the product service answer entities having the true relationship with the product service request entity; and provide an answer corresponding to the product service answer entity.

11. A method for knowledge graph construction, comprising: providing, by a computing device, a first number of entities, a second number of relationships, and a plurality of documents, each document in the plurality of documents comprising at least one sentence; transforming, by the computing device, each sentence in the at least one sentence into a third number of sentence embeddings; forming, by the computing device, a third number of primary capsule layers, each primary capsule layer in the third number of primary capsule layers corresponding to one sentence embedding in the third number of sentence embeddings; learning, by the computing device, the first number of entity capsule layers and the second number of relationship capsule layers from the third number of primary capsule layers using a set transformer, an i-th entity and a j-th entity from the first number of entity capsule layers and an m-th relationship from the second number of relationship capsule layers forming a head entity-tail entity-relationship triple; projecting, by the computing device, the ith entity in an entity space into an mth relation space to form an ith projection, projecting the jth entity in the entity space into the mth relation space to form a jth projection, determining that the ith entity and the jth entity have the mth relation if a sum of the ith projection and the mth relation substantially equals the jth projection; and constructing the knowledge graph using the determined mth relation.

12. The method of claim 11, wherein, The set transformer includes an encoder including a plurality of self-attention blocks (SABs) and a decoder including a multi-head attention pooling (PMA) block and a plurality of SABs.

13. The method of claim 11, wherein, The step of converting each of the at least one sentence into a third number of sentence embeddings includes: encoding tokens in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to a token in the at least one sentence; embedding each one-hot vector in the plurality of one-hot vectors into a word embedding; performing LSTM on the word embedding to obtain a plurality of feature vectors, each feature vector corresponding to a token in the at least one sentence; and performing self-structural attention on the plurality of feature vectors to obtain the third number of sentence embeddings.

14. The method of claim 11, wherein, The step of converting each of the at least one sentence into a third number of sentence embeddings includes: encoding tokens in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to a token in the at least one sentence; transforming the one-hot vectors by a transformer to obtain a plurality of feature vectors, each feature vector corresponding to a token in the at least one sentence; and performing self-structural attention on the plurality of feature vectors to obtain the third number of sentence embeddings.

15. The method of claim 11, wherein, The plurality of documents are product descriptions, the entities are a plurality of products, and the relations include matches between the plurality of products, the method further comprising: upon receiving a query product, querying the knowledge graph using the query product to obtain a query entity corresponding to the query product and at least one matching entity having the match relation with the query entity; and providing a matching product corresponding to the at least one matching entity.

16. The method of claim 11, wherein, The plurality of documents are product questions and answers, the entities are a plurality of product questions and a plurality of product answers, and the relations include true relations linking one product answer in the plurality of product answers to one product question in the plurality of product questions, the method further comprising: upon receiving a product question, querying the knowledge graph using the product question to obtain one of the product question entities and to obtain product answer entities having the true relation with the product question entity; and providing an answer corresponding to the product answer entity.

17. The method of claim 11, wherein, The plurality of documents are product service requests and answers, the entities are a plurality of product service request entities and a plurality of product service answer entities, the relationships include true relationships that link one product service request entity in the plurality of product service request entities to one product service answer entity in the plurality of product service answer entities, and the method further includes: Upon receiving a product service request, querying the knowledge graph using the product service request to obtain one product service request entity in the product service request entities and to obtain a product service answer entity in the product service answer entities that has the true relationship with the product service request entity; and providing an answer corresponding to the product service answer entity.

18. A non-transitory computer-readable medium storing computer-executable code, wherein, The computer executable code, when executed at a processor of a computing device, is configured to: provide a first number of entities, a second number of relationships, and a plurality of documents, each document in the plurality of documents including at least one sentence; transform each sentence in the at least one sentence into a third number of sentence embeddings; form a third number of primary capsule layers, each primary capsule layer in the third number of primary capsule layers corresponding to one sentence embedding in the third number of sentence embeddings; learn, using a set transformer, the first number of entity capsule layers and the second number of relationship capsule layers from the third number of primary capsule layers, an ith entity from the first number of entity capsule layers and a jth entity from the first number of entity capsule layers and an mth relationship from the second number of relationship capsule layers forming a head entity-tail entity-relationship triple; project the ith entity in an entity space into an mth relationship space to form an ith projection and project the jth entity in the entity space into the mth relationship space to form a jth projection, and determine that the ith entity and the jth entity exist the mth relationship if a sum of the ith projection and the mth relationship substantially equals the jth projection; and use the determined mth relationship to construct a knowledge graph. The set transformer includes an encoder and a decoder, the encoder includes a plurality of self-attention blocks (SABs), and the decoder includes a multi-head attention pooling (PMA) block and a plurality of SABs.

19. The non-transitory computer-readable medium of claim 18, wherein, The computer executable code is configured to transform each sentence in the at least one sentence into the third number of sentence embeddings by:

20. The non-transitory computer-readable medium of claim 18, wherein, encoding words in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to one token in the at least one sentence; embedding each one-hot vector in the plurality of one-hot vectors into a word embedding; performing LSTM on the word embedding to obtain a plurality of feature vectors, each feature vector corresponding to one token in the at least one sentence; and performing self-structural attention on the plurality of feature vectors to obtain the third number of sentence embeddings. The set transformer includes an encoder and a decoder, the encoder includes a plurality of self-attention blocks (SABs), and the decoder includes a multi-head attention pooling (PMA) block and a plurality of SABs. The computer executable code is configured to transform each sentence in the at least one sentence into the third number of sentence embeddings by: encoding words in the at least one sentence into a plurality of one-hot vectors, each one-hot vector in the plurality of one-hot vectors corresponding to one token in the at least one sentence; embedding each one-hot vector in the plurality of one-hot vectors into a word embedding; performing LSTM on the word embedding to obtain a plurality of feature vectors, each feature vector corresponding to one token in the at least one sentence; and performing self-structural attention on the plurality of feature vectors to obtain the third number of sentence embeddings.

Images