Knowledge graph embedding method based on semantic and relational structure fusion embedding

Abstract

The present invention utilizes the role of semantic relevance and relational structure relevance in association relationship reasoning, and proposes a knowledge graph embedding method based on the fusion of semantics and relational structure embedding. It is suitable for the embedding and reasoning of all knowledge graphs that contain both entity descriptions and association relationships. Compared with the knowledge graph embedding methods based purely on semantic embedding and purely on relational structure embedding, the present invention has good performance in knowledge graph association prediction tasks, entity prediction tasks and unknown entity sample prediction tasks.

Description

Technical Field

[0001] The present invention utilizes the role of semantic relevance and relational structure relevance in association relationship reasoning, and proposes a knowledge graph embedding method based on the fusion of semantics and relational structure embedding. It is suitable for the embedding and reasoning of all knowledge graphs that contain both entity descriptions and association relationships. Compared with the knowledge graph embedding methods based purely on semantic embedding and purely on relational structure embedding, the present invention has good performance in knowledge graph association prediction tasks, entity prediction tasks and unknown entity sample prediction tasks. Background Art

[0002] A knowledge graph is a large-scale semantic network knowledge base that uses triples (head entity, relationship, tail entity) to describe specific knowledge and represents and stores it in the form of a directed graph. It has the advantages of being semantically rich, structure-friendly, and easy to understand. Due to its excellent properties in expressing human prior knowledge, knowledge graphs have been widely and successfully applied in many fields such as natural language processing, question-answering systems, and recommendation systems in recent years. Knowledge graphs are constructed based on human prior knowledge, and the problem of missing associative structures is inevitable, which to a certain extent limits the application of knowledge graphs in downstream tasks. To solve this problem, knowledge graph reasoning and completion tasks have emerged, aiming to infer new facts based on existing facts in the knowledge graph, thereby making the knowledge graph more complete.

[0003] Knowledge graph embedding is one of the important methods to solve the problem of knowledge graph reasoning and completion. It embeds entities and relations in the knowledge graph into a multidimensional vector space, and maps the association of knowledge triples through the spatial relationship between vectors, thereby realizing the reasoning of knowledge graph association relationships. Semantic relevance and relational structure relevance are the two most commonly used implementation methods of knowledge graph embedding: (1) semantic-based relevance, that is, the relevance of related entities can be identified from textual descriptions such as names and descriptions; (2) relational structure-based relevance, that is, there is also a potential association relationship between two entities associated with the same entity. These two methods have their own advantages and disadvantages. Semantic-based embedding is based on word vectors or sentence vectors, and supports the prediction task of unknown entity samples, but can only infer whether there is a correlation, and it is difficult to accurately determine the type of relationship; while relational structure-based embedding requires a large number of identified association relationships and cannot support the prediction task of unknown entity samples, but has better support for accurate reasoning of relationships. Summary of the Invention

[0004] The present invention proposes a knowledge graph embedding method based on the fusion of semantics and relational structure embedding, which not only solves the problem that purely semantic-based embedding cannot support relational type reasoning, but also makes up for the deficiency that purely relational structure-based embedding does not support unknown entity sample prediction tasks.

[0005] The basic idea of ​​the present invention is to jointly supervise the training of entity and relationship embedding models based on the existing known semantic and relational structures. Through the fusion embedding training of entity semantic description and relational structure, it is embedded in the multi-dimensional vector space, so that the embedding vector can show both semantic relevance and relational structure relevance. Different from the commonly used vector splicing or simple vector operations for multi-information source embedding, the semantic embedding and relational structure embedding in the present invention are both relatively independent and interrelated. The entity embedding vector serves as both a parameter for relational structure embedding training and an output for semantic embedding training. Through two-stage linked back propagation and parameter update optimization, the fusion embedding of semantics and relational structure is achieved. The overall structure of the present invention is as follows Figure 1 shown.

[0006] Method flow:

[0007] Step 1: Extract the "entity description dataset" and "relationship structure dataset" of the knowledge graph. The "entity description dataset" is derived from the descriptive attributes of the entity, such as the entity name, entity description, and entity attributes; the "relationship structure dataset" is derived from the entity-relationship triple (h, r, t), where h, t∈E represent the head entity and tail entity, and r∈R represents the relationship type. E and R represent the abbreviation of the entity and the relationship type.

[0008] Step 2: Train the word embedding model. Train the word embedding model based on the "Entity Description Dataset" and build a word embedding model. The word embedding model stores the embedding vectors of the words in the "Entity Description Dataset";

[0009] Step 3: Pre-embedding the entity vector. Randomly select training data from the "Relational Structure Dataset". For each triple (h, r, t), obtain the corresponding entity descriptions of the head and tail h, t from the "Entity Description Dataset". Define the pre-embedding vector of the entity semantics as the mean of the word vectors of all words in the entity description.

[0010] Step 4: Semantic embedding. The head entity pre-vector pre_H_vector and the tail entity pre_T_vector are embedded into the head entity vector H_vector and the tail entity vector T_vector after passing through the semantic embedding network with the same structure and parameters.

[0011] Step 5: Relational structure embedding. The head entity vector H_vector and the tail entity vector T_vector are input together with the relation vector R_vector into the relational structure model for optimization training. The parameters of the head and tail entity vectors, the relation vector, and the semantic embedding network are optimized simultaneously to achieve joint training and fusion embedding of semantics and relational structures.

[0012] Technical implementation and network model such as Figure 2 shown.

[0013] Assuming the dimension of the desired embedding is V, a V-dimensional word embedding is performed on the words in the "Entity Description Dataset." Triples (h, r, t) are extracted from the "Relational Structure Dataset." Assuming the number of words in the head entity description is m, the word embedding model yields an m×V entity description matrix. Mean compression is performed to obtain a V-dimensional head entity pre-vector (pre_H_vector). The same operation is performed for the tail entity t to obtain a tail entity pre-vector (pre_T_vector). Using mean compression ensures that even if the head and tail entity descriptions have different word counts, the pre-vectors are of the same dimensionality, which is a prerequisite for using a unified semantic embedding network.

[0014] The semantic embedding method of the present invention is based on the assumption that words that appear in similar contexts have similar meanings. It achieves word embedding by predicting the current word w based on its surrounding words context(w), maximizing the conditional probability p(w|context(w)). Random negative sampling is also used to improve word embedding performance. The head entity pre-vector pre_H_vector and the tail entity pre-vector pre_T_vector are passed through a semantic embedding network with the same structure and parameters to obtain the V-dimensional head entity vector H_vector and tail entity vector T_vector, respectively.

[0015] The head entity vector H_vector and the tail entity vector T_vector are respectively input into the relational structure embedding network structure to participate in optimization training. The goal of the relational structure embedding stage is to embed the relational structure information of the head entity, tail entity and association relationship triples (h, r, t) in the "relational structure dataset" into the vector. The basic idea of ​​relational structure embedding is to understand the association relationship r between entities as a conversion operation between the head entity h and the tail entity t, that is: H_vector+R_vector≈T_vector. The geometric meaning is that the tail entity vector should be the nearest neighbor of the sum of the head entity vector and the relationship vector. The processing of the association relationship r is similar to the relational structure embedding method. The corresponding vector is extracted from the C×V dimensional relationship vector matrix and input into the relational structure embedding network structure to participate in optimization training, where C is the number of relationship types.

[0016] During model optimization training, h, r, t and the semantic embedding network model are supervised and backpropagated based on H_vector + R_vector = T_vector. During the relational structure embedding phase, only r and the relation type vector matrix are directly optimized. For h and t, backpropagation is continued to the semantic embedding network, where they are indirectly optimized by optimizing the semantic embedding network model. The present method does not continue backpropagation to avoid affecting the word embedding training process and the embedding of entity pre-vectors.

[0017] This method achieves the fusion embedding and reasoning of semantics and relational structures. Unlike the commonly used vector concatenation or simple vector operations for multi-source embedding, the semantic embedding and relational structure embedding in the secKG2vec method are both relatively independent and interrelated. The entity embedding vector serves as both a parameter for relational structure embedding training and an output for semantic embedding training. Through two stages of coordinated backpropagation and parameter update optimization, the fusion embedding of semantics and relational structures is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Overview of the secKG2vec Method

[0019] Figure 2 Technical implementation of the secKG2vec method DETAILED DESCRIPTION

[0020] (1) Data collection and dataset construction

[0021] The entities in the knowledge graph dataset required by the present invention must contain both complete text descriptions and identified relationships to ensure that the requirements for semantic embedding and relational structure embedding for the dataset are met simultaneously. Taking commonly used public security knowledge bases and corresponding knowledge graph embedding as examples, entity descriptions and relationships are first crawled from public security knowledge base websites such as Common Vulnerabilities & Exposures (CVE), Common Weakness Enumeration (CWE), and Common Attack Pattern Enumeration and Classification (CAPEC). Among them, CVE focuses on the "Current Description" and "Weakness Enumeration" field information; CWE focuses on the "Description", "Relationships", "Observed Examples", and "Related Attack Patterns" field information; CAPEC focuses on the "Description", "Relationships", and "Related Weaknesses" field information. Based on the "Description" field information, pre-processing is performed by deleting spaces, numbers, punctuation marks and some special symbols (such as "*", "#", "@", " / / ", " / . / ", etc.) to establish an "entity description dataset"; based on other field information, a "relationship structure dataset" is established. The recognized relationship types include: "ChildOf", "ParentOf", "PeerOf", "CanPrecede", "CanFollow", "Related", and the "irrelevant" relationship type "NonRelated" is added. The correspondence between the knowledge base field information and the two types of datasets (entity description dataset and relationship structure dataset) required to be constructed by the present invention is shown in Table 1.

[0022] Table 1 Construction of the dataset

[0023]

[0024]

[0025] (2) Implementation of semantic embedding technology

[0026] The semantic embedding of this invention is based on the assumption that words that appear in similar contexts have similar meanings. It implements word embedding by predicting the current word w based on its surrounding words context(w), that is, maximizing the conditional probability p(w|context(w)). At the same time, random negative sampling is used to improve word embedding performance. For a given context(w), word w is a positive sample, and other words are negative samples. For a given sample (context(w), w), we hope to maximize:

[0027]

[0028] Where NEG(w) represents the negative sample set, and g(w) is maximized by increasing the probability of positive samples while reducing the probability of negative samples. Based on the Sigmoid function, p(u|context(w)) is constructed as follows:

[0029]

[0030] Where, L w (u) is the sample label, the positive sample L w (u) = 1, negative sample L w (u) = 0; x w represents the sum of the word vectors in context(w), θ u represents the parameters to be trained corresponding to word u, and σ(X) represents the Sigmoid function.

[0031] For the corpus ￡ in the “Entity Description Dataset”, the objective function is defined based on formula (1) and formula (2):

[0032]

[0033] Use stochastic gradient to solve parameter updates:

[0034]

[0035] After model training, we get the word embedding model and the word vector v(w) of all words in the corpus ￡. Then, we build the semantic embedding vector of the entity based on the word embedding model. Let the word set D = {w1,w2,...,w m}, the word vector corresponding to each description word is v(w i ), then the entity pre-vector:

[0036]

[0037] It is worth noting that the entity pre-vector is completely dependent on the word embedding model and the entity description word, which means that the subsequent semantic embedding and relational structure embedding will not back-propagate to update the pre-vector, which is beneficial to protect the semantic and syntactic features captured by the word embedding. The entity pre-vector is input into the semantic neural network model as a pre-trained vector:

[0038] f(pre_vector,ω)(6)

[0039] Where ω is the parameter in the semantic embedding network model.

[0040] ω is a general term for the parameters in the semantic embedding network model, which is designed to convert the pre-embedding word vectors of a sentence into vectors of the same dimension with certain semantic features. The number of parameters in the model is determined by the vector dimension V, which is V*V. The semantic embedding stage uses a V-dimensional to V-dimensional fully connected approach to maintain the consistency of the embedding dimensions of each component. The parameter updates of the fully connected layer are based on the backpropagation during the relational structure embedding stage. That is, the relational structure between entities also supervises the semantic embedding, but this supervision does not affect the word embedding and the entity pre-embedding, maintaining a balance between semantic embedding and relational structure embedding. From an engineering perspective, more complex network models can be replaced without affecting the feasibility of the method. After the semantic embedding stage, the head entity and tail entity are embedded as V-dimensional H_vector and T_vector, respectively, and input into the neural network model of the relational structure embedding stage.

[0041] (3) Implementation of relational structure embedding technology

[0042] After the semantic embedding stage, the head and tail entity vectors H_vector and T_vector are used as input to the relational structure embedding stage. The goal of the relational structure embedding stage is to embed the relational structure information of the head entity, tail entity, and association relationship triples (h, r, t) in the "relational structure dataset" into vectors. Therefore, in addition to the head and tail entity vectors, the input of this stage should also include the association relationship vector. In keeping with the dimension of the entity vector, the association relationship embedding vector is also V-dimensional. Assuming the number of association relationship types is C, the association vector matrix is ​​C×V dimensional. During training, the corresponding association relationship vector is extracted based on the triple (h, r, t).

[0043] The basic idea behind relational structure embedding is to interpret the association relationship r between entities as a transformation operation between the head entity h and the tail entity t, i.e., H_vector + R_vector ≈ T_vector. Geometrically, the tail entity vector should be a neighbor of the sum of the head entity vector and the relation vector. The relational structure embedding efficiency function E is defined as:

[0044] E=||H_vector+R_vector-T_vector|| (7)

[0045] Based on E, a loss function is designed and gradient descent is used to optimize and update model parameters. During the relational structure embedding phase, the parameters updated are the entire association vector matrix, as well as the head and tail vectors H_vector and T_vector. H_vector and T_vector serve only as auxiliary parameters, passing gradient changes to the semantic embedding phase to update parameters in the semantic embedding network model.

[0046] (4) Embedded model optimization

[0047] According to formula (7) and L2 norm, the distance function d(h+r,t) is defined to measure the distance between h+r and t:

[0048]

[0049] Where h and t represent the head and tail entity vectors, i.e., H_vector and T_vector; r represents the relationship vector, i.e., R_vector.

[0050] The maximum margin method is used in relational structure embedding, and the objective function Loss is defined as:

[0051]

[0052] Where S is the triple in the “relational structure dataset”, S'(h,r,t) is the negative sampling triple obtained by replacing h and t in (h,r,t); γ is the interval distance parameter with a value greater than 0; [X] + represents a positive function, that is, when X>0, [X] + =X; when X≤0, [X] + = 0. Let L = γ + d(h + r, t) - d(h' + r, t'). The relational structure embedding stage mainly trains and optimizes the relation vector r. The update formula is:

[0053]

[0054] The semantic embedding stage mainly trains and optimizes the parameters ω in the semantic neural network model. The update formula is:

[0055]

[0056] Substitute formula (6) into L:

[0057] L=γ+d[f(p_h,ω)+r,f(p_t,ω)]-d[f(p_h',ω)+r,f(p_t',ω)] (12)

[0058] Then in formula (11):

[0059]

[0060] In the formula, p_* represents different pre-vectors. Through formulas (12) and (13), the training and update of the head and tail entity vectors in the relational structure embedding stage are further transferred to the network model in the semantic embedding stage, so that the identified entity relationships can jointly supervise the optimization of the relational structure embedding model and the semantic embedding model.

[0061] (5) Important parameter settings

[0062] This patent proposes a knowledge graph embedding method that integrates semantics and relational structures. Like other machine learning methods, the parameter settings involved in this method also need to be continuously tested and optimized according to specific application scenarios. Some important parameter settings are as follows:

[0063] 1) In each iteration, the training data is divided into multiple batches of size Batch, and one batch is trained at a time. Common values ​​Batch∈{32,64,128};

[0064] 2) The interval distance parameter γ is an important parameter for relational structure embedding (see Formula 9), and the commonly used value is γ∈{1,5,10};

[0065] 3) The learning rate η of stochastic gradient descent (see formulas 10 and 11), with a common value of η∈{0.01, 0.1, 0.5};

[0066] 4) The embedding dimension V of entities and relations, commonly used values ​​are V∈{32,64,128}.

[0067] To verify the effectiveness of the proposed method, a comparative experiment was conducted. The experiment used the Doc2vec method based purely on semantic embedding and the TransE method based purely on relational structure embedding as baselines. By comparing their performance in the association relationship reasoning task, the effectiveness of the proposed method was verified. The experiment defined two types of reasoning tasks: (1) association prediction, that is, predicting the association relationship given the head and tail entities; and (2) entity prediction, that is, predicting the tail entity given the head entity and the association relationship. 6,000 triples were randomly selected as the "relation structure dataset" for the experiment and the corresponding head and tail entity descriptions as the "entity description dataset" for the experiment, of which 4,800 (80%) were used for model training and 1,200 (20%) were used for model validation. The head and tail entities in the dataset cover a total of 1,265 entities. Two evaluation indicators were designed: 1) Mean Rank: represents the average ranking of the correct entities or relations in the prediction results. 2) Hits@n (%): represents the proportion of correctly predicted entities ranked in the top n (%) in the prediction results. The association prediction task uses Hits@3, which is the percentage of correctly predicted associations ranked in the top 3; the entity prediction task uses Hits@30%, which is the percentage of correctly predicted related entities ranked in the top 30%. A smaller Mean Rank and a higher Hits@n (%) indicate better prediction performance.

[0068] In the association prediction task experiment, all three methods participated in the comparative analysis and used the same training and test data, but different data types were selected as input based on the specific requirements of each method. For example, for the same triple, the Doc2vec method trained the association classifier using the entity description sentence embeddings corresponding to the head and tail entities; the TransE method did not need to query the "entity description dataset" and only trained and optimized the entity and relationship embedding vectors based on the relationship structure; the method of the present invention required both entity description information and relationship structure information. Training and testing were performed on the above datasets, and by setting different experimental parameters, the best results of each method were obtained. Mean Rank results: The TransE method and the method of the present invention both reached the optimal value of 3.723 and 3.496, respectively, when the parameters Batch = 32; γ = 5; η = 0.01; V = 64. The Doc2vec method did not involve any parameters and reached the optimal value of 3.663 when the parameters Batch = 32; V = 128 were used. Hits@3 results: The TransE method and the proposed method achieved optimal performance with parameters Batch=32; γ=5; η=0.01; and V=64, achieving 69.4% and 78.2% respectively. The Doc2vec method achieved optimal performance with parameters Batch=32 and V=128, achieving 74.5%. Comparing the optimal performance of the three methods, the proposed method (3.496) improved Mean Rank by 4.5% and 6.1% compared to the Doc2vec method (3.663) and the TransE method (3.723), respectively. In terms of Hits@3, the proposed method (77.2%) improved Hits@3 by 5% and 9.7% compared to the Doc2vec method (74.5%) and the TransE method (70.4%), respectively. This indicates that the proposed knowledge graph embedding method, based on the fusion of semantics and relational structure embedding, can effectively improve association prediction performance compared to the purely semantic embedding Doc2vec method and the purely relational embedding TransE method.

[0069] In the entity prediction task experiment, the Doc2vec method, which cannot embed relation types and is not comparable to the other two methods, was not included in the comparative analysis. The TransE method and the proposed method used the same data but different inputs for the experiments. Mean Rank results: The TransE method achieved optimal performance with parameters Batch = 32; γ = 5; η = 0.01; V = 128, achieving a score of 666.3. The proposed method achieved optimal performance with parameters Batch = 64; γ = 5; η = 0.01; V = 64, achieving a score of 563.5. Hits@30% results: The TransE method achieved optimal performance with parameters Batch = 32; γ = 5; η = 0.01; V = 128, achieving a score of 28.1%. The proposed method achieved optimal performance with parameters Batch = 64; γ = 5; η = 0.01; V = 64, achieving a score of 35.6%. Due to the large number of entities and the presence of complex relationships such as 1-N, N-1, and NN, the overall performance of the entity prediction experiment was inferior to that of the association prediction experiment. Despite this, the proposed method still shows significant improvement over the TransE method, which relies on pure relational structure embedding. In terms of Mean Rank, the proposed method (563.5 / 1265) improves by 15.4% compared to the TransE method (666.3 / 1265); in terms of Hits@30%, the proposed method (35.6%) improves by 26.9% compared to the TransE method (28.1%). This also shows that the advantages of semantic embedding in relevance prediction can indeed well complement the knowledge graph embedding that relies purely on relational structure.

[0070] Comparative experiments verify the advantages of the method of the present invention over the Doc2vec method based on semantic embedding and the TransE method based purely on relational structure embedding. This shows that the method of the present invention not only solves the problem that purely semantic-based embedding cannot support relational type reasoning, but also makes up for the deficiency that purely relational structure-based embedding does not support unknown entity sample prediction tasks. It performs well in knowledge graph association prediction tasks and entity prediction tasks.

Claims

1. A knowledge graph embedding method based on the fusion embedding of semantics and relational structures, characterized by: Step 1: Extract the "entity description dataset" and "relationship structure dataset" of the knowledge graph; the "entity description dataset" is derived from the descriptive attributes of the entity; the "relationship structure dataset" is derived from the entity-relationship triple (h, r, t), where h and t represent the head entity and the tail entity, and r represents the relationship type; Step 2: Train the word embedding model; perform word embedding training based on the "Entity Description Dataset" and build a word embedding model; the word embedding model stores the embedding vectors of the words in the "Entity Description Dataset"; Step 3: Entity pre-vector embedding: randomly select training data from the "Relational Structure Dataset"; for each triple (h, r, t), obtain the corresponding head and tail entity descriptions h, t from the "Entity Description Dataset" and define the pre-vector pre_vector of the entity semantic embedding as the mean of the word vectors of all words in the entity description; Step 4: Semantic embedding: The head entity pre-vector pre_H_vector and the tail entity pre_T_vector are embedded into the head entity vector H_vector and the tail entity vector T_vector after passing through the semantic embedding network with the same structure and parameters; Step 5: Relational structure embedding: The head entity vector H_vector and the tail entity vector T_vector are input together with the relation vector R_vector into the relational structure model for optimization training. At the same time, the parameters of the head and tail entity vectors, the relation vector, and the semantic embedding network are optimized to achieve joint training and fusion embedding of semantics and relational structures. The technical implementation and network model are as follows: (1) Implementation of semantic embedding technology Semantic embedding is based on the assumption that words that appear in similar contexts have similar meanings. It implements word embedding by predicting the current word w based on its surrounding words context(w), that is, maximizing the conditional probability p(w|context(w)). At the same time, random negative sampling is used to improve word embedding performance. For a given context(w), word w is a positive sample, and other words are negative samples. For a given sample (context(w), w), we hope to maximize: g(w)=∏ u∈{w}∪NEG(w) p(u|context(w)) (1) Where NEG(w) represents the negative sample set, and g(w) is maximized by increasing the probability of positive samples while reducing the probability of negative samples. Based on Sigmoid, p(u|context(w)) is constructed: Where, L w (u) is the sample label, the positive sample L w (u) = 1, negative sample L w (u) = 0; x w represents the sum of the word vectors in context(w), θ u Represents the parameters to be trained corresponding to word u; Sigmoid function is represented by σ(X); For the corpus in the "Entity Description Dataset" Based on formula (1) and formula (2), the objective function is defined as Use stochastic gradient to solve parameter updates: After model training, we get the word embedding model and corpus The word vector v(w) of all words in the word embedding model is constructed; then, the semantic embedding vector of the entity is constructed based on the word embedding model; let the word set D = {w1,w2,...,w m }, the word vector corresponding to each description word is v(w i ), then the entity pre-vector: The entity pre-vector pre_vector is used as a pre-trained vector to input into the semantic neural network model: f(pre_vector,ω) (6) where ω is the parameter in the semantic embedding network model; ω is a general term for the parameters in the semantic embedding network model. The number of parameters in the model is determined by the vector dimension V, which is V*V. The semantic embedding stage adopts a V-dimensional to V-dimensional full connection method to maintain the consistency of the embedding dimensions of each part. After the semantic embedding stage, the head entity and the tail entity are embedded as V-dimensional H_vector and T_vector respectively, and are input into the neural network model of the relational structure embedding stage respectively. (2) Implementation of relational structure embedding technology After the semantic embedding phase, the head and tail entity vectors H_vector and T_vector are used as input to the relational structure embedding phase. The goal of the relational structure embedding phase is to embed the relational structure information of the head entity, tail entity, and association relationship triples (h, r, t) in the "relational structure dataset" into vectors. Therefore, in addition to the head and tail entity vectors, the input of this phase should also include the association relationship vector. To keep consistent with the dimension of the entity vector, the association relationship embedding vector is also V-dimensional. Assuming the number of association relationship types is C, the association vector matrix is ​​C×V-dimensional. During training, the corresponding association relationship vector is extracted based on the triples (h, r, t). The basic idea of ​​relational structure embedding is to interpret the association relationship r between entities as a conversion operation between the head entity h and the tail entity t, that is, H_vector + R_vector ≈ T_vector. The geometric meaning is that the tail entity vector should be the nearest neighbor of the sum of the head entity vector and the relation vector. The relational structure embedding efficiency function E is defined as: E=||H_vector+R_vector-T_vector|| (7) Based on E, a loss function is designed and the model parameters are optimized and updated using the gradient descent method. The parameters updated in the relational structure embedding stage are the entire association vector matrix, as well as the head and tail vectors H_vector and T_vector. H_vector and T_vector serve only as auxiliary parameters, passing the gradient changes to the semantic embedding stage to update the parameters in the semantic embedding network model. (3) Embedding model optimization According to formula (7) and L2 norm, the distance function d(h+r,t) is defined to measure the distance between h+r and t: Where h, t represent the head and tail entity vectors, i.e. H_vector, T_vector; r represents the relationship vector, i.e. R_vector; The maximum margin method is used in relational structure embedding, and the objective function Loss is defined as: Where S is the triple in the "relational structure dataset", S'(h,r,t) is the negative sampling triple obtained by replacing h and t in (h,r,t); γ is the interval distance parameter with a value greater than 0; [X] + represents a positive function, that is, when X>0, [X] + =X; when X≤0, [X] + =0; let L=γ+d(h+r,t)-d(h'+r,t'); The relationship structure embedding stage trains and optimizes the relationship vector r, and the update formula is: The semantic embedding stage mainly trains and optimizes the parameters ω in the semantic neural network model. The update formula is: Substitute formula (6) into L: L=γ+d[f(p_h,ω)+r,f(p_t,ω)]-d[f(p_h',ω)+r,f(p_t',ω)] (12) Then in formula (11): In the formula, p_* represents different pre-vectors; through formulas (12) and (13), the training and update of the head and tail entity vectors in the relational structure embedding stage are further transferred to the network model in the semantic embedding stage, so that the identified entity relations can jointly supervise the optimization of the relational structure embedding model and the semantic embedding model.

Images