Knowledge graph representation learning method and system based on hybrid expert algorithm

Abstract

The present disclosure relates to the field of knowledge graph technology and proposes a knowledge graph representation learning method and system based on a hybrid expert algorithm, comprising the following steps: obtaining a triple to be processed, performing global embedding to obtain a global embedding representation of the triple; obtaining an expert embedding representation corresponding to each expert model for the triple to be processed, and connecting it with the global embedding to obtain a connected embedding representation of each expert; performing routing selection based on a routing mechanism for the triple to be processed, assigning models to different relationships, and obtaining a weight for each expert model; performing weighted mixing on the obtained connected embedding representation based on the weight obtained by routing selection to generate a final triple embedding. The present disclosure uses the collaborative learning of multiple expert models, combined with the routing mechanisms of gate routing and attention routing, to assign appropriate models to different relationship types to improve the performance of representation learning.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to knowledge graphs, and specifically, to a knowledge graph representation learning method and system based on a hybrid expert algorithm. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Knowledge graphs structure text in the form of triples, which is beneficial for knowledge-intensive tasks such as question-answering and dialogue systems. They are also used to enhance the reasoning capabilities of large language models. Representation learning on knowledge graphs aims to learn implicit embeddings of entities and relations and has become an indispensable step in knowledge graph applications.

[0004] The inventors found in their research that the existing general representation learning models focus on using linear, bilinear, complex or convolution operations to generate structural features. TransE regards relationships as transfers between head and tail entities, and is mainly applicable to simple transfer relationships, such as one-to-one (1-1), one-to-many (1-n), and many-to-one (n-1) relationships; DistMult is a bilinear model that is good at capturing relationship semantics and is more suitable for processing symmetric relationships; ComplEx uses complex-valued embeddings and can effectively process asymmetric relationships; ConvE uses convolutional neural networks to embed the interactions between entities and relationships, thereby showing stronger performance in processing complex relationships. Existing representation learning methods have certain limitations and cannot effectively process diverse and complex relationship types at the same time. Specifically, these methods often perform well in processing simple relationship types (such as one-to-one, symmetric relationships, etc.), but they are insufficient in dealing with complex, diverse and heterogeneous relationships. For example, there may be certain types of relationships that have symmetry, transferability or other more complex characteristics. At this time, existing representation learning methods may not be able to effectively integrate the characteristics of these complex relationships, resulting in limited reasoning capabilities of the model. Summary of the Invention

[0005] In order to solve the above problems, this paper proposes a knowledge graph representation learning method and system based on a hybrid expert algorithm. Through the collaborative learning of multiple expert models, combined with the routing mechanisms of gate routing and attention routing, appropriate models are assigned to different relationship types to improve the performance of representation learning.

[0006] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0007] One or more embodiments provide a knowledge graph representation learning method based on a hybrid expert algorithm, comprising the following steps:

[0008] Get the triple to be processed, perform global embedding to obtain the global embedding representation of the triple;

[0009] For the triples to be processed, the expert embedding representation corresponding to each expert model is obtained, and it is concatenated with the global embedding to obtain the concatenated embedding representation of each expert;

[0010] For the triples to be processed, routing is selected based on the routing mechanism, models are assigned to different relations, and the weight of each expert model is obtained;

[0011] The obtained connection embedding representation is weighted mixed based on the weights obtained from routing selection to generate the final triplet embedding.

[0012] One or more embodiments provide a knowledge graph representation learning system based on a hybrid expert algorithm, including:

[0013] A global embedding module is configured to obtain a triple to be processed and perform global embedding to obtain a global embedding representation of the triple;

[0014] The expert embedding module is configured to obtain the expert embedding representation corresponding to each expert model for the processed triples, and concatenate it with the global embedding to obtain the concatenated embedding representation of each expert;

[0015] The routing module is configured to select routes for the triples to be processed based on the routing mechanism, assign models to different relations, and obtain the weight of each expert model;

[0016] The fusion module is configured to perform weighted mixing on the obtained connection embedding representations based on the weights obtained by routing selection to generate the final triplet embedding.

[0017] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above-mentioned knowledge graph representation learning method based on the hybrid expert algorithm are completed.

[0018] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, completes the steps in the above-mentioned knowledge graph representation learning method based on a hybrid expert algorithm.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] This embodiment uses a representation learning method based on a hybrid expert algorithm to effectively address the limitations of existing methods in dealing with diverse and complex relationships, and significantly improves the model's modeling and reasoning capabilities for complex relationships. Compared to existing single-model methods, the introduction of a routing mechanism achieves adaptive optimization for different relationship types, which is suitable for simple relationships (such as symmetry, one-to-one relationships) and can efficiently handle complex relationships (such as transferability, many-to-many relationships). Through the synergy of expert models and weighted hybrids, it fully utilizes the advantages of multiple models and improves the representation capabilities of knowledge graph embedding. Moreover, it can be extended to handle other complex relationship types and has a wide range of applicability.

[0021] The advantages of the present disclosure and additional advantages will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0023] Figure 1 Schematic diagram of a hybrid expert algorithm model of a knowledge graph representation learning method based on a hybrid expert algorithm in Example 1 of the present disclosure;

[0024] Figure 2 It is a flowchart of the knowledge graph representation learning method based on the hybrid expert algorithm of Example 1 of the present disclosure. DETAILED DESCRIPTION

[0025] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0027] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0028] Example 1

[0029] In the technical solutions disclosed in one or more embodiments, Figures 1 to 2 As shown in FIG, a knowledge graph representation learning method based on a hybrid expert algorithm includes the following steps:

[0030] Step 1: Get the triple (h, r, t) to be processed, perform global embedding to obtain the global embedding representation of the triple (h g , r g , t g );

[0031] Step 2: For the triples to be processed, use the expert model to obtain the expert embedding representation (h e , r e , t e ), and concatenate with the global embedding to obtain the concatenated embedding representation of each expert;

[0032] Step 3: For the triple (h, r, t) to be processed, perform routing selection based on the routing mechanism, assign models to different relations, and obtain the weight of each expert model;

[0033] Step 4: The obtained connection embedding representation is weighted mixed based on the weights obtained by routing to generate the final triple embedding;

[0034] In this embodiment, in the above implementation, the triplet (h, r, t) to be processed is firstly global embedded to generate a global embedding representation (h g , r g , t g ), this process uses existing embedding methods to extract global features of entities and relations to ensure the comprehensiveness of the initial representation. Then, multiple predefined expert models are used, each of which generates a corresponding expert embedding representation (h e , r e , t e ), fully capturing the diverse characteristics of entities and relationships. The global embedding representation is concatenated with the expert embedding representation to obtain the final expert connected embedding representation. The routing mechanism dynamically calculates the weight of each expert model to assign the most appropriate model to each relationship. The routing mechanism automatically selects and adjusts the weights of expert models based on the characteristics of the relationship, achieving adaptive model optimization. Ultimately, the expert embeddings are weighted and mixed using the routing weights to generate the final embedding representation of the triple, ensuring efficient modeling and reasoning of complex relationships.

[0035] This embodiment uses a representation learning method based on a hybrid expert algorithm to effectively address the limitations of existing methods in dealing with diverse and complex relationships, and significantly improves the model's modeling and reasoning capabilities for complex relationships. Compared to existing single-model methods, the introduction of a routing mechanism achieves adaptive optimization for different relationship types, which is suitable for simple relationships (such as symmetry, one-to-one relationships) and can efficiently handle complex relationships (such as transferability, many-to-many relationships). Through the synergy of expert models and weighted hybrids, it fully utilizes the advantages of multiple models and improves the representation capabilities of knowledge graph embedding. Moreover, it can be extended to handle other complex relationship types and has a wide range of applicability.

[0036] First, let's explain the mixed-expert model. In the real world, a medical team typically consists of specialists such as internists, surgeons, and radiologists, each with expertise in a different field. When a patient seeks medical care, they are first evaluated at a triage desk and then assigned to the physician best qualified to treat their specific condition. During this process, not every physician is involved in the care of all patients; instead, each specialist specializes in their own area of ​​expertise. This model is called a mixed-expert model.

[0037] In some embodiments, the above steps 1 to 4 can be implemented by constructing a hybrid expert algorithm model, such as Figure 1 As shown, it includes multiple expert models and gating mechanism modules; multiple representation learning models are selected as expert models to capture different types of relationship features; the expert models are selected and integrated through the gating mechanism module;

[0038] In some embodiments, the representation learning model may include but is not limited to the TextEnc model, the TransE model, the DistMult model, the ComplEx model, and the ConvE model, and different types of relational features may be captured through different expert models:

[0039] (1) TextEnc: Uses BiGRU to encode text sequences and generate text embeddings of entities and relations to supplement semantic information;

[0040] TextEnc learns the sequence features of word sequences of entities and relations. For an entity h, its word sequence is For a relation r, its word sequence is Use BiGRU to encode these sequences:

[0041] I w =BiGUR(w i )for i∈{h,r,t} (1)

[0042] Among them, the last state of the forward and backward directions is spliced ​​to obtain the sequence embedding, that is, the text embedding of the entity h and the relation r is generated, which are h w and r w .

[0043] (2) TransE: Translation Embedding, is a linear translation embedding, assuming h + r ≈ t. The connection between entity h and relation r can be captured by an addition:

[0044]

[0045] in, h w and r w is the text embedding defined by formula (1).

[0046] (3) DistMult: uses a bilinear formula to capture the bilinear features of entities and relations;

[0047] The connection between entity h and relation r can be obtained by multiplication:

[0048]

[0049] (4) ComplEx: uses complex-valued embeddings, where the combination of complex-valued embeddings can handle various binary relations, including symmetric and antisymmetric relations. The connection between entity h and relation r can be expressed as:

[0050]

[0051] (5) ConvE: Use a 2D convolutional network to learn the potential connection between the head entity h and the relation r as follows:

[0052]

[0053] in, and Represent entity embedding vectors and relation embedding vectors Reshape; the reshape operation converts the vector Convert from one dimension D to two dimensions D = D1.D2;

[0054] Indicates h w and r w The result of concatenating the embedding vector after reshaping. ω The symbol represents a two-dimensional convolutional layer with a filter ω, which returns a feature map tensor Among them, C1 is the number of feature maps with dimension C2×C3. Then Reshape into a vector And project back using a linear layer ρ represents ReLU activation.

[0055] Furthermore, in order to guide the learning of each expert, this embodiment introduces a global entity embedding for each entity h A global relation embedding is introduced for each relation r Finally, the global embedding and the embedding representation output by each expert model are connected, that is, the connected embedding of the i-th expert is

[0056] In order to assign different triples to the most appropriate experts, the routing mechanisms set in this embodiment include gate routing and attention routing;

[0057] Gate Routing: Based on the input of entities and relations, the weight of each expert is generated through the gate network.

[0058] Furthermore, the gate routing mechanism includes sparse gate routing, dense gate routing and multi-gate routing. Each routing mechanism calculates the weight of experts in a different way and optimizes the selection of experts respectively;

[0059] For a triple Its gate routing function is:

[0060] gate(h,r)=Softmax(TopN(f(h,r),n)) (6)

[0061]

[0062] in, They represent the weight matrix and the weight matrix with noise added, σ is the softplus smooth activation function, and softmax is the softmax activation function. i represents the i-th value of f(h,r), gate(h,r) represents the gate routing function, f(h,r) calculates the score of the feature representation of entity h and relation r, and n is the n expert models set when calculating TopN;

[0063] According to the selection of TopN, gate routing is divided into sparse gate routing and dense gate routing (n=N). TopN represents the n expert models with the highest weights. Specifically;

[0064] Dense Gate Routing: Each expert is assigned a weight, and experts with higher weights contribute more to the final embedding;

[0065] like Figure 1 As shown in (a), four expert models are set as an example. The dense gate routing calculates weights for all expert models. The weights calculated by the dense gate routing are P = 0.41, P = 0.19, P = 0.32, and P = 0.08, which are multiplied by the corresponding model outputs and then weighted;

[0066] Sparse Gate Routing: selects the TopN expert models (the N expert models with the highest weights) and resets the weights of other experts to zero, thereby reducing computational overhead;

[0067] like Figure 1 As shown in (b), four expert models are set as an example. The two expert models with the highest weights are selected. The weights calculated by sparse gate routing are P = 0.41 and P = 0.32 respectively. The weights are multiplied by the corresponding model outputs and then weighted to obtain the weighted results;

[0068] Furthermore, this embodiment also proposes a multi-gate routing, designing different gate networks for entities and relationships, respectively, calculating the independent weights of the head entity and relationship, and then fusing them to form the final expert model weight:

[0069]

[0070] f(h,r)=f(h)+f(r) (11)

[0071] in, denote the weight matrices of the head entity and relation respectively, They represent the noise weight matrices of the head entity and relation, respectively, and f(h) and f(r) represent the feature representation scores of the head entity and relation, respectively.

[0072] like Figure 1 As shown in (c), four expert models are set as an example. After weighting the independent weights of the head entity and relationship, the two highest expert models are selected. The weights of the independent weights calculated by multi-gate routing are P = 0.35 and P = 0.45 respectively. The obtained weights are multiplied by the corresponding model output and then weighted to obtain the weighted result Y;

[0073] In some embodiments, attention routing is configured to calculate the relevance weight of each expert model and the current triplet through an attention network, including sparse attention routing, dense attention routing, and multiple attention routing;

[0074] Attention routing is an attention network that integrates the correlation between experts. The attention routing function is:

[0075] Att(h,r)=softmax(TopN(f(h,r),k)) (12)

[0076] Q=W Q [h; r], K = W K [h; r], V = W V [h;r] (13)

[0077]

[0078] in, τ represents the temperature hyperparameter;

[0079] W Q : represents the weight matrix used to calculate the query vector Q;

[0080] W K : represents the weight matrix used to calculate the key vector K;

[0081] W V : represents the weight matrix used to calculate the value vector V;

[0082] Q: represents the query vector, which is used to represent the target features in the attention mechanism;

[0083] K: represents the key vector, which is used to represent the context information related to the target feature;

[0084] V: represents the value vector, which represents the eigenvalue associated with the key vector;

[0085] k is the k expert models set when calculating TopN, and TopN is the same as formula (8).

[0086] Furthermore, according to the selection of topN, it can be divided into sparse attention routing and dense attention routing (n=N):

[0087] Dense Attention Routing: All expert models participate in weight calculation;

[0088] Sparse Attention Routing: only assigns weights to the top N experts with the highest relevance, reducing computational redundancy.

[0089] Furthermore, this embodiment also proposes multiple attention routing, designing different attention mechanisms for entities and relationships respectively, calculating the independent weights of the head entity and relationship respectively, and then fusing them to form the weight of the final expert model;

[0090] Q=Wh h,K=W r r (15)

[0091] In summary, the probability value of each expert is obtained, which will be used in the subsequent mixing process.

[0092] A further technical solution is to dynamically select a routing mechanism based on the input triples and calculate the weight of the expert model. The method for selecting the routing mechanism includes the following steps:

[0093] Step 31: Conduct experimental verification for triple data of each relationship type, select the optimal routing mechanism for the triple based on the relationship type, and construct a correspondence table between the triple relationship type and the routing mechanism;

[0094] Specifically, we compare different routing mechanisms for knowledge graph representation learning, and use the downstream task of link prediction to conduct test experiments, and calculate the index data, where the index data can use AR and H@N indicators;

[0095] Link prediction is a technique for predicting possible connections between unconnected nodes in a network graph (such as a social network or knowledge graph).

[0096] Downstream tasks are specific application tasks based on link prediction results, such as:

[0097] Recommendation system: recommends items of possible interest based on link prediction;

[0098] Knowledge reasoning: predicting possible facts or relationships in a knowledge graph;

[0099] Social network analysis: predicting potential friendships or community connections;

[0100] For a specific example, experiments were conducted on triplets of many-to-many relationships. Six different routing mechanisms were tested, and the results are shown in Table 1. The top two groups show the different routing mechanisms, while the bottom two groups focus on topN selection. The results in the table show that each gate routing mechanism achieves better results than its corresponding attention routing mechanism. Among the gate routing mechanisms, the order of sparse, dense, and multiple routing is dense > sparse > multiple. Therefore, the dense gate routing mechanism is selected as the optimal routing mechanism.

[0101] Table 1

[0102]

[0103] Step 32: for the obtained triple (h, r, t) to be processed, determine the relationship type of the triple by counting the entity connectivity;

[0104] Specifically, analyze the number of connections between each entity h and t, and calculate the number of tail entities h is connected to through relationship r (degree(h,r)). Determine the relationship type based on the number of connected tail entities.

[0105] degree(h,r)=1 and degree(t,r)=1, it is a one-to-one relationship;

[0106] If degree(h,r)>1 and degree(t,r)=1, it is a one-to-many relationship;

[0107] degree(h,r)=1 and degree(t,r)>1, it is a many-to-one relationship;

[0108] If degree(h,r)>1 and degree(t,r)>1, it is a many-to-many relationship;

[0109] Step 33: Select a routing mechanism corresponding to the triple to be processed according to the relationship type of the triple to be processed and the correspondence table between the triple relationship type and the routing mechanism;

[0110] In step 4, the routing mechanism is used to mix the representation learning experts and obtain the mixed similarity score β of each triple (h, r, t) m (.), if the routing mechanism is gate routing, the weighted formula is:

[0111]

[0112] Where, “.” represents the dot product similarity function, T∈E;

[0113] above Figure 1 In the training of the hybrid expert algorithm model, the discriminative embedding is generated by contrastive learning in the pre-training stage to obtain the pre-trained model. In the fine-tuning stage, the model is optimized through link prediction to improve the performance of representation learning.

[0114] A method for pre-training a hybrid expert algorithm model in a feasible implementation manner includes the following steps:

[0115] Step S11, obtaining a knowledge graph training dataset;

[0116] Step S12: construct positive samples and negative samples for the triples in the data set;

[0117] Specifically, positive samples are constructed by connecting the head entity h to multiple tail entities through relations r, or by connecting the head entity h to the tail entity t through multiple relations;

[0118] Connect the head entity h to multiple tail entities through the relationship r to obtain multiple positive samples pe , expressed as:

[0119]

[0120] Connect the head entity h to the tail entity t through multiple relationships to obtain multiple positive samples p r , expressed as:

[0121]

[0122] Specifically, negative sample construction includes:

[0123] Negative entity samples: Randomly select negative samples from candidate entities that do not meet the true triple conditions as negative entity samples

[0124]

[0125] Where: t j is a negative entity selected from the candidate entity list ε-ε(h,r); ε(h,r) is the entity list of the true answer (tail entity); that is, if the triple Then t i ∈ε(h,r);

[0126] Negative relationship samples: Randomly select negative samples from candidate relationships that do not meet the true triple conditions as negative relationship samples

[0127]

[0128] Among them, the negative relationship From the candidate relationship list Select from is a list of relations that satisfy the condition, i.e. if the triple but

[0129] Step S13: input the constructed positive samples and negative samples into the hybrid expert algorithm model respectively, capture the discriminative features of entities and relationships through contrastive learning and minimize the objective function of contrastive learning, optimize the model parameters, and obtain the pre-trained hybrid expert algorithm model;

[0130] Among them, the objective function of contrastive learning is defined as:

[0131]

[0132] Where τ is the temperature hyperparameter, n + represents the positive sample, n - represents a positive sample;

[0133] A feasible implementation method for fine-tuning a pre-trained hybrid expert algorithm model includes the following steps:

[0134] Step S21: define a fine-tuning task and use link prediction as a downstream task in the fine-tuning phase;

[0135] Step S22: Match the embeddings of entities and relations with the embeddings of all other entities to obtain the current prediction value.

[0136] For a test triple (h, r, t), the embeddings of entity h and relation r are Match with the embeddings of all entities and calculate the prediction value using the routing mechanism Take gate routing as an example to calculate weight:

[0137]

[0138] Among them, E i =[E i ;E g ] is the concatenation of the i-th expert embedding and the global embedding for all entities.

[0139] Step S23: Use a gradient descent algorithm (such as Adam optimizer) to update the model parameters and gradually minimize the loss function L FT , get the fine-tuned hybrid expert algorithm model;

[0140] When predicting the head entity h based on a pair (r, t), the method used is to reverse the relationship and obtain the reverse triple (t, r rev ,h) make predictions.

[0141] The binary cross entropy is used as the loss function to optimize the model parameters. The loss function is as follows:

[0142]

[0143] Among them, if t j ∈ε(h,r), then Y j =1, otherwise Y j = 0, ε(h,r) is the set of all true tail entities of a pair (h,r). When predicting the head entity h based on a pair (r,t), the relationship is reversed to obtain the reverse ternary (t,r rev ,h).

[0144] In this example, we introduce a hybrid expert model into the field of knowledge graph representation learning and propose a representation learning method that adapts to different relationship types. We design multiple routing mechanisms (including gate routing and attention routing) to select the optimal TopN expert model and assign appropriate models to different relationships.

[0145] In order to illustrate the effect of the method of this embodiment, a simulation experiment was carried out, which is described in detail as follows:

[0146] This example uses the ReVerb20K and ReVerb45K OpenKG benchmark test sets, which are built using ReVerb. The data statistics of the datasets are shown in Table 2. Ent, Rel, and cluster represent the number of entities, relations, and entity sets. Valid, Test, and Train represent the number of triples in the validation set, test set, and training set, respectively. ReVerb45K, with 27K entities and 21.6K relations, is larger and more sparse than ReVerb20K, with 11.1K entities and 11.1K relations. Entity clusters are gold-normalized clusters extracted using the Freebase entity linking tool. Entities in an entity cluster have the same meaning. Typically, entity clusters are only used for evaluation.

[0147] Table 2

[0148]

[0149] To evaluate on a single test triple, this example uses Mention Ranking, or MR for short, which refers to the minimum ranking position of the answer entity. To evaluate all test triples, this example uses three of the most widely used methods to integrate a single MR score:

[0150] (a) H@N: proportion of MR scores not higher than N;

[0151] (b) AR: the average of all MR scores;

[0152] (c) ARR: Calculate the inverse of each MR score and average all the inverses. A well-performing model should have higher H@N and ARR scores and lower AR scores.

[0153] To illustrate the effectiveness of the method of this embodiment, the model of this embodiment (abbreviated as: MOEE_PFT) is compared with some strong baselines, which are divided into three groups:

[0154] General: Uses general knowledge graph models to capture the structural features of open knowledge graphs, including TransE, DistMult, ComplEx, ConvE, and ConvTransE.

[0155] OpenKG: A baseline designed specifically for open knowledge graphs to capture structural and textual features. Includes CaReTransE, CaReConv, SimKG, OKGITBert, and OKGITRob.

[0156] Single: This includes a single baseline where each model has a kge model and TextEnc, for example, TTransE = TransE + TextEnc. It includes TDistMult, TComplEx, TTransE, TConvE, and TConvE_PFT, where _PFT indicates both a pre-training phase and a fine-tuning phase, otherwise only a fine-tuning phase.

[0157] Table 3 shows the experimental results of the algorithm of this embodiment and the baselines on the ReVerb20K and ReVerb45K datasets. The results in Table 3 show that the algorithm of this embodiment achieves substantial improvements over any baseline: First, this algorithm significantly outperforms any general method that captures structural features using only a single formula function; second, this algorithm also outperforms open knowledge graph models that capture both structural and textual features, including language enhancement methods (OKGITBert and OKGITRob); finally, this algorithm achieves significant performance improvements over any single baseline.

[0158] Table 3 Experimental results of the model of this embodiment and the existing baseline on the ReVerb20K and ReVerb45K datasets

[0159]

[0160] In Table 2, the explanations of each algorithm are as follows:

[0161] ConvTransE, or Convolutional TransE, combines the ideas of ConvE and TransE, using convolutional networks to process entity relationship embedding while retaining the translation characteristics of TransE.

[0162] CaReTransE: Canonicalization-infused Representations TransE is a variant of the CaRe (Canonicalization-infused Representations) framework that uses TransE as the base model. It aims to improve entity and relation embeddings in open knowledge graphs by integrating normalization information.

[0163] CaReConv: Canonicalization-infused Representations Conv, similar to CaReTransE, but the base model uses ConvE;

[0164] SimKG: Similarity-based Knowledge Graph, a similarity-based approach to processing open knowledge graphs, focusing on semantic similarity and matching of entities and relations. SimKGC: Simple Contrastive Knowledge Graph Completion, a text-based knowledge graph completion method that leverages a pre-trained language model to learn embedding representations of entities and relations.

[0165] OKGITBert: Open Knowledge Graph Integration with BERT, using BERT as a text embedding model to combine textual information with the Open Knowledge Graph structure, capturing both structural and textual features. Open Knowledge Graph Link Prediction with Implicit Types using BERT, leveraging a pre-trained BERT model to capture implicit type information of entities.

[0166] OKGITRob: Open Knowledge Graph Integration with RoBERTa is similar to OKGITBert, but uses RoBERTa as the text embedding model. RoBERTa is an improved version of BERT, capable of capturing better text semantic information. Open Knowledge Graph Link Prediction with Implicit Types using RoBERTa is similar to OKGITBert, using the RoBERTa model instead of BERT.

[0167] TTransE: TransE+TextEnc, which combines the TransE model with a text encoder (Text Encoder, TextEnc) to jointly optimize structure embedding and text embedding;

[0168] TDistMult: DistMult+TextEnc, which combines the knowledge graph embedding model DistMult with the text encoder (TextEnc) to optimize the model's representation capabilities;

[0169] TComplEx: ComplEx+TextEnc, which combines the ComplEx model with the text encoder (TextEnc) to handle asymmetric relations and enhance representation capabilities through joint text embedding;

[0170] TConvE: ConvE+TextEnc, which is a combination of the convolutional neural network-based knowledge graph embedding model ConvE and the text encoder (TextEnc);

[0171] TConvE_PFT: ConvE+TextEnc (Pretrain+Finetune), an extended version of TConvE with pretraining (Pretrain) and fine-tuning (Finetune) stages.

[0172] Table 4 Results of different relation types 1-1, 1-M, M-1, MM in ReVerb20K and ReVerb45K

[0173]

[0174] From the analysis of the results (H@1 scores) of the relation categories in Table 4, it can be seen that the algorithm of this embodiment (abbreviated as MOEE in Table 3) achieved the highest scores in almost all subtasks: (1) For the 1-1 category, the performance is significantly better than the baseline method when predicting the head and tail entities and the average of the two datasets; (2) For the M-1 and 1-M categories, the performance is better than (or equal to) the baseline method when predicting the tail entity; (3) For the MM category, better results are achieved when predicting the head entity on the ReVerb20K dataset and better performance is also achieved when predicting the tail entity on the ReVerb45K dataset. Experiments show that the performance of this method is better than any single representation learning model when processing various relation types (such as 1-1, 1-M, M-1, MM).

[0175] Example 2

[0176] Based on Example 1, this embodiment provides a knowledge graph representation learning system based on a hybrid expert algorithm, including:

[0177] A global embedding module is configured to obtain a triple to be processed and perform global embedding to obtain a global embedding representation of the triple;

[0178] The expert embedding module is configured to obtain the expert embedding representation corresponding to each expert model for the processed triples, and concatenate it with the global embedding to obtain the concatenated embedding representation of each expert;

[0179] The routing module is configured to select routes for the triples to be processed based on the routing mechanism, assign models to different relations, and obtain the weight of each expert model;

[0180] The fusion module is configured to perform weighted mixing on the obtained connection embedding representations based on the weights obtained by routing selection to generate the final triplet embedding.

[0181] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0182] Example 3

[0183] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, the steps in the knowledge graph representation learning method based on the hybrid expert algorithm of Example 1 are completed.

[0184] Example 4

[0185] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps in the knowledge graph representation learning method based on the hybrid expert algorithm of Example 1 are completed.

[0186] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

[0187] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Those skilled in the art should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A knowledge graph representation learning method based on a hybrid expert algorithm, characterized by: The steps include: Get the triple to be processed, perform global embedding to obtain the global embedding representation of the triple; For the triples to be processed, the expert embedding representation corresponding to each expert model is obtained, and it is concatenated with the global embedding to obtain the concatenated embedding representation of each expert; For the triples to be processed, routing is selected based on the routing mechanism, models are assigned to different relations, and the weight of each expert model is obtained; The obtained connection embedding representation is weighted mixed based on the weights obtained from the routing selection to generate the final triple embedding; Routing mechanisms include gate routing and attention routing; Gate routing mechanisms include sparse gate routing, dense gate routing, and multiple gate routing; Dense gate routing: Calculate weights for all expert models and assign a weight to each expert; Sparse gate routing: select the N expert models with the highest weights, and reset the remaining expert weights to zero; Multi-gate routing: Different gate networks are designed for entities and relationships respectively, and the independent weights of the head entity and relationship are calculated and then integrated to form the final weight of the expert model; Attention routing, which calculates the relevance weight of each expert model with the current triplet through the attention network, including sparse attention, dense attention routing, and multiple attention routing; Dense attention routing: weights are calculated for all expert models; Sparse attention routing: assign weights to the top N experts with the highest relevance; Multiple attention routing, designing different attention mechanisms for entities and relations respectively, calculating the independent weights of the head entity and relationship respectively and then fusing them to form the final weight of the expert model.

2. The knowledge graph representation learning method based on the hybrid expert algorithm according to claim 1, characterized in that: It also includes building a hybrid expert algorithm model, including multiple expert models and gating mechanism modules; selecting multiple representation learning models as expert models to capture different types of relationship features; and selecting and fusing expert models through the gating mechanism module.

3. The knowledge graph representation learning method based on the hybrid expert algorithm according to claim 1, characterized in that: Representation learning models include TextEnc model, TransE model, DistMult model, ComplEx model or / and ConvE model.

4. The knowledge graph representation learning method based on hybrid expert algorithm according to claim 1, characterized in that: Based on the input triples, a routing mechanism is dynamically selected and the weight of the expert model is calculated. The routing mechanism selection method includes the following steps: Conduct experimental verification for triple data of each relationship type, select the optimal routing mechanism for the triple based on the relationship type, and build a correspondence table between triple relationship types and routing mechanisms; For the obtained triples to be processed, the relationship type of the triples is determined by counting the entity connectivity; According to the relationship type of the triple to be processed and the correspondence table between the triple relationship type and the routing mechanism, a routing mechanism corresponding to the triple to be processed is selected.

5. The knowledge graph representation learning method based on the hybrid expert algorithm according to claim 2, characterized in that: The method for pre-training the hybrid expert algorithm model includes the following steps: Obtain the knowledge graph training dataset; For the triplets in the dataset, construct positive samples and negative samples respectively; Positive sample construction: connect the head entity h to multiple tail entities through the relationship r; or connect the head entity h to the tail entity t through multiple relationships; Randomly select negative samples from candidate entities that do not meet the true triple conditions; or randomly select negative samples from candidate relations that do not meet the true triple conditions; The constructed positive samples and negative samples are input into the hybrid expert algorithm model respectively. The discriminative features of entities and relations are captured through contrastive learning and the objective function of contrastive learning is minimized. The model parameters are optimized to obtain the pre-trained hybrid expert algorithm model. The method for fine-tuning the pre-trained hybrid expert algorithm model includes the following steps: Define the fine-tuning task and use link prediction as the downstream task of the fine-tuning stage; Match the embeddings of entities and relations with the embeddings of all other entities to get the current prediction value; The gradient descent algorithm is used to update the model parameters, gradually minimize the loss function, and obtain the fine-tuned hybrid expert algorithm model.

6. Knowledge graph representation learning system based on hybrid expert algorithm, characterized by: include: A global embedding module is configured to obtain a triple to be processed and perform global embedding to obtain a global embedding representation of the triple; The expert embedding module is configured to obtain the expert embedding representation corresponding to each expert model for the processed triples, and concatenate it with the global embedding to obtain the concatenated embedding representation of each expert; The routing module is configured to select routes for the triples to be processed based on the routing mechanism, assign models to different relations, and obtain the weight of each expert model; The fusion module is configured to perform weighted mixing on the obtained connection embedding representation based on the weights obtained by routing selection to generate the final triplet embedding; Routing mechanisms include gate routing and attention routing; Gate routing mechanisms include sparse gate routing, dense gate routing, and multiple gate routing; Dense gate routing: Calculate weights for all expert models and assign a weight to each expert; Sparse gate routing: select the N expert models with the highest weights, and reset the remaining expert weights to zero; Multi-gate routing: Different gate networks are designed for entities and relationships respectively, and the independent weights of the head entity and relationship are calculated and then integrated to form the final weight of the expert model; Attention routing, which calculates the relevance weight of each expert model with the current triplet through the attention network, including sparse attention, dense attention routing, and multiple attention routing; Dense attention routing: weights are calculated for all expert models; Sparse attention routing: assign weights to the top N experts with the highest relevance; Multiple attention routing, designing different attention mechanisms for entities and relations respectively, calculating the independent weights of the head entity and relationship respectively and then fusing them to form the final weight of the expert model.

7. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the knowledge graph representation learning method based on the hybrid expert algorithm described in any one of claims 1 to 5 are completed.

8. A computer-readable storage medium, characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of the knowledge graph representation learning method based on a hybrid expert algorithm as described in any one of claims 1 to 5.

Images