Knowledge graph representation learning framework optimization method based on key embedding

Abstract

The present invention discloses a knowledge graph representation learning framework optimization method based on key embedding, including: initializing a key embedding cache table, after which the key embedding cache table is empty, the key embedding cache table includes a first column and a second column, the first column identifies the ID of an entity or relationship in a subgraph, and the second column is the embedding of the entity or relationship; using preloading and filtering to dynamically construct the key embedding cache table during training iterations; using a constant partial delay or a dynamic partial delay method to update the embedding of entities and relationships in the subgraph stored in the working node using the key embedding cache table; pushing the updated embedding back to the parameter server. The present invention designs a representation learning framework optimization method, and in the knowledge graph representation learning training process, dynamically stores and updates high-frequency hit entities and relationships as key embeddings. At the cost of increasing a small amount of computing time, the communication cost of obtaining key embeddings is reduced, thereby improving the overall training efficiency.

Description

Technical Field

[0001] The present invention relates to the field of knowledge graphs, and more specifically, to the field of knowledge graph representation learning. Background Art

[0002] Knowledge graph is an important cornerstone for realizing the new generation of cognitive artificial intelligence. With the rapid development of artificial intelligence, knowledge graph has been widely used in many fields. The scale of knowledge graph data is increasing rapidly, with a scale of millions of nodes (10 6 ) and hundreds of millions of edges (10 8 ) knowledge graphs are widely present in various fields, with a scale of tens of millions of nodes (10 7 ) and one billion edges (10 9 ) are also beginning to appear.

[0003] In the semantic Web community, Resource Description Framework (RDF) is a set of technical specifications for markup languages ​​proposed by the World Wide Web Consortium (W3C). It is widely used to describe the characteristics of Web resources due to its simplicity and extensibility. RDF is developed based on XML syntax and XML Schema data types, which allows it to more richly describe and express the content and structure of resources. It has gradually developed into a general data model for representing various types of data and is also widely used to represent various domain knowledge. RDF triples are represented as (S, P, O), where S represents the subject, P represents the predicate, and O represents the object. The triple represents that the subject and the object are connected by the predicate to become an existing fact. Due to the simplicity and ease of use of RDF, many fields use RDF data models to manage and publish domain knowledge, such as social networks, bioinformatics, etc.

[0004] Since knowledge graphs are high-dimensional graph data, knowledge graph data represented by the RDF data model is difficult to be directly applied to machine learning tasks. To solve this problem, the current mainstream method is the knowledge graph representation learning algorithm, whose key idea is to embed high-dimensional and sparse knowledge graph data containing entities and relationships into a low-dimensional continuous vector space to simplify operations while retaining the inherent structure of the knowledge graph. The knowledge vectors obtained by the representation learning algorithm can be widely used in real-world tasks, such as semantic parsing, information extraction, knowledge question answering, and recommendation systems.

[0005] Knowledge graph representation learning models have shown high accuracy on small-scale benchmark datasets such as FB15k and WN18. However, the growing size of knowledge graphs poses new challenges to the performance of existing representation learning models. Most current models have large parameter sizes and high computational complexity, and the existing knowledge graph representation learning frameworks lack computing power. Therefore, there is an urgent need for an efficient, distributed and parallel knowledge graph representation learning framework optimization method to achieve efficient representation learning of large-scale knowledge graph data. Summary of the invention

[0006] In view of the above-mentioned prior art, the present invention proposes a knowledge graph representation learning framework optimization method based on key embedding, called HotKE. In combination with the characteristics of the knowledge graph representation learning framework, the cost model and optimization scheme of the distributed training process are designed to balance the computational cost and communication overhead. In the process of knowledge graph distributed representation learning, on the premise of not reducing the accuracy of the model, an embedding cache table is introduced to dynamically select high-frequency hit entities and relationships, extract and store the embeddings of these entities and relationships as key embeddings, so as to reduce the communication cost of obtaining key embeddings. By dynamically updating the embedding cache table, it is ensured that the difference between the local embedding cache and the global embedding can be limited to a given threshold. At the cost of a small increase in computing time, the communication time is significantly reduced, thereby improving the overall training efficiency.

[0007] In order to solve the above technical problems, the present invention proposes a knowledge graph representation learning framework optimization method based on key embedding, comprising the following steps:

[0008] Step 1: Initialize a key embedding cache table. After initialization, the key embedding cache table is empty. The key embedding cache table includes a first column and a second column. The first column identifies the ID of an entity or relationship in a subgraph, and the second column is the embedding of the entity or relationship.

[0009] Step 2: dynamically construct the key embedding cache table during the training iteration process by using preloading and filtering;

[0010] Step 3: Using a constant partial delay or a dynamic partial delay, the key embedding cache table is used to update the embeddings of entities and relationships in the working node storage subgraph;

[0011] Step 4: Push the updated embedding back to the parameter server.

[0012] Furthermore, the knowledge graph representation learning framework optimization method described in the present invention comprises:

[0013] In step 1, when the training process of knowledge graph representation learning begins, the knowledge graph is partitioned by a graph partitioning algorithm, and each working node stores a subgraph including some entities and relationships of the knowledge graph; based on the partitioning results, the parameter server on each working node initializes the embedding.

[0014] In step 2, the preloading process is as follows: the working node samples from its own subgraph to obtain a positive triple sample set, and generates a negative triple sample set; the working node removes duplicate entities and relationships contained in the positive triple sample set and the negative triple sample set, and then stores the entity and relationship lists as preloading results;

[0015] In step 2, the filtering process is that the working node calculates the frequency of entities and relationships in the preloaded list in the subgraph, arranges them in descending order of the frequency of each entity and relationship, selects high-frequency entities and relationships in the preloaded list, and pulls the embeddings corresponding to the high-frequency entities and relationships from the parameter server as key entity and relationship embeddings, thereby constructing the key embedding cache table.

[0016] In step 2, the high-frequency entities and relationships in the preloaded list refer to the top 1% of entities and relationships in the table.

[0017] In step three, a constant partial delay method is used to update the embeddings of entities and relationships in the subgraph stored in the working node using the key embedding cache table. The process is: the working node iterates for several rounds to train the embeddings of entities and relationships, and uses the constructed key embedding cache table to perform training optimization; if the entities and relationships used for training appear in the key embedding cache table, directly obtain their corresponding embeddings; if the entities and relationships used for training are not in the relationship embedding cache table, pull the corresponding embeddings of the entities and relationships from the parameter server; use the obtained embeddings to calculate the loss function and update the embeddings of the entities and relationships in the subgraph stored in the working node.

[0018] In step three, a dynamic partial delay method is adopted to update the embedding of entities and relationships in the subgraph stored in the working node using the key embedding cache table. The process is as follows: the working node iterates several rounds to train the embedding of entities and relationships, uses the constructed key embedding cache table to optimize the training, and dynamically adjusts the content of the key embedding cache table; if the entities and relationships used for training appear in the key embedding cache table, directly obtain their corresponding embeddings; if the entities and relationships used for training are not in the relationship embedding cache table, pull the corresponding embeddings of the entities and relationships from the parameter server; at the same time, when the training round exceeds a given threshold, dynamically adjust the embedding of entities and relationships in the key embedding cache table: pull the embeddings corresponding to the entities and relationships in the key embedding cache table at the current moment from the parameter server, and overwrite the original embeddings in the key embedding cache table with the newly acquired embeddings; use the acquired embeddings to calculate the loss function and update the embeddings of entities and relationships in the subgraph stored in the working node.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The knowledge graph representation learning framework optimization method based on key embedding proposed in the present invention expands the traditional parameter server architecture and reduces the communication overhead between working nodes by introducing a key embedding cache table based on the locality of parameter access. The present invention designs a key embedding synchronization algorithm, adopts a preloading mechanism to adaptively select key embeddings and dynamically update the cache embedding table, and reduces the communication overhead generated by parameter transmission by retaining high-frequency access embeddings, thereby improving communication efficiency; at the same time, two strategies, constant partial delay and dynamic partial delay, are designed to construct a key embedding cache table, limit the difference between cached key embeddings and global embeddings, and ensure the accuracy of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the overall architecture diagram of the HotKE method proposed in the present invention;

[0022] Figure 2 This is an example of constructing a key embedding cache table based on the CPS and DPS strategies of the present invention;

[0023] Figure 3 is a flow chart of the method proposed by the present invention when it is running on a working node;

[0024] Figure 4 It is the result of the convergence experiment of the present invention on the evaluation data set;

[0025] Figure 5 The results of the scalability experiment of the present invention in the evaluation data set;

[0026] Figure 6This is the result of the present invention testing the impact of the number of key embeddings on training in a large-scale dataset;

[0027] Figure 7 This is the result of the present invention testing the effect of synchronization period on training in large-scale data sets;

[0028] Figure 8 This is the result of the present invention testing the impact of entity key embedding ratio on training in a large-scale dataset. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention in any way.

[0030] The present invention proposes a knowledge graph representation learning framework optimization method based on key embedding, comprising the following steps:

[0031] Step 1: Initialize the key embedding cache table.

[0032] Parameter servers, working nodes and key embedding cache tables are configured on each physical machine in a distributed environment. Before the start of the training iteration, the knowledge graph is partitioned using a commonly used graph partitioning algorithm to reduce the number of edges between different working nodes. The entity and relationship embeddings are automatically divided into multiple parts and stored in the training cluster. Initialize the key embedding cache table on each working node. In the present invention, when the training process of the knowledge graph representation learning begins, the knowledge graph is partitioned by a graph partitioning algorithm, and each working node stores a subgraph including some entities and relationships of the knowledge graph; according to the partitioning results, the parameter server on each working node initializes the embedding. At this time, the key embedding cache table is empty, and the key embedding cache table contains a first column and a second column, the first column is an ID that can uniquely identify an entity or relationship in the subgraph, and the second column is the embedding of the entity or relationship.

[0033] Step 2: Construct the key embedding cache table.

[0034] Considering that the process of knowledge graph representation learning and training has a large number of cross-node embedding accesses, even after the knowledge graph is divided, the communication cost is still the main overhead of the knowledge graph representation learning training process. The key embedding cache table is adaptively constructed according to the training load. In the present invention, preloading and filtering are used to dynamically construct the key embedding cache table during the training iteration process.

[0035] 1) The preloading process is as follows: the working node samples from its own subgraph to obtain a positive triple sample set and generates a negative triple sample set; the working node deduplicates the entities and relationships contained in the positive triple sample set and the negative triple sample set and stores the entity and relationship lists as the preloading results.

[0036] 2) The filtering process uses the frequency of entities and relationships in the preloaded list as an indicator of their importance. The working node calculates the frequency of entities and relationships in the preloaded list in the subgraph, arranges them in descending order of the frequency of each entity and relationship, selects high-frequency entities and relationships in the preloaded list, and pulls the embeddings corresponding to the high-frequency entities and relationships from the parameter server as key entity and relationship embeddings, thereby constructing the key embedding cache table. Due to the heterogeneity of knowledge graph nodes, the frequency of relationships is usually higher than that of entities. Ignoring this node heterogeneity when constructing the key embedding cache table may result in a bias towards caching relationship embeddings. In order to solve this problem, the present invention fixes the ratio of entity and relationship embeddings in the key embedding cache table. In the present invention, the high-frequency entities and relationships in the preloaded list refer to the top 1% of entities and relationships in the table.

[0037] Step 3: Use the key embedding cache table to update the embeddings of entities and relationships in the working node storage subgraph.

[0038] According to the access frequency of different entity and relationship embeddings and the access locality characteristics of parameters during training, a key embedding cache table is introduced to reduce the number of network transmissions of high-frequency parameters. Based on the existing parameter server architecture, the present invention provides two optional methods for the delay strategy for updating the key embedding cache table, namely, a constant partial delay (CPS) method and a dynamic partial delay (DPS) method.

[0039] 1) In order to obtain high-quality key embeddings, the sampling process of knowledge representation learning is analyzed. When a uniform sampler generates samples from a knowledge graph, entities with higher degrees and relationships with more occurrences are more likely to be obtained. Based on this, the entire subgraph is preloaded and the access frequency of all entity and relationship embeddings is calculated. Since the memory space of each working node is limited, the highest frequency embedding vectors are screened out to build an embedding cache table. Caching high-frequency embeddings can reduce more network transmission. The key embeddings in the cache built based on the constant partial delay strategy are fixed and will not be replaced during the training process. In the present invention, the constant partial delay method is adopted, and the process of updating the embeddings of entities and relationships in the subgraph stored by the working node using the key embedding cache table is: the working node iterates for several rounds to train the embeddings of entities and relationships, and uses the constructed key embedding cache table for training optimization; if the entities and relationships used for training appear in the key embedding cache table, directly obtain their corresponding embeddings; if the entities and relationships used for training are not in the relationship embedding cache table, pull the corresponding embeddings of the entities and relationships from the parameter server; use the acquired embeddings to calculate the loss function and update the embeddings of the entities and relationships of the subgraph stored by the working node.

[0040] 2) In the constant partial delay strategy, it is assumed that the embedding access distribution in each training sample is similar to the global distribution, so the key embedding cache table is determined before training. In fact, due to the randomness of the sampling process, the key embeddings in some training samples may be those that appear less frequently in the global embedding, which makes it difficult for the cache built based on the CPS strategy to have a high hit rate in each iteration. To solve this problem, the dynamic partial delay strategy is introduced. During the training process, the DPS strategy preloads samples from several consecutive training rounds, filters the key embeddings according to the access frequency of the entity and relationship embeddings, and dynamically modifies the key embedding cache table. The DPS strategy can improve the cache hit rate because it ensures that the key embeddings reflect the access pattern of the embeddings in a shorter time and further improves the reduction of network transmission. For the CPS strategy, the key embedding cache table is fixed during the training iteration. For DPS, the key embedding cache table will be updated regularly to achieve a higher cache hit rate. In the present invention, a dynamic partial delay method is adopted, and the process of using the key embedding cache table to update the embedding of entities and relationships in the subgraph stored in the working node is: the working node iterates for several rounds to train the embedding of entities and relationships, uses the constructed key embedding cache table to optimize the training, and dynamically adjusts the content of the key embedding cache table; if the entities and relationships used for training appear in the key embedding cache table, directly obtain their corresponding embeddings, if the entities and relationships used for training are not in the relationship embedding cache table, then pull the corresponding embeddings of the entities and relationships from the parameter server; at the same time, when the training round exceeds a given threshold, the embedding of entities and relationships in the key embedding cache table is dynamically adjusted: the embeddings corresponding to the entities and relationships in the key embedding cache table at the current moment are pulled from the parameter server, and the newly obtained embeddings overwrite the original embeddings in the key embedding cache table; using the obtained embeddings, calculate the loss function, and update the embeddings of the entities and relationships in the subgraph stored in the working node.

[0041] Step 4: Push the updated embedding back to the parameter server.

[0042] Example

[0043] Figure 1 The overall architecture diagram of the knowledge graph representation learning framework optimization method based on key embedding proposed by the present invention is shown. Multiple server processes are started on each machine to achieve load balancing, and these servers access locally stored entity and relationship embeddings through shared memory. Before training starts, the knowledge graph training data is divided into several shards and sent to each machine. The parameter server is started and the model parameters are initialized. At the same time, the model parameter shards are stored in each machine respectively, and the trainer starts the process to execute the training process in parallel. Through the key embedding cache table, each working node can avoid a large amount of remote communication for key embedding.

[0044] In each iteration, the training process based on key embeddings includes:

[0045] 1) Sample a set of positive samples from the local subgraph and randomly replace the entities in it to construct negative samples;

[0046] 2) Load entity and relation embeddings from the key embedding cache table, and pull the remaining embeddings required for the current mini-batch from the parameter server;

[0047] 3) Perform forward calculation and back propagation to calculate the gradients of entity and relationship embeddings;

[0048] 4) Push all embedding gradients of this iteration to the parameter server, which updates the corresponding entity and relationship embeddings.

[0049] Figure 2 Workflow examples of the two proposed partial delay strategies, namely constant partial delay and dynamic partial delay, are shown. For a training process involving 6 iterative calculations, the CPS method selects key embeddings to build a cache and uses a fixed key embedding cache table in all 6 iterations. For the DPS method with a preloading round number of 3, the worker rebuilds the key embedding cache table every 3 iterations. In rounds 4-6, DPS selects embeddings with higher access frequency as key embeddings in rounds 4-6. Compared with CPS, the dynamic construction mechanism can cache locally high-frequency but globally low-frequency embeddings, so that the average cache hit rate of DPS is higher. The key difference between the existing knowledge graph representation learning and training system and the optimization method proposed in the present invention lies in the stage of pulling the embedding.

[0050] Considering that the scale of large-scale knowledge graphs can often reach tens of millions, while the batch size used in the training process is usually only thousands, the parameters of a single update are sparse with respect to the global parameters. Therefore, knowledge representation learning is not sensitive to parameter delays and can be further optimized by utilizing the characteristics of sparse updates. In order to reduce training time without sacrificing accuracy, efforts are made to maximize the access to local key embeddings and reduce the inconsistency between local key embeddings and their copies on parameter servers. Based on this goal, a key embedding synchronization algorithm is proposed for the distributed training process, which periodically synchronizes the key embeddings with the parameter server to prevent the expiration of the key embeddings from exceeding a given threshold. The specific process is:

[0051] 1) In the partial delay strategy, worker nodes asynchronously execute distributed training tasks. The knowledge graph is partitioned and samples are preloaded from the subgraph to build the key embedding cache table. During the iteration process, the latest version of the key embedding is obtained from the parameter server to update the cache.

[0052] 2) Subsequently, the worker node loads samples from the preload list, obtains embeddings from the cache and parameter server, and then calculates the loss function, updates the embedding value and pushes it back to the parameter server. The inconsistency between the key embedding cache table of the worker node and the global embedding in the parameter server will sacrifice accuracy.

[0053] 3) Before training begins, the worker node builds a key embedding cache table and synchronizes the latest version of key embeddings from the parameter server, and uses the key embeddings in subsequent training. During this process, even if these key embeddings are updated by other worker nodes, the current worker node will still use the key embeddings pulled during construction for training. In order to alleviate the problem of parameter inconsistency in asynchronous training, a bounded delay technique is introduced to constrain the inconsistency of key embeddings.

[0054] 4) Bounded delay technology uses lightweight synchronization to alleviate the problem of parameter inconsistency during the convergence process and has been widely used in distributed training algorithms and distributed systems. The key embedding cache table is synchronized every several rounds to reduce the difference between the key embedding cache table and the global embedding.

[0055] Figure 3 The flowchart of the distributed training and synchronization algorithm based on key embedding is shown. For a worker node, before training, the worker node divides the knowledge graph G into n subgraphs. When the iteration reaches the set preloading threshold D, the worker node i is loaded from the subgraph G. i Preload samples, entities, and relations in the preloaded list, and then build a key embedding cache table. When the number of iterations reaches the specified synchronization threshold K, the latest version of the key embedding is obtained from the parameter server to update the cache. Subsequently, the worker node loads samples from the preloaded list, obtains embeddings from the cache and parameter server, and then calculates the loss function. When the loss value When the update condition is met, the worker node calculates the gradient of the embedding through back propagation, and then pushes the gradient back to the parameter server. For the parameter server, methods for embedding parameter and gradient transmission are provided. During the training process, the parameter server continuously obtains elements in the message queue, uses the optimizer to update the embedding, or pushes the embedding to the target worker node.

[0056] Table 1 shows the experimental results of the proposed method (HotKE) on three datasets for a commonly used knowledge graph representation learning evaluation task, namely, the link prediction task.

[0057]

[0058] Two of the most advanced distributed graph representation learning systems, PyTorch-BigGraph (PBG) and DGL-KE, were selected as comparative benchmarks. All experiments were completed on a distributed cluster consisting of 4 machines, and all systems only used CPU for calculation without GPU. The experimental data sets were the knowledge representation learning benchmark data sets FB15K, WN18 and the large-scale real knowledge graph Freebase-86m. The knowledge representation learning model used for comparison adopted the more adaptable TransE model, and the evaluation indicators were all set as filter settings. On the benchmark data sets FB15K and WN18, DGL-KE and HotKE can achieve comparable training accuracy, slightly higher than PBG. Among them, the Hits@10 index is more than 0.8, indicating that the model trained by distributed knowledge representation learning can well predict the missing entities in the triples. In the link prediction task, the top ten triples in the candidate list have a probability of containing the correct result of more than 80%. On the large-scale knowledge graph Freebase-86m dataset, PBG, DGL-KE, HotKE-C and HotKE-D (two partial delay strategies) can achieve comparable accuracy. The training time of HotKE-C and HotKE-D is shorter than that of PBG and DGL-KE, which shows that the optimization scheme based on key embedding can effectively improve the training efficiency. The training time of HotKE-D is shorter than that of HotKE-C, indicating that the dynamic construction of the key embedding cache table can further improve the training efficiency compared with the fixed method. The results verify that the framework optimization method of the present invention can achieve comparable accuracy with the existing most advanced systems PBG and DGL-KE.

[0059] Figure 4 The convergence of each training framework on the FB15K, WN18 and Freebase-86m datasets is shown. The following conclusions can be drawn from the line graph: PBG, DGL-KE and HotKE can all converge smoothly, which shows that in a distributed environment, parallel training by multiple computing nodes does not significantly affect the convergence of the model. On datasets of different sizes, the training convergence speed of the framework optimized by the HotKE method is faster than that of PBG and DGL-KE, which shows that the distributed training algorithm based on key embedding can effectively reduce the system training time while ensuring the correctness of the algorithm. At the same time, on the large-scale dataset Freebase-86m, the training efficiency of HotKE is more significantly improved. This is because as the scale of the knowledge graph increases, the proportion of network communication time will increase, and the optimization effect on network communication will be more significant. It is verified that HotKE can achieve accuracy comparable to the existing state-of-the-art systems PBG and DGL-KE, while having higher training efficiency.

[0060] Figure 5 The speedup ratio of each training framework varies with the number of worker nodes and the training time required for each distributed framework on the dataset is shown. As the number of worker nodes increases, the speedup ratios of PBG, DGL-KE and HotKE all increase. When the number of worker nodes reaches 16, the speedup ratio tends to be stable, which shows that increasing the number of worker nodes can effectively reduce the training time. However, as the number of worker nodes increases, network communication gradually becomes a bottleneck restricting system performance. The speedup ratio of HotKE is higher than that of the benchmark methods PBG and DGL-KE, which verifies the effectiveness of the optimization strategy based on key embedding. At the same time, the speedup ratio of HotKE-D is higher than that of HotKE-C, which reflects the superiority of the dynamic partial delay strategy. On the large-scale knowledge graph Freebase-86m, the communication time of all distributed training frameworks accounts for more than 80%. The computing time of the HotKE framework is comparable to that of PBG and DGL-KE, and the communication time is reduced by 76%, and the total training time is reduced by 73%, which shows that the HotKE optimization strategy can effectively reduce the network communication time in the distributed training process and improve the training efficiency. The above results demonstrate that the HotKE framework has better scalability than the baseline methods PBG and DGL-KE.

[0061] Figure 6 The effect of the number of key embeddings K on the training results is shown. As the number of key embeddings k increases, the model accuracy does not change significantly, which indicates that when the number of key embeddings is small, using old embeddings for training will not have a significant impact on the model accuracy. As the cache space increases, the number of embeddings that need to be pulled remotely in each round of training decreases, which indicates that caching key embeddings for training on working nodes can reduce network communication.

[0062] Figure 7 The effect of the synchronization period on the training results is shown. As the synchronization period K increases, the average parameter transfer amount in each round of training decreases. When the synchronization period K < 16, the model accuracy remains almost unchanged. When the synchronization period K increases further, the model accuracy gradually decreases. This shows that the increase in the synchronization period means that the learning model will use more expired parameters for training, which will have an adverse effect on the accuracy. The above experimental results show that the model accuracy and training efficiency can be balanced by adjusting the synchronization period K. When K = 8, the framework optimized by the HotKE method performs best, and can reduce more training time without sacrificing accuracy.

[0063] Figure 8The influence of the content of key embeddings on the training results is shown. Under the premise that the number of key embeddings remains unchanged, the training accuracy of the model is not affected by increasing the proportion of entity embeddings, but the amount of parameter transmission in each round of training gradually increases, which indicates that the access to relation embeddings is more intensive than that to entity embeddings. Storing more relation embeddings in the key embedding cache table can achieve better optimization results.

[0064] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many modifications without departing from the purpose of the present invention, all of which are within the protection of the present invention.

Claims

1. A knowledge graph representation learning framework optimization method based on key embedding, characterized in that: include: Step 1: Initialize a key embedding cache table. After initialization, the key embedding cache table is empty. The key embedding cache table includes a first column and a second column. The first column identifies the ID of an entity or relationship in a subgraph, and the second column is the embedding of the entity or relationship. Step 2: dynamically construct the key embedding cache table during the training iteration process by using preloading and filtering; The preloading process is as follows: the working node samples from its own subgraph to obtain a positive triple sample set and generates a negative triple sample set; the working node removes duplicate entities and relationships contained in the positive triple sample set and the negative triple sample set and stores the entity and relationship list as the preloading result; The filtering process is as follows: the working node calculates the frequency of entities and relationships in the preloaded list in the subgraph, sorts them in descending order of the frequency of each entity and relationship, selects high-frequency entities and relationships in the preloaded list, and pulls the embeddings corresponding to the high-frequency entities and relationships from the parameter server as key entity and relationship embeddings, thereby constructing the key embedding cache table; the high-frequency entities and relationships in the preloaded list refer to the top 1% of entities and relationships in the table; Step 3: Using a constant partial delay or a dynamic partial delay, the key embedding cache table is used to update the embeddings of entities and relationships in the working node storage subgraph; Step 4: Push the updated embedding back to the parameter server.

2. The knowledge graph representation learning framework optimization method according to claim 1, characterized in that: In step 1, when the training process of knowledge graph representation learning begins, the knowledge graph is partitioned by a graph partitioning algorithm, and each working node stores a subgraph including some entities and relationships of the knowledge graph; based on the partitioning results, the parameter server on each working node initializes the embedding.

3. The knowledge graph representation learning framework optimization method according to claim 1, characterized in that: In step 3, the process of updating the embeddings of entities and relationships in the working node storage subgraph using the key embedding cache table by using a constant partial delay method is: The working node iterates several rounds to train the embeddings of entities and relations, and uses the constructed key embedding cache table to optimize the training; if the entities and relations used for training appear in the key embedding cache table, their corresponding embeddings are directly obtained; If the entities and relations used for training are not in the relation embedding cache table, the corresponding embeddings of the entities and relations are pulled from the parameter server; Using the acquired embeddings, calculate the loss function and update the embeddings of entities and relations of the subgraph stored by the worker nodes.

4. The knowledge graph representation learning framework optimization method according to claim 1, characterized in that: In step 3, the process of using the key embedding cache table to update the embedding of entities and relationships in the working node storage subgraph using the dynamic partial delay method is: The working node iterates several rounds to train the embeddings of entities and relationships, uses the constructed key embedding cache table to optimize the training, and dynamically adjusts the content of the key embedding cache table; if the entities and relationships used for training appear in the key embedding cache table, directly obtain their corresponding embeddings; if the entities and relationships used for training are not in the relationship embedding cache table, pull the corresponding embeddings of the entities and relationships from the parameter server; at the same time, when the training round exceeds a given threshold, dynamically adjust the embeddings of the entities and relationships in the key embedding cache table: pull the embeddings corresponding to the entities and relationships in the key embedding cache table at the current moment from the parameter server, and overwrite the original embeddings in the key embedding cache table with the newly acquired embeddings; Using the acquired embeddings, calculate the loss function and update the embeddings of entities and relations of the subgraph stored by the worker nodes.

Images