A knowledge graph completion method that integrates weights and temporal information

Abstract

The application discloses a kind of fusion weight and time information's knowledge graph completion method, it is related to natural language processing application technical field, the method is as follows: processing time sequence knowledge graph quadruple content;In weight learning module, with the aid of weight calculation mode, pre-training encoder and attention mechanism, perfect data embedding representation;Combined with text information processing mode, improve the ability of attention mechanism to capture changes in things, build time sequence attention mechanism module;Fusion two module, learn knowledge graph original data together, based on training data reasoning test data part, obtain the model capable of reasoning and completing the missing content of knowledge graph, can improve the integrity of the generated knowledge graph, and then drive to improve the correctness of representation, reduce error rate, improve the performance efficiency of the effect of time information knowledge graph completion task.

Description

Technical Field

[0001] This invention relates to the field of natural language processing application technology, and more specifically, to a knowledge graph completion method that integrates weight and temporal information. Background Technology

[0002] Knowledge graphs represent world knowledge as structured triples, serving as an important information storage method. However, most existing knowledge graphs face significant incompleteness due to various reasons. For example, in the Freebase knowledge graph, many facts lack temporal information, which limits the application of knowledge graphs in downstream artificial intelligence tasks such as machine translation and intelligent question answering.

[0003] To alleviate the problem of incomplete knowledge graphs, the task of automatic knowledge graph completion has been proposed. Knowledge Graph Completion (KGC) aims to predict and add missing knowledge from the knowledge graph based on existing knowledge, making the knowledge graph more complete. KGC provides more comprehensive knowledge content for downstream applications of knowledge graphs (such as question answering, relation prediction, text translation, and multimodal information processing).

[0004] As things in the real world develop, the facts recorded in knowledge graphs should also change over time, exhibiting a dynamic evolutionary nature to better align with natural laws. Since 2016, research on Temporal Knowledge Graphs (TKGs) containing temporal information has gradually unfolded.

[0005] Knowledge graphs contain various data types, with complex attributes for nodes and relationships. However, when performing knowledge graph completion tasks, it is difficult to effectively obtain local neighbor information of the target node to obtain a suitable data embedding representation. At the same time, because facts change over time, the trends of mutual influence and joint change among facts are difficult to capture, and the complex interaction relationships among massive amounts of data further increase the difficulty of the completion task. Summary of the Invention

[0006] This invention provides a knowledge graph completion method that integrates weight and temporal information. By collecting weight information and temporal change information, the performance of the completion task is optimized, thereby improving the completeness and reliability of the knowledge graph content.

[0007] To achieve the above objectives, the present invention provides the following technical solution.

[0008] A knowledge graph completion method that integrates weights and temporal information includes:

[0009] Construct a quadruple dataset and transform the original knowledge graph into a static knowledge graph subgraph divided by time sequence. Construct quadruple data format and integrate time into the original fact triples in the form of timestamps.

[0010] The quadruple data format is: s = (p, q, r, t);

[0011] Where s represents a fact, p represents the head entity of the fact, q represents the tail entity of the fact, r represents the relationship between two entities, and t represents the time when the fact occurred, stored in the form of a timestamp; an experimental dataset based on multi-source databases is constructed, training data and test data are divided according to temporal relationship, and a portion of data from a public large dataset is used as experimental data; time nodes are set according to time order, facts are divided according to time nodes, and multiple static knowledge subgraphs are formed.

[0012] The last subgraph in the temporal sequence is used as the test data in each dataset, and the remaining subgraphs are used as training data. A pre-trained model, attention mechanism, and weight learning method are combined to perform embedding reinforcement learning on the knowledge graph data. The pre-trained model BERT is used as the encoder to encode the original knowledge graph data. An attention mechanism is used to learn the basic embedding representation of the target data. Through weight calculation, appropriate weights are assigned based on the data of mutual influence between the selected target entity and its neighboring entities (distance, number of interactions, etc.). The basic embedding representation is combined with the information already containing weights to obtain a weighted reinforced embedding representation. A temporal attention mechanism is constructed, and using the attention mechanism and borrowing from text processing methods, positional information is used to simulate the chronological order. The process involves optimizing the model's ability to infer factual changes by using an attention mechanism to learn static knowledge subgraphs with temporal order. Positional information is assigned to different subgraphs based on temporal order, and then sequentially input into the attention mechanism. Positional information is used to simulate the sequential processing of text, training the knowledge graph completion model to capture factual evolution by fusing weights and temporal information. Data sets from multiple data sources are used in the model to continuously learn knowledge graph information, optimize model parameters, and adjust model performance to obtain a well-trained knowledge graph completion model and a completed knowledge graph. Multiple datasets are used as experimental data for the model, and data from different datasets cannot be used interchangeably. The trained model is then used to train on other datasets for continuous learning, updating model parameters, and optimizing model performance.

[0013] Unlike existing knowledge graph completion methods, the knowledge graph completion method that integrates weights and temporal information provided in this invention has the following advantages:

[0014] (1) Calculate and assign appropriate weights based on the different importance of neighbors of different nodes, combine local information in an appropriate way, capture potential semantic associations and implicit structural patterns, and further improve the quality of embedding representation by obtaining the influence of higher-order neighbor relationships. Start from the acquisition of basic entity information to study the development of completion tasks, focusing on improving node embedding representation, improving representation correctness, and reducing error rate.

[0015] (2) The temporal knowledge graph is represented by a fact quadruple based on timestamps. Existing data is used to simulate the changing trends of things in the real world. The framework of temporal attention mechanism is combined with text processing and weight information fusion method to learn the facts containing time information, simulate the dynamic evolution of data, obtain the different effects of historical information, and improve the performance efficiency of the temporal information knowledge graph completion task. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the knowledge graph completion method described in this invention;

[0017] Figure 2 This is a schematic diagram of the training method for the knowledge graph completion model described in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The method provided by this invention can be implemented within common machine learning frameworks (PyTorch, Tensorflow, etc.) and can be deployed on systems such as Windows and Linux.

[0020] The purpose of this invention is to provide a training method for a knowledge graph completion model that integrates weights and temporal information for factual relationships with potential mutual influence, and to use the trained model to complete the knowledge graph. Using this model, for each missing piece of knowledge, related textual data can be retrieved, and local semantic information and temporal information can be integrated to generate the final answer, thus completing the completion of the missing knowledge.

[0021] Among them, such as Figure 1 The aforementioned knowledge graph completion method, which integrates weights and temporal information, includes:

[0022] Construct a quadruple dataset and transform the original knowledge graph into static knowledge graph subgraphs divided according to time sequence;

[0023] Construct an experimental dataset based on multi-source databases, and divide the training data and test data according to time sequence relationships;

[0024] By integrating pre-trained models with attention mechanisms and weight learning methods, we can perform embedding reinforcement learning on knowledge graph data. We construct a temporal attention mechanism and use location information to simulate the chronological order, thereby optimizing the model's ability to predict factual changes. We use datasets from multiple data sources to continuously learn knowledge graph information, optimize model parameters, and adjust model performance to obtain a well-trained knowledge graph completion model and a completed knowledge graph.

[0025] This invention also provides a knowledge graph completion method that integrates weight information, comprising: collecting time information, organizing fact triples in chronological order, and constructing a temporal knowledge graph; adding time information as timestamps to the corresponding fact triples, and forming knowledge graph subgraphs by combining facts at the same time in the form of quadruples; arranging different knowledge graph subgraphs in chronological order, requiring that the data of all subgraphs, after fusion, can reflect all the data of the original knowledge graph; using the last knowledge graph subgraph in the chronological order as test data, and the remaining temporal knowledge graph subgraphs as training data; constructing training data and test data based on the data within the knowledge graph subgraphs at different time nodes; constructing a semantic network of entity nodes and relation edges through the data in the dataset; performing initial embedding representation learning on the training data through a pre-trained model; and for the target node, calculating the distance between its indeterminate-order neighbor nodes and the target, element-level information, and the degree of influence among multiple neighbors, and calculating the weights.

[0026] An attention mechanism is used to extract information about the target node and its local neighbors to obtain an embedded representation. The initial embedded representation of the target node is combined with weight information, and neighbor information is assigned weights and fused into the embedded representation to obtain an optimized embedded representation. The attention mechanism is then used to learn the target embedded representation at different times to initially obtain real-time trends. The model is processed by simulating sequential text input, assigning corresponding temporal position weights to data at different times. The embedded representations corresponding to different times are fused according to temporal weights and then fused with the aforementioned attention mechanism to construct a temporal attention mechanism. The four-tuple form of the target fact node on the test data is extracted, and four-tuples lacking a single element are used as test problems. Training is performed separately for different target fact nodes until all training data is learned, resulting in a trained knowledge graph completion model.

[0027] The model is used for prediction tasks. The prediction performance of the model is calculated. The parameter thresholds are set according to the test standards. The training process is repeated. The model parameters are continuously adjusted according to the test results to obtain a trained knowledge graph completion model. The trained model is then used for reasoning to obtain a complete knowledge graph that meets the standards.

[0028] like Figure 2 As shown, this embodiment of the invention provides a training method for a knowledge graph completion model that integrates weights and temporal information, including:

[0029] S101, the data processing module, constructs training and testing data, performs simple classification of existing knowledge graph data, divides facts according to time sequence, and organizes time-series knowledge graph subgraphs;

[0030] S102, the weight learning module, uses a pre-trained model to preprocess the training data and uses attention mechanisms and weight calculation methods to obtain optimized embedding representations of entities and relations.

[0031] S103, the temporal prediction module, uses an attention mechanism to integrate the weights assigned by information such as temporal distance to form a temporal attention mechanism, thereby obtaining the potential factual evolution trend;

[0032] S104, Fusion Training Module, uses the training set and test set obtained from the data processing module to train the fusion method and optimize the parameter performance so that the method can infer the missing information based on the original data;

[0033] S105, the parameter optimization module, processes different datasets to obtain different training and test data, applies the model to multiple datasets for learning, and continuously learns and automatically optimizes parameters using loss functions and other methods to obtain a well-trained knowledge graph completion model.

[0034] Furthermore, in the data processing module, constructing training and testing data includes: processing the original knowledge graph, dividing facts by time, and forming a static knowledge graph subgraph based on facts at the same time; arranging all static subgraphs in chronological order to form a temporal knowledge graph; setting the subgraph corresponding to the last time in the temporal sequence as test data, and setting the remaining subgraphs as training data; for different datasets, constructing quadruples according to timestamps, and inputting the data into the model in quadruple form; training and testing data from different data sources cannot be used interchangeably.

[0035] Furthermore, in the weight learning module, the pre-trained model, attention mechanism, and weight learning include:

[0036] For the data obtained from the data processing module, an initial embedding representation is first obtained using a pre-trained model. The pre-trained model uses high-performance models such as BERT as encoders to optimize the basic representation. The triples corresponding to the target entity and their local information are fed into the attention mechanism to obtain the basic embedding representation of the target entity. The distance and node density between the target entity and its neighbors of indeterminate order are calculated, and the element-level information crossover between the target entity and its neighbors is calculated to allocate weights accordingly. Among these, a reasonable threshold is set for the neighbors of indeterminate order to determine the maximum order, thereby optimizing the model training efficiency. The order of the neighbors to be interacted with is automatically adjusted based on the interaction time between the neighbors and the target entity to accurately combine local information. The optimized embedding representation of the target entity affected by the weights is calculated, and the basic embedding representation and the optimized embedding representation are fused to obtain the entity embedding representation with fused weights.

[0037] The attention mechanism utilizes the encoder portion of the widely used Transformer framework.

[0038] Furthermore, in the temporal prediction module, the use of the attention mechanism and temporal adaptive weights includes: learning the entity embedding representation with fused weights obtained in the weight learning module using the attention mechanism to obtain partial model parameters; dividing the entity data obtained in the weight learning module according to time sequence, using position information to simulate the time sequence, and combining it with the entity representation to obtain the combined embedding representation; wherein, using position information to simulate the time sequence means assigning the position information of the earlier occurrence at a time node to the position information of the later occurrence at a time node to the position information of the later occurrence, thereby simulating the text input in sequence (from left to right) processing mode, and learning the training data in an orderly manner; inputting the embedding representation combined with position information into the above attention mechanism to construct a temporal attention mechanism, learning the temporal evolution trend, adjusting the model parameters, and obtaining the trained model; wherein, the attention mechanism will simulate the text processing order, targeting a certain entity, in a timely manner... The corresponding local weight information and positional information are embedded into the attention mechanism in a sequential manner. Positional information can be used as an identifier for the attention mechanism to identify the input order. In positional information such as (1, 2, 3...10), position 1 corresponds to the earliest time node and is input into the attention mechanism first. The data corresponding to position 10 is the last fact to be input. For a target entity at a certain time point, the temporal weight can be automatically adjusted according to the positional information. Node data that is older than the target time node is assigned a lower weight, while data that is younger and more recently occurred is assigned a higher weight. This reflects the different importance of data at different times to the target time node entity. For example, if an entity at position 8 is the target entity for learning, the data corresponding to position 10 will have a greater impact than the data corresponding to position 1. In this way, temporal information is selectively combined to simulate human reading order and characteristics, and to learn the dynamic evolution trend of facts with forgetting characteristics, so that the model can obtain predictive reasoning ability.

[0039] Furthermore, in the aforementioned fusion training module, the process of fusing the weight learning module and the temporal prediction module includes: inputting the optimized entity embedding representation obtained from the fusion weight learning module into the temporal prediction module, whereby the temporal prediction module learns and simulates the evolution trend of things, training its inference ability; inputting all training data into the weight learning module and the temporal prediction module to train the knowledge graph completion model; processing the test data separately, selecting the fact quadruple corresponding to a target entity in the test data as the proposed question, or using a knowledge graph subgraph as the proposed question for the entire test, and inputting it into the training model so that the model outputs the inferred answer based on the question; adjusting the question quadruple, improving the test question based on real data, and preventing overfitting; wherein, missing parts and incomplete quadruple information in the test data are not used as questions for testing.

[0040] Furthermore, in the parameter optimization module, the use of different datasets and loss functions for parameter optimization includes: using partial data from various large open-source datasets, such as ICEWS, GDELT, YAGO, and WIKIDATA, as experimental data for model training and performance testing; using binary cross-entropy (BCE) loss and gradient descent to optimize model parameters and performance, with significant parameter adjustments for poorly performing models; the model performance testing section uses common unified evaluation metrics (Hits@n, HR, HRR) to compare the experimental model performance with others' results and optimize accordingly; using models trained on different datasets for training on other datasets to continuously learn from different data and improve model performance; continuous learning may lead to an increase in the number of model parameters and a decrease in the performance of the completed model, so a threshold processing method is used to terminate the learning of models whose performance changes after learning different datasets exceed a threshold, while models that meet the threshold continue learning to obtain a completed knowledge graph that meets the threshold standard requirements.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0042] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for knowledge graph completion with fusion of weights and temporal information, the method being applied to semantic analysis, characterized in that, Comprise: Build a quadruple data set, and convert the original knowledge graph into a static knowledge graph subgraph divided by time sequence; Build an experimental data set based on multiple source databases, and divide the training data and test data according to the time sequence relationship; Use the pre-training model and the attention mechanism, and the weight learning method to jointly embed and strengthen the knowledge graph data; Build a time sequence attention mechanism, use the attention mechanism and learn from the text processing method, use the position information to simulate the time sequence, and optimize the ability of the model to predict the fact change; Use the data set composed of multiple data sources for the model method, continuously learn the knowledge graph information, continuously optimize the model parameters, and adjust the model performance to obtain the trained knowledge graph completion model and obtain the completed knowledge graph; The experimental data set based on multiple source databases comprises: Use part of the data in the public large data set as experimental data; According to the time sequence, set up a time node, divide the facts according to the time node, and form multiple static knowledge subgraphs; The last subgraph in the time sequence relationship is used as test data in the respective data set, and the remaining subgraphs are used as training data; For a target entity, the quadruple of the target entity in the last subgraph in the time sequence relationship and missing some information is used as a test question, the accuracy of predicting the missing information is counted, and the performance of the knowledge graph completion model is calculated; The embedding and reinforcement learning of the knowledge graph data comprises: Use the pre-training model BERT as an encoder to encode and represent the original knowledge graph data; Use the attention mechanism to perform basic embedding representation learning on the target data; Through weight calculation, appropriate weights are allocated according to the mutual influence data between the selected target entity and its neighbor entities; Combine the basic embedding representation with the information containing the weights to obtain the reinforced embedding representation containing the weights; The time sequence attention mechanism comprises: Use the attention mechanism to learn the static knowledge subgraph with time sequence arrangement; Assign position information to different subgraphs based on the time sequence, and input the above attention mechanism in order; Through the position information, simulate the sequential processing method of the text, and train the knowledge graph completion model to capture the fact evolution ability. 2.The method of claim 1, wherein, The construction of the quadruple data set and the conversion of the original knowledge graph into a static knowledge graph subgraph divided by time sequence comprises: Build a quadruple data form, and integrate time into the original fact triple in the form of a timestamp; The quadruple data form: s = (p, q, r, t); Wherein, s represents a fact, p represents the head entity in the fact, q represents the tail entity in the fact, r represents the relationship between the two entities, and t represents the time when the fact occurs, which is stored in the form of a timestamp. 3.The method of claim 1, wherein, The data set composed of multiple data sources is used for the model method, which comprises: Use multiple data sets as model experimental data, and the data in different data sets cannot be used crosswise; Continue to use the trained model for training of other data sets, continuously learn, update the model parameters, and optimize the model performance.

4. A training method of a knowledge graph completion model fusing weight and time information, comprising: S101, a data processing module, constructing training data and test data, classifying the data of the existing knowledge graph, dividing the facts in chronological order, and organizing the time sequence knowledge graph subgraph; S102, a weight learning module, using a pre-trained model to preprocess the training data, using an attention mechanism and a weight calculation method to obtain an optimized embedding representation of entities and relationships; S103, a time prediction module, using an attention mechanism, fusing the weight allocated by the time sequence distance information, constructing a time sequence attention mechanism, and obtaining the potential evolution trend of the facts; S104, a fusion training module, training the fusion method using the training set and test set obtained by the data processing module, optimizing the parameter performance, so that the method can predict missing information based on the original data; S105, a parameter optimization module, based on different data sets, different training data and test data are obtained, the model is used for learning on multiple data sets, and the loss function method is used for continuous learning and automatic optimization of parameters to obtain a trained knowledge graph completion model; In the weight learning module, the pre-trained model, the attention mechanism and the weight learning include: S1021, for the data obtained by the data processing module, first use the pre-trained model to obtain the initial embedding representation; S1022, the pre-trained model uses the BERT high-performance model as the encoder to optimize the basic representation form; S1023, the target entity corresponding triple and its local information are input into the attention mechanism to obtain the embedding representation of the basic target entity; S1024, the distance and node density between the target entity and the neighbor information of indefinite order are calculated, and the cross degree of the target entity and the neighbor information element level information is calculated to allocate the weight; S1025, the optimized embedding representation of the target entity affected by the weight is calculated, the basic embedding representation and the optimized embedding representation are fused to obtain the entity embedding representation with fusion weight; In the time prediction module, the use of the attention mechanism and the time adaptive weight includes: S1031, for the entity embedding representation with fusion weight obtained in the weight learning module, the attention mechanism is used for learning to obtain part of the model parameters; S1032, the entity data obtained in the weight learning module is divided in time sequence, the position information is used to simulate the time sequence, and the combined embedding representation is obtained by combining the entity representation; S1033, the embedding representation combined with the position information is input into the above attention mechanism to construct the time sequence attention mechanism, learn the time evolution trend, adjust the model parameters, and obtain the trained model.

5. The training method of the knowledge graph completion model fusing weight and time information according to claim 4, wherein in the fusion training module, the fusion process of the weight learning module and the time prediction module includes: S1041, input the optimized entity embedding representation obtained by the fusion weight learning module into the time prediction module, the time prediction module learns and simulates the evolution trend of things, and trains the prediction ability; S1042, input all the training data into the weight learning module and the time prediction module to train the knowledge graph completion model; S1043, the test data is processed alone, and a fact quadruple corresponding to a certain target entity in the test data is selected as a proposed question or the entire test knowledge graph subgraph is selected as a proposed question, which is input into the training model, so that the model outputs a guessed answer based on the question; S1044, adjusting the question quadruple, improving the test question based on the real data, and preventing overfitting.

Images