Knowledge graph completion method, device, electronic device and storage medium

Abstract

The present application relates to a knowledge graph completion method, device, electronic device and storage medium. The method comprises: obtaining corpus data to be processed; performing predictive processing on the corpus data to be processed through a prediction model to obtain a prediction result corresponding to the corpus data to be processed, wherein the prediction model comprises at least one differentiable neural computer DNC model, and when there are multiple DNC models, the multiple DNC models are in a stacked structure; completing the knowledge graph to be completed according to the corpus data to be processed and the prediction result to obtain a completed knowledge graph. The use of this method can improve the accuracy of knowledge graph completion.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a knowledge graph completion method, device, electronic device and storage medium. Background Art

[0002] Knowledge Graph (KG) has become an extremely important way of knowledge representation in the big data era due to its excellent interpretability and powerful semantic recognition and processing capabilities.

[0003] One of the common defects of knowledge graphs is their incompleteness, which limits the development and application of knowledge graph technology. Knowledge reasoning for knowledge graphs can be based on the existing knowledge in the knowledge graph to obtain new knowledge and achieve the role of knowledge graph completion (KGC), which is of great significance to make up for the defects of knowledge graph incompleteness. Therefore, knowledge reasoning for knowledge graphs has attracted the attention of researchers at home and abroad and has become a hot research issue in the knowledge graph technology system.

[0004] However, due to the richness of data, the source of knowledge graph construction is not sufficient, and due to the limited performance of knowledge extraction methods and knowledge reasoning methods, some implicit knowledge cannot be fully mined, which leads to low reasoning accuracy in the process of knowledge graph completion and poor accuracy of the completed knowledge graph. Summary of the invention

[0005] Based on this, it is necessary to provide a knowledge graph completion method, device, electronic device and storage medium that can improve the accuracy of knowledge graph completion in response to the above technical problems.

[0006] In a first aspect, the present application provides a knowledge graph completion method, the method comprising:

[0007] Get the corpus data to be processed;

[0008] Performing prediction processing on the corpus data to be processed by a prediction model to obtain a prediction result corresponding to the corpus data to be processed, wherein the prediction model includes at least one differentiable neural computer (DNC) model, and when there are multiple DNC models, the multiple DNC models are in a stacked structure;

[0009] The knowledge graph to be completed is completed according to the corpus data to be processed and the prediction results to obtain a completed knowledge graph.

[0010] According to the knowledge graph completion method provided by the embodiment of the present disclosure, since the DNC model separates the neural network parameters and memory capacity, the network capacity can be increased, and the DNC model can be stacked to increase the hyperparameter scale and the model representation space. It can have better reasoning and question-answering effects, and is oriented to question-answering system tasks. It effectively reduces the average word error rate and the comprehensive average word error rate, and improves the knowledge graph completion accuracy.

[0011] In one embodiment, the DNC model includes a controller and a memory, and the prediction model is used to perform prediction processing on the corpus data to be processed to obtain a prediction result corresponding to the corpus data to be processed, including:

[0012] For any time step t in the corpus data to be processed, a content-based addressing operation is used to read the memory information corresponding to time step t-1 from the memory to obtain the corresponding memory read vector r t-1 ;

[0013] By means of the controller, the corpus data X corresponding to the time step t in the corpus data to be processed is processed t , the memory read vector r t-1 Perform prediction processing to obtain the corresponding controller output vector V t ;

[0014] Using content-based addressing operations, the memory information corresponding to the time step t is read from the memory to obtain the corresponding memory read vector r t ;

[0015] The controller output vector V t and the memory read vector r t , get the output vector Y corresponding to the time step t t ;

[0016] When the time step t is the last time step in the corpus data to be processed, determine the output vector Y corresponding to the time step t t is the final vector representation corresponding to the corpus data to be processed;

[0017] The prediction result corresponding to the corpus data to be processed is obtained through the final vector representation corresponding to the corpus data to be processed.

[0018] According to the knowledge graph completion method provided by the embodiment of the present disclosure, redundant structures in the DNC model can be removed, unnecessary system resource consumption can be reduced, and model efficiency can be enhanced.

[0019] In one embodiment, the content-based addressing operation is used to read the memory information corresponding to the time step from the memory to obtain the corresponding memory read vector r t ,include:

[0020] According to the corpus data X corresponding to the time step t t , updating the memory information stored in the memory;

[0021] Using content-based addressing operations, the memory information corresponding to the time step t is read from the updated memory to obtain the corresponding memory read vector r t .

[0022] According to the knowledge graph completion method provided by the embodiment of the present disclosure, each DNC model in the prediction model will remove the temporal memory link and its related components, and the addressing method of the memory storage matrix will only use content-based addressing, which can improve the model prediction efficiency.

[0023] In one of the embodiments, the memory includes a project memory and a relation memory, the project memory is used to store input corpus data through gated associative memory, and the relation memory is used to store the relations between the corpus data.

[0024] According to the knowledge graph completion method provided by the embodiment of the present disclosure, by analyzing the original attention mechanism, a dual memory structure is introduced to enhance the attention mechanism, thereby enhancing the reasoning ability of the model.

[0025] In one embodiment, the memory read vector includes project memory information and relationship memory information, and the content-based addressing operation is used to read the memory information corresponding to the time step t from the updated memory to obtain the corresponding memory read vector r t ,include:

[0026] Using content-based addressing operations, the project memory information corresponding to the time step t is read from the updated project memory, and the relationship memory information corresponding to the time step t is read from the updated relationship memory.

[0027] According to the knowledge graph completion method provided by the embodiment of the present disclosure, by analyzing the original attention mechanism, a dual memory structure is introduced to enhance the attention mechanism, thereby enhancing the reasoning ability of the model.

[0028] In one embodiment, the corpus data X according to the time step t t , updating the memory information stored in the memory, including:

[0029] According to the corpus data X at time step tt Updating the project memory information at time step t-1 stored in the project memory to obtain an updated project memory, wherein the updated project memory stores the project memory information at time step t;

[0030] According to the corpus data X at time step t t and the relation memory information of time step t-1 stored in the relation memory, and determining the target relation vector;

[0031] According to the target relationship vector, the item memory information at the time step t and the corpus data X at the time step t t , determine the target item vector;

[0032] The relation memory information of time step t-1 stored in the relation memory is updated according to the target item vector to obtain an updated relation memory, wherein the updated relation memory stores the relation memory information of time step t.

[0033] According to the knowledge graph completion method provided by the embodiment of the present disclosure, the redundant structure of the DNC model of the differentiable neural computer is simplified, and the dual memory structure is introduced to enhance the attention mechanism by analyzing the original attention mechanism. In addition, the model is stacked and adopts a double-layer structure to enhance the reasoning ability of the model.

[0034] In a second aspect, the present application further provides a knowledge graph completion device, the device comprising:

[0035] The acquisition module is used to obtain the corpus data to be processed from the knowledge graph to be completed;

[0036] A prediction module, used for performing prediction processing on the corpus data to be processed by using a prediction model to obtain a prediction result corresponding to the corpus data to be processed, wherein the prediction model includes at least one differentiable neural computer (DNC) model, and when there are multiple DNC models, the multiple DNC models are in a stacked structure;

[0037] A completion module is used to complete the knowledge graph to be completed according to the corpus data to be processed and the prediction results to obtain a completed knowledge graph.

[0038] In one embodiment, the DNC model includes a controller and a memory, and the prediction module is further used to:

[0039] For any time step t in the corpus data to be processed, a content-based addressing operation is used to read the memory information corresponding to time step t-1 from the memory to obtain the corresponding memory read vector r t-1 ;

[0040] By means of the controller, the corpus data X corresponding to the time step t in the corpus data to be processed is processed t , the memory read vector r t-1 Perform prediction processing to obtain the corresponding controller output vector V t ;

[0041] Using content-based addressing operations, the memory information corresponding to the time step t is read from the memory to obtain the corresponding memory read vector r t ;

[0042] The controller output vector V t and the memory read vector r t , get the output vector Y corresponding to the time step t t ;

[0043] When the time step t is the last time step in the corpus data to be processed, determine the output vector Y corresponding to the time step t t is the final vector representation corresponding to the corpus data to be processed;

[0044] The prediction result corresponding to the corpus data to be processed is obtained through the final vector representation corresponding to the corpus data to be processed.

[0045] In one embodiment, the prediction module is further used to:

[0046] According to the corpus data X corresponding to the time step t t , updating the memory information stored in the memory;

[0047] Using content-based addressing operations, the memory information corresponding to the time step t is read from the updated memory to obtain the corresponding memory read vector r t .

[0048] In one of the embodiments, the memory includes a project memory and a relation memory, the project memory is used to store input corpus data through gated associative memory, and the relation memory is used to store the relations between the corpus data.

[0049] In one embodiment, the prediction module is further used to:

[0050] Using content-based addressing operations, the project memory information corresponding to the time step t is read from the updated project memory, and the relationship memory information corresponding to the time step t is read from the updated relationship memory.

[0051] In one embodiment, the memory read vector includes item memory information and relationship memory information, and the prediction module is further used to:

[0052] According to the corpus data X at time step t t Updating the project memory information at time step t-1 stored in the project memory to obtain an updated project memory, wherein the updated project memory stores the project memory information at time step t;

[0053] According to the corpus data X at time step t t and the relation memory information of time step t-1 stored in the relation memory, and determining the target relation vector;

[0054] According to the target relationship vector, the item memory information at the time step t and the corpus data X at the time step t t , determine the target item vector;

[0055] The relation memory information of time step t-1 stored in the relation memory is updated according to the target item vector to obtain an updated relation memory, wherein the updated relation memory stores the relation memory information of time step t.

[0056] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements any of the above-mentioned knowledge graph completion methods when executing the computer program.

[0057] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, any of the above-mentioned knowledge graph completion methods is implemented.

[0058] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements any of the above-mentioned knowledge graph completion methods.

[0059] The above-mentioned knowledge graph completion method, device, electronic device and storage medium can obtain the corpus data to be processed, perform prediction processing on the corpus data to be processed through the prediction model, and obtain the prediction result corresponding to the corpus data to be processed. The prediction model includes at least one differentiable neural computer DNC model. In the case of multiple DNC models, the multiple DNC models are in a stacked structure. The knowledge graph to be completed is completed according to the corpus data to be processed and the prediction result, and the completed knowledge graph can be obtained. According to the knowledge graph completion method, device, electronic device and storage medium provided by the embodiments of the present disclosure, since the DNC model separates the neural network parameters and memory capacity, the network capacity can be increased, and by stacking the DNC model, the hyperparameter scale can be increased, the model representation space can be increased, and the average word error rate and the comprehensive average word error rate can be effectively reduced, so the prediction accuracy can be effectively improved, and the completion accuracy of the knowledge graph can be improved.

[0060] For example, when the completed knowledge graph is applied to question-answering system tasks, it can have better reasoning and question-answering effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flowchart of a knowledge graph completion method in one embodiment;

[0062] Figure 2 is a schematic diagram of a DNC model in one embodiment;

[0063] Figure 3 is a schematic diagram of a DNC model in one embodiment;

[0064] Figure 4 is a flowchart of a knowledge graph completion method in one embodiment;

[0065] Figure 5 is a flowchart of a knowledge graph completion method in one embodiment;

[0066] Figure 6 is a schematic diagram of a DNC model in one embodiment;

[0067] Figure 7 is a flowchart of a knowledge graph completion method in one embodiment;

[0068] Figure 8 is a schematic diagram of a knowledge graph completion method in one embodiment;

[0069] Fig. 9 is a schematic diagram of a knowledge graph completion method in one embodiment;

[0070] Fig.10 It is a structural block diagram of a knowledge graph completion device in one embodiment;

[0071] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0073] Figure 1 This is a flow chart of a knowledge graph completion method according to an exemplary embodiment. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0074] Step 102: Obtain corpus data to be processed.

[0075] In the disclosed embodiment, the corpus data to be processed may be corpus data that has been preprocessed into word vector representations. Taking a question-answering system as an example, the corpus data to be processed may be word vector representations corresponding to input sentence information for answer prediction.

[0076] Step 104, predictively process the corpus data to be processed by using a prediction model to obtain a prediction result corresponding to the corpus data to be processed, wherein the prediction model includes at least one differentiable neural computer DNC model. When there are multiple DNC models, the multiple DNC models are in a stacked structure.

[0077] In the disclosed embodiment, the corpus data to be processed can be input into the prediction model for processing to obtain a prediction result corresponding to the corpus data to be processed. Taking the question-answering system as an example, the prediction result can be the answer corresponding to the corpus data to be processed.

[0078] Among them, the prediction model includes at least one DNC (Differentiable Neural Computer) model. When there are multiple DNC models, the multiple DNC models are in a stacked structure. The DNC model mainly includes a controller and auxiliary storage-related components such as memory. Since the essence of the DNC controller is a recurrent neural network, the DNC model can be equivalently expanded in time. And because its architecture has independent auxiliary storage components such as memory, the DNC model can selectively eliminate and restore time coupling, that is, the introduction of auxiliary storage components such as memory enables the DNC model to have a decoupling function in time.

[0079] Reference Figure 2As shown in the figure, from a temporal perspective, the connections between the hidden layers of the sequentially unfolded DNC model network need to pass through the memory, that is, there are interactive operations with the memory at every moment; from a spatial perspective, the addition of auxiliary storage structures such as memory reduces the temporal dependence of the model.

[0080] In the case where the prediction model includes multiple DNC models, the multiple DNC models are in a stacked structure, and the stacked DNC models are referred to after the time series is expanded. Figure 3 As shown in the figure. The stacking of DNC models significantly increases the scale of the hyperparameters of the prediction model, so that the prediction model has sufficiently large parameters to form a potentially large representation space. In the stacked DNC model, the configuration and operation of a single layer can be independent of each other. Through coordination and cooperation, the whole can form a more efficient system than running alone. That is, the stacking of DNC models makes the configuration of the model more flexible and can build a more powerful heterogeneous system. In turn, stacking DNC models will bring better learning and reasoning effects.

[0081] Step 106, completing the knowledge graph to be completed according to the corpus data to be processed and the prediction results, to obtain the completed knowledge graph.

[0082] In the disclosed embodiment, after obtaining the prediction results of the corpus data to be processed, corresponding triples can be constructed based on the corpus data to be processed and its prediction results, and then the knowledge graph to be completed can be completed through the constructed triples.

[0083] Based on the knowledge graph completion method provided by the embodiment of the present disclosure, the corpus data to be processed can be obtained, and the corpus data to be processed can be predicted and processed by the prediction model to obtain the prediction result corresponding to the corpus data to be processed. The prediction model includes at least one differentiable neural computer DNC model. In the case of multiple DNC models, the multiple DNC models are in a stacked structure. The knowledge graph to be completed is completed according to the corpus data to be processed and the prediction result, and the completed knowledge graph can be obtained. According to the knowledge graph completion method provided by the embodiment of the present disclosure, since the DNC model separates the neural network parameters and the memory capacity, the network capacity can be increased, and by stacking the DNC model, the hyperparameter scale can be increased, the model representation space can be increased, and the average word error rate and the comprehensive average word error rate can be effectively reduced, so the prediction accuracy can be effectively improved, and the completion accuracy of the knowledge graph can be improved. Exemplarily, when the completed knowledge graph is applied to the task of the question-answering system, it can have a good reasoning and question-answering effect.

[0084] In an exemplary embodiment, the DNC model includes a controller and a memory, referring to Figure 4 As shown, in the above step 104, the prediction model is used to perform prediction processing on the corpus data to be processed to obtain the prediction result corresponding to the corpus data to be processed, including:

[0085] Step 402: for any time step t in the corpus data to be processed, use content-based addressing operations to read the memory information corresponding to time step t-1 from the memory to obtain the corresponding memory read vector r t-1 ;

[0086] Step 404: The controller processes the corpus data X corresponding to time step t in the corpus data to be processed. t 、Memory read vector r t-1 Perform prediction processing to obtain the corresponding controller output vector V t ;

[0087] Step 406, using content-based addressing operations, read the memory information corresponding to time step t from the memory to obtain the corresponding memory read vector r t ;

[0088] Step 408: Output vector V via controller t and memory read vector r t , get the output vector Y corresponding to time step t t ;

[0089] Step 410: when time step t is the last time step in the corpus data to be processed, determine the output vector Y corresponding to time step t t is the final vector representation corresponding to the corpus data to be processed;

[0090] Step 412: Obtain the prediction result corresponding to the corpus data to be processed through the final vector representation corresponding to the corpus data to be processed.

[0091] For example, the output of the DNC model is the weighted sum of the output vector of the controller network and the memory read vector, that is, the model effect of the DNC is affected by the joint action of the controller and the memory. For example, when applied to question-answering systems, since the requirements for recovery sequences in question-answering systems are very low, the DNC model mainly uses content-based addressing in tasks such as question-answering systems. At the same time, for question-answering system data, temporal memory links are the main driving factor for system resource consumption.

[0092] In order to make the model more efficient, the DNC model in the embodiment of the present application can remove the sequential memory link and its related components. After removal, the memory is addressed only using content-based addressing. In the content-based addressing process, the search key vector issued by the controller calculates the cosine similarity with the element stored in each position in the storage matrix in the memory to evaluate the content similarity based on the cosine similarity. The similarity score corresponding to the cosine similarity is essentially a weight used to control the read / write head, and then read the memory information in the memory through the read / write head to obtain the corresponding memory read vector r t-1 .

[0093] Exemplarily, the DNC model removes the sequential memory link structure, removes the release gate and the allocation gate (essentially a parameter item), and the addressing mechanism used in the interaction process between the controller and the memory only retains the content-based addressing. Before the structural simplification, the interaction process between the controller and the memory uses an addressing mechanism that combines multiple methods, for example: when reading memory information, content-based addressing and sequential memory links are used for positioning; when writing memory information, content-based addressing and dynamic addressing are used for positioning. Among them, dynamic addressing is combined with a release list, which is a storage location in the memory that has been released or not yet used. The calculation of the release list involves the release gate. The allocation gate participates in the calculation of write weight, read weight, and sequential memory link. That is, after the structural simplification, the DNS model mainly includes a controller and a memory.

[0094] For example, taking the prediction model using the LSTM (Long Short-Term Memory Network) network as a controller, the corpus data X corresponding to the time step t in the corpus data to be processed is t and the memory read vector r obtained from the memory t-1 Processing is performed to obtain the controller output vector V t . Controller (·) represents the controller processing process, which can be expressed as formula (1).

[0095] V t =controller([X t ; r t-1 ]) Formula (I)

[0096] Exemplarily, the controller network uses the cell state (also called unit state, internal state) C t To record memory information, the gating mechanism is used to modify the cell state and control the accumulation rate of memory information. The gating mechanism converts the received vector into a real number vector between 0 and 1. There are three multiplication gates involved: the forget gate f t , input gate i t And the output gate ot . Forget Gate f t Determines the degree to which the cell state discards memory information, that is, the cell state C at the previous moment t-1 How much can be retained until the current moment C t , refer to formula (II); input gate i t Determines the extent to which the cell state preserves new input information, that is, how much of the network input at the current moment can be retained to the cell state C t , refer to formula (III); output gate o t Determines the degree to which the cell state outputs memory information, that is, the cell state C t How much can be output to the current output value h t , refer to formula (IV). The corpus data at time t is X t , let x t =[X t ;w t-1 r t-1 ], x t Represents the network input information combined with the read vector corresponding to time step t-1 in the memory.

[0097] f t =σ(W f [h t-1 ;x t ]+b f ) Formula (II)

[0098] i t =σ(W i [h t-1 ;x t ]+b i ) Formula (III)

[0099] o t =σ(W o [h t-1 ;x t ]+b o ) Formula (IV)

[0100] where σ(·) is the sigmoid function, W f , W i , W o Represent the weight matrix respectively, […;…] represents the connection operation of the vector, b f 、b i 、b o They represent the bias terms of the forget gate respectively.

[0101] Cell State C t Combined with the cell state C at the previous moment t-1 And the candidate cell states generated by the current input That is, the historical memory information and the current new memory information are combined, refer to the following formula (5) and formula (6).

[0102]

[0103]

[0104] Where tanh(·) is the tanh activation function, ⊙ is the Hadamard Product operation, i.e., element-wise multiplication, W C The weight matrix used to characterize the cell state, b c The bias term used to characterize the cell state, I t The input gate vector representing time step t.

[0105] The output gate and the cell state jointly determine the hidden state (or external state) h t , the controller network generates an output vector V at each moment t , refer to the following formula (VII) and formula (VIII).

[0106] h t =o t ⊙tanh(C t ) Formula (VII)

[0107]

[0108] Where L represents the number of model stacking layers, that is, the number of DNC models, and W y The weight matrix used to characterize the controller output.

[0109] At each moment, the prediction model converts the controller output vector V t and memory read vector r t Combined, we get the output vector Y t , refer to the following formula (IX). When the time step t is the last time step in the corpus data to be processed, the output vector Y t It is regarded as the final vector representation of the entire sequence (or corpus data to be processed).

[0110] Y t =V t +W r [r t 1 ; r t 2 ;...;r t R Formula (IX)

[0111] Among them, R is used to represent the memory read vector r tThe number of .

[0112] Y t Then normalize it through the softmax classifier to obtain the probability vector of each word vector, and obtain the word vector with the highest probability as the prediction result, referring to the following formula (10).

[0113]

[0114] Among them, Y a Used to represent the probability that the output belongs to the word vector a; y a Represents the value of the output vector position a; where a and j are ergodic variables, referring to the position number of the output vector.

[0115] According to the knowledge graph completion method provided by the embodiment of the present disclosure, redundant structures in the DNC model can be removed, unnecessary system resource consumption can be reduced, and model efficiency can be enhanced.

[0116] In an exemplary embodiment, referring to Figure 5 As shown, in step 406, a content-based addressing operation is used to read the memory information corresponding to the time step from the memory to obtain the corresponding memory read vector r t ,include:

[0117] Step 502: based on the corpus data X at time step t t , update the memory information stored in the memory;

[0118] Step 504: Using content-based addressing operations, read the memory information corresponding to time step t from the updated memory to obtain the corresponding memory read vector r t .

[0119] In one embodiment, the memory includes a project memory and a relation memory. The project memory can be used to store input corpus data through gated associative memory, and the relation memory is used to store the relations between corpus data.

[0120] The knowledge graph completion method provided by the embodiment of the present disclosure analyzes the original attention mechanism, introduces a dual memory structure to enhance the attention mechanism, and enhances the reasoning ability of the model.

[0121] In the disclosed embodiment, after analyzing the attention mechanism of the DNC model, it is found that the original auxiliary storage setting cannot store the relational representation, which is not conducive to the reuse of the relational representation. In addition, the calculation of the memory information relational representation is not rich enough, which restricts the expression effect of the information. In view of these shortcomings, the present invention enhances the original attention mechanism and makes changes to the auxiliary storage components. Specifically, an associative memory model based on neurodynamics is introduced to store the relational memory separately, and the original memory component is updated to a dual memory structure, that is, the memory includes a project memory and a relational memory.

[0122] Item memory refers to the memory of whether a thing or event has occurred, and is used to store the input corpus data accordingly; relational memory refers to the memory of the relationship between specific details such as the location of things and the order of things, that is, it is used to store the relationship between the input corpus data. Among them, the item memory can selectively encode and store the input corpus data by using gated associative memory. Among them, the essence of gated associative memory is content-based addressing.

[0123] In addition, the project memory can operate with the memory information in the relational memory to generate new relational memory information to update the memory information stored in the relational memory. Since the memory information in the relational memory is of higher dimension, the interaction (operation process) between the two memories uses the outer product operation to construct a gated associative memory to represent the project and the relationship between the projects, which combines the stacked dual memory model structure with the enhanced attention mechanism, such as Figure 6 shown.

[0124] According to the knowledge graph completion method provided by the embodiment of the present disclosure, each DNC model in the prediction model will remove the temporal memory link and its related components, and the addressing method of the memory storage matrix will only use content-based addressing, which can improve the model prediction efficiency.

[0125] In one example, the memory read vector includes the item memory information and the relationship memory information. In step 406, the memory information corresponding to the time step t is read from the updated memory using the content-based addressing operation to obtain the corresponding memory read vector r t ,include:

[0126] Using content-based addressing operations, the item memory information corresponding to time step t is read from the updated item memory, and the relation memory information corresponding to time step t is read from the updated relation memory.

[0127] In the disclosed embodiment, the read weights of the two memories are defined as formula (XI).

[0128]

[0129] in To read the key vector, is the reading intensity, M t It is used to represent the storage matrix corresponding to the time step t of the memory. CD(·) is the content-based addressing operation, which can be specifically referred to as shown in formula (XII).

[0130]

[0131] The dual memory module receives the gate state and input vector of the controller network. x , generate the read vector r in the project memory through the SAM operator i and the read vector r in the relational memory r , and the dual memory matrix state and (The matrix state at time step t in the project memory is The matrix at time step t in the relational memory is ) is used to update. The dual memory component processing process is represented by two_memory(·), and the details can be referred to as shown in formula (13).

[0132] (r i , r r ;M i , M r )=two_memory(f;I;x) Formula (13)

[0133] Among them, f and I are used to represent the controller's forget gate vector and input gate vector, respectively. i is the matrix state in the project memory, M r is the matrix state in relational memory.

[0134] According to the knowledge graph completion method provided by the embodiment of the present disclosure, by analyzing the original attention mechanism, a dual memory structure is introduced to enhance the attention mechanism, thereby enhancing the reasoning ability of the model.

[0135] In one embodiment, referring to Figure 7 As shown, in the above step 502, according to the corpus data X at time step t t , update the memory information stored in the memory, including:

[0136] Step 702: based on the corpus data X at time step t t Updating the project memory information at time step t-1 stored in the project memory to obtain an updated project memory, wherein the updated project memory stores the project memory information at time step t;

[0137] Step 704: based on the corpus data X at time step t t and the relation memory information of time step t-1 stored in the relation memory, to determine the target relation vector;

[0138] Step 706: based on the target relationship vector, the item memory information at time step t, and the corpus data X at time step t t , determine the target item vector;

[0139] Step 708, updating the relation memory information of time step t-1 stored in the relation memory according to the target item vector to obtain an updated relation memory, wherein the updated relation memory stores the relation memory information of time step t.

[0140] In the embodiment of the present disclosure, the and Represents the memory information stored in the project memory and the relationship memory at time step t. The memory information can be stored in the form of a storage matrix state. At time t, and The update of can be expressed as the following formula (fourteen) to formula (seventeen).

[0141]

[0142]

[0143]

[0144]

[0145] where fnn i (·) represents a feed-forward neural network, represents the outer product operation, η1 and η2 are hyperparameters, and SAM(·) is the gated associative memory operation. The gated associative memory operation can store the state of the project memory Linearly map to different spaces to generate query matrix Q, key matrix K, and value matrix V respectively. The SAM(·) operation returns a high-dimensional relational representation, referring to the following formulas (18) to (21).

[0146]

[0147]

[0148]

[0149]

[0150] Among them, a and j are the positions of the vector sequence in the corresponding matrix, nkv Used to characterize the number of vectors in the key matrix / value matrix.

[0151] Among them, the self-attention mechanism using the query-key-value mode usually uses the dot product operation to process the QKV to obtain the attention score, which indicates the importance of the encoding of each position to other positions. In the above-mentioned dual-memory structure attention mechanism, the gated associative memory (Self-attentive Associative Memory, SAM) operator combined with the outer product operation is used. The outer product operation can retain the element-level interaction information between vectors and has rich representation learning capabilities, which is crucial for constructing the representation of high-dimensional relationships.

[0152] The knowledge graph completion method provided by the embodiment of the present disclosure simplifies the redundant structure of the DNC model of the differentiable neural computer, and introduces a dual memory structure to enhance the attention mechanism by analyzing the original attention mechanism. In addition, the model is stacked and adopts a double-layer structure to enhance the reasoning ability of the model.

[0153] In one embodiment of the present disclosure, normalization technology can be applied to the controller to effectively improve the training efficiency of the prediction model. Specifically, normalization can usually bring at least the following optimizations: better scale invariance and smoother optimization terrain. Specifically, the input of the prediction model is calculated layer by layer, and the update of the model parameters during training will cause a large change in the distribution of the output near the output layer. This instability phenomenon is called internal covariate shift. By normalizing the input of each layer to a standard normal distribution, it can have better scale invariance and can be initialized more efficiently. Optimization Landscape refers to the surface shape of the loss function in high-dimensional space. Normalization technology makes the optimization terrain of the network smoother, while making the gradient change more stable and improving the convergence speed.

[0154] Exemplarily, since the input distribution of the recurrent neural network has the characteristic of dynamic change, the present application can select the layer normalization (LN) method to normalize the input of each layer. Layer normalization is applied to the intermediate layer before the activation function to normalize the input of all neurons in the intermediate layer.

[0155] After analysis, it is known that the performance of the DNC model is related to the memory impact rate. When the model does not converge, the memory usually has a smaller impact rate; when the memory impact rate is larger, the model generally achieves better performance results. Therefore, it is necessary to ensure a high memory impact rate. By using the dropout method in the connection between the controller and the output, the number of connections can be reduced, thereby enhancing the high memory impact rate. The dropout method is a network regularization method. The main function of the regularization method is to improve the generalization ability of the neural network, limit the complexity of the model, and avoid overfitting. Reference Figure 8 As shown in Figure 2, the dropout method achieves regularization by randomly dropping some neurons and their corresponding connections.

[0156] In one embodiment of the present disclosure, the output of the DNC model is a weighted sum of the output of the controller and the memory read vector, that is, the output of the DNC model is affected by the controller and the memory. When determining the influence rate of the memory on the DNC model, the ratio of the output of the memory to the overall output of the DNC model can be defined as the memory influence rate, which is used to indicate the effect of the memory on the output of the DNC model. For details, refer to the following formula (XXII).

[0157]

[0158] Among them, the mean(·) function represents the mean of the elements of the vector, in M Indicates the memory impact rate.

[0159] Some researchers have shown that high memory usage is strongly correlated with good task results. Therefore, the memory impact rate, as an indicator of component participation, can measure the overall performance of the model. Fig. 9 As shown in the figure, in the experiment on the sample of task 1 of the question-answering dataset bAbI, the release gate and allocation gate of the DNC model play a relatively low role, which indicates that the DNC model mainly uses content-based addressing in the question-answering system task, and the temporal memory link has little impact on the experimental results. This is because the question-answering system task has a low requirement for the recovery sequence. At the same time, for the question-answering system data, the temporal memory link is the main driver of system resource consumption.

[0160] Therefore, for question-answering system tasks, in order to make the model more efficient, the DNC model removes the temporal memory links and their related components, and the addressing method of the memory storage matrix will only use content-based addressing.

[0161] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0162] Based on the same inventive concept, the embodiment of the present application also provides a knowledge graph completion device for implementing the above-mentioned knowledge graph completion method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more knowledge graph completion device embodiments provided below can refer to the limitations of the knowledge graph completion method above, and will not be repeated here.

[0163] In one embodiment, Fig.10 As shown, a knowledge graph completion device is provided, including: an acquisition module 1002, a prediction module 1004 and a completion module 1006, wherein:

[0164] An acquisition module 1002 is used to acquire the corpus data to be processed from the knowledge graph to be completed;

[0165] A prediction module 1004 is used to perform prediction processing on the corpus data to be processed by using a prediction model to obtain a prediction result corresponding to the corpus data to be processed, wherein the prediction model includes at least one differentiable neural computer (DNC) model. When there are multiple DNC models, the multiple DNC models are in a stacked structure.

[0166] The completion module 1006 is used to complete the knowledge graph to be completed according to the corpus data to be processed and the prediction results to obtain a completed knowledge graph.

[0167] Based on the knowledge graph completion device provided by the embodiment of the present disclosure, it is possible to obtain the corpus data to be processed, and the corpus data to be processed is predicted and processed by the prediction model to obtain the prediction result corresponding to the corpus data to be processed. The prediction model includes at least one differentiable neural computer DNC model. In the case of multiple DNC models, the multiple DNC models are in a stacked structure. The knowledge graph to be completed is completed according to the corpus data to be processed and the prediction result, and the completed knowledge graph can be obtained. According to the knowledge graph completion device provided by the embodiment of the present disclosure, since the DNC model separates the neural network parameters and the memory capacity, the network capacity can be increased, and by stacking the DNC model, the hyperparameter scale can be increased, the model representation space can be increased, and the average word error rate and the comprehensive average word error rate can be effectively reduced, so the prediction accuracy can be effectively improved, and the completion accuracy of the knowledge graph can be improved. Exemplarily, when the completed knowledge graph is applied to the task of the question-answering system, it can have a good reasoning and question-answering effect.

[0168] In one embodiment, the DNC model includes a controller and a memory, and the prediction module 1004 is further configured to:

[0169] For any time step t in the corpus data to be processed, a content-based addressing operation is used to read the memory information corresponding to time step t-1 from the memory to obtain the corresponding memory read vector r t -1;

[0170] By means of the controller, the corpus data X corresponding to the time step t in the corpus data to be processed is processed t , the memory read vector r t-1 Perform prediction processing to obtain the corresponding controller output vector V t ;

[0171] Using content-based addressing operations, the memory information corresponding to the time step t is read from the memory to obtain the corresponding memory read vector r t ;

[0172] The controller output vector V t and the memory read vector r t , get the output vector Y corresponding to the time step t t ;

[0173] When the time step t is the last time step in the corpus data to be processed, determine the output vector Y corresponding to the time step t t is the final vector representation corresponding to the corpus data to be processed;

[0174] The prediction result corresponding to the corpus data to be processed is obtained through the final vector representation corresponding to the corpus data to be processed.

[0175] In one embodiment, the prediction module 1004 is further configured to:

[0176] According to the corpus data X corresponding to the time step t t , updating the memory information stored in the memory;

[0177] Using content-based addressing operations, the memory information corresponding to the time step t is read from the updated memory to obtain the corresponding memory read vector r t .

[0178] In one embodiment, the memory read vector includes project memory information and relational memory information, and the memory includes project memory and relational memory. The project memory is used to store input corpus data through gated associative memory, and the relational memory is used to store the relations between the corpus data.

[0179] In one embodiment, the prediction module 1004 is further configured to:

[0180] Using content-based addressing operations, the project memory information corresponding to the time step t is read from the updated project memory, and the relationship memory information corresponding to the time step t is read from the updated relationship memory.

[0181] In one embodiment, the prediction module 1004 is further configured to:

[0182] According to the corpus data X at time step t t Updating the project memory information at time step t-1 stored in the project memory to obtain an updated project memory, wherein the updated project memory stores the project memory information at time step t;

[0183] According to the corpus data X at time step t t and the relation memory information of time step t-1 stored in the relation memory, and determining the target relation vector;

[0184] According to the target relationship vector, the item memory information at the time step t and the corpus data X at the time step t t , determine the target item vector;

[0185] The relation memory information of time step t-1 stored in the relation memory is updated according to the target item vector to obtain an updated relation memory, wherein the updated relation memory stores the relation memory information of time step t.

[0186] Each module in the above-mentioned knowledge graph completion device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0187] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a knowledge graph completion method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0188] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0189] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0190] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0191] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0192] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0193] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0194] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0195] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A knowledge graph completion method, characterized in that: The method comprises: Get the corpus data to be processed; The prediction model is used to perform prediction processing on the corpus data to be processed to obtain a prediction result corresponding to the corpus data to be processed, wherein the prediction model includes at least one differentiable neural computer DNC model, and when there are multiple DNC models, the multiple DNC models are in a stacked structure, and the DNC model includes a controller and a memory, wherein the memory includes a project memory and a relationship memory, wherein the project memory is used to store the input corpus data through a gated associative memory, and the relationship memory is used to store the relationship between the corpus data; Completing the knowledge graph to be completed according to the corpus data to be processed and the prediction results to obtain a completed knowledge graph; The step of performing prediction processing on the corpus data to be processed by using a prediction model to obtain a prediction result corresponding to the corpus data to be processed includes: For any time step t in the corpus data to be processed, a content-based addressing operation is used to read the memory information corresponding to time step t-1 from the memory to obtain the corresponding memory read vector r t-1 ; By means of the controller, the corpus data X corresponding to the time step t in the corpus data to be processed is processed t , the memory read vector r t-1 Perform prediction processing to obtain the corresponding controller output vector V t ; Using content-based addressing operations, the memory information corresponding to the time step t is read from the memory to obtain the corresponding memory read vector r t ; The controller output vector V t and the memory read vector r t , get the output vector Y corresponding to the time step t t ; When the time step t is the last time step in the corpus data to be processed, determine the output vector Y corresponding to the time step t t is the final vector representation corresponding to the corpus data to be processed; Obtaining a prediction result corresponding to the corpus data to be processed through a final vector representation corresponding to the corpus data to be processed; The memory read vector includes the project memory information and the relationship memory information. The content-based addressing operation is used to read the memory information corresponding to the time step t from the updated memory to obtain the corresponding memory read vector r t ,include: Using content-based addressing operations, the project memory information corresponding to the time step t is read from the updated project memory, and the relationship memory information corresponding to the time step t is read from the updated relationship memory.

2. The method according to claim 1, characterized in that The content-based addressing operation is used to read the memory information corresponding to the time step from the memory to obtain the corresponding memory read vector r t ,include: According to the corpus data X corresponding to the time step t t , updating the memory information stored in the memory; Using content-based addressing operations, the memory information corresponding to the time step t is read from the updated memory to obtain the corresponding memory output vector r t .

3. The method according to claim 2, characterized in that The corpus data X according to the time step t t , updating the memory information stored in the memory, including: According to the corpus data X at time step t t Updating the project memory information at time step t-1 stored in the project memory to obtain an updated project memory, wherein the updated project memory stores the project memory information at time step t; According to the corpus data X at time step t t and the relation memory information of time step t-1 stored in the relation memory, and determining the target relation vector; According to the target relationship vector, the item memory information at the time step t and the corpus data X at the time step t t , determine the target item vector; The relation memory information of time step t-1 stored in the relation memory is updated according to the target item vector to obtain an updated relation memory, wherein the updated relation memory stores the relation memory information of time step t.

4. The method according to any one of claims 1 to 3, characterized in that The sequential memory link structure is removed from the DNC model, and the release gate and the allocation gate are removed. The addressing mechanism used in the interaction process between the controller and the memory is content-based addressing.

5. A knowledge graph completion device, characterized in that: The device comprises: The acquisition module is used to obtain the corpus data to be processed from the knowledge graph to be completed; A prediction module, used for performing prediction processing on the corpus data to be processed through a prediction model to obtain a prediction result corresponding to the corpus data to be processed, wherein the prediction model includes at least one differentiable neural computer DNC model, and when there are multiple DNC models, the multiple DNC models are in a stacked structure, and the DNC model includes a controller and a memory, wherein the memory includes a project memory and a relationship memory, wherein the project memory is used for storing input corpus data through a gated associative memory, and the relationship memory is used for storing the relationship between the corpus data; A completion module, used to complete the knowledge graph to be completed according to the corpus data to be processed and the prediction result, to obtain a completed knowledge graph; The step of performing prediction processing on the corpus data to be processed by using a prediction model to obtain a prediction result corresponding to the corpus data to be processed includes: For any time step t in the corpus data to be processed, a content-based addressing operation is used to read the memory information corresponding to time step t-1 from the memory to obtain the corresponding memory read vector r t-1 ; By means of the controller, the corpus data X corresponding to the time step t in the corpus data to be processed is processed t , the memory read vector r t-1 Perform prediction processing to obtain the corresponding controller output vector V t ; Using content-based addressing operations, the memory information corresponding to the time step t is read from the memory to obtain the corresponding memory read vector r t ; The controller output vector V t and the memory read vector r t , get the output vector Y corresponding to the time step t t ; When the time step t is the last time step in the corpus data to be processed, determine the output vector Y corresponding to the time step t t is the final vector representation corresponding to the corpus data to be processed; Obtaining a prediction result corresponding to the corpus data to be processed through a final vector representation corresponding to the corpus data to be processed; The memory read vector includes the project memory information and the relationship memory information. The content-based addressing operation is used to read the memory information corresponding to the time step t from the updated memory to obtain the corresponding memory read vector r t ,include: Using content-based addressing operations, the project memory information corresponding to the time step t is read from the updated project memory, and the relationship memory information corresponding to the time step t is read from the updated relationship memory.

6. The device according to claim 5, characterized in that The content-based addressing operation is used to read the memory information corresponding to the time step from the memory to obtain the corresponding memory read vector r t ,include: According to the corpus data X corresponding to the time step t t , updating the memory information stored in the memory; Using content-based addressing operations, the memory information corresponding to the time step t is read from the updated memory to obtain the corresponding memory output vector r t .

7. The device according to claim 6, characterized in that The corpus data X according to the time step t t , updating the memory information stored in the memory, including: According to the corpus data X at time step t t Updating the project memory information at time step t-1 stored in the project memory to obtain an updated project memory, wherein the updated project memory stores the project memory information at time step t; According to the corpus data X at time step t t and the relation memory information of time step t-1 stored in the relation memory, and determining the target relation vector; According to the target relationship vector, the item memory information at the time step t and the corpus data X at the time step t t , determine the target item vector; The relation memory information of time step t-1 stored in the relation memory is updated according to the target item vector to obtain an updated relation memory, wherein the updated relation memory stores the relation memory information of time step t.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images