A knowledge graph embedding method based on U-shaped network
Through the knowledge graph embedding method based on U-shaped network, the problem of extracting complex relationships and high-level information in the prior art is solved, and more efficient and accurate knowledge graph embedding is achieved, reducing computing costs.
Patent Information
- Application Number
- CN202411916693.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-04-01
AI Technical Summary
The existing knowledge graph embedding technology is difficult to extract complex potential relationships between vectors, insufficient number of neural network layers cannot extract higher-level information, incomplete input of triple signal, and lack of appropriate loss functions lead to limitations of the existing technology.
Using the knowledge graph embedding method based on U-shaped network, by constructing a U-shaped network and initializing it, selecting triples as positive and negative samples, constructing high-dimensional input data, calculating the loss function and updating the network parameters, realizing the embedding of the knowledge graph.
This method can efficiently extract high-level information between triplets in the knowledge graph, integrate features at different levels, improve recognition accuracy, reduce calculation amount, improve response speed, and save calculation costs.
Smart Images
Figure CN119740647A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of knowledge graph technology, and in particular to a knowledge graph embedding method based on a U-type network. Background Art
[0002] The knowledge graph represents different things and their relationships in the real world in the form of triples (h, r, t), where h and t are the head entity and tail entity respectively, and they belong to the same entity set, and r is a set of different relationships. Each triple represents a correct relationship r between a head entity h and a tail entity t. Due to the wide range of sources of knowledge graphs, the set of triples is extremely large, and entities and relationships are abstract concepts and difficult to quantify. Knowledge graph embedding refers to the process of mapping entities and relationships in the knowledge graph to a low-dimensional vector space, that is, replacing entities and relationships with a finite-length data vector, thereby converting abstract entities and relationships into corresponding vectors. After the vector is constructed, the relationship between the corresponding entities can be represented by calculating the distance between the vectors, which is convenient for tasks such as entity linking and similarity calculation. It can be clearly seen that the shorter the length of the embedded vector, the faster and more efficient the calculation. At present, knowledge embedding mainly uses three technologies: translation model, semantic matching model and neural network model. The translation model projects entities and relations into a certain space, and then calculates the distance between them, that is, scores. The true triples have a small distance and a low score, while the wrong triples have a long distance. The semantic matching model models the relationship as a matrix, or constructs an additional core tensor, and uses the product of the head entity, the tail entity relationship, and the core tensor as the scoring function. The neural network inputs the head entity and the tail entity (or the head entity and the relationship) into the neural network, and multiplies the output vector with the relationship vector (or the tail entity vector) to score. However, the current knowledge embedding technology has the following problems: First: Using translation models and semantic matching embedding techniques cannot extract the complex underlying relationships between vectors.
[0003] Second: When using neural network embedding technology, the number of network layers is too small, and it is also impossible to extract higher-level information between vectors.
[0004] Third: When a triplet signal is used as input, the three vectors are usually directly constructed into a two-dimensional vector, and the extracted information is not complete.
[0005] Fourth: There is a lack of a suitable loss function to link the input triplet vector with the output signal of the neural network, which results in most existing neural network embedding techniques only being able to use vectors generated by existing embedding techniques as input. Summary of the invention
[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a knowledge graph embedding method based on a U-type network to solve the above-mentioned problems.
[0007] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a knowledge graph embedding method based on a U-shaped network, comprising the following steps: S1: triples that initialize the knowledge graph; S2: constructing a U-shaped network and initializing the U-shaped network; S3: Select a triplet as a positive sample, obtain the corresponding negative sample, and construct high-dimensional input data; S4: input the high-dimensional input data as an input signal into the U-shaped network to obtain a corresponding output signal; S5: Calculate a loss function according to the input signal and the output signal; S6: updating the parameters and input signal of the U-type network according to the loss function, and normalizing the input signal; S7: According to the updated U-shaped network and the input signal, determine whether the requirements for stopping training are met. If so, use the input signal as the embedding result of the knowledge graph, otherwise return to step S3.
[0008] Furthermore, the S1 includes the following sub-steps: S11: Each entity and relationship in the entity set and relationship set is randomly generated and uniformly distributed, with a dimension of The vector representation of is:
[0009] in, are vectors of entities and relations, For each vector containing the components, is the length of the vector; S12: Normalize each vector by dividing it by its second-order norm, using the formula:
[0010]
[0011] in, is the normalized vector, is the second-order norm of the vector; S13: Extract corresponding entity and relationship vectors to form the initial triple data set, and complete the triples of the initialized knowledge graph.
[0012] Furthermore, the U-type network in S2 includes the following sub-steps: S21: Input the input signal of the U-type network to the first-layer residual module, extract and output signal C1; S22: dividing the signal C1 into two paths, the first path continues to be input into the second-layer residual module, and the second path is temporarily stored for later use; S23: dividing the signal C2 output by the second-layer residual module into two paths, the first path continues to be input into the third-layer residual module, and the second path is temporarily stored for later use; S24: inputting the signal C3 output by the third layer residual module into the first layer transposed convolution module for interpolation, and outputting the interpolated signal d1; S25: adding the signal d1 to the second signal temporarily stored in the second-layer residual module, inputting the signal to the second-layer transposed convolution module, and outputting the interpolated signal d2; S26: Add the signal d2 to the second signal temporarily stored in the first-layer residual module and input it to the third-layer transposed convolution module to obtain the output signal of the U-type network.
[0013] Furthermore, S3 includes the following sub-steps: S31: randomly extract a triplet as a positive sample, randomly select a component in the triplet, and replace it with another randomly selected component of the same type to form a corresponding negative sample, and the negative sample is not in the known triplet set; S32: Each triplet in the positive and negative samples is 1-dimensional and has a length of Vector , and Convert to two-dimensional A vector of , , is the length of the two-dimensional vector, is the width of the two-dimensional vector; S33: The three The two-dimensional vectors are stacked into a dimension of A three-dimensional vector of S34: The three-dimensional vector of the positive sample is in front, and the three-dimensional vector of the negative sample is in the back, stacked into a dimension of The four-dimensional vector of is used to complete the construction of high-dimensional input data.
[0014] Furthermore, the loss function in S5 is for:
[0015]
[0016]
[0017]
[0018]
[0019] in, is the first loss function, is the second loss function, and is a normal number, is the coefficient of the U-type network, is the second-order norm, is a normal number, is the input vector of the U-type network, is the output vector of the U-type network , Output vector for the U-network No. Quantity, and Both are U-shaped network output vectors The amount.
[0020] Furthermore, the corresponding The output error matrix is:
[0021]
[0022]
[0023]
[0024]
[0025]
[0026] in, is the output error of the positive sample, is the output error of negative samples, for The output error matrix consists of the errors of positive and negative samples; The error backpropagation gradient descent training corresponds to The output error matrix is:
[0027] in, for The output error matrix is composed of the errors of positive and negative samples. is the matrix transpose, is the output error matrix Dimensional component; The corresponding The output error matrix plus the corresponding The output error matrix of constitutes the complete error matrix for backpropagation training.
[0028] Further, in S6, updating the parameters and input signal of the U-type network according to the loss function comprises the following sub-steps: S61: Calculate the gradient of each layer of the U-type network according to the output error matrix; S62: Based on the gradient of each layer of the U-shaped network, calculate the error of the input layer transmitted back through the U-shaped network ; S63: Based on input layer error And the output error of the U-type network, calculate the gradient of the input signal passed back, the formula is:
[0029]
[0030]
[0031]
[0032]
[0033]
[0034] in, is the gradient of the input triple matrix passed back, is the error matrix passed back to the input layer through the U-shaped matrix; S64: Update the parameters and input signal of the U-type network based on the gradient.
[0035] The beneficial effects of the present invention are as follows: the present invention uses a U-shaped network to embed the knowledge graph. Compared with the traditional method, deep learning has the advantages of extracting high-level information. At the same time, the U-shaped network used in the present invention can efficiently integrate features of different levels to achieve knowledge graph embedding, which can be more efficient and universal, and has a higher recognition accuracy rate for triples. Moreover, using the present invention, the dimensions of the entity and relationship vectors obtained are smaller, which can reduce the amount of calculation in subsequent use, improve the response speed, and save calculation costs. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a flow chart of a knowledge graph embedding method based on U-type network.
[0037] Figure 2 Figure 2 is a U-shaped network structure diagram for training knowledge graph embedding.
[0038] Figure 3 This is the structure diagram of the residual module.
[0039] Figure 4 Schematic diagram of the process of generating input high-dimensional matrix.
[0040] Figure 5 Schematic diagram of the error matrix calculation process.
[0041] Figure 6 Schematic diagram of the calculation process of error backpropagation gradient descent. DETAILED DESCRIPTION
[0042] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.
[0043] like Figure 1 As shown, a knowledge graph embedding method based on a U-type network includes the following steps: S1: triples that initialize the knowledge graph; S2: constructing a U-shaped network and initializing the U-shaped network; S3: Select a triplet as a positive sample, obtain the corresponding negative sample, and construct high-dimensional input data; S4: input the high-dimensional input data as an input signal into the U-shaped network to obtain a corresponding output signal; S5: Calculate a loss function according to the input signal and the output signal; S6: updating the parameters and input signal of the U-type network according to the loss function, and normalizing the input signal; S7: According to the updated U-shaped network and the input signal, determine whether the requirements for stopping training are met. If so, use the input signal as the embedding result of the knowledge graph, otherwise return to step S3.
[0044] In the knowledge graph, triples (h, r, t) are usually used to represent the relationship between specific things in the real world, h and t are entities, and r is the relationship between two entities. For example, a triple ("China", "capital", "Beijing") indicates that the capital of China is Beijing, where China and Beijing are entities and the capital is a relationship. However, such triples cannot be used for calculations, so such triples need to be represented by numerical vectors. For example, the triple above can be represented by three 1×10 vectors. However, there are many triples in the real world, and each entity and relationship needs to be represented by a unique vector. At the same time, it is hoped that triples with similar relationships are closer, such as the triple relationship above should be close to the triple ("France", "capital", "Paris"). However, it is not realistic to manually assign a unique vector that satisfies the above relationship to each entity and relationship. In order to complete this task, there are currently a variety of automatic implementation methods, all of which first assign a random initial value to all entities and relationships, and then complete it by minimizing a specific cost function. This patent uses a U-shaped network to complete this task.
[0045] The S1 includes the following sub-steps: S11: Each entity and relationship in the entity set and relationship set is randomly generated and uniformly distributed, with a dimension of The vector representation of is:
[0046] in, are vectors of entities and relations, For each vector containing the components, is the length of the vector; S12: Normalize each vector by dividing it by its second-order norm, using the formula:
[0047]
[0048] in, is the normalized vector, is the second-order norm of the vector; S13: Extract corresponding entity and relationship vectors to form the initial triple data set, and complete the triples of the initialized knowledge graph.
[0049] like Figure 2 As shown, it is a U-shaped network diagram. Figure 3 The figure is a schematic diagram of the residual module of the U-shaped network. The residual module compresses and extracts the features of the input signal, and then interpolates it through transposed convolution, so that the output signal size is the same as the input signal. The output of the residual module is added to the transposed convolution output through a direct connection structure, so that the output can integrate multi-scale information.
[0050] The U-type network in S2 includes the following sub-steps: S21: Input the input signal of the U-type network to the first-layer residual module, extract and output signal C1; S22: dividing the signal C1 into two paths, the first path continues to be input into the second-layer residual module, and the second path is temporarily stored for later use; S23: dividing the signal C2 output by the second-layer residual module into two paths, the first path continues to be input into the third-layer residual module, and the second path is temporarily stored for later use; S24: inputting the signal C3 output by the third layer residual module into the first layer transposed convolution module for interpolation, and outputting the interpolated signal d1; S25: adding the signal d1 to the second signal temporarily stored in the second-layer residual module, inputting the signal to the second-layer transposed convolution module, and outputting the interpolated signal d2; S26: Add the signal d2 to the second signal temporarily stored in the first-layer residual module and input it to the third-layer transposed convolution module to obtain the output signal of the U-type network.
[0051] The U-shaped network designed in the present invention is used for knowledge graph embedding. During training, the model parameters are adjusted through the error back propagation algorithm, and the input triplet vector is updated at the same time according to the designed loss function. The deep learning network is effectively used to mine the deep information between triplets, making the knowledge graph embedding more reasonable.
[0052] The S3 includes the following sub-steps: S31: randomly extract a triplet as a positive sample, and randomly select a component in the triplet ( or or ), and replace it with another randomly selected component of the same type to form a corresponding negative sample, and the negative sample is not in the known triple set; S32: Each triplet in the positive and negative samples is 1-dimensional and has a length of Vector , and Convert to two-dimensional A vector of , , is the length of the two-dimensional vector, is the width of the two-dimensional vector; S33: The three The two-dimensional vectors are stacked into a dimension of A three-dimensional vector of S34: The three-dimensional vector of the positive sample is in front, and the three-dimensional vector of the negative sample is in the back, stacked into a dimension of The four-dimensional vector of , completes the construction of high-dimensional input data, such as Figure 4 shown.
[0053] It should be noted that the size of the constructed high-dimensional matrix is not , but , because negative samples are only needed when training the network, and only an input size of .
[0054] Assume that the head vector is replaced by , then use Represents a complete input vector; inputs the vector consisting of positive and negative samples into the U-type network and calculates the output of the U-type network , which is the same size as the input vector, although in the input vector and is not replaced, but in the output vector , .
[0055] The loss function in S5 for:
[0056]
[0057]
[0058]
[0059]
[0060] in, is the first loss function, is the second loss function, The second-order norm of each output vector is kept at 1, and the minimization coefficient is also added to the loss function as a regularization term. and is a normal number, is the coefficient of the U-type network, is the second-order norm, is a positive constant used to control the distance between the positive and negative sample loss functions. is the input vector of the U-type network, is the output vector of the U-type network , Output vector for the U-network No. Quantity, and Both are U-shaped network output vectors The amount.
[0061] The corresponding The output error matrix is:
[0062]
[0063]
[0064]
[0065]
[0066]
[0067] in, is the output error of the positive sample, is the output error of negative samples, for The output error matrix consists of the errors of positive and negative samples; The error backpropagation gradient descent training corresponds to The output error matrix is:
[0068] in, for The output error matrix is composed of the errors of positive and negative samples. is the matrix transpose, is the output error matrix Dimensional component; The corresponding The output error matrix plus the corresponding The output error matrix of constitutes the complete error matrix for backpropagation training.
[0069] Both the positive sample error and the negative sample error are two-dimensional matrices. However, when using the error feedback training algorithm, a high-dimensional error with the same dimension as the output signal is required. According to the chain derivation rule, the corresponding high-dimensional error matrix can be obtained using the above formula. The specific process is as follows: Figure 5 shown.
[0070] The step of updating the parameters and input signal of the U-type network according to the loss function in S6 includes the following sub-steps: S61: Calculate the gradient of each layer of the U-type network according to the output error matrix; S62: Based on the gradient of each layer of the U-shaped network, calculate the error of the input layer transmitted back through the U-shaped network ; S63: Based on input layer error The output error of the U-shaped network is used to calculate the gradient of the input signal. According to the loss function, the gradient consists of two parts. The first part is the gradient of the input layer after the U-shaped network is returned. , the other part is based on The gradient of the input signal directly obtained by the function is as follows:
[0071]
[0072]
[0073]
[0074]
[0075]
[0076] in, is the gradient of the input triple matrix passed back, is the error matrix passed back to the input layer through the U-shaped matrix, and is the positive and negative sample error matrix output by the U-type network. The specific process is as follows Figure 6 As shown; S64: Update the parameters and input signal of the U-type network based on the gradient.
[0077] The present invention provides an embedding technology using a U-shaped network and an improved distance-based loss function, transforming the input triple vector to another space through a U-shaped network instead of a simple linear projection, so that the distance embedding model and the neural network embedding model are effectively integrated, and the network can be trained and the input signal can be changed at the same time. Specifically: The present invention takes triplet vectors as image input and uses a U-shaped network to effectively extract high-level information between triplets.
[0078] The present invention designs a loss function, which includes the output signal and input signal of the network. The network parameters and the input signal can be trained simultaneously by minimizing the loss function.
[0079] The present invention designs a training method for updating input signals based on the loss function.
[0080] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the invention.
Claims
1. A knowledge graph embedding method based on U-type network, characterized in that: The following steps are involved: S1: triples that initialize the knowledge graph; S2: constructing a U-shaped network and initializing the U-shaped network; S3: Select a triplet as a positive sample, obtain the corresponding negative sample, and construct high-dimensional input data; S4: input the high-dimensional input data as an input signal into the U-shaped network to obtain a corresponding output signal; S5: Calculate a loss function according to the input signal and the output signal; S6: updating the parameters and input signal of the U-type network according to the loss function, and normalizing the input signal; S7: According to the updated U-shaped network and the input signal, determine whether the requirements for stopping training are met. If so, use the input signal as the embedding result of the knowledge graph, otherwise return to step S3.
2. According to the U-type network-based knowledge graph embedding method of claim 1, it is characterized in that: The S1 includes the following sub-steps: S11: Each entity and relationship in the entity set and relationship set is randomly generated and uniformly distributed, with a dimension of The vector representation of is: in, are vectors of entities and relations, For each component contained in a vector, is the length of the vector; S12: Normalize each vector by dividing it by its second-order norm, using the formula: in, is the normalized vector, is the second-order norm of the vector; S13: Extract corresponding entity and relationship vectors to form the initial triple data set, and complete the triples of the initialized knowledge graph.
3. According to the U-type network-based knowledge graph embedding method of claim 1, it is characterized in that: The U-type network in S2 includes the following sub-steps: S21: Input the input signal of the U-type network to the first-layer residual module, extract and output signal C1; S22: dividing the signal C1 into two paths, the first path continues to be input into the second-layer residual module, and the second path is temporarily stored for later use; S23: dividing the signal C2 output by the second-layer residual module into two paths, the first path continues to be input into the third-layer residual module, and the second path is temporarily stored for later use; S24: inputting the signal C3 output by the third layer residual module into the first layer transposed convolution module for interpolation, and outputting the interpolated signal d1; S25: adding the signal d1 to the second signal temporarily stored in the second-layer residual module, inputting the signal to the second-layer transposed convolution module, and outputting the interpolated signal d2; S26: Add the signal d2 to the second signal temporarily stored in the first-layer residual module and input it to the third-layer transposed convolution module to obtain the output signal of the U-type network.
4. According to the U-type network-based knowledge graph embedding method of claim 2, it is characterized in that: The S3 includes the following sub-steps: S31: randomly extract a triplet as a positive sample, randomly select a component in the triplet, and replace it with another randomly selected component of the same type to form a corresponding negative sample, and the negative sample is not in the known triplet set; S32: Each triplet in the positive and negative samples is 1-dimensional and has a length of Vector , and Convert to two-dimensional A vector of , , is the length of the two-dimensional vector, is the width of the two-dimensional vector; S33: The three The two-dimensional vectors are stacked into a dimension of A three-dimensional vector of S34: The three-dimensional vector of the positive sample is in front, and the three-dimensional vector of the negative sample is in the back, stacked into a dimension of The four-dimensional vector of is used to complete the construction of high-dimensional input data.
5. A knowledge graph embedding method based on U-type network according to claim 4, characterized in that: The loss function in S5 for: in, is the first loss function, is the second loss function, and is a normal number, is the coefficient of the U-type network, is the second-order norm, is a normal number, is the input vector of the U-type network, is the output vector of the U-type network , Output vector for the U-network No. Quantity, and Both are U-shaped network output vectors The amount.
6. A knowledge graph embedding method based on U-type network according to claim 5, characterized in that: The corresponding The output error matrix is: in, is the output error of the positive sample, is the output error of negative samples, for The output error matrix consists of the errors of positive and negative samples; The error backpropagation gradient descent training corresponds to The output error matrix is: in, for The output error matrix is composed of the errors of positive and negative samples. is the matrix transpose, is the output error matrix Dimensional component; The corresponding The output error matrix plus the corresponding The output error matrix of constitutes the complete error matrix for backpropagation training.
7. A knowledge graph embedding method based on U-type network according to claim 6, characterized in that: The step of updating the parameters and input signal of the U-type network according to the loss function in S6 includes the following sub-steps: S61: Calculate the gradient of each layer of the U-type network according to the output error matrix; S62: Based on the gradient of each layer of the U-shaped network, calculate the error of the input layer transmitted back through the U-shaped network ; S63: Based on input layer error And the output error of the U-type network, calculate the gradient of the input signal passed back, the formula is: in, is the gradient of the input triple matrix passed back, is the error matrix passed back to the input layer through the U-shaped matrix; S64: Update the parameters and input signal of the U-type network based on the gradient.