Non-deterministic knowledge graph embedding method based on attention network extended meta path

Abstract

The application designs a non-deterministic knowledge graph embedding method based on attention network expansion path, and relates to the field of data mining; the relationship of related words in the non-deterministic knowledge graph ConceptNet is used to expand the deterministic knowledge graph, so that it becomes a deterministic knowledge graph with non-deterministic relationship, which is called non-deterministic knowledge graph embedding; when expanding the deterministic knowledge graph, not only the entities in the heterogeneous information network are expanded, but also more relationships between the entities are mined, and these relationships are the attributes with probability; then the embedding of the non-deterministic knowledge graph is integrated into KGAT, the embedding information of the neighbor nodes is recursively propagated by using the attention mechanism of KGAT, so that a node has the embedding information of multiple layers of neighbor nodes in the embedding space, thereby the embedding characteristics of the node are improved, and finally the final prediction value is obtained by aggregating the representations of entities and items.

Description

Technical Field

[0001] This invention relates to the field of data mining, specifically to a nondeterministic knowledge graph embedding method based on attention network-enhanced meta-paths. Background Technology

[0002] In recent years, knowledge graphs, as a structured representation of human knowledge, have attracted widespread attention from academia and industry. Composed of three important parts—entities, relations, and semantic descriptions—knowledge graphs provide a structured representation of real-world entities and relations, offering a novel knowledge representation method for intelligent systems in the real world and enabling them to solve complex tasks. In previous research, real-world data was often viewed as deterministic knowledge graphs with defined relations, such as social networks, search engines, and recommendation systems. Applications based on these knowledge graphs have thus achieved excellent performance. However, real-world data often contains non-deterministic relations. Graphs containing non-deterministic relations are called non-deterministic knowledge graphs. These graphs not only include non-deterministic relations between entities but also non-deterministic relations between entities and items or attributes. Applications driven by this knowledge graph can therefore produce more diverse and interpretable results.

[0003] The Aho-Corasick automaton, developed by Bell Labs in 1975, is used to match substrings from a finite set of "dictionaries" within an input string. It differs from ordinary string matching in that it matches all dictionary strings simultaneously. The algorithm has an amortized time complexity of approximately linear, roughly the length of the string plus the number of matches. This algorithm primarily relies on constructing a finite state machine (similar to adding mismatch pointers to a Trie tree). These additional mismatch pointers allow for backtracking when a string search fails (e.g., if the word "cat" fails to match in the Trie tree, but another word "cart" exists, the mismatch pointer will point to the prefix "ca"), redirecting to other branches of a prefix and avoiding repeated prefix matching, thus improving algorithm efficiency.

[0004] The Double-Array Aho-Corasick Automaton (DAAC) is an improvement on the Aho-Corasick Automaton. The Double-Array Trie structure, proposed by Jun-Ichioge in 1989, is a compressed form of the Trie structure, using only two linear arrays to represent the Trie tree. This structure effectively combines the high retrieval speed of the Digital Search Tree with the compact space structure of the linked Trie representation. Essentially, a Double-Array Trie is a Deterministic Finite-State Automaton (DFA), where each node represents a state of the automaton. State transitions occur based on different variables, and a query operation is completed when the final state is reached or no transition is possible. The relationships between characters in all keys within the Double-Array Trie are represented by simple mathematical addition, which not only improves retrieval speed but also eliminates the large number of pointers used in linked structures, saving storage space.

[0005] UKGE (Uncertain Knowledge Graph Embedding) is a model for embedding uncertain knowledge graphs. This model effectively utilizes the uncertain relationships between entities, preserving the structure and uncertainty information of relational facts in the embedding space. Unlike previous models that used binary classification techniques to represent relational facts, UKGE learns embeddings based on the confidence scores of uncertain relational facts, further improving its accuracy. It also introduces probabilistic soft logic to infer the confidence scores of unseen relational facts during training.

[0006] Patent CN112348190B proposes the SUKE model, utilizing the existing deterministic embedding model DistMult. This model preserves both the structured and non-deterministic information of knowledge and comprises two components: an evaluator and a confidence generator. The evaluator assesses the reasonableness of facts based on their structural and non-deterministic features, filtering out unreasonable facts to obtain candidate facts. The confidence generator generates confidence scores for candidate facts, representing the probability of entities engaging in a specific relationship. The evaluator defines a structural score and a non-deterministic score for each triple, used for fact reasonableness assessment. The confidence generator generates a confidence score for each triple, used for confidence prediction. The combination of these two components effectively enhances the embedding capability of knowledge graphs, thus enabling the task of link prediction in the recommendation domain using non-deterministic knowledge graphs.

[0007] Patent CN113010690B proposes a method to obtain entity structure embeddings and relation structure embeddings by loading entity vectors and relation vectors into entity embedding matrices and relation embedding matrices, respectively. Then, a pre-trained word vector model is used to obtain word vectors, and the word embedding matrices are queried to obtain entity description word vectors and relation word vectors. Next, the entity description word vectors are input into a BiLSTM network, and dot product attention is introduced into the output layer of the BiLSTM network. The relation word vectors are then averaged to obtain the relation embedding. The entity structure embeddings are projected onto the relation space to obtain the entity structure embedding projection. Finally, the entity description embeddings and the entity structure embedding projections are added together to obtain the entity embedding. Finally, textual information from a cybersecurity knowledge base is used to enhance the entity representation capabilities, thereby improving the accuracy of entity link prediction and ultimately contributing to the effective enhancement of recommendation performance.

[0008] The method proposed in patent CN113010690B, while fully considering entity description information embedding and relation embedding to improve the representational power of entity embedding through information augmentation and achieving good performance in expanding entity information in knowledge graphs, thus improving the accuracy of entity link prediction, fails to consider the non-deterministic nature of relationships between entities in the real world and does not analyze the influence of adjacent entities in the knowledge graph. Therefore, it exhibits a phenomenon where link prediction capability is higher locally but lower globally, resulting in recommendation results lacking diversity, interpretability, and accuracy.

[0009] The method proposed in patent CN112348190B, while considering the non-deterministic information in the graph structure (evaluator assesses the reasonableness of facts by considering structural and non-deterministic features, thus filtering out unreasonable facts), and with a confidence generator producing a confidence score for each triple for confidence prediction, fails to consider the impact of each meta-path in the knowledge graph on adjacent meta-paths. Specifically, it doesn't consider the influence of a node's neighboring nodes, doesn't truly integrate the features of neighboring nodes, and doesn't take into account the role of multi-layered graphs in link prediction. Therefore, the accuracy of the resulting recommendations is not particularly high.

[0010] Therefore, semantic augmentation of deterministic knowledge graphs, that is, extending nondeterministic relations into deterministic knowledge structures, can improve the poor diversity, low accuracy, and weak interpretability of deterministic knowledge graph-driven applications (such as recommendation systems). Thus, it is necessary to propose a nondeterministic knowledge graph embedding model that enriches the meta-path semantic information in the deterministic knowledge graph for recommendation. Summary of the Invention

[0011] To address the shortcomings of existing technologies, this invention proposes a non-deterministic knowledge graph embedding method based on attention networks to extend meta-paths in deterministic knowledge graphs using non-deterministic semantics. The resulting model is named UKGAT (Uncertain Knowledge Graph Attention). This method uses the relationships between related words in the non-deterministic knowledge graph ConceptNet to extend the deterministic knowledge graph, thus creating a deterministic knowledge graph with non-deterministic relationships; this is called non-deterministic knowledge graph embedding. Extending the deterministic knowledge graph not only expands the entities in the heterogeneous information network but also uncovers more relationships between entities, which have probabilistic attributes. The non-deterministic knowledge graph embedding is then integrated into UKGAT, and the attention mechanism of UKGAT recursively propagates the embedding information of neighboring nodes, allowing a node to possess embedding information from multiple layers of neighboring nodes in the embedding space, thereby improving the node's embedding features. Finally, the final predicted value is obtained by aggregating the representations of entities and items.

[0012] A nondeterministic knowledge graph embedding method based on attention network-enhanced meta-paths includes the following steps:

[0013] Step 1: Obtain two publicly available datasets containing comment information for recommendation purposes. Based on these datasets, create a related word lexicon and optimize the lexicon for subsequent expansion of the heterogeneous information network.

[0014] Step 1.1: Obtain two publicly available datasets containing comment information that can be used for recommendation. Extract comment data from the two datasets. Segment each comment data to obtain the original keyword set. Then remove stop words and duplicate keywords from the original keyword set to form comment keyword set 1.

[0015] Step 1.2: Associate related words with keywords, that is, use non-deterministic knowledge graphs to supplement the non-deterministic relationship between keywords and related words, and select the knowledge structure of ConceptNet as the auxiliary knowledge base for associating related words with keywords;

[0016] Step 1.3: The real purpose of building the thesaurus is to find the correlation between words and form the triple format between words. Therefore, it is necessary to retrieve the keywords of keyword set 1 in the non-deterministic knowledge graph ConceptNet in advance and form triples (head entity, relation, tail entity).

[0017] Step 1.4: Considering that a keyword has many related words, in order to match more related words based on a keyword, the API interface provided by ConceptNet, a non-deterministic knowledge graph, is used to query the related words, relevance relationships, and relevance probabilities of each keyword in keyword set 1. Keywords for which no related words were found are deleted, and finally a dictionary in the form of many-to-many relationship between keywords and related words is formed, called dictionary 1. The dictionary also records the relevance relationship between keywords and related words, relevance probabilities, user ID, product ID, language of the related relationship, and number of related relationships.

[0018] Step 1.5: Optimize dictionary 1 using a double-array AC automaton, i.e., a double-array Trie tree; where the entire tree is represented by two one-dimensional arrays BASE and CHECK, and the construction process of the BASE and CHECK arrays needs to satisfy the following state transition formula 1:

[0019] BASE[s]+c=t

[0020] CHECK[t] = s (1)

[0021] Where CHECK represents an array of predecessor node addresses, BASE represents an array of successor node base addresses, s and t both represent states, which have been numericalized and are initially set to 0; C corresponds to the encoding value of a letter in the character table.

[0022] The formula performs state transitions in two simple steps: first, the value of t is calculated by BASE[s]+c, and then the correctness of BASE[s]+c is determined by CHECK[t]=s.

[0023] Step 1.6: In keyword set 1, each keyword has a different probability of appearing. Sort the n keywords according to the word list order K = {k1, k2, ..., k} n}, using these keywords, construct an optimal search binary tree to minimize the total cost of querying elements; for each keyword k i There is a probability p for each. i Correspondingly, the probability p index starts from 1 and corresponds to each keyword;

[0024] Step 1.7: For a search tree, a search is successful when the searched element is already in the tree; a search fails when it is not. A "dummy leaf node" is used to represent a failed search, therefore n+1 dummy leaf nodes are needed {d0 < d1 < ... < d2}. n}, corresponding to d i The probability sequence is Q = {q0, q1, ..., q} n}; its d0 represents the failed result of searching for an element less than k1, dn This indicates that the search element is greater than k. n Failure scenarios; d i (0 < i < n) indicates that the search node is in k i and k (i+1) The failure scenarios during this period; therefore, the following formula exists:

[0025]

[0026] Where, p i Let q represent the probability of the i-th keyword. i This represents the probability of the i-th virtual leaf node;

[0027] Step 1.8: Determine the expected cost of a single search within a given binary search tree T based on the probability of each key and each dummy key (i.e., dummy leaf node) being searched; let the actual cost of a single search be the number of nodes checked, i.e., the depth of the nodes found in T plus 1; therefore, the expected cost of a single search within T is:

[0028]

[0029] Where E[T] represents the expected cost of a single search within T, and depth T (k i ) represents the k-th element within T. i The depth of the node;

[0030] Step 1.9: Define e[i,j] as the search term for a tree containing the keyword k. i ,......k j The expected cost of the optimal binary search tree is discussed in the following categories:

[0031] When j = i-1, it means that there is only virtual key d at this time. i-1 Therefore, e[i,i-1]=q i-1 ;

[0032] When j≥i, it is necessary to start from k i ,......k j Choose a root k from the middle r Then use the keyword k i ,......,k r-1 To construct an optimal binary search tree as the left subtree, use the key k. r+1 ,......,k j To construct an optimal binary search tree as the right subtree;

[0033] Define a tree with key k i ,......k jThe sum of probabilities of the subtrees is defined as w(i,j):

[0034]

[0035] Where i,j,r∈N + k r Let k represent the r-th keyword. r Here, it acts as the root of an optimal search binary subtree; p l Let q represent the probability of the l-th keyword. l This represents the probability of the l-th dummy leaf node;

[0036] Step 1.10: Therefore, if k r It is a tree containing the keyword k i ,......,k r-1 If the root of the optimal subtree is found, then:

[0037] e[i,j]=p r +(e[i,r-1]+w(i,r-1))+(e[r+1,j]+w(r+1,j))

[0038] w(i,j)=w(i,r-1)+p r +w(r+1,j) (5)

[0039] Where p r Represents the root node k r The probability of;

[0040] Step 1.11: Therefore, e[i,j] is rewritten as:

[0041] e[i,j]=e[i,r-1]+e[r+1,j]+w(i,j) (6)

[0042] Step 1.12: The final recursive formula:

[0043]

[0044] Step 2: Expand the two datasets obtained, which are also called the relationships between entities, items, and entities and items in the heterogeneous information network, to obtain the final expanded heterogeneous information network;

[0045] Step 2.1: Perform word segmentation, stop word removal, and deduplication preprocessing on the comments in the two datasets. Since the preprocessing of the comments was already completed when the dictionary was created, this step uses the results of Step 1.1.

[0046] Step 2.2: Retrieve one review data point from the dataset. iAfter step 2.1, the keyword sequence [kw1,kw2,...,kw] is obtained. n ], where kw n It is a review i The nth keyword in the query, kw n The relevant information in the dictionary yields the related word sequence [rw1,rw2,...,rw] n ], correlation [r1,r2,...,r n ] and related probabilities [rp1,rp2,...,rp n ];

[0047] Step 2.3: Keyword kw i With related word sequence [rw1,rw2,...,rw n Each related word in ]rw j They all form a triple. ij =(kw i ,r j ,rw j The probability that the triple is true is rp. j , i,j∈[1,n];

[0048] Step 2.4: Each comment in the dataset is used to form several triples. ij The subgraph G of the comments is formed by these triples. sub ={(h,r,t)|h,t∈E,r∈R}, where h and t are the head node and tail node, respectively, E is the entity set; r represents the relationship between h and t, and R represents the relation set; directly subgraph G sub The edges are concatenated into a heterogeneous information network. During the concatenation process, duplicate edges are not concatenated, resulting in the final expanded heterogeneous information network. This expanded heterogeneous information network can also be understood as an expanded knowledge graph. This knowledge graph contains not only the original user entities, items, and deterministic relationships between user entities and items in the two datasets, but also expanded entities, expanded items, uncertain relationships between expanded entities and expanded items, uncertain relationships between expanded entities and original entities, uncertain relationships between expanded items and original items, uncertain relationships between original entities and expanded items, and uncertain relationships between expanded entities and original items. Each relationship is represented as a path, with two attributes: relationship and probability.

[0049] Step 3: Convert the reasonable values ​​of triples in the heterogeneous information network into confidence scores; for a given triple l, the reasonable value g(l) ∈ R. 1 This indicates how likely it is that the relational fact represented by the triple is true; the higher this value, the higher the confidence score corresponding to the triple.

[0050] Step 3.1: Given a triple l = (h, r, t) and their embeddings e h ,e r ,e t The estimated value of a triple is calculated using the following formula:

[0051]

[0052] Among them, The dot (·) represents element-wise multiplication, while the dot (·) represents the inner product. This function, derived from DistMult, captures e under the condition of relation r. h and e t The relationship between them;

[0053] Step 3.2: To convert the estimated value into a confidence score, the Logistic mapping function is used. This function converts the estimated score g(l) into a confidence score f(l).

[0054] f(l)=φ(g(l))∈[0,1] (9)

[0055] The Logistic mapping function is a common S-shaped curve:

[0056]

[0057] Step 3.3: For a given observable triple l, there is a confidence score s for considering it true. l , where s l ∈[0,1], using two observable triples to infer unobservable relational facts, let I represent the interaction relation I(l) of a triple, where the confidence score is calculated as follows:

[0058]

[0059] Where ζ + Let represent the set of observed triples, and ζ - Let s represent the set of unobservable triples. l represents the confidence score of the observed triples, while f(l) represents the confidence score of the unobserved triples obtained through the mapping function;

[0060] Step 3.4: After expanding the heterogeneous information network, both deterministic and non-deterministic triples exist. For deterministic triples, the true confidence score s of the triples is calculated. l The variance of the predicted confidence score f(l) is used to reflect the embedding ability of the UKGAT model:

[0061]

[0062] The UKGAT is an improvement on KGAT, which utilizes a lexicon and embeds the uncertain knowledge graph UKGE.

[0063] Step 3.5: For nondeterministic triples l, minimize the confidence score f(l) obtained by logical reasoning PSL for each triple, using the variance of the distance as the loss function:

[0064]

[0065] Where ψ γ (f(l)) represents the mapping function for calculating the weighted distance, and γ represents a given PSL rule;

[0066] Step 3.6: By simultaneously solving equations (6) and (7), the following objective function is obtained:

[0067]

[0068] Step 4: Use the attention mechanism to recursively propagate the information of nodes in the multi-layer graph to embed it into the representation of entities and items, and aggregate the representations of entities and items in the graph to obtain the predicted value.

[0069] Step 4.1: Consider an entity h, using S h = {(h,r,t)|(h,r,t)∈G} represents the set of triples, where h is the head entity, also known as the self-network. To characterize the first-order connectivity structure of entity h, the linear structure of the self-network of entity h is calculated using the following formula:

[0070]

[0071] Where π(h,r,t) represents the attenuation factor for each propagation on the edge (h,r,t), and represents the amount of information propagated from t through r to h each time.

[0072] Step 4.2: To calculate the attenuation factor π(h,r,t), the tanh nonlinear activation function is used, as shown in the following formula:

[0073] π(h,r,t)=(W r e t ) T ·tanh(W r e h +e r (16)

[0074] Among them W r ∈R k×dis the transition matrix of relation r, used to project entities from the d-dimensional entity space to the k-dimensional relation space; π(h,r,t) represents the attention score of the triple (h,r,t), which depends on the distance between entities h and t in the relation r space;

[0075] Step 4.3: Normalize the attenuation factor obtained in the previous step, using the following formula:

[0076]

[0077] Step 4.4: Represent entity e h and self-network representation As a new representation of entity h This constitutes a single-layer self-network; an aggregator—a bidirectional interactive aggregator—is selected, which carefully considers e h and The interaction features between them are designed using the following formula:

[0078]

[0079] Where W1, W2 ∈ R d′×d It is a trainable matrix;

[0080] Step 4.5: To further stack more propagation layers to explore higher-order connection information, information propagated from higher-hop neighbors is collected; in layer l, the entity representation is recursively formulated as:

[0081]

[0082] The propagation information of entity h in the self-network at step l is as follows:

[0083]

[0084] It is a representation of entity t generated from the previous information propagation steps, storing information from its l-1 hop neighbors; It is set to e during the initial information propagation iteration. h It further facilitates the representation of entity h at layer l;

[0085] Step 4.6: After executing layer l, multiple representations of user node u are obtained, namely... Similar to item node i, we get Since the output of layer l is an aggregation of messages from the tree rooted at u or i, the outputs of different layers emphasize connectivity information of different orders. Finally, a layer aggregation mechanism is used to connect the representation of each step into a vector, as shown below:

[0086]

[0087] Here, ‖ is the join operation, which not only enriches the initial embedding by performing the embedding propagation operation, but also allows the propagation strength to be controlled by adjusting l;

[0088] Step 4.7: Finally, perform the inner product of the user and item representations to obtain the predicted score:

[0089]

[0090] Beneficial technical effects of the present invention:

[0091] Compared with existing technologies, the technical solution proposed in this invention uses the semantic augmentation theory of heterogeneous information network metapath to improve the semantic information of nodes in the graph, and uses the high-order connectivity and attention mechanism of multi-layer knowledge graph to improve the embedding information of nodes, thereby improving the accuracy of recommendation results. At the same time, it can better explain why the recommendation is made based on deterministic and non-deterministic relationships between entities. Since there are non-deterministic relationships in heterogeneous information networks, the results are more diverse. Attached Figure Description

[0092] Figure 1 Flowchart of the nondeterministic knowledge graph embedding method based on attention network augmented meta-path in this invention;

[0093] Figure 2 This invention's embodiments are based on heterogeneous information networks using Amazon datasets;

[0094] Figure 3 A schematic diagram of a nondeterministic knowledge graph embedding framework based on attention network augmented meta-paths in an embodiment of the present invention;

[0095] Figure 4 This invention presents a mind map of related terms from the non-deterministic knowledge graph ConceptNet (https: / / conceptnet.io / ). Detailed Implementation

[0096] The present invention will be further described below with reference to the accompanying drawings and embodiments;

[0097] like Figure 1 As shown, the nondeterministic knowledge graph embedding method based on attention network-enhanced meta-paths includes the following steps:

[0098] Step 1: Obtain two publicly available datasets containing review information for recommendation purposes. Create a related word thesaurus based on review data from Amazon Book and Yelp 2018 and optimize the thesaurus for subsequent expansion of the heterogeneous information network.

[0099] Step 1.1: Obtain two publicly available datasets containing comment information that can be used for recommendation. Extract comment data from the two datasets. Segment each comment data to obtain the original keyword set. Then remove stop words and duplicate keywords from the original keyword set to form comment keyword set 1.

[0100] Step 1.2: Identifying related words for keywords is a crucial step in building the Amazon vocabulary model. This involves using a non-deterministic knowledge graph to supplement the uncertain relationships between keywords and related words. Therefore, choosing a suitable non-deterministic knowledge graph is essential for expanding the vocabulary. Common non-deterministic knowledge graphs include ConceptNet, NELL, and STRING. STRING contains a large amount of information related to biological knowledge. NELL is a never-ending language learner (NELL), which learns knowledge from the internet daily and learns to reason about that knowledge. ConceptNet is a knowledge base that connects words and phrases of natural language with labeled edges, aiming to help applications better understand human-language. After comparison, the knowledge contained in ConceptNet meets the needs of this invention and is easy to use. The knowledge structure of ConceptNet is as follows: Figure 4 As shown.

[0101] We chose ConceptNet (https: / / conceptnet.io / ) as the auxiliary knowledge base for generating related words for keywords. It organizes and represents a large amount of common-sense information in the form of a graph, covering various fields and topics, including natural language processing, artificial intelligence, and machine learning. This helps computers better understand the connections between language and concepts. Because ConceptNet contains rich semantic relationships, such as inclusion, relevance, and similarity, these relationships help computers better understand the connections between keywords and related words, enriching the semantic information of meta-paths in deterministic knowledge graphs. Therefore, using ConceptNet to generate related words for keywords becomes crucial.

[0102] Step 1.3: The real purpose of building the thesaurus is to find the correlation between words and form the triple format between words. Therefore, it is necessary to retrieve the keywords of keyword set 1 in the non-deterministic knowledge graph ConceptNet in advance and form triples (head entity, relation, tail entity).

[0103] Step 1.4: Considering that a keyword has many related words, in order to match more related words based on a keyword, the API interface provided by ConceptNet, a non-deterministic knowledge graph, is used to query the related words, relevance relationships, and relevance probabilities of each keyword in keyword set 1. Keywords for which no related words were found are deleted, and finally a dictionary in the form of many-to-many relationship between keywords and related words is formed, called dictionary 1. The dictionary also records the relevance relationship between keywords and related words, relevance probabilities, user ID, product ID, language of the related relationship, and number of related relationships.

[0104] Step 1.5: Dictionary 1 contains a large number of keywords. To improve the efficiency of word lookup, i.e., to improve the speed of matching word patterns, a double-array AC automaton, i.e., a double-array Trie tree, is used to optimize Dictionary 1. The entire tree is represented by two one-dimensional arrays, BASE and CHECK. The construction process of the BASE and CHECK arrays needs to satisfy the following state transition formula 1:

[0105] BASE[s]+c=t

[0106] CHECK[t] = s (1)

[0107] Where CHECK represents an array of predecessor node addresses, BASE represents an array of successor node base addresses, s and t both represent states, which have been numericalized and are initially set to 0; C corresponds to the encoding value of a letter in the character table.

[0108] The formula performs state transitions in two simple steps: first, the value of t is calculated by BASE[s]+c, and then the correctness of BASE[s]+c is determined by CHECK[t]=s.

[0109] Step 1.6: In keyword set 1, each keyword has a different probability of appearing. Sort the n keywords according to the word list order K = {k1, k2, ..., k} n}, using these keywords, construct an optimal search binary tree to minimize the total cost of querying elements; for each keyword k i There is a probability p for each. i Correspondingly, the probability p index starts from 1 and corresponds to each keyword;

[0110] Step 1.7: For a search tree, a search is successful when the searched element is already in the tree; a search fails when it is not. A "dummy leaf node" is used to represent a failed search, therefore n+1 dummy leaf nodes are needed {d0 < d1 < ... < d2}. n}, corresponding to d i The probability sequence is Q = {q0, q1, ..., q}n}; its d0 represents the failed result of searching for an element less than k1, d n This indicates that the search element is greater than k. n Failure scenarios; d i (0 < i < n) indicates that the search node is in k i and k (i+1) The failure scenarios during this period; therefore, the following formula exists:

[0111]

[0112] Where, p i Let q represent the probability of the i-th keyword. i This represents the probability of the i-th virtual leaf node;

[0113] Step 1.8: Determine the expected cost of a single search within a given binary search tree T based on the probability of each key and each dummy key (i.e., dummy leaf node) being searched; let the actual cost of a single search be the number of nodes checked, i.e., the depth of the nodes found in T plus 1; therefore, the expected cost of a single search within T is:

[0114]

[0115] Where E[T] represents the expected cost of a single search within T, and depth T (k i ) represents the k-th element within T. i The depth of the node;

[0116] Step 1.9: Define e[i,j] as the search term for a tree containing the keyword k. i ,......k j The expected cost of the optimal binary search tree is discussed in the following categories:

[0117] When j = i-1, it means that there is only virtual key d at this time. i-1 Therefore, e[i,i-1]=q i-1 ;

[0118] When j≥i, it is necessary to start from k i ,......k j Choose a root k from the middle r Then use the keyword k i ,......,k r-1 To construct an optimal binary search tree as the left subtree, use the key k. r+1 ,......,k j To construct an optimal binary search tree as the right subtree;

[0119] Define a tree with key k i,......k j The sum of probabilities of the subtrees is defined as w(i,j):

[0120]

[0121] Where i,j,r∈N + k r Let k represent the r-th keyword. r Here, it acts as the root of an optimal search binary subtree; p l Let q represent the probability of the l-th keyword. l This represents the probability of the l-th dummy leaf node;

[0122] Step 1.10: Therefore, if k r It is a tree containing the keyword k i ,......,k r-1 If the root of the optimal subtree is found, then:

[0123] e[i,j]=p r +(e[i,r-1]+w(i,r-1))+(e[r+1,j]+w(r+1,j))

[0124] w(i,j)=w(i,r-1)+p r +w(r+1,j) (5)

[0125] Where p r Represents the root node k r The probability of;

[0126] Step 1.11: Therefore, e[i,j] is rewritten as:

[0127] e[i,j]=e[i,r-1]+e[r+1,j]+w(i,j) (6)

[0128] Step 1.12: The final recursive formula:

[0129]

[0130] Step 2: Expand the relationships between entities, between items, and between entities and items in the two acquired datasets, Amazon-Book and Yelp2018 (also known as heterogeneous information networks), to obtain the final expanded heterogeneous information network;

[0131] Step 2.1: Perform word segmentation, stop word removal, and deduplication preprocessing on the comments in the two datasets. Since the preprocessing of the comments was already completed when the dictionary was created, this step uses the results of Step 1.1.

[0132] Step 2.2: Retrieve one review data point from the dataset. i After step 2.1, the keyword sequence [kw1,kw2,...,kw] is obtained. n ], where kw n It is a review i The nth keyword in the query, kw n The relevant information in the dictionary yields the related word sequence [rw1,rw2,...,rw] n ], correlation [r1,r2,...,r n ] and related probabilities [rp1,rp2,...,rp n ];

[0133] Step 2.3: Keyword kw i With related word sequence [rw1,rw2,...,rw n Each related word in ]rw j They all form a triple. ij =(kw i ,r j ,rw j The probability that the triple is true is rp. j , i,j∈[1,n];

[0134] Step 2.4: Each comment in the dataset is used to form several triples. ij The subgraph G of the comments is formed by these triples. sub ={(h,r,t)|h,t∈E,r∈R}, where h and t are the head node and tail node, respectively, E is the entity set; r represents the relationship between h and t, and R represents the relation set; directly subgraph G sub The edges are spliced ​​into the heterogeneous information network; duplicate edges are not spliced ​​during the splicing process; the final expanded heterogeneous information network is obtained; the final expanded heterogeneous information network diagram is shown below. Figure 2 As shown; this expanded heterogeneous information network can also be understood as an expanded knowledge graph. This knowledge graph not only contains the original user entities, items, and deterministic relationships between user entities and items in the Amazon-Book or Yelp2018 datasets, but also includes expanded entities, expanded items, uncertain relationships between expanded entities and expanded items, uncertain relationships between expanded entities and original entities, uncertain relationships between expanded items and original items, uncertain relationships between original entities and expanded items, and uncertain relationships between expanded entities and original items; each relationship is represented in the form of a path, with two attributes: relationship and probability.

[0135] Step 3: Convert the reasonable values ​​of triples in the heterogeneous information network into confidence scores; for a given triple l, the reasonable value g(l) ∈ R. 1 This indicates how likely it is that the relational fact represented by the triple is true; the higher this value, the higher the confidence score corresponding to the triple.

[0136] Step 3.1: Given a triple l = (h, r, t) and their embeddings e h ,e r ,e t The estimated value of a triple is calculated using the following formula:

[0137]

[0138] Among them, The dot (·) represents element-wise multiplication, while the dot (·) represents the inner product. This function, derived from DistMult, captures e under the condition of relation r. h and e t The relationship between them;

[0139] Step 3.2: To convert the estimated value into a confidence score, the Logistic mapping function is used. This function converts the estimated score g(l) into a confidence score f(l).

[0140] f(l)=φ(g(l))∈[0,1] (9)

[0141] The Logistic mapping function is a common S-shaped curve:

[0142]

[0143] Step 3.3: For a given observable triple l, there is a confidence score s for considering it true. l , where s l ∈[0,1], using two observable triples to infer unobservable relational facts, let I represent the interaction relation I(l) of a triple, where the confidence score is calculated as follows:

[0144]

[0145] Where ζ + Let represent the set of observed triples, and ζ - Let s represent the set of unobservable triples. l represents the confidence score of the observed triples, while f(l) represents the confidence score of the unobserved triples obtained through the mapping function;

[0146] Step 3.4: After expanding the heterogeneous information network, both deterministic and non-deterministic triples exist. For deterministic triples, the true confidence score s of the triples is calculated. l The variance of the predicted confidence score f(l) is used to reflect the embedding ability of the UKGAT model:

[0147]

[0148] The UKGAT is an improvement on KGAT, which utilizes a lexicon and embeds the uncertain knowledge graph UKGE.

[0149] Step 3.5: For nondeterministic triples l, minimize the confidence score f(l) obtained by logical reasoning PSL for each triple, using the variance of the distance as the loss function:

[0150]

[0151] Where ψ γ (f(l)) represents the mapping function for calculating the weighted distance, and γ represents a given PSL rule;

[0152] Step 3.6: By simultaneously solving equations (6) and (7), the following objective function is obtained:

[0153]

[0154] Step 4: Use the attention mechanism to recursively propagate the information of nodes in the multi-layer graph to embed it into the representation of entities and items, and aggregate the representations of entities and items in the graph to obtain the predicted value.

[0155] This predicted value refers to the matching score or association score between the entity and the item; it can also be understood as the probability of recommending this item to this entity. A higher predicted value indicates better recommendation performance of the UKGAT model; it also indicates a higher probability that the entity and the item match. The reason for obtaining this predicted value is for use in subsequent recommendation tasks. By improving the model's recommendation metrics, a good recommendation metric can demonstrate that the model has good recommendation performance in real-world recommendation scenarios.

[0156] Step 4.1: Consider an entity h, using S h = {(h,r,t)|(h,r,t)∈G} represents the set of triples, where h is the head entity, also known as the self-network. To characterize the first-order connectivity structure of entity h, the linear structure of the self-network of entity h is calculated using the following formula:

[0157]

[0158] Where π(h,r,t) represents the attenuation factor for each propagation on the edge (h,r,t), and represents the amount of information propagated from t through r to h each time.

[0159] Step 4.2: To calculate the attenuation factor π(h,r,t), the tanh nonlinear activation function is used, as shown in the following formula:

[0160] π(h,r,t)=(W r e t ) T ·tanh(W r e h +e r (16)

[0161] Among them W r ∈R k×d is the transition matrix of relation r, used to project entities from the d-dimensional entity space to the k-dimensional relation space; π(h,r,t) represents the attention score of the triple (h,r,t), which depends on the distance between entities h and t in the relation r space. For example, closer entities can propagate more information.

[0162] Step 4.3: Normalize the attenuation factor obtained in the previous step, using the following formula:

[0163]

[0164] Step 4.4: Represent entity e h and self-network representation As a new representation of entity h This is a single-layer self-network; an aggregator is selected—a bi-interaction aggregator—which carefully considers e h and The interaction features between them are designed using the following formula:

[0165]

[0166] Where W1, W2 ∈ R d′×d It is a trainable matrix; in summary, the advantage of the embedded propagation layer is that it explicitly uses first-order connectivity information to associate user, item, and knowledge entity representations.

[0167] Step 4.5: To further stack more propagation layers to explore higher-order connection information, information propagated from higher-hop neighbors is collected; in layer l, the entity representation is recursively formulated as:

[0168]

[0169] The propagation information of entity h in the self-network at step l is as follows:

[0170]

[0171] It is a representation of entity t generated from the previous information propagation steps, storing information from its l-1 hop neighbors; It is set to e during the initial information propagation iteration. h It further facilitates the representation of entity h at layer l;

[0172] Step 4.6: After executing layer l, multiple representations of user node u are obtained, namely... Similar to item node i, we get Nondeterministic knowledge graph embedding frameworks based on attention networks and extended metapaths, such as... Figure 3 As shown, the model consists of an extended CKG model, an attention embedding propagation layer, and a prediction layer. Since the output of the l-th layer is an aggregation of messages from a tree rooted at u or i, the outputs of different layers emphasize connectivity information of different orders. Finally, a layer aggregation mechanism is used to connect the representation of each step to a vector, as shown below:

[0173]

[0174] Here, ‖ is the join operation, which not only enriches the initial embedding by performing the embedding propagation operation, but also allows the propagation strength to be controlled by adjusting l;

[0175] Step 4.7: Finally, perform the inner product of the user and item representations to obtain the predicted score:

[0176]

[0177] This invention provides a nondeterministic knowledge graph embedding method based on KGAT, which augments meta-path semantic information. It not only inherits the KGAT model's end-to-end simulation of high-order connectivity between nodes and recursively transmits embedding information (which can be users, items, or attributes) from neighboring nodes to refine node embeddings, but also employs an attention mechanism that distinguishes importance during information propagation. Furthermore, by augmenting the meta-path semantics in heterogeneous information networks, it allows relationships between entities to be both deterministic and nondeterministic, thereby better capturing latent semantic relationships between entities and integrating their structured knowledge into machine learning. Experiments on two real-world datasets demonstrate the effectiveness of the UKGAT model, outperforming existing models in capturing semantic relationships between entities and making it more suitable for many knowledge-driven applications.

[0178] A comparison of the overall performance of the UKGAT model of this invention;

[0179]

[0180] After improvements to the KGAT model, the UKGAT model's recall@20 performance on the Amazon-Book and Yelp2018 datasets was improved by 3.16% and 10.97% respectively compared to the original KGAT. Furthermore, UKGAT demonstrated superior performance with increasing layer count.

[0181] By using the heterogeneous information network meta-path semantic augmentation theory of this invention, the information of entities in the knowledge graph (including relationships between entities, relationships between entities and items, and relationships between items) can be made more complete, which greatly helps the subsequent recommendation task, making the recommendation results more interpretable and diverse, and improving the accuracy of the recommendation. It takes into account the high-order connectivity of entities with multi-hop relationships in the knowledge graph, which is more conducive to the information propagation of neighbor nodes. This invention combines the attention mechanism to recursively propagate the embedding information of neighbor nodes, so that a node has the embedding information of multiple layers of neighbor nodes in the embedding space, thereby improving the embedding features of the node.

Claims

1. A nondeterministic knowledge graph embedding method based on attention network-enhanced meta-paths, characterized in that, Includes the following steps: Step 1: Obtain two publicly available datasets containing comment information for recommendation purposes. Based on these datasets, create a related word lexicon and optimize the lexicon for subsequent expansion of the heterogeneous information network. Step 1.1: Obtain two publicly available datasets containing comment information that can be used for recommendation. Extract comment data from the two datasets. Segment each comment data to obtain the original keyword set. Then remove stop words and duplicate keywords from the original keyword set to form comment keyword set 1. Step 1.2: Associate related words with keywords, that is, use non-deterministic knowledge graphs to supplement the non-deterministic relationship between keywords and related words, and select the knowledge structure of ConceptNet as the auxiliary knowledge base for associating related words with keywords; Step 1.3: The real purpose of building the thesaurus is to find the correlation between words and form the triple format between words. Therefore, it is necessary to retrieve the keywords of keyword set 1 in the non-deterministic knowledge graph ConceptNet in advance and form triples (head entity, relation, tail entity). Step 1.4: Considering that a keyword has many related words, in order to match more related words based on a keyword, the API interface provided by ConceptNet, a non-deterministic knowledge graph, is used to query the related words, relevance relationships, and relevance probabilities of each keyword in keyword set 1. Keywords for which no related words were found are deleted, and finally a dictionary in the form of many-to-many relationship between keywords and related words is formed, called dictionary 1. The dictionary also records the relevance relationship between keywords and related words, relevance probabilities, user ID, product ID, language of the related relationship, and number of related relationships. Step 1.5: Optimize dictionary 1 using a double-array AC automaton, i.e., a double-array Trie tree; where the entire tree is represented by two one-dimensional arrays BASE and CHECK, and the construction process of the BASE and CHECK arrays needs to satisfy the following state transition formula 1: （1） Where CHECK represents an array of predecessor node addresses, BASE represents an array of successor node base addresses, s and t both represent states, which have been numericalized and are initially set to 0; C corresponds to the encoding value of a letter in the character table. The formula performs state transitions in two simple steps: first, the value of t is calculated by BASE[s]+c, and then the correctness of BASE[s]+c is determined by CHECK[t]=s. Step 1.6: In keyword set 1, each keyword has a different probability of appearing. Sort the n keywords according to the order of the word list. We construct an optimal binary search tree using these keywords to minimize the total cost of querying elements; for each keyword... There is a probability for each. Correspondingly, probability The subscripts start from 1 and correspond to each keyword; Step 1.7: For a search tree, a search is successful when the searched element is already in the tree; a search fails when it is not. A "dummy leaf node" is used to represent a failed search. Therefore, it is necessary to... virtual leaf nodes , corresponding to The probability sequence is ;That Indicates that the search element is less than The result of the failure Indicates that the search element is greater than Failure scenarios; Indicates the search node is in and The failure scenarios during this period; therefore, the following formula exists: （2） in, Indicates the first The probability of each keyword. Indicates the first The probability of a virtual leaf node; Step 1.8: Determine the expected cost of a single search within a given binary search tree T based on the probability of each key and each dummy key (i.e., dummy leaf node) being searched; let the actual cost of a single search be the number of nodes checked, i.e., the depth of the nodes found in T plus 1; therefore, the expected cost of a single search within T is: （3） in, Indicates in The expected cost of a single search within the scope, express Inner The depth of the node; Step 1.9: Definition To search for a tree containing the keyword The expected cost of the optimal binary search tree is then discussed in separate cases. Define a tree with keywords The subtree, the sum of probabilities is defined as : （4） in , Indicates the first One keyword, Here, it serves as the root of an optimal search binary subtree; Indicates the first The probability of each keyword. Indicates the first The probability of a virtual leaf node; Step 1.10: Therefore, if It is a tree containing keywords If the root of the optimal subtree is found, then: （5） in Represents the root node The probability of; Step 1.11: Therefore Rewritten as: （6） Step 1.12: The final recursive formula: （7） Step 2: Expand the two datasets obtained, which are also called the relationships between entities, items, and entities and items in the heterogeneous information network, to obtain the final expanded heterogeneous information network; Step 3: Convert the reasonable values ​​of triples in heterogeneous information networks into confidence scores; for a given triple... Then the reasonable value of this triplet This indicates how likely it is that the relational fact represented by the triple is true; the higher this value, the higher the confidence score corresponding to the triple. Step 4: Use the attention mechanism to recursively propagate the information of nodes in the multi-layer graph to embed it into the representation of entities and items, and aggregate the representations of entities and items in the graph to obtain the predicted value.

2. The nondeterministic knowledge graph embedding method based on attention network augmented meta-paths according to claim 1, characterized in that, The classification discussion described in step 1.9 specifically includes: when When this happens, it means that only virtual keys are available at this time. Therefore ; when At that time, it is necessary to start from Choose a root Then use keywords To construct an optimal binary search tree as the left subtree, use the key... To construct an optimal binary search tree as the right subtree.

3. The nondeterministic knowledge graph embedding method based on attention network augmented meta-paths according to claim 1, characterized in that, Step 2 is as follows: Step 2.1: Perform word segmentation, stop word removal, and deduplication preprocessing on the comments in the two datasets. Since the preprocessing of the comments was already completed when the dictionary was created, this step uses the results of Step 1.

1. Step 2.2: Retrieve one comment from the dataset. The keyword sequence is obtained through step 2.

1. ,in yes The Middle Search using keywords The relevant information in the dictionary is used to obtain a sequence of related words. Relationship and related probabilities ; Step 2.3: Keywords Related word sequences Each related word in They all form a triplet The probability that the triple is true is , ; Step 2.4: Each comment in the dataset forms several triples. These triples form a subgraph of comments. Where h and t are the head node and tail node respectively, E is the entity set; r represents the relationship between h and t, and R represents the relation set; directly subgraph The edges are concatenated into a heterogeneous information network. During the concatenation process, duplicate edges are not concatenated. This results in the final expanded heterogeneous information network, which can also be understood as an expanded knowledge graph. This knowledge graph contains not only the original user entities, items, and deterministic relationships between user entities and items in the two datasets, but also expanded entities, expanded items, uncertain relationships between expanded entities and expanded items, uncertain relationships between expanded entities and original entities, uncertain relationships between expanded items and original items, uncertain relationships between original entities and expanded items, and uncertain relationships between expanded entities and original items. Each relationship is represented as a path, with two attributes: relationship and probability.

4. The nondeterministic knowledge graph embedding method based on attention network augmented meta-paths according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Given a triplet and their embedding The estimated value of a triple is calculated using the following formula: （8） Among them, This represents element-wise product, while This function, which represents the inner product, is derived from DistMult and captures the inner product in relational functions. Under the conditions and The relationship between them; Step 3.2: To convert the estimated values ​​into confidence scores, the Logistic regression function was used. This function converts the estimated scores... Convert to confidence score : （9） The Logistic mapping function is a common S-shaped curve: （10） Step 3.3: For a given observable triplet Each has a confidence score for being considered true. ,in Using two observable triples to infer unobservable relational facts, To represent the interaction relationship of a triple. The confidence score is calculated using the following formula: （11） in This represents the set of observed triples, while Represents the set of unobservable triples. This represents the confidence score of the observed triples, while This represents the confidence score for unobservable events obtained through the mapping function; Step 3.4: After expanding the heterogeneous information network, both deterministic and non-deterministic triples exist. For deterministic triples, the true confidence score of the triple is calculated. With the predicted confidence score The variance is used to reflect the embedding ability of the UKGAT model: （12） The UKGAT is an improvement on KGAT, which utilizes a lexicon and embeds the uncertain knowledge graph UKGE. Step 3.5: For nondeterministic triples Minimize the confidence score obtained by logical reasoning PSL for each triple. The variance of the distance will be used as the loss function: （13） in This represents the mapping function for calculating weighted distances. This represents a given PSL rule; Step 3.6: By simultaneously solving equations (6) and (7), the following objective function is obtained: （14）。 5. The nondeterministic knowledge graph embedding method based on attention network augmented meta-paths according to claim 1, characterized in that, Step 4 is as follows: Step 4.1: Consider an entity ,use Denotes the set of triples, where It is the head entity, also known as the self-network, used to represent entities. First-order connected structure, computational entity The linear structure of the self-network; Step 4.2: To calculate the attenuation factor ,use The non-linear activation function is shown in the following formula: （16） in It is a relationship The transition matrix is ​​used to transfer entities from Projecting a dimensional entity space onto In a dimensional relational space; Represents a triplet Attention score, attention score depends on the relationship Entities in space and entity The distance; Step 4.3: Normalize the attenuation factor obtained in the previous step, using the following formula: （17） Step 4.4: Represent the entity and self-network representation As an entity New representation This is a self-network of one layer; an aggregator—a bidirectional interactive aggregator—is selected, which is carefully considered... and The interaction features between them are designed using the following formula: （18） in, It is a trainable matrix; Step 4.5: To further stack more propagation layers to explore higher-order connection information, collect information propagated from higher-hop neighbors; in the... Within the layer, the entity representation is recursively formulated as follows: （19） Among them, in the first Entities in the self-network of steps The information being disseminated is as follows: （20） Entities generated from previous information dissemination steps The representation, storage comes from its Jump to neighbor information; It was set during the initial information propagation iteration. It further helps entities exist Layer representation; Step 4.6: During execution After the layer, the user node was obtained. Multiple representations, namely Similar to item nodes ,get ; due to the The output of the layer is or as the root The tree of layers aggregates messages, thus the outputs of different layers emphasize connectivity information of different orders; finally, a layer aggregation mechanism is used to connect the representation of each step into a vector, as shown below: （21） Here, || represents the join operation, which not only enriches the initial embedding by performing embedding propagation operations, but also allows for adjustments... To control the intensity of transmission; Step 4.7: Finally, perform the inner product of the user and item representations to obtain the predicted score: （22）。 6. The nondeterministic knowledge graph embedding method based on attention network augmented meta-paths according to claim 5, characterized in that, The computational entity described in step 4.1 The linear structure of the self-network is specifically as follows: （15） in Indicates in The attenuation factor for each propagation along the edge, representing each propagation from... go through spread to The amount of information disseminated.

Images