A knowledge graph question answering method based on a double-cascade graph attention framework

Abstract

The application discloses a knowledge graph question answering method based on a double-cascade graph attention framework, and comprises the following steps: S1, preprocessing and coding of a question through an initialization layer to obtain a word-level question vector and a sentence vector of the question; S2, calculating the semantic contribution degree of each word and fusing the semantic contribution degree into the initial question vector; and S3, calculating a relationship score and calculating a tail entity score by using a head entity and the relationship score. The method can effectively solve the problem of insufficient understanding of question and relationship semantics in multi-hop reasoning and improve the performance of reasoning.

Description

Technical Field

[0001] This invention relates to the field of language processing technology, specifically to a knowledge graph question answering method based on a bi-cascaded graph attention framework. Background Technology

[0002] Knowledge graph question answering (KBQA) is a key upstream task for applications such as intelligent search, personalized recommendation, and human-computer dialogue systems. Its goal is to accurately locate the answer from the knowledge base by deeply analyzing the question and combining it with the structure and content of the knowledge graph. In multi-hop KBQA scenarios, the grammatical structure of the questions is usually quite complex, involving many entities and relationships. Solving these questions often requires multi-hop reasoning, which demands not only strong semantic understanding capabilities but also the ability to handle relational reasoning across multiple entities. This area has attracted widespread research attention.

[0003] Challenges in KBQA tasks include information loss during encoding and compression, lack of fine-grained question analysis, and weak interpretability of the reasoning process. Recent research has primarily focused on semantic parsing-based methods, embedding-based methods, and path-based methods. For example, Lan proposed a query graph generation method, but it faces problems such as a large query space and insufficient query graph effectiveness. The EmbedKGQA method proposed by Saxena et al. is a typical embedding-based method; its performance is limited by the quality of the knowledge base embeddings and heavily relies on the embedding of entities and relations. The path-based method proposed by Shi et al. can significantly improve the interpretability of multi-hop reasoning, but it lacks comprehensive attention to the semantic information of the question. Overall, path-based methods can capture complex relational paths and efficiently handle multi-hop reasoning problems, making them a popular mainstream approach.

[0004] Despite positive progress in existing research, there are still shortcomings in the understanding of the interaction between word and sentence-level information in complex problems, as well as the information transfer at different reasoning time steps. Therefore, these shortcomings prevent a comprehensive understanding of the semantics of the problem, particularly hindering reasoning for multi-hop problems. Furthermore, capturing the complex relationships between the problem and the answer during the reasoning process is crucial. Summary of the Invention

[0005] This invention aims to solve the technical problems existing in the prior art. In particular, it innovatively proposes a knowledge graph question answering method based on a dual-cascaded graph attention framework, which can effectively solve the problem of insufficient understanding of question and relation semantics in multi-hop reasoning and improve reasoning performance.

[0006] To achieve the above objectives, this invention provides a knowledge graph question answering method based on a bi-cascaded graph attention framework, comprising the following steps:

[0007] S1: The question is preprocessed and encoded through the initialization layer to obtain the word-level question vector and sentence vector of the question;

[0008] S2: Calculate the semantic contribution of each word and integrate it into the initial question vector;

[0009] S3: Calculate the relation score, and use the head entity and relation score to calculate the tail entity score.

[0010] In the above scheme, step S1 further includes the following steps:

[0011] S1-1: The input question is pre-trained using BERT to obtain a word-level question vector;

[0012] S1-2: Combine the word-level question vector Q0 obtained after BERT pre-training with the word vector Q at the current step size. t (t∈[0,T]) are fused to obtain the self-attention matrix M s The formula is as follows:

[0013] M s =MLP(Q t )×Q0

[0014] Among them, M s MLP(·) represents a self-attention matrix, MLP(·) represents a linear fully connected layer, t is the number of steps in the current inference stage, and T is the total number of inference steps.

[0015] S1-3: The self-attention matrix M s The key information distribution M of the problem under the current number of hops is obtained by transforming using Softmax. s' The formula is as follows:

[0016] M s' =Softmax(M s )

[0017] Among them, M s' This represents the distribution of key information about the problem at the current inference stage, where Softmax(·) is the activation function.

[0018] S1-4: M s' Embedded into Q t In the process, we obtain the inference representation information R. t The formula is as follows:

[0019] R t =M s' ×Q t .

[0020] In the above scheme, step S2 also includes the following:

[0021] S2-1: Perform word-level reasoning on the input question;

[0022] S2-2: Calculate the weights of the recurrent self-attention;

[0023] S2-3: Integrating sentence vector reasoning;

[0024] S2-4: Semantic information q represented by fine-grained information t Processed via VGAT.

[0025] In the above scheme, step S2-1 further includes the following steps:

[0026] When performing the first inference, execute S2-1-1; otherwise, execute S2-1-2.

[0027] S2-1-1: The word-level question vector Q0 obtained after BERT pre-training and the inference representation information R0 from the first inference stage are processed and transformed through the word inference layer, as shown in the following formula:

[0028] q0 = Q0 + R0

[0029] Where q0 represents the initial semantic information;

[0030] S2-1-2: Store the reasoning representation information for the current reasoning stage;

[0031] The formula is as follows:

[0032] U0 = R0

[0033] Where U0 is the initial storage unit; return to step S2-1;

[0034] S2-1-3: Store the reasoning representation information for the current reasoning stage;

[0035] The formula is as follows:

[0036] U t =Z t ×U t-1 +R t

[0037] Among them, U t U is the storage unit for the current inference phase. t-1 Z is the storage unit from the previous inference phase. t Z represents the key information extracted in the previous reasoning phase. t The calculation is performed using the following formula:

[0038] Z t =Sigmoid(MLP(U t-1 ))

[0039] Where MLP(·) represents a linear fully connected layer, and Sigmoid(·) is the activation function;

[0040] S2-1-4: Calculate the semantic information q of the current reasoning stage. t ;

[0041] q t =q0+Z t ×U t-1

[0042] Where q0 represents the initial semantic information, U t-1 This is the storage unit for the previous inference phase;

[0043] When t>0, the information storage unit also needs to be integrated into step t-1 via Z. t The filtered information is used to obtain a more fine-grained dynamic problem representation q. t This representation is then incorporated into the initialization semantic information q0.

[0044] In the above scheme, step S2-2 also includes the following:

[0045] S2-2-1: Calculate the initial attention weights;

[0046] S2-2-1-1: Distribution of key information M for the problem s' Perform one-dimensional summation and normalization to obtain the attention score;

[0047] S2-2-1-2: Multiply the attention score by the attention mask att_mask to ensure that the model only pays attention to the attention scores of valid words;

[0048] S2-2-1-3: The attention scores are summed in two dimensions again and normalized to obtain the final initial attention weights;

[0049] That is, the calculation is performed using the following formula:

[0050]

[0051] in, To initialize the attention weights, ∑ col To sum by column, ∑ row To sum by row, att_mask is the attention mask.

[0052] S2-2-2: Calculate the final attention weights

[0053] In each iteration, the final weights obtained from the previous calculation are used as the initial values ​​for the next calculation, and the weights are calculated using the following formula:

[0054]

[0055] The number of iterations, n, is a hyperparameter.

[0056] S2-2-3: Calculate the problem vector for predicting the relationship;

[0057] The formula is as follows:

[0058]

[0059] in, The problem vector is for predicting relationships.

[0060] In the above scheme, step S2-3 also includes the following:

[0061] S2-3-1: Calculate the initial relation prediction vector;

[0062] The sentence vector q obtained after BERT pre-training s With the problem vector The initial relation prediction vector r0 is obtained by adding the vectors together, as shown in the following formula:

[0063]

[0064] q s Let the sentence vector of the question be...

[0065] S2-3-2: Store the question vector of the predicted relationship in the current inference stage;

[0066] The formula is as follows:

[0067]

[0068] in, For the initial storage unit, This represents the relation representation selected in the previous reasoning phase, and The calculation is performed using the following formula:

[0069]

[0070] Where MLP(·) represents a linear fully connected layer, and Sigmoid(·) is the activation function;

[0071] S2-3-3: Calculate the final relation prediction vector r t ;

[0072] The formula is as follows:

[0073]

[0074] Where, r t This is the final relation prediction vector.

[0075] In the above scheme, step S2-4 also includes the following steps:

[0076] S2-4-1: Let the nodes in the vector graph be:

[0077] N={token1,token2,token3,...,token i}, where i∈[1,N], token i This represents the feature representation of the i-th node;

[0078] S2-4-2: Construct the self-associative feature adjacency matrix e-adj;

[0079] S2-4-3: Convert using the following formula:

[0080] W k =e-adj

[0081] W k It is the weight matrix of the corresponding input linear transformation;

[0082] S2-4-4: Calculate the attention of all nodes to their neighboring nodes;

[0083] The calculation is performed using the following formula:

[0084]

[0085] Where, α ij For the attention between vertices i and j, a T Let be the transpose of a 2F'-dimensional vector; LeakyReLU is the activation function, exp is the exponential function, W is the trainable parameters, Ni is the set of neighboring nodes of the i-th node; || denotes a connection.

[0086] S2-4-5: Calculate the final representation of the i-th node using its neighboring nodes;

[0087] The calculation is performed using the following formula:

[0088]

[0089] Where K represents the number of attention mechanisms; The attention between vertices i and j is caused by the k-th attention mechanism (α). k The normalized attention coefficient calculated; W k It is the weight matrix of the corresponding input linear transformation; σ represents the coefficients;

[0090] S2-4-6: Aggregation via VGAT layer;

[0091] The calculation is performed using the following formula:

[0092]

[0093] Where sum_token represents summation;

[0094] S2-4-7: Problem vector relating the aggregated features of the VGAT layer to the predictions To merge

[0095] The formula is as follows:

[0096]

[0097] r v This represents the relation vector after fusion through the VGAT layer;

[0098] S2-4-8: The final relation prediction vector r is obtained by combining a dual-cascaded framework and a vector graph attention layer. final ;

[0099] The formula is as follows:

[0100] The formula is as follows:

[0101] r final =r v +r t

[0102] Where, r v r represents the relation vector after fusion through the VGAT layer. final This represents the final relation prediction vector.

[0103] In the above scheme, step S3 also includes the following steps:

[0104] S3-1: Perform multi-label classification on the relationships in the knowledge graph to obtain the answer entity vector;

[0105] The calculation is performed using the following formula:

[0106] e t =f(e t-1 ,Sigmoid(MLP(r final )))

[0107] Among them, e t Let f(·) be the vector of answer entities, and f(·) denotes the score of all entities at the current inference stage.

[0108] S3-2: Predict the final answer;

[0109] The calculation is performed using the following formula:

[0110]

[0111] Where Y is the predicted final answer, e t The entity vector for the answer; r final For relation prediction vectors;

[0112] S3-3: Training on the final answer;

[0113] Training can be performed using the following formula:

[0114] L=α(Yy) 2 +βD kl (Y,y)

[0115] Where y is the true value, Y is the predicted final answer, L is the loss function, and D is the final value. kl (·) represents the kl divergence, and α and β are adjustable parameters.

[0116] In summary, the beneficial effects of this invention are as follows: It proposes a dual-cascade reasoning framework that integrates word-level information to enhance question representation and embeds sentence information to enhance relation representation. The two levels are connected by a recurrent self-attention mechanism, which can effectively solve the problem of insufficient understanding of question and relation semantics in multi-hop reasoning and improve reasoning performance. Between each reasoning time step, the attention weights are calculated multiple times and the question representation is continuously updated through a recurrent self-attention mechanism, allowing for more granular capture of the semantic information of the question. In addition, a vector graph attention network for relation enhancement is designed to effectively alleviate the problem of non-critical path interference during reasoning. Attached Figure Description

[0117] Figure 1 This is a schematic diagram of the structure of the present invention.

[0118] Figure 2 This is a schematic diagram of the VGAT layer structure.

[0119] Figure 3 This is a statistical chart showing the rise and fall of the score for the nth inference problem after adding the VGAT layer.

[0120] Figure 4 This is a flowchart illustrating the vector graph attention network.

[0121] Figure 5 This is a bar chart showing the number of increases in relation scores on the MetaQA dataset.

[0122] Figure 6 This is a bar chart showing the number of relationships whose scores decreased on the MetaQA dataset. Detailed Implementation

[0123] The present invention will be further described below with reference to the embodiments and accompanying drawings:

[0124] like Figures 1-6 As shown, a knowledge graph question answering method based on a bi-cascaded graph attention framework includes the following steps:

[0125] S1: The question is preprocessed and encoded through the initialization layer to obtain the word-level question vector and sentence vector of the question;

[0126] S1-1: The input question is pre-trained using BERT to obtain a word-level question vector;

[0127] S1-2: Combine the word-level question vector Q0 obtained after BERT pre-training with the word vector Q at the current step size. t (t∈[0,T]) are fused to obtain the self-attention matrix M s The formula is as follows:

[0128] M s =MLP(Q t )×Q0

[0129] Among them, M s MLP(·) represents a self-attention matrix, MLP(·) represents a linear fully connected layer, t is the number of steps in the current inference stage, and T is the total number of inference steps.

[0130] S1-3: The self-attention matrix M s The key information distribution M of the problem under the current number of hops is obtained by transforming using Softmax. s' The formula is as follows:

[0131] M s' =Softmax(M s )

[0132] Among them, M s' This represents the distribution of key information about the problem at the current inference stage, where Softmax(·) is the activation function.

[0133] S1-4: M s' Embedded into Q t In the process, we obtain the inference representation information R. t The formula is as follows:

[0134] R t =M s' ×Q t

[0135] S2: Calculate the semantic contribution of each word and integrate it into the initial question vector;

[0136] S2-1: Perform word-level reasoning on the input question; execute S2-1-1 when performing the first reasoning, otherwise execute S2-1-2.

[0137] S2-1-1: The word-level question vector Q0 obtained after BERT pre-training and the inference representation information R0 from the first inference stage are processed and transformed through the word inference layer, as shown in the following formula:

[0138] q0 = Q0 + R0

[0139] Where q0 represents the initial semantic information;

[0140] S2-1-2: Store the reasoning representation information for the current reasoning stage;

[0141] The formula is as follows:

[0142] U0 = R0

[0143] Where U0 is the initial storage unit; return to step S2-1;

[0144] S2-1-3: Store the reasoning representation information for the current reasoning stage;

[0145] The formula is as follows:

[0146] U t =Z t ×U t-1 +R t

[0147] Among them, U t U is the storage unit for the current inference phase. t-1 Z is the storage unit from the previous inference phase. t Z represents the key information extracted in the previous reasoning phase. t The calculation is performed using the following formula:

[0148] Z t =Sigmoid(MLP(U t-1 ))

[0149] Where MLP(·) represents a linear fully connected layer, and Sigmoid(·) is the activation function;

[0150] S2-1-4: Calculate the semantic information q of the current reasoning stage. t ;

[0151] q t =q0+Z t ×U t-1

[0152] Where q0 represents the initial semantic information, U t-1This is the storage unit for the previous inference phase;

[0153] When t>0, the information storage unit also needs to be integrated into step t-1 via Z. t The filtered information is used to obtain a more fine-grained dynamic problem representation q. t This representation is then incorporated into the initialization semantic information q0.

[0154] S2-2: Calculate the weights of the recurrent self-attention;

[0155] S2-2-1: Calculate the initial attention weights;

[0156] S2-2-1-1: Distribution of key information M for the problem s' Perform one-dimensional summation and normalization to obtain the attention score;

[0157] S2-2-1-2: Multiply the attention score by the attention mask att_mask to ensure that the model only pays attention to the attention scores of valid words;

[0158] S2-2-1-3: The attention scores are summed in two dimensions again and normalized to obtain the final initial attention weights;

[0159] That is, the calculation is performed using the following formula:

[0160]

[0161] in, To initialize the attention weights, ∑ col To sum by column, Σ row For row-wise summation, att_mask is the attention mask;

[0162] In this way, the semantic information in the question weight matrix can be accurately captured and applied to reasoning.

[0163] S2-2-2: Calculate the final attention weights

[0164] In each iteration, the final weights obtained from the previous calculation are used as the initial values ​​for the next calculation, and the weights are calculated using the following formula:

[0165]

[0166] The number of iterations, n, is a hyperparameter that varies depending on the dataset. In this example, n is 3 on the WebQSP and MetaQA datasets.

[0167] S2-2-3: Calculate the problem vector for predicting the relationship; assign weights... Integrating into the representation q of dynamic problems t In this process, a problem vector for predicting relationships is obtained, which improves the model's expressive power and reasoning accuracy;

[0168] The formula is as follows:

[0169]

[0170] in, The question vector for predicting the relationship;

[0171] S2-3: Integrating sentence vector reasoning;

[0172] S2-3-1: Calculate the initial relation prediction vector;

[0173] The sentence vector q obtained after BERT pre-training s With the problem vector The initial relation prediction vector r0 is obtained by adding the vectors together, as shown in the following formula:

[0174]

[0175] q s Let the sentence vector of the question be...

[0176] S2-3-2: Store the question vector of the predicted relationship in the current inference stage;

[0177] The formula is as follows:

[0178]

[0179] in, For the initial storage unit, This represents the relation representation selected in the previous reasoning phase, and The calculation is performed using the following formula:

[0180]

[0181] Where MLP(·) represents a linear fully connected layer, and Sigmoid(·) is the activation function;

[0182] S2-3-3: Calculate the final relation prediction vector r t ;

[0183] The formula is as follows:

[0184]

[0185] Where, r t Predict the final relation vector;

[0186] S2-4: Semantic information q represented by fine-grained information t Processed via VGAT;

[0187] The problem vector q represented by fine-grained information t Each token in the algorithm is treated as a virtual node, and the hidden layer output of each token is considered as the node feature. We use q... t The token represents the construction of a self-associative feature adjacency matrix e-adj, such as Figure 2 As shown. Although graph neural networks typically aggregate information from neighboring nodes to learn global features, the association matrix ensures that each node has at least its own information, preserving the node's own feature information while considering its contextual information. Then, q... t The e-adj is embedded into the GAT layer to obtain the problem representation after VGAT.

[0188] GAT uses an attention mechanism to calculate the weights between each node in the graph and its neighboring nodes. The training process of GAT depends on the relationships between node pairs and is not affected by the specific network structure.

[0189] S2-4-1: Let the nodes in the vector graph be:

[0190] N={token1,token2,token3,...,token i}, where i∈[1,N], token i This represents the feature representation of the i-th node;

[0191] S2-4-2: Construct the self-associative feature adjacency matrix e-adj;

[0192] S2-4-3: Convert using the following formula:

[0193] W k =e-adj

[0194] W k W is the weight matrix corresponding to the linear transformation of the input; the self-correlation feature adjacency matrix e-adj is regarded as the weight matrix W. k ;

[0195] S2-4-4: Calculate the attention of all nodes to their neighboring nodes;

[0196] The calculation is performed using the following formula:

[0197]

[0198] Where, α ij For the attention between vertices i and j, a TLet be the transpose of a 2F'-dimensional vector; LeakyReLU is the activation function, exp is the exponential function, W is the trainable parameters, Ni is the set of neighboring nodes of the i-th node; || denotes a connection.

[0199] S2-4-5: Calculate the final representation of the i-th node using its neighboring nodes;

[0200] The calculation is performed using the following formula:

[0201]

[0202] Where K represents the number of attention mechanisms; The attention between vertices i and j is caused by the k-th attention mechanism (α). k The normalized attention coefficient calculated; W k It is the weight matrix of the corresponding input linear transformation; σ represents the coefficients;

[0203] S2-4-6: Aggregation via VGAT layer;

[0204] The calculation is performed using the following formula:

[0205]

[0206] Here, sum_token represents summation.

[0207] S2-4-7: Problem vector relating the aggregated features of the VGAT layer to the predictions To merge

[0208] The formula is as follows:

[0209]

[0210] r v This represents the relation vector after fusion through the VGAT layer;

[0211] S2-4-8: The final relation prediction vector r is obtained by combining a dual-cascaded framework and a vector graph attention layer. final ;

[0212] The formula is as follows:

[0213] r final =r v +r t

[0214] Where, r v r represents the relation vector after fusion through the VGAT layer. final This represents the final relation prediction vector;

[0215] S3: Calculate the relation score, and use the head entity and relation score to calculate the tail entity score;

[0216] To enhance the expressive power of the current path and improve the accuracy of relation prediction, this technical solution adopts a method of directly fusing question vectors to process relation vectors. This fusion operation aims to enhance the semantics of relations, further improve the representational power of the current path, and enable the model to better understand the association between questions and relations, thereby improving the accuracy of relation prediction.

[0217] S3-1: Perform multi-label classification on the relationships in the knowledge graph to obtain the answer entity vector;

[0218] The calculation is performed using the following formula:

[0219] e t =f(e t-1 ,Sigmoid(MLP(r final )))

[0220] Among them, e t Let f(·) be the vector of answer entities, and f(·) denotes the score of all entities at the current inference stage.

[0221] After T rounds of inference iterations, the relation prediction vector r is calculated. t This is used to predict the final answer entity. A relation prediction vector r is calculated in each iteration. final This includes the predicted relations after reasoning. The relation prediction vector rt is used to predict the answer entity. If the correct answer is obtained in round t-1 of reasoning, the semantics of the question are considered incorrect in round t of reasoning. This means that in each iteration, the model focuses on the parts for which the correct answer was not found in the previous iteration and attempts to find a more accurate answer in the next iteration.

[0222] S3-2: Predict the final answer;

[0223] The calculation is performed using the following formula:

[0224]

[0225] Where Y is the predicted final answer, e t The entity vector for the answer; r final For relation prediction vectors;

[0226] S3-3: Training on the final answer;

[0227] Training can be performed using the following formula:

[0228] L=α(Yy) 2 +βD kl (Y,y)

[0229] Where y is the true value, Y is the predicted final answer, L is the loss function, and D is the final value. kl (·) represents the kl divergence, a statistical measure of the difference between two probability distributions P and Q, where α and β are adjustable parameters.

[0230] This technical solution encodes the question using a pre-trained model, then starts with the question's topic entity and searches for the answer entity by hopping along relation triples, calculating the relation score and entity score for the next hop. Unlike existing technologies, this solution constructs a two-tiered inference framework with recurrent self-attention. The first layer of the framework is word-level, integrating word-level question vectors into the initial inference information. Each hop updates the inference information and filters noise to more accurately calculate relation scores. The second layer is sentence-level, embedding sentence-level question vectors into relation vectors to deepen the model's understanding of the overall semantics of the question. Between each inference time step, we propose a recurrent self-attention mechanism to capture fine-grained semantics by repeatedly learning word-level question vectors, enhancing subsequent relation vector representations. Simultaneously, we propose a graph attention network based on virtual nodes of the question vectors to model the question vectors, fully capturing the potential connections between distant entities in complex questions.

[0231] To better illustrate this technical solution, WebQuestionsSP (WebQSP) is used as the training set for validation. WebQuestionsSP contains 2998 training questions, 100 validation questions, and 1639 test questions. The questions are sourced from the FreeBase knowledge base, and each question involves one or two rounds of reasoning. This dataset has a large knowledge graph with millions of entities and triples. Following trims the knowledge graph to include only mentioned relations and 2-hop triples of mentioned entities, and also adds reverse relations. The processed knowledge graph contains 1.8 million entities, 1144 relations, and 11.4 million triples. MetaQA is a large multi-hop knowledge graph question-answering dataset. Its knowledge graph originates from the film domain, with over 400,000 questions generated from more than a dozen templates. Each question has a maximum of three hops and contains 43,000 entities, 9 relations, and 135,000 triples.

[0232] The DCGA framework was implemented using the PyTorch deep learning framework. For the WebQSP dataset, BERT (bert-base-uncased) was used as the pre-trained model for question encoding. This model, downloaded from the HuggingFace website, has a 12-layer Transformer encoder with 110M parameters. The hyperparameter T represents the range of inferences within T inferences; for the WebQSP dataset, T=2. For the MetaQA dataset, BIGRU was used as the question encoder, with T=3. The model was trained using the RAdam optimizer with a learning rate of 0.001. The batch size was 128 for MetaQA and 16 for WebQSP. The evaluation metrics were Hits@1 and F1. The model was trained on a single RTX 3090 GPU.

[0233] Figure 4 This is a schematic diagram illustrating the role of the Vector Graph Attention Network (VGAT) in our inference process. Since the Freebase Knowledge Base website is no longer in service, we use entity codes to represent entities.

[0234] As shown in Table 1, on MetaQA, TransferNet and GFC achieve 100% on Hit@1 in both 2-inference and 3-inference iterations, while our DCGA achieves 100% in 2-inference iterations. In 1-inference iterations, DCGA reaches 97.6%, surpassing TransferNet's 97.5% and matching GFC. In 3-inference iterations, DCGA reaches 99.9%. We believe this is because DCGA increases model complexity, while the problems in MetaQA are relatively simple, with fewer constrained entities and similarity relationships. Therefore, when filtering noise, it's easier to filter out some key information, leading to performance fluctuations.

[0235] On the WebQSP dataset, DCGA achieves a Hit@1 score of 78.9%, outperforming GFC and RE-KBQA, exceeding the large model ChatGpt by 17.7%, and surpassing StructGpt and ReasoningLM by 6.3% and 0.4% respectively, reaching the state-of-the-art (SOTA) score on this dataset. DCGA's F1 score on the WebQSP dataset is 75.9%, higher than RnG-KBQA's 75.6%. We analyze the reasons for this through case studies, such as... Figure 4As shown, after VGAT enhancement, the relationship score in one-hop reasoning is improved compared to the case without VGAT. This indicates that VGAT can effectively capture the relationships between nodes in single-hop reasoning, thereby improving the model's accuracy. In two-hop reasoning, VGAT's advantage is even more pronounced. It lowers the r3 score in the non-critical path "m.014j017→m.03btywc→m.014y6", while maintaining the r3 score in the critical path "m.014j017→m.02_96cv→m.014y6". This is because the model measures the matching degree between the relationships and entities involved in the reasoning path and the question through semantic cues of the question. Reasoning paths that are unrelated to or weakly related to the question score lower than those that are related to the question. This shows that VGAT can better focus on the critical path in reasoning, thereby reducing the impact of noise and improving the accuracy of reasoning.

[0236] Table 1: Hit@1 scores in MetaQA and WebQSP, and F1 scores in WebQSP compared to 12 baseline models.

[0237]

[0238] We replicated the results of Jiang et al.'s LLMs and other baseline results from Xie et al. DCGA significantly outperformed the baseline models. Table 2: Hit@1 values ​​for 1-pass and 2-pass inference of TransferNet, GFC, and DCGA models on WebQSP.

[0239]

[0240] As shown in Table 2, compared to GFC, DCGA improves performance by 1.6% in single-step reasoning problems and by 3.1% in two-step reasoning problems. This means the improvement in single-step reasoning is slightly lower than that in two-step reasoning problems, indicating that the DCGA method is more suitable for multi-step reasoning problems. We believe this is because single-step reasoning problems only consider the relationships between connected nodes and do not need to capture more distant connections and complex information propagation between nodes, while multi-step reasoning problems require multiple information propagations. Since the dual-cascade reasoning framework of DCGA in this paper can effectively enhance key reasoning information during entity-hopping reasoning, it can improve the performance of multi-hop question answering.

[0241] In many real-world scenarios, knowledge graphs are often incomplete, requiring models to have stronger reasoning capabilities. We tested the WebQSP dataset on an incomplete knowledge graph (50% KG) and compared the performance of DCGA with other baseline methods, as shown in Tables 3 and 4.

[0242] Table 3: Hit@1 values ​​for different models on the MetaQA dataset at 50% KG.

[0243]

[0244] Table 4: Hit@1 values ​​for different models on the WebQSP dataset at 50% KG.

[0245]

[0246] Because incomplete knowledge graphs lack certain entity, relation, or attribute information, it may be difficult to find matching entities or relations during reasoning, leading to inaccurate representations learned by embedding-based methods. As shown in Table 4, DCGA achieves a performance of 58.8%, which is better than TransferNet's 52.4%, indicating that DCGA in this paper still has stable and good reasoning capabilities in incomplete knowledge graphs.

[0247] To further verify the impact of our proposed solution, we conducted ablation experiments on the WebQSP dataset, and the results are shown in the table. After removing the double concatenation framework (without the double concatenation framework), Hit@1 decreased by 1.2%, and F1 decreased by 6.7%, indicating that the embedding of sentence vectors in our paper enriches the question vectors for relational path prediction. After removing the vector graph attention network (without the vector graph attention network), Hit@1 decreased by 1.4%, and F1 decreased by 6.8%. This significant decrease reflects the importance of introducing question semantics to the overall performance of the model and also demonstrates the effectiveness of the proposed vector graph attention network. After removing the recurrent self-attention (without removing the recurrent self-attention), Hit@1 decreased by 1.5%, and F1 decreased by 6.0%, indicating that the model's performance in accuracy and inference is significantly impaired, especially in fine-grained inference capabilities.

[0248] Table 5: Ablation studies conducted on WebQSP.

[0249]

[0250] To further verify the impact of VGAT on the inference process, we conducted a statistical analysis of the relation scores for each inference question on the MetaQA dataset using our proposed method. Relation scores are crucial to the inference results; in the MetaQA test set, the ratio of first-order, second-order, and third-order inferences is 1:1.5:1.4. We statistically analyzed the relation scores for each question at different inference orders, as shown in the following formula:

[0251]

[0252] a1 represents a first-order reasoning problem, a2 represents a second-order reasoning problem, a3 represents a third-order reasoning problem, Qn represents the number of n-order reasoning problems, d represents the reasoning methods for different order of reasoning, D = [1,3], j represents different relation types, rv represents the relation score when using VGAT, and rb represents the relation score when not using VGAT.

[0253] Therefore, from Figure 5 We can see that among the improvements in relation scores, first-order reasoning > second-order reasoning > third-order reasoning. This is because the accuracy of information gradually decreases with each inference. From... Figure 6 It was found that among the relationships with reduced scores, regardless of whether it was a single-reasoning or multi-reasoning problem, the relationships with the largest score decrease during a single reasoning step were combined with... Figure 6 It can be seen that in single-inference problems, the number of relation scores for each inference is higher than that of the model without VGAT; in quadratic and tertiary inference problems, the second and third inferences show the greatest improvement in scores. Since the nth inference in an n-inference problem is directly related to the final entity selection, Vector Graph Attention Network (VGAT) can handle complex long-range dependencies across entities and significantly improve the relation scores on the inference path through contextual information passed layer by layer in multiple inferences, thereby enhancing the accuracy and robustness of the final inference.

[0254] like Figure 3 As shown, taking the three-reasoning problem as an example, the number of relationships that decrease in the first reasoning is the largest. Relationships in the first and second reasoning are weakened and filtered. The first two reasonings are usually used to narrow down the range of candidate entities and guide the model to gradually approach the correct answer, while the third reasoning is to confirm the final entity. The score of the last reasoning has a key impact on the quality of the final reasoning. VGAT also suppresses relationships that are irrelevant to the question. Therefore, in the three-reasoning problem, the number of relationships that increase in the third reasoning is the largest.

Claims

1. A knowledge graph question answering method based on a bi-cascaded graph attention framework, characterized in that: Includes the following steps: S1: The question is preprocessed and encoded through the initialization layer to obtain the word-level question vector and sentence vector of the question; S2: Calculate the semantic contribution of each word and integrate it into the initial question vector; S2-1: Perform word-level reasoning on the input question; When performing the first inference, execute S2-1-1; otherwise, execute S2-1-3. S2-1-1: The word-level question vectors obtained after BERT pre-training are processed through a word inference layer. and the reasoning representation information in the first reasoning stage The following formulas are used for processing and transformation: ； in, Semantic information that has not been initialized; S2-1-2: Store the reasoning representation information for the current reasoning stage; The formula is as follows: ； in, The initial storage unit is used; return to step S2-1; S2-1-3: Store the reasoning representation information for the current reasoning stage; The formula is as follows: ； in, This is the storage unit for the current inference stage. This is the storage unit from the previous inference phase. This represents the key information extracted in the previous reasoning phase, and The calculation is performed using the following formula: ； in, Represents a linear fully connected layer. For activation functions; S2-1-4: Calculate the semantic information of the current reasoning stage. ; ； in, For initial semantic information, This is the storage unit for the previous inference phase; When t > 0, the information storage unit also needs to incorporate the data from step t - 1. The filtered information is used to obtain a more granular representation of the dynamic problem. This representation is then incorporated into the initialization semantic information. middle; Return to step S2-1-3 and repeat S2-1-3 to S2-1-4 until the nth reasoning is completed; S2-2: Calculate the weights of recurrent self-attention; S2-3: Integrating sentence vector reasoning; S2-4: Semantic information represented by fine-grained information Processed via VGAT; S3: Calculate the relation score, and use the head entity and relation score to calculate the tail entity score; S3-1: Perform multi-label classification on the relationships in the knowledge graph to obtain the answer entity vector; The calculation is performed using the following formula: ； in, Let t be the entity vector of the answer at step t. This indicates that the scores of all entities at the current inference stage are calculated; S3-2: Predict the final answer; The calculation is performed using the following formula: ； in, For the final answer of the prediction, The entity vector of the answer; For relation prediction vectors; S3-3: Training on the final answer; Training can be performed using the following formula: ； in, For the true value, For the final answer of the prediction, For loss function, Let kl divergence be the inequality. and This is an adjustable parameter.

2. The knowledge graph question answering method based on a bi-cascaded graph attention framework according to claim 1, characterized in that: Step S1 also includes the following steps: S1-1: The input question is pre-trained using BERT to obtain a word-level question vector; S1-2: The word-level question vectors obtained after BERT pre-training , and the word vector at the current step size By performing fusion, t∈[0,T], we obtain the self-attention matrix. The formula is as follows: ； in, For self-attention matrix, This represents a linear fully connected layer, where t is the number of steps in the current inference stage and T is the total number of inference steps. S1-3: The self-attention matrix pass The transformation is performed to obtain the key information distribution of the problem under the current number of hops. The formula is as follows: ； in, This indicates the distribution of key information about the problem at the current stage of reasoning. For activation functions; S1-4: Will Embedded into In the process, we obtain the inference representation information. The formula is as follows: 。 3. The knowledge graph question answering method based on a bi-cascaded graph attention framework according to claim 1, characterized in that: Step S2-2 also includes the following: S2-2-1: Calculate the initial attention weights; S2-2-1-1: Distribution of key information regarding the problem Perform one-dimensional summation and normalization to obtain the attention score; S2-2-1-2: Integrating Attention Score with Attention Mask Multiply to ensure the model only focuses on the attention scores of valid words; S2-2-1-3: The attention scores are summed in two dimensions again and normalized to obtain the final initial attention weights; The calculation is performed using the following formula: ； in, To initialize attention weights, To sum by column, To sum by row, For attention masking; S2-2-2: Calculate the final attention weights ; In each iteration, the final weights obtained from the previous calculation are used as the initial values ​​for the next calculation, and the weights are calculated using the following formula: ； The number of iterations, n, is a hyperparameter. S2-2-3: Calculate the problem vector for predicting the relationship; The formula is as follows: ； in, The problem vector is for predicting the relationship.

4. The knowledge graph question answering method based on a bi-cascaded graph attention framework according to claim 1, characterized in that: Step S2-3 also includes the following: S2-3-1: Calculate the initial relation prediction vector; The sentence vectors obtained after BERT pre-training With the problem vector Add them together to obtain the initial relation prediction vector. The formula is as follows: ； The sentence vector of the question; S2-3-2: Store the question vector of the predicted relationship in the current inference stage; The formula is as follows: ； in, For the initial storage unit, This represents the relation representation selected in the previous reasoning phase, and The calculation is performed using the following formula: ； in, Represents a linear fully connected layer. For activation functions; S2-3-3: Calculate the final relationship prediction vector ; The formula is as follows: ； in, This is the final relation prediction vector.

5. The knowledge graph question answering method based on a bi-cascaded graph attention framework according to claim 1, characterized in that: Step S2-4 also includes the following steps: S2-4-1: Let the nodes in the vector graph be: ,in, , Indicates the first Feature representation of each node; S2-4-2: Constructing a self-correlated feature adjacency matrix ; S2-4-3: Convert using the following formula: ； It is the weight matrix of the corresponding input linear transformation; S2-4-4: Calculate the attention of all nodes to their neighboring nodes; The calculation is performed using the following formula: ； in, As vertices and Attention between them It is the transpose of a 2F'-dimensional vector; For activation function, It is an exponential function. For trainable parameters, It is the first The set of neighboring nodes of a given node; Indicates a connection; S2-4-5: Calculate the first... through adjacent nodes The final representation of each node; The calculation is performed using the following formula: ； in, Indicates the number of attention mechanisms; It is the vertex and Attention between them is caused by the kth attention mechanism ( The normalized attention coefficient was calculated. It is the weight matrix of the corresponding input linear transformation; Indicates coefficient; S2-4-6: Aggregation via VGAT layer; The calculation is performed using the following formula: ； in, To express summation; S2-4-7: Problem vector relating the aggregated features of the VGAT layer to the predictions To merge The formula is as follows: ； This represents the relation vector after fusion through the VGAT layer; S2-4-8: The final relation prediction vector is obtained through a dual-cascaded framework and a vector graph attention layer. ; The formula is as follows: ； in, This represents the relation vector after fusion through the VGAT layer. This represents the final relation prediction vector.

Images