Knowledge-based dual-stream attention multi-hop reasoning method and system for visual question answering

Abstract

The present invention proposes a knowledge-based visual question-answering dual-stream attention multi-hop reasoning method and system, which relates to the field of machine understanding. The method includes obtaining questions, images, and corresponding image knowledge bases to obtain a knowledge hypergraph and a question hypergraph, and then obtaining knowledge hypergraph features and question hypergraph features; calculating the association matrix of the knowledge hypergraph guided by the question hypergraph, obtaining the knowledge hyperedges most relevant to the question hyperedges, calculating the weight of each knowledge hyperedge, and extracting the first aggregated information of the knowledge hyperedge centered on the question hyperedge; calculating the association matrix of the question hypergraph guided by the knowledge hypergraph, obtaining the question hyperedges most relevant to the knowledge hyperedges, calculating the weight of each question hyperedge, and extracting the second aggregated information of the question hyperedge centered on the knowledge hyperedge; inputting the first aggregated information and the second aggregated information into the answer prediction layer, and outputting the predicted answer. The present invention effectively improves the adaptive reasoning ability of the model and reduces the interference of redundant information on the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine reading comprehension, and in particular relates to a knowledge base-based visual question answering dual-stream attention multi-hop reasoning method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Knowledge-based visual question answering (KBVQA) aims to answer factual questions by combining image visual information with relevant knowledge bases. It has attracted widespread attention in recent years. Specifically, the core task of KBVQA is to understand the semantic information of the question and extract the most relevant knowledge facts from the image visual information and relevant knowledge bases to infer the answer to the question. As an extension of the VQA task, KBVQA aims to answer more complex real-world questions. Therefore, it has attracted increasing attention not only in academic research but also in industry.

[0004] Early VQA tasks utilized visual information from a given image to answer questions. However, when answering complex questions, the information in the image is insufficient. Wang, Shah, and others proposed a benchmark dataset for KBVQA, which requires reasoning about images based on large knowledge bases, such as knowledge facts from Freebase and DBPedia. To address this task, two pioneering works, Wang et al., proposed a logic parsing-based approach. This approach uses predefined query templates to convert questions into knowledge graph logical queries and executes the generated queries on the knowledge graph to search for answers.

[0005] Most existing methods for solving KBVQA use knowledge graph completion, multi-hop reasoning, etc., but there are some defects. Specifically, most methods based on knowledge graph completion extract the head entity of the visual content and mine the multimodal relationship between the image and the question to complete the tail entity answer. Methods based on multi-hop reasoning need to answer questions based on understanding and reasoning the complete text, and usually require a full understanding of the contextual semantic information of the question. Therefore, Zhu et al. represented the retrieved knowledge facts as a graph and performed graph reasoning by using graph convolution for message passing schemes. However, when the semantics of the question are too complex, how to better learn the high-order semantics between the question and the knowledge base is also a challenge.

[0006] To address this issue, Hypergraph Transformer et al. proposed a multi-hop reasoning framework based on hypergraph encoding. This framework encodes the question and image knowledge facts into a hypergraph structure, enabling better learning of their high-level semantics. It then infers the answer by matching the correlations between the two hypergraph edges. However, as graph traversals increase, the number of facts extracted from the knowledge base also increases, leading to a large amount of redundant information and unnecessary knowledge facts in the reasoning process. This interferes with the correct reasoning process and reduces the adaptive reasoning capability of multi-hop reasoning. Summary of the Invention

[0007] To overcome the shortcomings of the above-mentioned prior art, the present invention provides a knowledge base-based visual question answering dual-stream attention multi-hop reasoning method and system. On the basis of hypergraph encoding, the self-attention weight matrix of the question semantics is added, and the similarity weight matrix between the question hypergraph and the image knowledge hypergraph is calculated. The maximum pooling layer and hard attention mechanism are adopted to effectively improve the adaptive reasoning ability of the model and reduce the interference of redundant information on the model.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of the present invention provides a knowledge base-based visual question answering dual-stream attention multi-hop reasoning method.

[0010] The knowledge base-based dual-stream attention multi-hop reasoning method for visual question answering includes the following steps:

[0011] Obtain questions, images, and corresponding image knowledge base;

[0012] Input the question and image into the hypergraph encoding layer for encoding, and obtain the knowledge hypergraph and question hypergraph respectively;

[0013] The knowledge hypergraph and question hypergraph are preliminarily processed using the attention mechanism to obtain knowledge hypergraph features and question hypergraph features;

[0014] Based on the features of the knowledge hypergraph and the problem hypergraph, the association matrix of the knowledge hypergraph guided by the problem hypergraph is calculated to obtain the knowledge hyperedge most relevant to the problem hyperedge, the weight of each knowledge hyperedge is calculated, and the first aggregate information of the knowledge hyperedge centered on the problem hyperedge is extracted; at the same time, the association matrix of the problem hypergraph guided by the knowledge hypergraph is calculated to obtain the problem hyperedge most relevant to the knowledge hyperedge, the weight of each problem hyperedge is calculated, and the second aggregate information of the problem hyperedge centered on the knowledge hyperedge is extracted;

[0015] The first aggregate information and the second aggregate information are spliced ​​and fused, and the spliced ​​and fused aggregate information is input into the answer prediction layer to output the predicted answer corresponding to the question.

[0016] Preferably, the knowledge hypergraph and the question hypergraph are preliminarily processed using an attention mechanism, specifically including:

[0017] A multi-head attention mechanism is used to increase the attention weight of the problem hypergraph itself, and a guided attention layer is used to learn the correlation between knowledge hyperedges and problem hyperedges. Then, the self-attention layer is used to learn the semantic relationship within the problem hyperedges and knowledge hyperedges to obtain the knowledge hypergraph features and problem hypergraph features.

[0018] Preferably, the specific process of obtaining the first aggregate information is:

[0019] First, based on the knowledge hypergraph features and the problem hypergraph features, the association matrix A between the problem-guided problem hypergraph and the knowledge hypergraph is calculated. q ,Evaluate the relevance of each hyperedge in the knowledge hypergraph with each hyperedge in the problem hypergraph;

[0020] Then calculate the correlation A q The row-wise max pooling layer on the matrix obtains the knowledge hyperedge that is most relevant to the problem hyperedge;

[0021] Use one-hot encoding classification distribution to calculate the weight of each knowledge hyperedge;

[0022] According to the weight distribution obtained by the one-hot encoding classification distribution, only the information of the knowledge hyperedge centered on the problem hyperedge at the index position with a weight value of 1 is extracted and aggregated into the first aggregate information Z k .

[0023] Preferably, the specific process of obtaining the second aggregate information is:

[0024] Based on the features of the knowledge hypergraph and the problem hypergraph, calculate the association matrix A between the knowledge hypergraph guided by the knowledge hypergraph and the problem hypergraph k ;

[0025] Then calculate the association matrix A k The row max pooling layer above obtains the question hyperedge that is most relevant to the knowledge hyperedge;

[0026] Use one-hot encoding classification distribution to calculate the weight of each problem hyperedge;

[0027] According to the weight distribution obtained by the one-hot encoding classification distribution, only the information of the problem hyperedge centered on the knowledge hyperedge at the index position with a weight value of 1 is extracted and aggregated into the second aggregate information Z q .

[0028] Preferably, the first aggregate information Z obtained k and the second aggregate information Z q Perform concatenation and fusion and input into a single feed-forward layer to obtain a connected representation z.

[0029] Preferably, in the task of answering visual questions, a multilayer perceptron is used as an answer classifier; based on the answer calculated by similarity, the dot product similarity between z and the answer candidate set is calculated, and according to the values ​​calculated by the dot product similarity, the index position of the z array where the largest value among these values ​​is located is selected, and according to this index position, the word with the same index corresponding to the position is selected from the answer candidate set array as the answer.

[0030] Preferably, the question and the image are input into the hypergraph coding layer for encoding respectively to obtain the knowledge hypergraph and the question hypergraph, specifically:

[0031] Treat the large amount of knowledge facts in the knowledge base as a huge knowledge graph and construct a hypergraph through graph traversal;

[0032] A triple is defined as the basic unit of graph traversal. Each graph traversal contains three nodes {head, predicate, tail}. Then, multiple interrelated relational facts are encoded through multi-hop graph traversal. n-hop graph traversal merges n facts into a hyperedge, thus obtaining a knowledge hypergraph.

[0033] Each word in the question sentence is defined as a node with edges to adjacent nodes. The multi-hop graph traversal is the same as the knowledge hypergraph. A phrase is regarded as a hyperedge in the question hypergraph, thus obtaining the question hypergraph.

[0034] The second aspect of the present invention provides a knowledge base-based visual question answering dual-stream attention multi-hop reasoning system.

[0035] A knowledge-based dual-stream attention multi-hop reasoning system for visual question answering, including:

[0036] The acquisition module is configured to: acquire questions, images and corresponding image knowledge base;

[0037] The encoding module is configured to: input the question and the image into the hypergraph encoding layer for encoding, thereby obtaining a knowledge hypergraph and a question hypergraph;

[0038] The attention module is configured to: perform preliminary processing on the knowledge hypergraph and the question hypergraph using the attention mechanism to obtain knowledge hypergraph features and question hypergraph features;

[0039] The aggregate information acquisition module is configured to: calculate the association matrix of the knowledge hypergraph guided by the problem hypergraph based on the knowledge hypergraph features and the problem hypergraph features, obtain the knowledge hyperedges most relevant to the problem hyperedges, calculate the weight of each knowledge hyperedge, and extract the first aggregate information of the knowledge hyperedge centered on the problem hyperedge; simultaneously calculate the association matrix of the problem hypergraph guided by the knowledge hypergraph, obtain the problem hyperedges most relevant to the knowledge hyperedges, calculate the weight of each problem hyperedge, and extract the second aggregate information of the problem hyperedge centered on the knowledge hyperedge;

[0040] The fusion and prediction module is configured to: splice and fuse the first aggregate information and the second aggregate information, input the spliced ​​and fused aggregate information into the answer prediction layer, and output the predicted answer corresponding to the question.

[0041] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the knowledge base-based visual question answering dual-stream attention multi-hop reasoning method as described in the first aspect of the present invention.

[0042] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, it implements the steps in the knowledge base-based visual question answering dual-stream attention multi-hop reasoning method as described in the first aspect of the present invention.

[0043] One or more of the above technical solutions have the following beneficial effects:

[0044] To address the problem of inaccurate facts extracted from image-related knowledge bases, this paper proposes a knowledge base-based visual question answering dual-stream attention multi-hop reasoning method and system. By utilizing a dual-stream attention multi-hop reasoning network, on the basis of hypergraph encoding, the self-attention weight matrix of the question semantics is added, and the similarity weight matrix between the question hypergraph and the image knowledge hypergraph is calculated, which effectively improves the model's adaptive reasoning ability in multi-hop reasoning.

[0045] To address the problem of redundant information interfering with the reasoning process in multi-hop reasoning methods, the present invention sets up a classifier with row maximum pooling and one-hot encoding in the module, and adopts the maximum pooling layer and hard attention mechanism to effectively improve the adaptive reasoning ability of the model and reduce the interference of redundant information on the model, further avoiding the occurrence of interference problems.

[0046] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0048] Figure 1 This is a flow chart of the method of the first embodiment.

[0049] Figure 2 Attention visualization heatmap for the first example. DETAILED DESCRIPTION

[0050] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0051] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0052] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0053] The overall idea proposed by the present invention is:

[0054] The overall architecture of the DSAMR model proposed in this invention is as follows: Figure 1 As shown in Figure 3, it mainly consists of the following four parts: hypergraph encoding layer, guided attention layer, two-stream attention layer and answer prediction layer.

[0055] First, given a question-image pair and the corresponding image knowledge base, the hypergraph encoding layer constructs a query-aware knowledge hypergraph and a question hypergraph, respectively. The knowledge hypergraph incorporates both image and knowledge base information. Second, the guided attention layer is used to associate high-level semantics between the question and knowledge hypergraph. Third, the two-stream attention layer is used to further enhance the deep semantic relevance between the knowledge hypergraphs. Finally, the output of the two-stream attention network is subjected to feature fusion and input into the answer selection layer to select the answer.

[0056] Glossary:

[0057] KBVQA (Knowledge-based Visual Question Answering): knowledge-based visual question answering;

[0058] VQA (Visual Question Answering): Visual Question Answering;

[0059] DSAMR (Dual-Stream Attention Multi-hop Reasoning): dual-stream attention multi-hop reasoning;

[0060] PQ (PathQuestion): path reasoning benchmark dataset;

[0061] PQL (PathQuestion-Large): a large-scale path reasoning benchmark dataset;

[0062] ORG (Original): original semantics;

[0063] PRP (Paraphrased): translation semantics;

[0064] DA (Dual-Stream Attention): dual-stream attention mechanism;

[0065] SA (Self Attention): self-attention mechanism;

[0066] RNN (Recurrent Neural Network): recurrent convolutional network.

[0067] Example 1

[0068] This embodiment discloses a knowledge base-based dual-stream attention multi-hop reasoning method for visual question answering.

[0069] like Figure 1 As shown in the figure, the knowledge base-based visual question answering dual-stream attention multi-hop reasoning method includes the following steps:

[0070] Obtain questions, images, and corresponding image knowledge base;

[0071] Input the question and image into the hypergraph encoding layer for encoding, and obtain the knowledge hypergraph and question hypergraph respectively;

[0072] The knowledge hypergraph and question hypergraph are preliminarily processed using the attention mechanism to obtain knowledge hypergraph features and question hypergraph features;

[0073] Based on the features of the knowledge hypergraph and the problem hypergraph, the association matrix of the knowledge hypergraph guided by the problem hypergraph is calculated to obtain the knowledge hyperedge most relevant to the problem hyperedge, the weight of each knowledge hyperedge is calculated, and the first aggregate information of the knowledge hyperedge centered on the problem hyperedge is extracted; at the same time, the association matrix of the problem hypergraph guided by the knowledge hypergraph is calculated to obtain the problem hyperedge most relevant to the knowledge hyperedge, the weight of each problem hyperedge is calculated, and the second aggregate information of the problem hyperedge centered on the knowledge hyperedge is extracted;

[0074] The first aggregate information and the second aggregate information are spliced ​​and fused, and the spliced ​​and fused aggregate information is input into the answer prediction layer to output the predicted answer corresponding to the question.

[0075] Specifically:

[0076] 1.1 Hypergraph Coding Layer

[0077] We use the hypergraph construction method used in Hypergraph Transformer to construct the problem hypergraph and knowledge hypergraph respectively. Structurally, a directed hypergraph H = {v, ε} is composed of a set of nodes H = {v1,…,v |v|} and a set of hyperedge graphs ε={h1,…,h |ε| Each node is represented as a w-dimensional embedding vector, i.e. Each hyperedge connects any number of nodes, and the nodes themselves also have an order.

[0078] Knowledge Hypergraph H k ={v k , ε k}It consists of a node set v k and hyperedge set ε k They represent entities in a set of knowledge facts and a set of hyperedges. Each hyperedge connects the vertex v′ k in We regard the large number of knowledge facts in the knowledge base as a huge knowledge graph, and construct a hypergraph through graph traversal. We define a triple as the basic unit of graph traversal, that is, each graph traversal contains three nodes {head, predicate, tail}. Then, we encode multiple interrelated relational facts through multi-hop graph traversal, so n-hop graph traversal merges n facts into one hyperedge.

[0079] We reconstruct the question sentence into a node set v q and hyperedge set ε q The problem hypergraph H q , define each word in the question sentence as a node with edges to adjacent nodes, then the multi-hop graph traversal is the same as the knowledge hypergraph, and an n-gram phrase is regarded as a hyperedge in the question hypergraph.

[0080] 1.2 Dual-Stream Attention Module

[0081] We believe that high-order semantic interactions between questions and knowledge bases are important for reasoning about answers. After obtaining the question hypergraph and knowledge hypergraph, we first use the multi-head attention mechanism in the Transformer method to increase the attention weight of the question itself, then use Guided-attention to learn the correlation between knowledge hyperedges and question hyperedges, and then use Self-attention to learn the semantic relationships within the question and knowledge hyperedges.

[0082] In order to mine the correlation between question hyperedges and knowledge hyperedges, we use the attention mechanism of Q, K and V. k For the problem of hyperedge E q The attention mechanism parameters Q, K and V are set to and Among them, W are parameters that can be learned, d v are the dimension values ​​of Q and V.

[0083]

[0084] In addition, the guiding attention for the question hyperedge and knowledge hyperedge is set as Attention(Q q ,K k ,V k ) and Attention(Q k ,K q ,V q ), the knowledge hypergraph in Self-attention is set to Attention(Q k ,K k ,V k ), the problem hypergraph is set to Attention(Q q ,K q ,V q ).

[0085] In order to further enhance the correlation between the question hypergraph and the knowledge hypergraph, we obtain the knowledge hypergraph feature E after the initial processing of the attention mechanism of Q, K and V. k* and the problem hypergraph feature E q* Finally, the association weight matrix of the knowledge hypergraph guided by the problem hypergraph and the association weight matrix of the problem hypergraph guided by the knowledge hypergraph are calculated respectively.

[0086] First, calculate the association matrix A between the question-guided problem hypergraph and the knowledge hypergraph q =(W q1 E q* ) T (W k1 E k*) to evaluate the relevance of each hyperedge in the knowledge hypergraph to each hyperedge in the problem hypergraph, where W q1 and W k1 All are learnable parameters. At the same time, the features are mapped from 256 dimensions to 512 dimensions, and then A is calculated. q The row max pooling layer above obtains the knowledge hyperedge most relevant to the problem hyperedge. The calculation formula of hyperedge is as follows:

[0087]

[0088] Based on Based on this, we introduce the hard attention mechanism to select the knowledge hyperedge that is most relevant to the problem hyperedge. Compared with the soft attention mechanism, the hard attention mechanism provides a more stable and explanatory knowledge hyperedge. Then, we use the Gumbel-Softmax unique hot encoding classification distribution to classify each E k The weight calculation formula is as follows:

[0089]

[0090] We sample from the Gumbel(0,1) distribution, where τ is the temperature coefficient of Softmax. Finally, we extract the problem hyperedge E q* Knowledge hyperedge E k* The information Aggregation

[0091] Information before aggregation is Get, then I k Polymerization to obtain Z k .

[0092] Similarly, the feature information extraction of the problem hypergraph guided by the knowledge hypergraph is consistent with the above method, and the association matrix A between the knowledge hypergraph guided by the knowledge hypergraph and the problem hypergraph is calculated. k , by learning the parameter W q2 and W k2 Map the features from 256 dimensions to 512 dimensions, and then calculate A k The row max pooling layer on the top layer obtains the question hyperedge that is most relevant to the knowledge hyperedge. Based on Based on the Gumbel-Softmax unique hot encoding classification distribution, each E q* Perform weight calculation, and finally, we extract the knowledge hyperedge E k* The problem centered on the hyperedge E q* The information Aggregation

[0093] Information before aggregation is Get, then I q Polymerization to obtain Z q .

[0094] 1.3 Answer Prediction Layer

[0095] Based on the most relevant knowledge hyperedge and question hyperedge, we obtain the aggregated information Z q and Z k The concatenation and fusion are performed and then input into a single feed-forward layer to obtain a connected representation z.

[0096] In order to adapt to the prediction of questions in different tasks, we consider two answer prediction layers. In the visual question answering task, multi-layer perceptron is commonly used as the answer classifier p = ψ(z). For the answer based on similarity calculation, we calculate Z and the answer candidate set The dot product similarity between them p = zC T Here, |A| refers to multiple candidate answers, w represents the dimension of each candidate answer, and the most similar answer in z is selected as the answer of the candidate answer. For the framework training, only the supervision of QA pairs is used and the reasoning path of the ground truth is not annotated. Therefore, the cross entropy between the predicted p and the ground truth t is used as the loss function.

[0097] 2.1 Dataset and Evaluation Metrics

[0098] To verify the effectiveness of the proposed model, we conducted experiments on a public knowledge-based visual question answering dataset, KVQA[3], and two knowledge-based text question answering datasets, PathQuestion and PathQuestion-Large

[10] . The KVQA dataset is a large-scale benchmark dataset for complex VQA, which contains 183,007 pairs of images and their corresponding text description titles from Wikipedia, and provides 174,006 knowledge facts for 39,414 named entities based on Wikidata. The questions in the KVQA dataset are composed of two types: original questions (ORG) in the original dataset and paraphrased questions (PRP) generated from the original questions using an online paraphrase tool. The PQ and PQL datasets are mainly used to evaluate the multi-hop reasoning ability of the knowledge-based question answering task in this paper. PQ and PQL contain 7,106 and 2,625 question-answer pairs from 4,050 and 9,844 knowledge facts in Freebase, respectively. The statistics of the datasets are shown in Table 1.

[0099] Table 1 Statistics of each data set

[0100]

[0101] During the experiment, for the KVQA dataset, we use the accuracy of one to three hops on ORG and PRP questions as the evaluation metric to measure the performance of the model. For the PQ and PQL datasets, we only use the accuracy of two hops, three hops, and mixed questions as the evaluation metric to measure the performance of the model.

[0102] 2.2 Experimental Setup

[0103] The experimental environment includes Intel Core i9-10900X CPU, DDR4 16GB*2 memory, NVIDIA QuadroRTX6000 GPU, and the operating system is Ubuntu 20.04.

[0104] In terms of model parameter setting, we use a multi-head attention mechanism with 4 attention heads and 256 hidden units, and a Transformer-encode model with 4 attention heads, 2 hidden layers, and 256 hidden units. During the model training phase, we use the Adam optimizer and set the learning rate to 1e for different data sets. -4 and 1e -5 The ConsineAnnealingWarmUpRestarts annealing strategy is used to adjust the learning rate. When training the KVQA dataset, a multi-layer perceptron is used as the answer predictor. For the training of other datasets, the dot product similarity method is used as the answer predictor.

[0105] 2.3 Comparison Model

[0106] To evaluate the performance of the DSAMR model on the KBVQA task, we conducted comparative experiments with the following baselines:

[0107] 1) BLSTM: A variant of the recurrent neural network (RNN) model, known as a bidirectional long short-term memory (BLSTM) network. BLSTM can process both forward and reverse input sequences simultaneously and merge information from both, thereby improving sequence modeling performance.

[0108] 2) MemNN: MemNN, also known as Memory Networks, is a memory-based neural network model. The main idea of ​​this model is to use external memory, namely memory, to store and retrieve information and integrate it into the neural network for processing. It is mostly used for question-answering tasks.

[0109] 3) GCN: This paper proposes a graph convolution method to encode the question and knowledge graph separately. It uses a graph convolutional network to aggregate and convolve the feature information of neighboring nodes to update the feature representation of each node, thereby achieving node representation learning.

[0110] 4) GGNN: A recurrent neural network model for processing graph data is proposed, which encodes questions and knowledge graphs respectively, and extracts features from node and edge information to model the graph.

[0111] 5) HAN: We propose a novel heterogeneous graph neural network based on an attention mechanism. By learning importance at both the node level and the semantic level, we can fully consider the importance of nodes and meta-paths. The model generates node embeddings by hierarchically aggregating meta-path-based neighborhood features.

[0112] 6) BAN: A proposed model for visual question answering tasks. This model uses bilinear confluence and an attention mechanism to process the image and question. It then uses bilinear confluence to interact the features of the image and question to generate a new feature vector. The attention mechanism is then used to weight this new feature vector, and finally, the correct answer is predicted.

[0113] 7) Hypergraph Transformer: A knowledge-based visual question answering and hypergraph encoding multi-hop reasoning model is proposed. The model encodes the question and knowledge base into a hypergraph structure respectively, captures the matching correlation between question hyperedges and question hyperedges through a Transformer-based attention mechanism, and then predicts the answer based on a multi-hop reasoning method.

[0114] 2.4 Results and Analysis

[0115] On the KVQA dataset, we experimentally compared the DSAMR model with different baseline models. The results are shown in Table 2, and the best ones are highlighted in bold.

[0116] Table 2 Comparison of experimental results of different models on the KVQA dataset

[0117]

[0118] The performance results of the BLSTM, MemNN, and MenNN* models can be found in the Hypergraph-Transformer report. Our proposed DSAMR model achieved the best average performance in the KBVQA task compared to all baseline models, and also achieved excellent results in other metrics.

[0119] First, as can be seen from the table, since the (GCN, GGNN) models encode the question and knowledge graph separately, they do not consider the correlation between the question and knowledge. Therefore, they show a relatively low performance of less than 74% compared to the accuracy of the (MemNN, HAN, and BAN) models. However, the (MemNN, HAN, and BAN) models generally consider the question-guided knowledge graph attention mechanism. This shows that models with attention mechanisms can deeply explore the potential correlation between questions and knowledge graphs, and to a certain extent have a positive guiding effect on answer prediction, thus achieving better performance on the KBVQA task.

[0120] Secondly, for models with guided attention, the Hypergraph-Transformer model based on hypergraph encoding generally outperforms the (MemNN, HAN, and BAN) models. Hypergraph encoding methods use triples to encode questions, with each triple corresponding to a question graph node. Similarly, triple graph traversal is used to construct a knowledge hypergraph for knowledge graphs. This method generates a lot of redundant information during the hypergraph encoding process and is unnecessary for answer prediction. However, hypergraph-based methods can better learn the high-level semantic correlation between questions and knowledge bases, which is crucial for KBVQA tasks.

[0121] Finally, for the hypergraph encoding-based Hypergraph-Transformer model, our proposed method outperforms other models in average evaluation metrics, achieving suboptimal performance only slightly below the most recent model in the 3-hop metric on the ORG dataset. Specifically, on the KVQA dataset, the DSAMR model outperforms the Hypergraph-Transformer model by 1.0% (ORG) and 1.2% (PRP) in the 1-hop metric, 0.7% (ORG) and 0.2% (PRP) in the 2-hop metric, and 0.1% (ORG), 0.4% (PRP), and 0.6% in the 3-hop and Main metrics, respectively. This result demonstrates the effectiveness of our proposed DSAMR model on the KBVQA task.

[0122] 2.5 Evaluation of Multi-hop Reasoning Question Answering Task

[0123] The state-of-the-art methods we compared to used the number of graph traversals to evaluate the performance of original questions (ORG) and paraphrase questions (PRP). To evaluate the multi-hop reasoning ability of the model, we conducted experiments on the PathQuestion (PQ) dataset and the PathQuestion-Large (PQL) dataset. The results are shown in Table 3, with the best results in bold.

[0124] 1) Seq2Seq: is an end-to-end sequence training method that makes minimal assumptions about the sequence structure.

[0125] 2) KV-MemNN: This is a modification and extension of the Memory Network (MemNN). The query phase is key-based, and the read phase is value-based.

[0126] 3) IRN: An Interpretable Reason Network (IRN) model is proposed to solve multi-relational question answering, completing the question answering process of multi-relational questions through multi-hop reasoning.

[0127] 4) Embed: A model is proposed to learn the vector representation of questions and candidate answers, and then the answer is selected by calculating the relevance of the vector representation of the question and candidate questions.

[0128] 5) Subgraph: A weakly supervised embedding model is proposed to embed questions and triples into vector space respectively, and then calculate the similarity between them to determine which triple is most likely to be the answer to the question.

[0129] 6) MINERVA: proposed the use of reinforcement learning to reason about path relationships in a knowledge base.

[0130] 7) IRN-weak: It is an improvement and extension of the IRN model, and proposes a joint knowledge graph to achieve automatic question answering.

[0131] 8) SRN: A reinforcement learning-based neural network model is proposed to transform the answer form of multi-relational questions into a sequential decision problem.

[0132] Table 3 Comparison of experimental results of different models on the KVQA dataset

[0133]

[0134] Table 3 shows DSAMR's performance in the multi-hop question answering task. Compared with all baseline models, it achieves the best results in all metrics. It performs particularly well on the PQL dataset, which covers more knowledge facts but has fewer QA entities, making it more challenging than the PQ dataset.

[0135] 2.6 Problem of Self-Attention Weight and the Impact of Two-Stream Attention Layer

[0136] In order to verify the impact of the self-attention weight of the question and the dual-stream attention layer on the model performance, we conducted ablation experiments on the above two modules respectively based on the original model. The results are shown in Table 4, where "SA" represents the self-attention layer for the question and "DA" represents the dual-stream attention layer.

[0137] Table 4 Comparison of experimental results of different models on the KVQA dataset

[0138]

[0139] Table 4 shows that after removing the self-attention layer and the dual-stream attention layer for the question, the model's performance degraded to varying degrees. Specifically, without DA+SA, the model's overall performance declined more than without Da or SA. This suggests that increasing the attention weight for the question and introducing the dual-stream attention module help the model better capture the relevant semantic features between the question and the knowledge base, enabling the model to better select knowledge content relevant to the question, thus achieving optimal performance.

[0140] Figure 2 This is a method of visualizing the attention weights from the attention module using a heatmap. This figure proves that our proposed method can more accurately find the knowledge information needed to answer a given question from the knowledge base. As shown in the figure, the arrows represent the knowledge information of the two reasoning paths, and the correct reasoning path on the left is darker in the matrix heatmap above, which means that our model can accurately locate the correct reasoning path and output the correct answer.

[0141] Example 2

[0142] This embodiment discloses a knowledge base-based visual question answering dual-stream attention multi-hop reasoning system.

[0143] A knowledge-based dual-stream attention multi-hop reasoning system for visual question answering, including:

[0144] The acquisition module is configured to: acquire questions, images and corresponding image knowledge base;

[0145] The encoding module is configured to: input the question and the image into the hypergraph encoding layer for encoding, thereby obtaining a knowledge hypergraph and a question hypergraph;

[0146] The attention module is configured to: perform preliminary processing on the knowledge hypergraph and the question hypergraph using the attention mechanism to obtain knowledge hypergraph features and question hypergraph features;

[0147] The aggregate information acquisition module is configured to: calculate the association matrix of the knowledge hypergraph guided by the problem hypergraph based on the knowledge hypergraph features and the problem hypergraph features, obtain the knowledge hyperedges most relevant to the problem hyperedges, calculate the weight of each knowledge hyperedge, and extract the first aggregate information of the knowledge hyperedge centered on the problem hyperedge; simultaneously calculate the association matrix of the problem hypergraph guided by the knowledge hypergraph, obtain the problem hyperedges most relevant to the knowledge hyperedges, calculate the weight of each problem hyperedge, and extract the second aggregate information of the problem hyperedge centered on the knowledge hyperedge;

[0148] The fusion and prediction module is configured to: splice and fuse the first aggregate information and the second aggregate information, input the spliced ​​and fused aggregate information into the answer prediction layer, and output the predicted answer corresponding to the question.

[0149] Example 3

[0150] The purpose of this embodiment is to provide a computer-readable storage medium.

[0151] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the knowledge-based visual question answering dual-stream attention multi-hop reasoning method as described in Example 1 of the present disclosure.

[0152] Example 4

[0153] The purpose of this embodiment is to provide an electronic device.

[0154] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the knowledge-based visual question answering dual-stream attention multi-hop reasoning method described in Example 1 of the present disclosure are implemented.

[0155] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0156] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0157] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A dual-stream attention multi-hop reasoning method for visual question answering based on a knowledge base, characterized by: The following steps are involved: Obtain questions, images, and corresponding image knowledge bases; obtain questions and images from knowledge-based visual question answering datasets and knowledge-based text question answering datasets; The question and image are respectively input into the hypergraph encoding layer for encoding to obtain the knowledge hypergraph and question hypergraph. The question and image are respectively input into the hypergraph encoding layer for encoding to obtain the knowledge hypergraph and question hypergraph. Specifically, the large number of knowledge facts in the knowledge base are regarded as a huge knowledge graph, and the hypergraph is constructed through graph traversal. A triple is defined as the basic unit of graph traversal. Each graph traversal contains three nodes {head, predicate, tail}. Then, multiple interrelated relational facts are encoded through multi-hop graph traversal. The n-hop graph traversal merges n facts into a hyperedge, thereby obtaining a knowledge hypergraph. Each word in the question sentence is defined as a node with edges to adjacent nodes. The multi-hop graph traversal is the same as the knowledge hypergraph. A phrase is regarded as a hyperedge in the question hypergraph, thereby obtaining a question hypergraph. The knowledge hypergraph and question hypergraph are preliminarily processed using the attention mechanism to obtain knowledge hypergraph features and question hypergraph features; The knowledge hypergraph and question hypergraph are preliminarily processed using the attention mechanism, specifically: using a multi-head attention mechanism to increase the attention weight of the question hypergraph itself, using a guided attention layer to learn the correlation between knowledge hyperedges and question hyperedges, and then using a self-attention layer to learn the semantic relationships within question hyperedges and knowledge hyperedges to obtain the knowledge hypergraph features and question hypergraph features; Based on the features of the knowledge hypergraph and the problem hypergraph, the association matrix of the knowledge hypergraph guided by the problem hypergraph is calculated to obtain the knowledge hyperedge most relevant to the problem hyperedge, the weight of each knowledge hyperedge is calculated, and the first aggregate information of the knowledge hyperedge centered on the problem hyperedge is extracted; at the same time, the association matrix of the problem hypergraph guided by the knowledge hypergraph is calculated to obtain the problem hyperedge most relevant to the knowledge hyperedge, the weight of each problem hyperedge is calculated, and the second aggregate information of the problem hyperedge centered on the knowledge hyperedge is extracted; The specific process of obtaining the first aggregate information is as follows: first, based on the knowledge hypergraph features and the question hypergraph features, the association matrix A between the question-guided question hypergraph and the knowledge hypergraph is calculated. q , evaluate the correlation between each hyperedge in the knowledge hypergraph and each hyperedge in the problem hypergraph; then calculate the association A q The row max pooling layer on the matrix obtains the knowledge hyperedge most relevant to the problem hyperedge; the one-hot encoding classification distribution is used to calculate the weight of each knowledge hyperedge; according to the weight distribution obtained by the one-hot encoding classification distribution, only the information of the knowledge hyperedge centered on the problem hyperedge at the index position with a weight value of 1 is extracted and aggregated into the first aggregate information Z k ; The specific process of obtaining the second aggregate information is: based on the knowledge hypergraph features and the problem hypergraph features, calculate the association matrix A between the knowledge hypergraph guided by the knowledge hypergraph and the problem hypergraph k ; Then calculate the association matrix A k The row maximum pooling layer on the top is used to obtain the question hyperedge that is most relevant to the knowledge hyperedge; the one-hot encoding classification distribution is used to calculate the weight of each question hyperedge; according to the weight distribution obtained by the one-hot encoding classification distribution, only the information of the question hyperedge centered on the knowledge hyperedge at the index position with a weight value of 1 is extracted and aggregated into the second aggregate information Z q ; The first aggregate information and the second aggregate information are spliced ​​and fused, and the spliced ​​and fused aggregate information is input into the answer prediction layer to output the predicted answer corresponding to the question.

2. The knowledge base-based visual question answering dual-stream attention multi-hop reasoning method according to claim 1 is characterized in that: The first aggregate information Z k and the second aggregate information Z q Perform concatenation and fusion and input into a single feed-forward layer to obtain a connected representation z.

3. The knowledge base-based visual question answering dual-stream attention multi-hop reasoning method according to claim 2 is characterized in that: In the visual question answering task, a multilayer perceptron is used as the answer classifier; Based on the answer calculated by similarity, calculate the dot product similarity between z and the answer candidate set. According to the values ​​calculated by dot product similarity, select the index position of the z array where the largest value among these values ​​is located. Based on this index position, select the word with the same index corresponding to the position from the answer candidate set array as the answer.

4. A knowledge-based dual-stream attention multi-hop reasoning system for visual question answering, characterized by: include: The acquisition module is configured to: acquire questions, images and corresponding image knowledge base; Obtain questions and images from the knowledge-based visual question answering dataset and the knowledge-based text question answering dataset; The encoding module is configured to: input the question and image into the hypergraph encoding layer for encoding, respectively, to obtain a knowledge hypergraph and a question hypergraph; input the question and image into the hypergraph encoding layer for encoding, respectively, to obtain a knowledge hypergraph and a question hypergraph. Specifically, the large number of knowledge facts in the knowledge base are regarded as a huge knowledge graph, and a hypergraph is constructed through graph traversal; a triple is defined as the basic unit of graph traversal, each graph traversal contains three nodes {head, predicate, tail}, and then multiple interrelated relational facts are encoded through multi-hop graph traversal. The n-hop graph traversal merges n facts into a hyperedge, thereby obtaining a knowledge hypergraph; each word in the question sentence is defined as a node with edges to adjacent nodes. The multi-hop graph traversal is the same as the knowledge hypergraph. A phrase is regarded as a hyperedge in the question hypergraph, thereby obtaining a question hypergraph; The attention module is configured to: perform preliminary processing on the knowledge hypergraph and the question hypergraph using the attention mechanism to obtain knowledge hypergraph features and question hypergraph features; The knowledge hypergraph and question hypergraph are preliminarily processed using the attention mechanism, specifically: using a multi-head attention mechanism to increase the attention weight of the question hypergraph itself, using a guided attention layer to learn the correlation between knowledge hyperedges and question hyperedges, and then using a self-attention layer to learn the semantic relationships within question hyperedges and knowledge hyperedges to obtain the knowledge hypergraph features and question hypergraph features; The aggregation information acquisition module is configured to: calculate the association matrix of the knowledge hypergraph guided by the problem hypergraph based on the knowledge hypergraph features and the problem hypergraph features, obtain the knowledge hyperedge most relevant to the problem hyperedge, calculate the weight of each knowledge hyperedge, and extract the first aggregation information of the knowledge hyperedge centered on the problem hyperedge; at the same time, calculate the association matrix of the problem hypergraph guided by the knowledge hypergraph, obtain the problem hyperedge most relevant to the knowledge hyperedge, calculate the weight of each problem hyperedge, and extract the second aggregation information of the problem hyperedge centered on the knowledge hyperedge; the specific acquisition process of the first aggregation information is: first, based on the knowledge hypergraph features and the problem hypergraph features, calculate the association matrix A of the problem hypergraph guided by the problem and the knowledge hypergraph q , evaluate the correlation between each hyperedge in the knowledge hypergraph and each hyperedge in the problem hypergraph; then calculate the association A q The row max pooling layer on the matrix obtains the knowledge hyperedge most relevant to the problem hyperedge; the one-hot encoding classification distribution is used to calculate the weight of each knowledge hyperedge; according to the weight distribution obtained by the one-hot encoding classification distribution, only the information of the knowledge hyperedge centered on the problem hyperedge at the index position with a weight value of 1 is extracted and aggregated into the first aggregate information Z k ; The specific process of obtaining the second aggregate information is: based on the knowledge hypergraph features and the problem hypergraph features, calculate the association matrix A between the knowledge hypergraph guided by the knowledge hypergraph and the problem hypergraph k ; Then calculate the association matrix A k The row maximum pooling layer on the top is used to obtain the question hyperedge that is most relevant to the knowledge hyperedge; the one-hot encoding classification distribution is used to calculate the weight of each question hyperedge; according to the weight distribution obtained by the one-hot encoding classification distribution, only the information of the question hyperedge centered on the knowledge hyperedge at the index position with a weight value of 1 is extracted and aggregated into the second aggregate information Z q ; The fusion and prediction module is configured to: splice and fuse the first aggregate information and the second aggregate information, input the spliced ​​and fused aggregate information into the answer prediction layer, and output the predicted answer corresponding to the question.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the knowledge base-based visual question answering dual-stream attention multi-hop reasoning method as described in any one of claims 1 to 3 are implemented.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the knowledge base-based visual question answering dual-stream attention multi-hop reasoning method as described in any one of claims 1 to 3 are implemented.

Images