A visual question answering method and system based on external knowledge

Abstract

The application discloses a visual question answering method and system based on external knowledge, belongs to the technical field of visual question answering, and can solve the problems of limited model explainability and zero sample. The method comprises the following steps: S1, acquiring a target image and a target question, and constructing a scene knowledge graph; S2, acquiring an entity score vector and a relationship score vector of each step of T-step routing according to entities and relationships in the scene knowledge graph; S3, acquiring an entity total score vector and a relationship total score vector after T-step routing according to the entity score vector and the relationship score vector of each step; and S4, performing answer prediction according to the entity total score vector and the relationship total score vector to obtain a total prediction answer score. The application is used for visual question answering.

Description

Technical Field

[0001] This invention relates to a visual question answering method and system based on external knowledge, belonging to the field of visual question answering technology. Background Technology

[0002] Knowledge-Based Visual Question Answering (KB-VQA) requires a model to answer questions by incorporating knowledge beyond the scope of a given image and question. While successful in Visual Question Answering (VQA) tasks, KB-VQA presents a greater challenge for models to achieve human-like cross-modal scene understanding using external knowledge. Therefore, a core challenge of KB-VQA is the appropriate representation and utilization of knowledge within this cross-modal context.

[0003] Early research focused on retrieving relevant supporting facts from knowledge graphs (such as ConceptNet) for KB-VQA. For example, GloVe embeddings were used for fact scoring, selecting entities from the most relevant facts as the answer, or graph neural networks were applied to classify nodes (answers) in the retrieved knowledge subgraph. These methods achieved promising results on simple datasets (such as FVQA), but they lacked the ability to handle more complex problems requiring more sophisticated knowledge. To address this, in addition to explicit knowledge from knowledge graphs, implicit knowledge from pre-trained visual language models has been utilized to better understand the content of a given image and question. However, due to their black-box nature, the interpretability of these models is limited. Furthermore, all of these models suffer from the zero-shot problem because their answer classifiers do not handle questions with unseen answers well. Summary of the Invention

[0004] This invention provides a visual question answering method and system based on external knowledge, which can solve the problems of limited interpretability and zero-sample existence of existing models.

[0005] On one hand, the present invention provides a visual question answering method based on external knowledge, the method comprising:

[0006] S1. Obtain the target image and target question, and construct a scene knowledge graph;

[0007] S2. Based on the entities and relationships in the scenario knowledge graph, obtain the entity score vector and relationship score vector for each step of the T-step routing;

[0008] S3. Based on the entity score vector and the relationship score vector at each routing step, obtain the total entity score vector and the total relationship score vector after T routing steps;

[0009] S4. Based on the total score vector of the entities and the total score vector of the relationships, predict the answer to obtain the overall predicted answer score.

[0010] Optionally, S2 specifically includes:

[0011] S21. Obtain the entity score vector of the entity in the scene knowledge graph at the t-th step of routing;

[0012] S22. Calculate the probability score of each triple between different entities at the t-th step of routing to obtain the transition matrix and relation score vector of the t-th step of routing;

[0013] S23. Based on the entity score vector and transition matrix of the t-th step route, obtain the entity score vector of the (t+1)-th step route; where 0≤t≤T.

[0014] Optionally, S22 specifically includes:

[0015] S221. Obtain the problem keyword for the t-th step of routing, and calculate the probability score of each triple between different entities at the t-th step of routing based on the problem keyword;

[0016] S222. Obtain the sum of the probability scores of all triples of relationships between different entities as the transition probability scores between different entities, and obtain the transition matrix of the t-th step of the route based on the transition probability scores.

[0017] S223. Obtain the sum of the probability scores of all triples under each relation, and use it as the relation score vector for the t-th step of routing.

[0018] Optionally, S1 specifically includes:

[0019] S11. Obtain the target image and target problem, and construct a scene graph;

[0020] S12. Align the scene graph with entities in the external knowledge graph, and retrieve knowledge triples related to the scene graph from the external knowledge graph;

[0021] S13. Add the knowledge triples to the scene graph to obtain the scene knowledge graph.

[0022] Optionally, after S12, S1 further includes:

[0023] S14. Filter the knowledge triples related to the target problem in the knowledge triples to obtain the filtered knowledge triples.

[0024] Correspondingly, S13 specifically refers to:

[0025] The filtered knowledge triples are added to the scene graph to obtain the scene knowledge graph.

[0026] Optionally, S3 specifically includes:

[0027] S31. Obtain the hop count probability distribution of the T-step route;

[0028] S32. Calculate the total entity score vector and the total relationship score vector based on the jump probability distribution, the entity score vector, and the relationship score vector.

[0029] Optionally, S4 specifically includes:

[0030] S41. Obtain the probability of the predicted answer type;

[0031] S42. Based on the type probability, the entity total score vector, and the relationship total score vector, predict the answer to obtain the overall predicted answer score.

[0032] On the other hand, the present invention provides a visual question-answering system based on external knowledge, the system comprising:

[0033] A module is built to acquire the target image and target question, and to construct a scene knowledge graph;

[0034] The single-step scoring module is used to obtain the entity score vector and relation score vector for each step of the T-step routing based on the entities and relations in the scenario knowledge graph.

[0035] The total score module is used to obtain the total entity score vector and the total relationship score vector after T-step routing based on the entity score vector and the relationship score vector at each routing step.

[0036] The prediction module is used to predict the answer based on the total score vector of the entity and the total score vector of the relationship, and obtain the overall predicted answer score.

[0037] Optionally, the single-step scoring module specifically includes:

[0038] The acquisition unit is used to acquire the entity score vector of the entity in the scene knowledge graph at the t-th step of routing.

[0039] The calculation unit is used to calculate the probability score of each triple between different entities at the t-th step of routing, and to obtain the transition matrix and relation score vector of the t-th step of routing;

[0040] An iterative unit is used to obtain the entity score vector of the (t+1)th step route based on the entity score vector and the transition matrix of the t-th step route; where 0≤t≤T.

[0041] Optionally, the computing unit specifically includes:

[0042] The calculation subunit is used to obtain the question keyword for the t-th step of routing and calculate the probability score of each triple between different entities at the t-th step of routing based on the question keyword.

[0043] The transition matrix acquisition sub-unit is used to obtain the sum of the probability scores of all triples of relationships between different entities, which is used as the transition probability score between different entities, and the transition matrix of the t-th step routing is obtained based on the transition probability score.

[0044] The relation score acquisition sub-unit is used to obtain the sum of probability scores of all triples under each relation, which serves as the relation score vector for the t-th step of routing.

[0045] The beneficial effects that this invention can produce include:

[0046] This invention provides a visual question answering method based on external knowledge. For a target image and a target question, it first constructs a scene knowledge graph, specifically by combining the scene graph with retrieved external knowledge triples related to the given target image and question. Based on the scene knowledge graph, it performs step-by-step entity routing to predict the answer. Specifically, in each routing step, the method retains an entity score vector representing the probability of each entity being activated as an answer, and a transition matrix representing the transition probability from one entity to another. Simultaneously, the method focuses on certain keywords in the question at each routing step, and the calculation of the transition matrix references these key keywords. Then, based on the entity scores and transition matrix of the current routing step, entity routing is performed using transition entity scores. After a fixed number of routing steps, the method can predict which steps include the answer and select the answer from the entities and relationships along the routing path. The method of this invention explicitly provides a step-by-step reasoning process, demonstrating good interpretability. Furthermore, instead of classifying candidate answers, this invention selects answers from the constructed scene knowledge graph, which effectively handles questions with unseen answers and avoids the zero-shot problem. Meanwhile, this invention proposes an interpretable graph routing network (GRN) to implement the above method, which explicitly reasones about the constructed scene knowledge graph and predicts the answer step by step. Attached Figure Description

[0047] Figure 1 A schematic diagram of the framework of the GRN model provided in an embodiment of the present invention;

[0048] Figure 2 The diagram shows the results of the GRN model provided in three examples according to embodiments of the present invention. Detailed Implementation

[0049] The present invention will now be described in detail with reference to the embodiments, but the present invention is not limited to these embodiments.

[0050] Knowledge-based VQA requires models to combine visual information with external knowledge for answer reasoning. Early work followed a knowledge retrieval and application process, retrieving structured knowledge related to images and questions from knowledge graphs. For example, retrieved facts were embedded into image and question features for answer classification. Alternatively, graph neural networks were used to infer answer entities on retrieved knowledge subgraphs. These models performed well on early, simple datasets, but their performance declined on more complex questions due to limitations in knowledge coverage. To improve performance, implicit knowledge from pre-trained visual language models was incorporated as a supplement. For example, MUKEA used the pre-trained visual language model LXMERT to encode questions and images for knowledge extraction and accumulation. These methods achieved considerable improvements on KB-VQA, but lacked interpretability because they understood and answered questions in a black-box manner. Furthermore, all these methods are based on answer classification and do not handle questions with unseen answers well, resulting in the zero-shot problem.

[0051] Scene graphs are graphical representations of images, containing information about abstract visual concepts and the relationships between them. For VQA tasks, scene graphs have an advantage over typical object features extracted from images because they explicitly organize visual knowledge in a structured way. Previous research has utilized scene graphs for VQA. For example, by constructing a fully connected scene graph, using language-guided graph network layers for message passing, and classifying answers based on node features. Alternatively, graph reasoning on real-world scene graphs has been performed using GAT, and explicit sequential reasoning on scene graphs has been achieved using reinforcement learning. These studies focus on visual reasoning based on scene abstraction, without considering external knowledge.

[0052] This invention provides a knowledge-based visual question answering method, the method comprising:

[0053] S1. Obtain the target image and target question, and construct a scene knowledge graph.

[0054] S1 specifically includes:

[0055] S11. Obtain the target image and target problem, and construct a scene graph.

[0056] S12. Align the scene graph with entities in the external knowledge graph and retrieve knowledge triples related to the scene graph from the external knowledge graph.

[0057] Specifically, visual entities in the scene graph are aligned with entities in the external knowledge graph based on entity names, and knowledge triples related to the scene graph are retrieved from the external knowledge graph.

[0058] S13. Add the knowledge triples to the scene graph to obtain the scene knowledge graph.

[0059] Following S12, S1 also includes:

[0060] S14. Filter the knowledge triples that are related to the target problem in the knowledge triples to obtain the filtered knowledge triples.

[0061] Correspondingly, S13 specifically refers to:

[0062] The filtered knowledge triples are added to the scene graph to obtain the scene knowledge graph.

[0063] KB-VQA requires incorporating external knowledge beyond the given image and question to predict answers. To incorporate this knowledge, this invention constructs a scene knowledge graph, as shown below. (Reference) Figure 1 As shown, firstly, this invention designs a heuristic method to retrieve relevant knowledge from an external KG (Knowledge Graph). Specifically, it utilizes the scene graph of image I, represented as G. S ={E S ,R S ,T S}, where E S and R S These are visual entities and the relationships between them. S Representing the three elements of a scene {(h,r,t)} (h∈E) S ,t∈E S ,r∈R S Given a scene graph G. S First, visual entity E S Entities in KG are aligned based on entity name, G KG ={E KG ,R KG ,T KG}(where E) KG ,R KG and T KG Let these represent the three elements: entity, relation, and knowledge, respectively. Then, retrieve the relevant knowledge triples {(h,r,t)|h∈E} from the KG. S ∩E KG ,t∈E KG ,r∈R KG This makes the head entity of the retrieved triple a visual entity.

[0064] To further filter knowledge triples relevant to a given target question, this invention removes triples containing relations or tail entities not mentioned in the question text. Finally, the retrieved and filtered knowledge triples are added to the scene graph G. S In this process, a scene knowledge graph G = {E, R, T} is obtained for a given target image I and target question T (where E, R, and T represent the three elements of entity, relation, and knowledge, respectively). The constructed scene knowledge graph G combines the scene graph with relevant external knowledge to answer the question.

[0065] Based on scene knowledge graphs, this invention proposes an interpretable graph routing network model, GRN, which explicitly routes data along entity scores. Figure 1 Entity routing is performed step by step. At each step, the GRN retains an entity score vector a. t ∈[0,1] N , representing the activation probability of an entity, and a transition matrix W. t ∈[0,1] N×N , representing the transition probability from one entity to another. At each step, GRN focuses on a specific part of the question (i.e., the question keywords) and routes entities based on the transition matrix calculated with reference to the question keywords, using the scores of the transition entities. After a fixed number of steps (T steps), GRN stops entity routing. It then first determines which steps include the answer, and then selects the answer from the entities and relations along the routing path to predict the answer.

[0066] S2. Based on the entities and relationships in the scenario knowledge graph, obtain the entity score vector and relationship score vector for each step of the T-step routing.

[0067] S2 specifically includes:

[0068] S21. Obtain the entity score vector of the entity in the scene knowledge graph at the t-th step of the route.

[0069] For the t-th step of GRN, retain the entity score vector. in Represents entity e i The activation probability is given by N, where N represents the total number of entities in the scene knowledge graph G. Initially, when t = 0, the activation probability is calculated. Where entity e i If entities are mentioned in the question text using n-gram matching (where n represents the total number of entities mentioned in the question text), then... otherwise Note that this invention sets an activation threshold. For each step, only the score a i Entity e with ≥δ i Activated.

[0070] S22. Calculate the probability score of each triple between different entities at the t-th step of routing to obtain the transition matrix and relation score vector of the t-th step of routing.

[0071] S22 specifically includes:

[0072] S221. Obtain the question keyword for the t-th step of routing, and calculate the probability score of each triple between different entities at the t-th step of routing based on the question keyword;

[0073] S222. Obtain the sum of the probability scores of all triples of relationships between different entities as the transition probability scores between different entities, and obtain the transition matrix of the t-th step of the route based on the transition probability scores.

[0074] S223. Obtain the sum of the probability scores of all triples under each relation, and use it as the relation score vector for the t-th step of routing.

[0075] Having identified the activated entities, we can now consider transitions from these entities and compute the transition matrix. For the transition matrix... Calculated elements Indicates from entity e i to e j The transition probability scores are calculated. It is assumed that at each step, the GRN model focuses on certain keywords in the question and selects the transition direction based on these keywords. Therefore, the question keywords can be calculated first. In step t, the embedding status of the attention layer is considered:

[0076] m,(h1,…,h |Q| )=ViLT(Q,I;θ), (1)

[0077]

[0078] α t =Softmax(k t ·[h1,...,h ∣Q∣ ] · (3)

[0079]

[0080] Where m represents the multimodal features of a given question Q and image I based on a pre-trained visual language model ViLT. t It is a mapping function (the superposition of linear and Tanh layers). k t Based on word embedding h i Calculate the weight α of each word t The attention key. Finally, the key word for the question is q. t It is obtained by weighted summation, which is the weighted sum of word embeddings in step t.

[0081] Next, in step t, based on the question keyword q t Calculate the transition probability score from the activated entity to other entities. Specifically, for an activated entity e... i From it to another entity e j The transition probability fraction is e i to e j Each triple (e i ,r,e j The sum of probability fractions for () ∈ T. Formally, the relation is r. k The triple (e i ,r k ,e j The probability fraction of ) is:

[0082]

[0083] In the formula, The keyword q represents the problem. t With relation r k Semantic similarity between them The keyword q represents the problem. t With tail entity e j The semantic similarity between them. μ is a learnable gating parameter used to control whether it is a relation or a tail entity pair triple (e i ,r k ,e j The transition probability score contributes more. Formally, μ is calculated based on the question keywords of the current step, since keywords can correspond to relations or entities.

[0084] μ = Sigmoid(MLP(q) t (6)

[0085] By utilizing the gating parameter μ, this invention can obtain more accurate triplet transition scores. Key words: q t With relation r k semantic similarity between Calculated as:

[0086]

[0087] Where, r k The r obtained from pre-trained BERT k Relational embedding. ⊙ refers to element-wise product. Similarly, the key word in the question is q. t With tail entity e j semantic similarity between for:

[0088]

[0089] Where e j It is entity e j The entity embeddings are obtained from the pre-trained BERT.

[0090] After calculating each triplet (e) i ,r k ,e j probability fractions Then, all triples (K is e) i With e j The sum of all probability scores for the relationship between e and e is used to calculate the probability of e. i to e j transition probability fraction

[0091]

[0092] If e i to e j If there are no direct triples between them, then For inactive entities, the transition probability score from them to any other entity is set to 0.

[0093] After obtaining the entity score vector {a o ,...,a t ,...,a T Afterwards, this invention still needs to calculate the relation score for answer selection, because a relation r∈R can also be an answer to the question. Therefore, a relation score vector s is constructed at each step t. t ∈R M (M represents the total number of relations in graph G), where M represents the probability of the relation being activated as the answer. Formally, all triples (e... i ,r k ,e j The relationship between )∈T and step t: r k The transition probability scores are summed to calculate r. k Relationship score

[0094]

[0095] Note the relational score vector s t It is also normalized to maintain

[0096] S23. Based on the entity score vector and transition matrix of the t-th step route, obtain the entity score vector of the (t+1)-th step route; where 0≤t≤T.

[0097] With entity score vector at and the transition matrix W at step t t The entity fraction can be multiplied by W. t Perform the transition to obtain the entity score vector at step t+1:

[0098] a t+1 =a t W t (11)

[0099] Note that a t+1 Will be normalized to maintain if If the value is greater than the activation threshold δ, then entity e i It will be activated at step t+1.

[0100] After T steps, GRN terminates the entity routing process and predicts the answer from the entities and relations on the routing path.

[0101] S3. Based on the entity score vector and relationship score vector of each routing step, obtain the total entity score vector and total relationship score vector after T routing steps.

[0102] S3 specifically includes:

[0103] S31. Obtain the hop count probability distribution of the T-step route;

[0104] S32. Based on the jump probability distribution, entity score vector, and relationship score vector, calculate the total entity score vector and the total relationship score vector.

[0105] After a fixed T-step routing, the entity score vector (a) is obtained. 0 ,a 1 ,…,a T ) and relational score vector (s 0 ,s 1 ,…,s T This invention first predicts the probability distribution of the number of hops h (1≤h≤T) that may contain the answer, and then selects the answer based on the entity score vector and the relation score vector, as follows:

[0106] c = Softmax(MLP(m)), (12)

[0107]

[0108]

[0109] where c∈[0,1] T Let h represent the probability distribution over the number of jumps. Then, calculate the total entity score vector a′ and the total relation score vector s′ for the final answer selection.

[0110] S4. Based on the total entity score vector and the total relation score vector, predict the answer to obtain the overall predicted answer score.

[0111] S4 specifically includes:

[0112] S41. Obtain the probability of the predicted answer type;

[0113] S42. Based on the type probability, the total entity score vector, and the total relation score vector, predict the answer to obtain the overall predicted answer score.

[0114] Since the answer could be an entity or a relation, the type of the answer is predicted first. This invention uses u = {u0, u1} to represent the probability that the answer is an entity or a relation, and calculates the probability u of the predicted answer type as follows.

[0115] u = Softmax(MLP(m)). (15)

[0116] Therefore, the overall predicted answer score o∈R N+M The calculation method is as follows:

[0117]

[0118]

[0119]

[0120] By using the total entity score vector a′∈R N The total score vector s′∈R of the relation N By unifying the same space by filling it with zeros, we obtain... and The overall predicted answer score o can be based on the type probability u. and Perform a weighted average.

[0121] This model can explicitly provide the step-by-step reasoning process and has good interpretability. Furthermore, compared to answer classification models, GRN can effectively handle questions without visible answers, avoiding the zero-shot problem.

[0122] This invention aligns the standard answer A to question Q with a specific entity or relation, and constructs a one-hot objective score vector y∈{0,1}. N+M .

[0123]

[0124]

[0125] y = [yent ;y rel ] (twenty one)

[0126] The training objective of this invention is the L2 Euclidean distance between o and y:

[0127]

[0128] In this embodiment of the invention, cross-entropy loss can also be used as the training objective. However, L2 Euclidean distance is more efficient and stable.

[0129] The GRN model of this invention is not affected by intermediate supervision signals because the model is differentiable and can be trained in an end-to-end manner.

[0130] This invention proposes a novel interpretable graph routing network (GRN) for KB-VQA, which derives answers by progressively routing entities across a constructed scene knowledge graph. At each step, the GRN centers on a specific question keyword, learns transition probability scores based on the keyword, and transfers entity scores along the graph. The GRN model of this invention handles zero-starting-point problems well. Furthermore, the GRN explicitly provides the reasoning process step-by-step, exhibiting good interpretability.

[0131] Another embodiment of the present invention provides a visual question-answering system based on external knowledge, the system comprising:

[0132] A module is built to acquire the target image and target question, and to construct a scene knowledge graph;

[0133] The single-step scoring module is used to obtain the entity score vector and relation score vector for each step of the T-step routing based on the entities and relations in the scene knowledge graph.

[0134] The total score module is used to obtain the total entity score vector and total relation score vector after T steps of routing, based on the entity score vector and relation score vector of each routing step.

[0135] The prediction module is used to predict the answer based on the total entity score vector and the total relation score vector, and obtain the overall predicted answer score.

[0136] Specifically, the single-step scoring module includes:

[0137] The acquisition unit is used to acquire the entity score vector of the entity in the scene knowledge graph at the t-th step of routing.

[0138] The calculation unit is used to calculate the probability score of each triple between different entities at the t-th step of routing, and to obtain the transition matrix and relation score vector of the t-th step of routing;

[0139] An iterative unit is used to obtain the entity score vector of the (t+1)th step route based on the entity score vector and transition matrix of the t-th step route; where 0≤t≤T.

[0140] The computing unit specifically includes:

[0141] The computational sub-unit is used to obtain the question keyword for the t-th step of the route and calculate the probability score of each triple between different entities when the route is taken at the t-th step based on the question keyword.

[0142] The transition matrix acquisition sub-unit is used to obtain the sum of the probability scores of all triples of relationships between different entities, which is used as the transition probability score between different entities, and the transition matrix of the t-th step of the route is obtained based on the transition probability score.

[0143] The relation score acquisition sub-unit is used to obtain the sum of probability scores of all triples under each relation, which serves as the relation score vector for the t-th step of routing.

[0144] For a detailed description of each module in the above system, please refer to the description of each step in the method. It will not be repeated here. The above system can achieve the same functionality as the method.

[0145] The effectiveness of the GRN proposed in this invention will be evaluated below.

[0146] This invention evaluates the GRN on the benchmark dataset KRVQA. KRVQA is the largest knowledge-based VQA dataset used to evaluate the multi-step reasoning capabilities of KB-VQA models. It contains 32,910 images, 157,201 QA pairs, and a knowledge graph containing 193,449 knowledge triples, 2,339 relation types, and 102,343 distinct entities. For each image, KRVQA provides its scene graph annotations from their original dataset, VisualGenome. Based on the reasoning steps, questions are categorized into one-step and two-step questions, containing 68,448 and 88,753 samples, respectively. This invention splits the questions and images into training, validation, and test sets at 60%, 20%, and 20% ratios, respectively. The GRN model is trained on the training split, and results are reported on the test split. It was found that the answers to a total of 3% of the questions in the test set were not present in the training set. The models for these questions were evaluated to investigate the effectiveness of the GRN under the zero-score setting. This invention uses top-1 accuracy as the evaluation metric for all models.

[0147] Benchmarks: To comprehensively evaluate the proposed GRN, this invention compares it with the following strong baselines, including traditional VQA models, knowledge-based VQA models, and scene graph-based VQA models. For a fair comparison, the constructed scene knowledge graph is used as input to the scene graph-based VQA model, which contains relevant knowledge retrieved for KB-VQA.

[0148] Traditional VQA model:

[0149] FiLM: It employs characteristic linear modulation to achieve better visual reasoning.

[0150] MFH: A generalized multimodal factorial high-order pooling method that can more effectively fuse multimodal features, thereby achieving better VQA performance.

[0151] MCAN: A deep modular co-concern network for VQA.

[0152] Knowledge-based VQA model:

[0153] MCAN+know.: Improves MCAN by embedding knowledge into the co-attention module of KB-VQA.

[0154] ViLT: A widely used pre-trained vision and language model. ViLT is fine-tuned on VQAv2 and further fine-tuned on KRVQA.

[0155] MUKEA: A novel framework for multimodal knowledge extraction and accumulation. It uses a pre-trained LXMERTLXMERT model to encode questions and images, and extracts multimodal knowledge triples.

[0156] DMMGR: It uses a key-value memory network to infer answers on spatially perceived image graphs.

[0157] Scene diagram based on VQA model:

[0158] LCGN: Constructing a language guidance graph network on a scene graph for answer classification.

[0159] LCGN+ Image Augmentation. Image features are obtained using CLIP and concatenated with feature embeddings for answer classification in LCGN.

[0160] GraphVQA: A graph attention network is designed on the scene graph to classify answers by combining graph embeddings with question embeddings.

[0161] GraphVQA+ Image Augmentation. Similarly, image features are learned using CLIP and fused into the feature embeddings in GraphVQA for answer classification.

[0162] Experimental Details: The maximum number of inference steps (question hops) in KRVQA is 2, so T = 2 was set. A pre-trained visual language model, ViLT, was used as the question-image encoder, with a hidden dimension of 768. BERT was used to initialize the embeddings of relations and entities in the scene knowledge graph. For the visual scene graph, the ground truth scene graph provided by the KRVQA dataset was used. For the knowledge graph, the supporting knowledge graph provided by KRVQA was used. Reverse relations were added to the scene knowledge graph. Since question hops and answer types are provided in KRVQA, they can be used as auxiliary targets. Their cross-entropy losses were calculated and added to equation (22) after multiplying by a factor of 0.01. The model was optimized using RAdam with a learning rate of 0.001 for 20 epochs. Dynamic learning weight decay and early stopping were employed to better train the model.

[0163] Table 1 shows the accuracy of different methods on KRVQA. From Table 1, it can be observed that:

[0164] (1) The GRN proposed in this invention consistently outperforms all baseline models on all types of problems (especially KB-related problems). Compared with the best baseline, GRN improves the top-1 accuracy by approximately 33.51%. This verifies the effectiveness of GRN in KB-VQA.

[0165] (2) Knowledge-based models are generally superior to traditional VQA models, which suggests that external knowledge is useful for KB-VQA.

[0166] (3) Scene graph-based VQA models (LCGN and GraphVQA) constructed based on the constructed scene knowledge graph generally outperform knowledge-based VQA models (MCAN+know., ViLT, MUKEA, and DMMGR). This indicates that scene graphs can facilitate the understanding of images, thereby improving visual question answering. These scene graph-based models, combined with image embedding (+image augmentation), can achieve a slight improvement (approximately +1.0%), but their best accuracy (33.03%) is still significantly lower than the model GRN (66.53%) of this invention.

[0167] This verifies that, in addition to the constructed scene knowledge graph and image features, the GRN model primarily benefits from a graph routing network that explicitly performs step-by-step entity routing on the scene knowledge graph.

[0168] Table 1. Performance comparison of different models on KRVQA.

[0169]

[0170] The best results are indicated in bold.

[0171] Table 2 shows the accuracy of GRN on a subset of zero-shot tests in KRVQA and some strong baselines. As shown in Table 2, GRN successfully handles questions with unknown answers, while other models suffer from zero-shot problems, even achieving zero accuracy. This is because these baselines are based on answer classifiers trained on the training data, which cannot handle questions where the answer never appeared in the training data. In contrast, GRN performs entity routing on the scene knowledge graph and selects answers from entities and relations along the routing path. Therefore, it can handle questions with unknown answers.

[0172] To verify the contributions of certain modules in GRN, GRN can be compared with the following variants:

[0173] (1) w / o auxiliary response type loss, eliminating auxiliary response type loss;

[0174] (2) eliminate the jump loss of auxiliary problems;

[0175] (3) w / o double auxiliary loss, that is, removing the two auxiliary losses;

[0176] (4) w / o image features, replace the image with a completely black image and remove image features;

[0177] (5) No BERT initialization, random initialization is used for entity and relation embedding.

[0178] Table 3 shows the results of the ablation study. It can be observed that:

[0179] (1) GRN outperforms all variants, proving the effectiveness of each module.

[0180] (2) The performance of the GRN without auxiliary question jump loss is lower than that of the GRN without auxiliary answer type loss, which means that question jump information is more important than answer type information. Eliminating these two auxiliary losses will lead to a further performance degradation.

[0181] (3) Image features are relatively important in GRN models because the accuracy of GRN without image features drops by about 6.73%.

[0182] (4) The BERT initialization of entity and relation embedding is relatively less important (accuracy decreased by about 1.57%).

[0183] Table 2 Precision on the zero-sample test subset

[0184]

[0185] Table 3 Ablation experiments of KRVQA

[0186]

[0187] This invention is in Figure 2 The image visualizes the intermediate results of the GRN for three multi-hop problems. For the first problem, the GRN starts with the entity "Public toilet" mentioned in the problem text, then focuses on the keywords "belong to" and "in," ultimately arriving at the correct answer entity "Drain." For the second problem, the GRN finds the correct answer "Instance of," which is a relation. This is because the GRN proceeds along the triples, considering both the relation and the tail entity at each step, and selects the answer from both the relation and the entity. The third problem selects from a subset of zero-point tests. The GRN successfully finds the answers to the problems, demonstrating that it can handle these problems indiscriminately.

[0188] In summary, the contributions of this invention can be summarized as follows:

[0189] (1) This invention proposes a novel interpretable graph routing network model (GRN) for KB-VQA, which explicitly addresses the contextual knowledge of the constructed environment. Figure 1 Proceed with reasoning step by step.

[0190] (2) In the GRN model, a scene knowledge graph is first constructed, and the scene graph is combined with the retrieved knowledge triples. Based on the graph, GRN performs entity routing step by step according to the calculated transition matrix, and selects the answer from the entities and relations on the routing path, thus avoiding the zero-shot problem.

[0191] (3) Experiments on the benchmark dataset KRVQA show that GRN significantly improves the performance of KB-VQA and provides good interpretability.

[0192] The above description is merely a few embodiments of this application and is not intended to limit this application in any way. Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any changes or modifications made by those skilled in the art without departing from the scope of the technical solution of this application using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A visual question answering method based on external knowledge, characterized in that, The method includes: S1. Obtain the target image and target question, and construct a scene knowledge graph; S2. Based on the entities and relationships in the scenario knowledge graph, obtain the entity score vector and relationship score vector for each step of the T-step routing; S3. Based on the entity score vector and the relationship score vector at each routing step, obtain the total entity score vector and the total relationship score vector after T routing steps; S4. Based on the total entity score vector and the total relation score vector, predict the answer to obtain the overall predicted answer score; S1 specifically includes: S11. Obtain the target image and target problem, and construct a scene graph; S12. Align the scene graph with entities in the external knowledge graph, and retrieve knowledge triples related to the scene graph from the external knowledge graph; S13. Add the knowledge triples to the scene graph to obtain the scene knowledge graph; S2 specifically includes: S21. Obtain the entity score vector of the entity in the scene knowledge graph at the t-th step of routing; the entity score vector of the entity at the t-th step of routing. ,in Representing entities activation probability, Representation of scene knowledge graph The total number of entities in the middle; S22. Obtain the question keyword for the t-th step of routing, and calculate the probability score of each triple between different entities in the t-th step of routing based on the question keyword; obtain the sum of the probability scores of triples of all relations between different entities as the transition probability score between different entities, and obtain the transition matrix of the t-th step of routing based on the transition probability score; obtain the sum of the probability scores of all triples under each relation as the relation score vector of the t-th step of routing. S23. Based on the entity score vector and transition matrix of the t-th step route, obtain the entity score vector of the (t+1)-th step route; where 0≤t≤T.

2. The method according to claim 1, characterized in that, Following S12, S1 further includes: S14. Filter the knowledge triples related to the target problem in the knowledge triples to obtain the filtered knowledge triples. Correspondingly, S13 specifically refers to: The filtered knowledge triples are added to the scene graph to obtain the scene knowledge graph.

3. The method according to claim 1, characterized in that, S3 specifically includes: S31. Obtain the hop count probability distribution of the T-step route; S32. Calculate the total entity score vector and the total relationship score vector based on the jump probability distribution, the entity score vector, and the relationship score vector.

4. The method according to claim 1, characterized in that, S4 specifically includes: S41. Obtain the probability of the predicted answer type; S42. Based on the type probability, the entity total score vector, and the relationship total score vector, predict the answer to obtain the overall predicted answer score.

5. A visual question-answering system based on external knowledge, characterized in that, The system includes: A module is built to acquire the target image and target question, and to construct a scene knowledge graph; The single-step scoring module is used to obtain the entity score vector and relation score vector for each step of the T-step routing based on the entities and relations in the scenario knowledge graph. The total score module is used to obtain the total entity score vector and the total relationship score vector after T-step routing based on the entity score vector and the relationship score vector at each routing step. The prediction module is used to predict the answer based on the total score vector of the entity and the total score vector of the relationship, and obtain the overall predicted answer score; The construction module is specifically used for: acquiring the target image and the target question, and constructing a scene graph; aligning the scene graph with entities in an external knowledge graph, and retrieving knowledge triples related to the scene graph from the external knowledge graph; and adding the knowledge triples into the scene graph to obtain a scene knowledge graph. The single-step scoring module specifically includes: The acquisition unit is used to acquire the entity score vector of the entity in the scene knowledge graph at the t-th step of routing; the entity score vector of the entity at the t-th step of routing. ,in Representing entities activation probability, Representation of scene knowledge graph The total number of entities in the middle; The calculation unit is used to obtain the question keyword for the t-th step of routing, and calculate the probability score of each triple between different entities in the t-th step of routing based on the question keyword; obtain the sum of the probability scores of triples of all relations between different entities as the transition probability score between different entities, and obtain the transition matrix of the t-th step of routing based on the transition probability score; obtain the sum of the probability scores of all triples under each relation as the relation score vector of the t-th step of routing. An iterative unit is used to obtain the entity score vector of the (t+1)th step route based on the entity score vector and the transition matrix of the t-th step route; where 0≤t≤T.

Images