A system for open visual question answering based on path reasoning

Abstract

The application provides a system for open visual question answering based on path reasoning, comprising: a schema graph construction module, configured to construct a second schema graph according to an input image and question text raised for the image; a pruning network, comprising a node encoder and a pruning layer, the node encoder being configured to extract an embedding vector of a node according to the image, the question text and a type of each node; the pruning layer being configured to prune the second schema graph to obtain a third schema graph; a path reasoning network, comprising a path encoder, a bilinear layer and an output layer, the path encoder being configured to determine an embedding vector of a candidate reasoning path, the bilinear layer being configured to determine a matching probability of the embedding vector of each candidate reasoning path and a multi-modal context feature, and the output layer being configured to determine a reasoning result according to the matching probability corresponding to each candidate reasoning path.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural networks, in particular to the technical field of visual question answering, and more particularly to a system for open visual question answering based on path reasoning. BACKGROUND

[0002] With the development of computer vision and natural language processing technology, the knowledge-based visual question answering (KB-VQA) task has become a research hotspot. The KB-VQA task requires reasoning on a given question and image, and obtaining relevant information from a knowledge base to obtain an answer.

[0003] The model for solving this problem is usually a two-stage operation based on the knowledge retriever-answerer framework, that is, first, the knowledge retriever obtains relevant information from the knowledge base, and then inputs the obtained relevant information, the question and the image into the answerer to predict the answer.

[0004] The knowledge retrieval stage usually uses manual rules for operation, and is not trained end-to-end with the answerer. For text-form knowledge, the dense passage retrieval model (DPR) has proven to be effective in retrieving relevant knowledge fragments. Visual knowledge can be obtained by applying visual models (such as object segmentation models, image description models, etc.) on the input image.

[0005] The framework of the answerer can be divided into two categories: answer generator and answer classifier. Pre-trained answer generators are mainly based on the given image, question and knowledge to generate answers. However, they usually rely on large-scale pre-training corpus, so the computing cost is high. Therefore, most of the current KB-VQA methods use answer classifiers. Answer classifiers predict the final answer by classifying in a pre-defined answer space (equivalent to having a pre-defined answer candidate set). Most studies use pre-trained visual-linguistic models to integrate visual and linguistic information in the input question and retrieved knowledge. In order to improve the explainability of the visual question answering (VQA) model, researchers have proposed different solutions. The graph neural network (GNN) based model infers the answer by propagating and aggregating on the question-image related graph. The reasoning process of these models can be explained by decoding the attention weights of the graph neural network. However, all these models are limited by the pre-defined answer candidate set and cannot fully exploit the knowledge contained in the entire knowledge base. In addition, when reasoning directly from the input question and image, it is done through hidden parameters that cannot be intuitively understood, which is a black box operation, and the reasoning process cannot be explained.

[0006] Open KB-VQA tasks aim to move beyond the limitations of predefined answer candidate sets. To address the open KB-VQA problem, extractive readers and answer generators can be used. Extractive readers first retrieve relevant knowledge fragments from an existing corpus based on the question, then extract the answer by predicting the start and end positions of the answer within the retrieved knowledge fragments. However, this approach is unsuitable for direct application to VQA tasks due to the lack of answer position annotations, and the high cost of annotating the start and end positions of answers in large amounts of text. Another approach to implementing open VQA is to generate answers from scratch. Some large-scale pre-trained models (such as GPT-3) can already generate plausible answers. The success of these models largely relies on implicit knowledge stored during model training, such as knowledge from large-scale language models, as well as explicit knowledge retrieved for each question and joint inference networks. Both approaches face a common problem: over-reliance on recall rates that retrieve knowledge fragments containing true answers. Model performance is highly sensitive to the quality and quantity of retrieved knowledge. For example, in order to achieve a sufficient knowledge hit rate, extractive readers require a large number of knowledge paragraphs, which also introduces a lot of interference, potentially reducing the accuracy of the reader.

[0007] Therefore, the existing technology has the following unresolved issues:

[0008] 1. Answer classifiers predict the final answer by classifying within a predefined answer space (equivalent to a predefined set of candidate answers, which contains far less knowledge than the knowledge base), and cannot fully exploit the knowledge contained in the entire knowledge base;

[0009] 2. The reasoning process is a black box operation and cannot be explained. Summary of the Invention

[0010] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide an open visual question answering system based on path reasoning.

[0011] The objective of this invention is achieved through the following technical solution:

[0012] According to a first aspect of the present application, there is provided a system for open visual question answering based on path reasoning, comprising: a schema graph construction module configured to obtain an input image and a question text raised for the image, extract nodes and relations from the image and the question text to construct a first schema graph, retrieve neighbor nodes within a preset hop number of nodes in the first schema graph from a preset knowledge base to expand the first schema graph, and obtain a second schema graph incorporating external knowledge information; a pruning network comprising a node encoder and a pruning layer, wherein the node encoder is configured to extract an embedding vector of each node according to the image, the question text, and a type of each node; the pruning layer is configured to prune the second schema graph according to a weighted sum of a similarity between an embedding vector of each node and a multi-modal context feature and a distance between each node and a key node, and obtain a third schema graph, wherein the multi-modal context feature is a fusion feature obtained by cross-extracting features of the image and the text, and the key node is a node extracted from the image and the question text; a path reasoning network comprising a path encoder, a bilinear layer, and an output layer, wherein the path encoder is configured to determine an embedding vector of each candidate reasoning path according to a text feature of the question text, an image feature of the image, and a splicing vector of embedding vectors of nodes on each candidate reasoning path based on a plurality of candidate reasoning paths extracted from the third schema graph; the bilinear layer is configured to determine a matching probability between an embedding vector of each candidate reasoning path and the multi-modal context feature; and the output layer is configured to determine a reasoning result according to the matching probability corresponding to each candidate reasoning path.

[0013] Optionally, the schema graph construction module is configured to: extract a plurality of keywords and their relations from the question text, the keywords being any one of a noun, a verb, an adjective, an adverb, and a phrase; perform a first matching between the plurality of keywords and their relations and entities and relations in the knowledge base, and create an initial schema graph containing text nodes and their relations based on the entities and relations matched in the first matching; identify objects and relations in the image, perform a second matching between the identified objects and relations and the entities and relations in the knowledge base, and add the entities and relations matched in the second matching to the initial schema graph to obtain the first schema graph containing text nodes, image nodes, and their relations; and retrieve neighbor nodes within two hops of nodes in the first schema graph from the knowledge base, and add the retrieved neighbor nodes and their relations to the first schema graph to obtain the second schema graph.

[0014] Optionally, the knowledge base is a ConceptNet knowledge graph or a Freebase knowledge graph.

[0015] Optionally, the node encoder is configured to: obtain a first embedding model to extract text features from the question text, extract image features from the image, and extract multi-modal context features from the text and the image; obtain a second embedding model to extract fusion text features from fusion text, the fusion text being obtained by splicing the question text and text corresponding to neighbor nodes expanded from the node; for each node, determine an embedding vector of the node by using at least one perception layer according to the multi-modal context features, entity features corresponding to the node, a type of the node, and the fusion text features extracted by the second embedding model from the fusion text.

[0016] Optionally, the first embedding model is a ViLBERT model, a VLBERT model, a UNITER model, or a ViLT model.

[0017] Optionally, the second embedding model is a BERT model, a TinyBERT model, a GPT2 model, or a GPT3 model.

[0018] Optionally, the pruning layer is configured to determine the weighted sum in the following manner:

[0019] s prune = (1-θ p )s cos + θ p s bfs

[0020] wherein s cos represents similarity between an embedding vector of a node and multi-modal context features corresponding to the image and the question text, s bfs represents a distance score between the node and a key node, θ p represents a weight corresponding to s bfs .

[0021] Optionally, each candidate reasoning path in the plurality of candidate reasoning paths is obtained in the following manner: starting from a starting node in the third mode graph, performing a random walk of k steps based on a path constructed by a relationship, and obtaining a candidate reasoning path when walking to an ending node, wherein the starting node at each time is a node randomly selected from all key nodes, and the ending node at each time is an answer node.

[0022] Optionally, the k is greater than or equal to 3.

[0023] Optionally, the system is trained in the following manner: training data is obtained, including a plurality of samples and labels, wherein each sample includes a sample image and sample question text, and the label indicates the sample corresponding answer truth value; the samples in the training data are input into the system to obtain the sample corresponding second mode graph and the embedding vector of each node thereof, and the matching probability of a plurality of candidate reasoning paths corresponding to each sample; the total loss is determined according to the triple loss and the binary classification cross-entropy loss of the samples in the training data, and the trainable parameters of the node encoder, the path encoder and the bilinear layer are updated according to the total loss, wherein the triple loss of each sample is the average of the sub-losses of all triplets that can be constructed according to the second mode graph of the sample, in each triplet, the anchor sample is the multimodal context feature of the sample, the positive sample is the embedding vector of the node corresponding to an answer truth value, and the negative sample is the embedding vector of the node corresponding to a non-answer truth value; the binary cross-entropy loss of each sample is the average of the binary cross-entropy sub-losses calculated according to the matching probability of all candidate reasoning paths corresponding to the sample and the label. BRIEF DESCRIPTION OF DRAWINGS

[0024] The embodiments of the present application will be further described below with reference to the drawings, in which:

[0025] Figure 1 A system module schematic diagram of the system for open visual question answering based on path reasoning according to the embodiments of the present application;

[0026] Figure 2 An effect schematic diagram of reasoning by the system according to the embodiments of the present application;

[0027] Figure 3 A comparison schematic diagram of the effect of reasoning by the system according to the embodiments of the present application and the effect of reasoning by the prior art. DETAILED DESCRIPTION

[0028] In order to make the objects, technical schemes and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0029] As mentioned in the background section, the prior art has the following problems to be solved: 1. The answer classifier predicts the final answer by classifying in a predefined answer space (equivalent to having a predefined answer candidate set, which contains much less knowledge than the knowledge base), which cannot fully exploit the knowledge contained in the entire knowledge base; 2. The reasoning process is a black box operation, and the reasoning process cannot be explained.

[0030] To this end, the application is based on a knowledge base, first constructs a first pattern graph for an image and a text (question text) asking a question for the image, then retrieves neighbor nodes within a preset hop number of nodes in the first pattern graph from the knowledge base to expand the first pattern graph (equivalent to a first rough screening), obtains a second pattern graph that integrates external knowledge information, and thus takes the entire knowledge base as a possible source of answers to predict the final answer with the knowledge of the entire knowledge base;

[0031] In addition, by pruning the network, on the one hand, embedding vectors of various nodes can be obtained, and on the other hand, some nodes with low similarity and long distance between embedding vectors corresponding to the image and the text can be pruned (equivalent to a second fine selection), and a third pattern graph is obtained.

[0032] Finally, through the path reasoning network, multiple candidate reasoning paths are selected from the third pattern graph in a random walk manner, the embedding vector of each candidate reasoning path is determined according to the text features of the text, the image features of the image and the splicing vector of the embedding vectors of the nodes on each candidate reasoning path, so as to more accurately describe the candidate reasoning path; the matching probability of the embedding vector of each candidate reasoning path and the multi-modal context feature is determined by using a bilinear layer, so as to determine the reasoning result; thus, the reasoning result can be obtained based on the reasoning path, and the explainability and accuracy of the reasoning result are improved.

[0033] In order to better understand the application, the overall structure and function of the system of the application based on path reasoning open visual question answering (hereinafter referred to as the system of the application or the system) will be briefly described first. The system of the application is constructed by using a neural network, and the schematic diagram is shown in Figure 1 The system of the application comprises:

[0034] The pattern graph construction module 100 is used to extract nodes and relationships from the input image and the text asking a question for the image to construct a first pattern graph, retrieve neighbor nodes within n hops (n hops are preset hops, such as 2 hops) of nodes in the first pattern graph from a preset knowledge base to expand the first pattern graph, and obtain a second pattern graph (equivalent to finding out various nodes and relationships representing the semantics of the question to construct a first pattern graph; then, based on the knowledge base, n-hop neighbor nodes of the nodes in the first pattern graph in the knowledge base are retrieved and added to the first pattern graph, and these added nodes are potential answer nodes, so the answer space is not fixed for different inputs, and therefore it is called open visual question answering);

[0035] The pruning network 200 is used for calculating embedding vectors of respective nodes and pruning the second pattern graph to obtain a third pattern graph (since the second pattern graph is selected based on the hop number of neighbor nodes, the neighbor nodes covered may be more, and the relevance of some neighbor nodes to the problem may also be relatively weak, so the second pattern graph is pruned).

[0036] The path reasoning network 300 is used for extracting a plurality of candidate reasoning paths from the third pattern graph, and determining a reasoning result according to the candidate reasoning paths (since reasoning is performed in the manner of reasoning paths, the explainability of the reasoning process is increased).

[0037] In order to better understand the present application, the application scenario is described first.

[0038] The system of the present application needs to give an image I, its related text question Q, and a knowledge base (knowledge graph) The vertex set V is represented by The entity is represented by The edge set E is represented by The relationship is represented by. The system of the present application performs an open visual question answering task, and aims to retrieve the answer to Q and I and the reasoning path from the entity in the knowledge graph.

[0039] The structure of the system (including the pattern graph construction module, the pruning network, and the path reasoning network) and the training process of the system are described in detail below.

[0040] I. Structure of the system

[0041] (1) Pattern graph construction module

[0042] The pattern graph construction module is used for obtaining an input image and text of a question raised for the image, extracting nodes and relationships from the image and the question text to construct a first pattern graph, retrieving neighbor nodes within n hops of the nodes in the first pattern graph from a preset knowledge base to expand the first pattern graph, and obtaining a second pattern graph.

[0043] In other words, the pattern graph construction process includes extracting nodes from the question text and the image (visual information) to obtain a pattern graph of the question text and the image (i.e., the first pattern graph), and then expanding the pattern graph to an ideal size by expanding the adjacent nodes, thereby obtaining a comprehensive pattern graph of the question text and the image combined with the structure of the knowledge graph (i.e., the second pattern graph).

[0044] In short, the pattern graph construction process includes the following steps:

[0045] Text Graph Construction: The question text is parsed to extract keywords and their relationships. Keyword types include nouns, verbs, adjectives, adverbs, and phrases. Then, the keywords and their relationships are matched against entities and relationships in a pre-defined knowledge base. The matched entities and relationships are added to the schema graph, and the nodes corresponding to the matched entities form the question node set. For example, see again Figure 1 Input an image containing a sandwich and the question text: What condiments would be good on this sandwich? Parse the keywords "condiment," "good," and "sandwich" and their relationships. Figure 1 (Not shown). It should be understood that, Figure 1 The image shows the case where the input is in English. During implementation, the configuration can also be tailored to the needs of the implementer, allowing the input to be in Chinese or other languages.

[0046] Scene graph construction: The image is represented as a scene graph containing objects and relationships. Then, the scene graph is simplified, retaining only objects and relationships with a confidence level higher than a preset confidence threshold. These objects and relationships are matched against entities and relationships in a pre-defined knowledge base, and the matched entities and relationships are added to the pattern graph. The nodes corresponding to the matched entities form the image node set v. v .

[0047] External knowledge search: For matched text and image nodes, search the knowledge base for their one-hop and two-hop neighbor nodes, add the neighbor nodes to the pattern graph, and the neighbor nodes added to the pattern graph form a neighbor node set. In addition, these neighboring nodes can be filtered and sorted according to certain rules, and finally the filtered neighboring nodes can be added to the pattern graph.

[0048] By following these three steps, a schema diagram containing the problem, image, and external knowledge information can be successfully constructed. (i.e., the second pattern graph), where the node set This pattern diagram will help to better understand the relationship between the problem and the image, and to uncover potential answers from external knowledge.

[0049] There are different ways to construct a pattern diagram, which are illustrated below:

[0050] Based on the order in which nodes and relationships are extracted from the image and question text, the construction of the first pattern graph can be carried out using the following different implementation methods:

[0051] Illustratively, the nodes and relations in the question text can be extracted first. According to an embodiment of the present application, the schema graph construction module is configured to: extract a plurality of keywords and their relations from the question text, the keywords being any one of a noun, a verb, an adjective, an adverb and a phrase; perform a first matching of the plurality of keywords and their relations with entities and relations in a knowledge base (in this embodiment, the ConceptNet knowledge graph), create an initial schema graph containing text nodes and their relations according to the entities and relations matched in the first matching; identify objects and relations in the input image, perform a second matching of the identified objects and relations with the entities and relations in the knowledge base, add the entities and relations matched in the second matching to the initial schema graph to obtain the first schema graph, the first schema graph containing text nodes, image nodes and their relations; from the knowledge base, retrieve neighbor nodes within 2 hops of the nodes in the first schema graph; add the retrieved neighbor nodes and their relations to the first schema graph to obtain the second schema graph. The technical solution of this embodiment can achieve at least the following beneficial technical effects: this embodiment respectively extracts nodes and relations from the question text and the image using corresponding means to better describe the semantic relationship between the question in the question text and the image; in addition, by extending neighbor nodes from the nodes corresponding to the question text and the image through the knowledge base, the situation of selecting from a closed answer range can be avoided, thereby forming an open question and answer result.

[0052] When extracting a plurality of keywords and their relations from the question text, a natural language processing algorithm can be used. According to an embodiment of the present application, an entity relation extraction algorithm in natural language processing is used to extract a plurality of keywords and their relations from the question text.

[0053] When identifying the objects and relationships in the input image, a pre-trained scene generator can be used. According to an embodiment of the present application, the objects and relationships in the input image are identified using a pre-trained scene generator, and objects and relationships with a confidence higher than a pre-set confidence threshold are retained to obtain a scene graph (i.e. the entities and relationships matched for the second time in the previous embodiment). The scene graph is a structured representation of the scene, which can express the objects (such as objects, people, animals, etc.) in the scene and the relationships between the objects. Illustratively, the pre-trained scene generator can refer to Tang, Kaihua, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased SceneGraph Generation from Biased Training. In Proceedings of the 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3713-3722. It should be understood that there are other scene graph generation techniques in the prior art, or scene graph generation techniques defined by the implementer in the future, as long as the objects and their relationships in the scene can be extracted, which can be applied to the present application, and the present application does not limit this.

[0054] Illustratively, the nodes and relationships in the image can also be extracted first. According to another embodiment of the present application, the schema graph construction module is configured to: identify the objects and relationships in the input image, match the identified objects and relationships with the entities and relationships in the knowledge base for the first time, and create an initial schema graph containing image nodes and their relationships according to the entities and relationships matched for the first time; extract a plurality of keywords and their relationships from the question text, the keywords being any one of nouns, verbs, adjectives, adverbs and phrases; match the plurality of keywords and their relationships with the entities and relationships in the knowledge base (Freebase knowledge graph) for the second time, and add the entities and relationships matched for the second time to the initial schema graph to obtain the first schema graph, the first schema graph containing text nodes, image nodes and their relationships; retrieve neighbor nodes within 3 hops of the nodes in the first schema graph from the knowledge base; and add the retrieved neighbor nodes and their relationships to the first schema graph to obtain the second schema graph.

[0055] Preferably, the preset knowledge base is a node-link knowledge graph, such as ConceptNet knowledge graph or Freebase knowledge graph; more preferably, the preset knowledge base is ConceptNet knowledge graph. The ConceptNet knowledge graph is preferably used as external knowledge in the present application, which contains 516782 entities and 42 different relationships.

[0056] In the present application, the number of nodes within n hops is included; for example, the neighbor nodes within 2 hops include the neighbor nodes within 1 hop and the neighbor nodes within 2 hops. Optionally, n in n hops can also be set to 4 or 5, etc. in addition to 2 hops and 3 hops, which can be set according to the needs of the implementer and / or in combination with the computing power of the processing device.

[0057] Since there can be many nodes per hop in some cases, improvements can also be made to avoid excessive computation. According to an embodiment of the present application, when the number of neighbor nodes within n hops exceeds the preset upper limit of neighbor nodes, the neighbor nodes are screened according to the edge weights of the neighbor nodes in the knowledge base and / or the degree indicators of the neighbor nodes; the screened neighbor nodes are added to the first pattern graph to obtain the second pattern graph. The edge weights are the edge weights set in the knowledge base, and the degree indicator refers to the number of edges connecting the neighbor nodes to all key nodes. For example, in the case of n = 2, the first 500 1-hop neighbor nodes can be selected after sorting the 1-hop neighbor nodes according to the edge weights and the degree indicators of the neighbor nodes; then, the next-hop neighbor nodes (i.e. 2-hop neighbor nodes) are retrieved based on the 500 1-hop neighbor nodes, and the first 500 2-hop neighbor nodes are selected after sorting the 2-hop neighbor nodes according to the edge weights and the degree indicators of the neighbor nodes. It should be understood that, in addition to this, the total number of nodes in the second pattern graph can also be used to determine how many nodes there are in each hop. For example, the total number is set to 1000, the maximum number of 1-hop neighbor nodes is set to 500, and the number of 2-hop neighbor nodes is set to the difference between the total number and the number of 1-hop neighbor nodes. For example, there are 60 key nodes this time, and after 500 1-hop neighbor nodes are selected, only 440 2-hop neighbor nodes can be selected; for another example, there are 30 key nodes this time, and only 200 1-hop neighbor nodes are selected, so 770 2-hop neighbor nodes can be selected this time.

[0058] (2) Pruning network

[0059] The pruning network includes a node encoder for extracting an embedding vector of a node and a pruning layer for pruning the second pattern graph to obtain a third pattern graph.

[0060] The node encoder and the pruning layer will be described below.

[0061] 2.1) Node encoder

[0062] According to one embodiment of the present application, the node encoder is configured to extract the embedding vector of the node according to the image, the question text and the type of each node. For example, the node encoder is configured to: obtain a first embedding model to extract text features from the question text, image features from the image and multi-modal context features from the question text and the image; obtain a second embedding model to extract fusion text features from the fusion text, which is obtained by splicing the question text and the text corresponding to the neighbor nodes expanded from the node; for each node, determine the embedding vector of the node according to the multi-modal context features, the entity features corresponding to the node, the type of the node and the fusion text features extracted from the fusion text by the second embedding model, by using a two-layer perception machine (one layer or three layers can also be used). Optionally, the first embedding model is a ViLBERT model, a VLBERT model, a UNITER model or a ViLT model; preferably, the first embedding model is a ViLBERT model. Optionally, the second embedding model is a BERT model, a TinyBERT model, a GPT2 model or a GPT3 model; preferably, the second embedding model is a TinyBERT model. The technical scheme of this embodiment can at least achieve the following beneficial technical effects: In addition to the multi-modal context features, the pre-trained Trans-E features corresponding to the node, the type of the node and the fusion text features are also used to extract the embedding vector of the node, which can strengthen the information encoded in the text and the candidate answer words, and improve the accuracy of the subsequent reasoning.

[0063] The process of obtaining the embedding vector of the node will be described below by taking the first embedding model as the ViLBERT model and the second embedding model as the TinyBERT model as an example.

[0064] According to one example of the present application, the process of obtaining the node embedding representation includes:

[0065] K1, using the ViLBERT model as the first embedding model, extracting text features t, image features v and multi-modal context features z from the given question text Q and image I, which is represented as:

[0066] z,v,t=VilBERT(Q,I)

[0067] wherein VilBERT(Q,I) represents generating image features text features and multi-modal context features d represents the dimension of the features; the multi-modal context features z are fusion features obtained by cross-extracting image and text features. In the VilBERT model, the multi-modal context features z are the output of the common attention transformer (Co-TRM) layer thereof;

[0068] K2, obtaining entity features e corresponding to the node pair i , for example, is a pre-trained Trans-E feature, wherein the pre-trained Trans-E feature is an entity feature extracted by a Trans-E embedding model for an entity corresponding to a node pair in a preset knowledge base (i.e., a Trans-E feature, or a Trans-E embedding vector), some knowledge bases have an embedding vector table formed by pre-trained Trans-E features of entities in the knowledge base, and the pre-trained Trans-E feature corresponding to the node can be obtained by looking up the table according to the number of the node;

[0069] K3, obtaining the type p of the node i The type of the node is divided into a text node, an image node, and a neighbor node (in this case, the type of the node can be represented by a three-bit one-hot code, such as 001 representing a text node, 010 representing an image node, and 100 representing a neighbor node; here, only an example is given, and other representation methods can also be used, of course, the type of the node can also be further refined, such as according to the part of speech of the node in the text node, according to different object categories corresponding to the node in the image node, and according to the entity category of the entity corresponding to the node in the neighbor node, for further refined division);

[0070] K4, using a TinyBERT model as a second embedding model to obtain a TinyBERT fused text encoding, and obtaining a fused text feature u i ;

[0071] K5, using a two-layer perception machine to determine the embedding vector of the node according to the multi-modal context feature z, the entity feature e i corresponding to the node, the type p i of the node, and the fused text feature u i extracted by the second embedding model, and the formula is represented as follows:

[0072]

[0073] wherein z represents the multi-modal context feature, e i represents the entity feature encoded by the Trans-E model, p i represents the type of the node, u i represents the fused text feature encoded by the TinyBERT, || represents splicing, and f n (·) represents a node encoder (such as a two-layer perception machine or a three-layer perception machine).

[0074] It should be understood that a person skilled in the art can also make adjustments to obtain other embodiments that are feasible in principle, for example, the node encoder does not use u i , and the corresponding The present application does not limit this.

[0075] 2.2) Pruning layer

[0076] The pruning layer is configured to prune the second mode graph based on a weighted sum of a similarity between an embedding vector of each node and multi-modal context features corresponding to the image and the text and a distance score of each node to a key node, to obtain a third mode graph, the key node being a node extracted from the image and the text, i.e., an image node or a text node. For example, the pruning layer is configured to determine the weighted sum in the following manner:

[0077] s prune = (1 - θ p ) s cos + θ p s bfs

[0078] s cos s bfs s p s bfs s prune s cos s bfs s cos s bfs s cos s bfs s

[0079] (3) Path reasoning network

[0080] The path reasoning network comprises a path encoder configured to encode each of the plurality of candidate reasoning paths extracted from the third schema graph to obtain an embedding vector of the path, a bilinear layer configured to calculate a matching probability of each of the embedding vectors of the candidate reasoning paths and the multi-modal context features, and an output layer configured to determine the reasoning result according to the matching probability corresponding to each of the candidate reasoning paths.

[0081] The path encoder, the bilinear layer and the output layer will be introduced respectively as follows.

[0082] 3.1) Path encoder

[0083] The path encoder is configured to determine, based on the plurality of candidate reasoning paths extracted from the third schema graph, the embedding vector of each of the candidate reasoning paths according to the text features of the question text, the image features of the image and the splicing vector of the embedding vectors of the nodes on each of the candidate reasoning paths.

[0084] According to an embodiment of the present application, each of the plurality of candidate reasoning paths is obtained by sampling (path sampling) in the following manner: starting from the starting node in the third schema graph, performing a random walk of k steps (for example, 3 steps, 4 steps or 5 steps) based on the relationship construction, and obtaining a candidate reasoning path when the walk reaches the ending node, wherein the starting node at each time is a node randomly selected from all the key nodes, and the ending node at each time is the answer node.

[0085] Illustratively, the candidate reasoning path is defined with the question or image node as the starting point and the answer node as the ending point, that is, starting from any key node (the text node corresponding to the question text Q or the image node corresponding to the image V), a random walk of k steps is performed to sample N (for example, 100, 150 or 200) candidate reasoning paths, and the ending node is considered as the predicted answer node. Then, the path encoder is used to encode each of the candidate reasoning paths to obtain its embedding vector. For example, the embedding vector of the candidate reasoning path j is The calculation is as follows:

[0086]

[0087] wherein, represents the fusion representation of the k nodes in the candidate reasoning path, p n represents the splicing vector of the embedding vectors of the nodes on the candidate reasoning path, f t (·) represents a perception machine, t represents the text features, v represents the image features, and f p (·) represents the path encoder. represents the splicing vector of the embedding vectors of the nodes on the candidate reasoning path, f t (·) represents a perception machine, t represents the text features, v represents the image features, and f nThe processed vector. Namely: represents the use of a path encoder f p (·) t, v, The concatenation vector is processed to obtain the embedding vector of the candidate reasoning path For dimensional transformation in the processing process, p n ∈R k×d represents the concatenation vector represented by k nodes in the path, and d represents the dimension of the vector. represents the linear mapping of a two-layer perceptron (which can also be a three-layer MLP). represents the linear mapping of a two-layer perceptron (which can also be a three-layer MLP).

[0088] 3.2) Bilinear layer

[0089] The bilinear layer is used to determine the matching probability of the embedding vector of each candidate reasoning path and the multi-modal context feature.

[0090] According to one embodiment of the present application, the matching probability is determined in the following manner:

[0091]

[0092] wherein a represents the predicted answer node, Q represents the question text, I represents the image, represents the third mode graph, f bi (·, ·) represents the bilinear layer, and z represents the multi-modal context feature. represents the embedding vector of the candidate reasoning path j.

[0093] 3.3) Output layer

[0094] The output layer is used to determine the reasoning result according to the matching probability corresponding to each candidate reasoning path.

[0095] The reasoning result can be selected in a threshold manner. According to one embodiment of the present application, the output layer is used to determine the reasoning result according to the matching probability corresponding to each candidate reasoning path and a preset matching threshold. For example, the preset matching threshold is set to 0.8 or 0.9, etc. Taking the case where the preset matching threshold is set to 0.8 as an example, the output layer outputs the (one or more) candidate reasoning paths with a matching probability greater than 0.8 as the reasoning result.

[0096] The reasoning result can be selected in a ranked manner. According to an embodiment of the present application, an output layer is configured to determine the reasoning result according to the ranking result of the matching probabilities of the candidate reasoning paths. For example, a predetermined number of candidate reasoning paths are selected as the reasoning result output according to the ranking result in descending order of the matching probabilities. For example, the predetermined number is 1, 2, 3, 5, 7, etc. Taking the case where the predetermined number is 3 as an example, the output layer outputs three candidate reasoning paths selected in descending order of the matching probabilities as the reasoning result output.

[0097] Referring again to Figure 1 , assuming that the predetermined number is 1 after the input image and the question text are inputted, the output reasoning path is: sandwich→condiment→ketchup (sandwich→condiment→ketchup).

[0098] In addition, the output layer can also be improved. For example, the knowledge base can be a concept node linked knowledge graph, which includes entities, relationships between entities, and concepts of entities (explanatory content, such as a encyclopedia), such as the ConceptNet knowledge graph. In this case, the output layer can also output the concepts of the nodes contained in the reasoning path according to the user's request or directly output the reasoning path when outputting the reasoning result.

[0099] II. Training process of the system

[0100] Since the system of the present application is constructed by using a neural network, some modules need to be trained.

[0101] According to an embodiment of the present application, the system is trained in the following manner:

[0102] Obtain training data, which includes a plurality of samples and labels, wherein each sample includes a sample image and a sample question text, and the label indicates the true value of the answer corresponding to the sample;

[0103] Input the samples in the training data into the system to obtain the embedding vectors of each node of the second mode graph corresponding to the samples and the matching probabilities of the plurality of candidate reasoning paths corresponding to each sample, wherein:

[0104] The mode graph construction module is configured to: obtain the input sample image and the sample question text proposed for the sample image, extract nodes and relationships from the sample image and the sample question text to construct a first mode graph, retrieve neighbor nodes within a predetermined hop number of the nodes in the first mode graph from a predetermined knowledge base to expand the first mode graph, and obtain a second mode graph corresponding to the sample;

[0105] The pruning network is configured to: extract an embedding vector of each node by using a node encoder according to the sample image, the sample question text, and the type of each node; and prune a second mode graph corresponding to the sample to obtain a third mode graph corresponding to the sample by using a pruning layer according to a weighted sum of similarity between the embedding vector of each node and the multi-modal context feature and distance between each node and a key node, the key node being a node extracted from the sample image and the sample question text.

[0106] The path reasoning network is configured to: determine an embedding vector of each candidate reasoning path of the sample based on a plurality of candidate reasoning paths extracted from the third mode graph corresponding to the sample by using a path encoder according to a text feature of the sample question text, a sample image feature of the sample image, and a splicing vector of the embedding vector of each node on each candidate reasoning path; and determine a matching probability of the embedding vector of each candidate reasoning path of the sample and the multi-modal context feature by using a bilinear layer.

[0107] The total loss is determined according to a triplet loss of a sample in the training data and a binary cross entropy loss, and the trainable parameters of the node encoder, the path encoder, and the bilinear layer are updated according to the total loss, wherein the triplet loss of each sample is a mean value of sub-losses of all triplets that can be constructed according to the second mode graph of the sample, in each triplet, the anchor sample is the multi-modal context feature of the sample, the positive sample is an embedding vector of a node corresponding to an answer true value, and the negative sample is an embedding vector of a node corresponding to a non-answer true value; and the binary cross entropy loss of each sample is a mean value of binary cross entropy sub-losses calculated by the matching probability of all candidate reasoning paths corresponding to the sample and a label.

[0108] Illustratively, the sub-loss of each triplet is determined in the following manner:

[0109]

[0110] wherein triplet(·,·,·) represents a triplet loss function, z represents the multi-modal context feature, represents the positive sample, represents the negative sample.

[0111] Illustratively, the binary cross entropy sub-loss calculated by the matching probability of each candidate reasoning path and a label is determined in the following manner:

[0112]

[0113] wherein BCE(·,·) represents a binary cross entropy loss function, s represents the matching probability of a candidate reasoning path, and l gtdenotes the indicator value obtained according to the label, where l gt is 1, otherwise 0.

[0114] When training the system, the learning rate of some components can be set according to the needs of the implementer. For example, the learning rate of the first embedding model in the node encoder can be set to 10 -5 , the learning rate of the second embedding model, f n (·) in the node encoder can be set to 10 -4 , the learning rate of f bi (·,·) and d p (·) can be set to 10 -4 . And the initial parameters of the first embedding model can be set to the existing pre-trained ViLBERT. It should be understood that the above is only an example, and the learning rate of each module can be adjusted. For example, the learning rate of the first embedding model is changed to 10 -6 , and the others remain unchanged (or others are changed to 10 -5 ).

[0115] After training is completed, the system can be inputted with an image and a question text for the image, and the system performs path inference to output an inference result (for example: one or more inference paths are selected as the inference result based on a matching probability threshold or a matching probability ranking result).

[0116] In order to intuitively show the effect, the inventors have conducted experiments. The output effect of the system of the present application for some input images and question texts is shown in Figure 2 . Among them, Q (Question) refers to the input question text, GT (Ground Truth) refers to the label, and PI (Path Inference) refers to the output inference result.

[0117] From top to bottom, for the first image:

[0118] Q: What condiments would be good on this sandwich? (What condiments would be good on this sandwich?)

[0119] GT: ketchup, mustard (ketchup, mustard)

[0120] PI: sandwich→condiment→ketchup (sandwich→condiment→ketchup)

[0121] sandwich→food→ketchup (sandwich→food→ketchup);

[0122] For the second image:

[0123] Q: What sport is this?

[0124] GT: snowboard

[0125] PI: mountain→ snowy→ ski

[0126] For the 3rd image:

[0127] Q: What military branch to the men in the picture belong to?

[0128] GT: navy

[0129] PI: military→ air_force→ navy

[0130] From Figure 2 it can be seen that the output of the system of the present application clearly and intuitively presents the reasoning process, and increases the explainability of the reasoning process.

[0131] The inventors also compared the present application with the prior art: Multi-Modal Answer Validation for Knowledge-Based VQA (MAVEx), and the comparison results are as shown in Figure 3 .

[0132] From top to bottom, for the 1st image:

[0133] Q: Name the type of birds shown in this picture?

[0134] GT: seagull

[0135] MAVEx output: seagull

[0136] PI (output of the system of the present application):

[0137] beach→ sea→ seagull

[0138] lamp→ leg→ seagull

[0139] For the 2nd image:

[0140] Q: Why are these men dressed like this?

[0141] GT: military, soldier

[0142] MAVEx output: cold

[0143] PI (System Output of this Invention):

[0144] jacket→boy→military

[0145]

[0146]

[0147] from Figure 3 As can be seen, MAVEx only outputs a single final result, such as seagull, and the reasoning process is not interpretable; the output of the system of this invention is interpretable, and the overall performance of reasoning is also improved.

[0148] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order, as long as the required function can be achieved.

[0149] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0150] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0151] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments covered by the claims. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the spirit and scope of the described embodiments. It is intended that the scope of the application should only be limited by the appended claims.

Claims

1. A system for open visual question answering based on path reasoning, characterized in that, The system comprises: a schema graph construction module, configured to acquire an input image and question text proposed for the image, extract nodes and relations from the image and the question text to construct a first schema graph, retrieve neighbor nodes within a preset hop number of nodes in the first schema graph from a preset knowledge base to extend the first schema graph, and obtain a second schema graph fused with external knowledge information; a pruning network comprising a node encoder and a pruning layer, wherein the node encoder is configured to extract an embedding vector of each node according to the image, the question text, and a type of each node; the pruning layer is configured to prune the second schema graph according to a weighted sum of a similarity between an embedding vector of each node and a multi-modal context feature and a distance between each node and a key node, and obtain a third schema graph, wherein the multi-modal context feature is a fused feature obtained by cross-extracting features of the image and the text, and the key node is a node extracted from the image and the question text; a path reasoning network comprising a path encoder, a bilinear layer, and an output layer, wherein the path encoder is configured to determine an embedding vector of each candidate reasoning path according to a text feature of the question text, an image feature of the image, and a spliced vector of embedding vectors of nodes on the candidate reasoning path based on a plurality of candidate reasoning paths extracted from the third schema graph, the bilinear layer is configured to determine a matching probability of the embedding vector of each candidate reasoning path and the multi-modal context feature, the output layer is configured to determine a reasoning result according to the matching probability corresponding to each candidate reasoning path; wherein the system is trained in the following manner: obtain training data comprising a plurality of samples and labels, wherein each sample comprises a sample image and a sample question text, and the label indicates a true value of a corresponding answer of the sample; input the samples in the training data into the system to obtain a second schema graph and an embedding vector of each node thereof, and a matching probability of a plurality of candidate reasoning paths corresponding to each sample; determine a total loss according to a triple loss and a binary cross-entropy loss of each sample in the training data, and update trainable parameters of the node encoder, the path encoder, and the bilinear layer according to the total loss, wherein the triple loss of each sample is a mean value of sub-losses of all triplets that can be constructed according to the second schema graph of the sample, in each triplet, an anchor sample is the multi-modal context feature of the sample, a positive sample is an embedding vector of a node corresponding to an answer true value, and a negative sample is an embedding vector of a node corresponding to a non-answer true value; and the binary cross-entropy loss of each sample is a mean value of binary cross-entropy sub-losses calculated from matching probabilities of all candidate reasoning paths corresponding to the sample and the label.

2. The system of claim 1, wherein, The schema graph construction module is configured to: extract a plurality of keywords and their relations from the question text, wherein the keywords are any one of a noun, a verb, an adjective, an adverb, and a phrase; perform a first matching between the plurality of keywords and their relations and entities and relations in the knowledge base, and create an initial schema graph containing text nodes and their relations according to the matched entities and relations in the first matching; identifying objects and relations in the image, secondly matching the identified objects and relations with entities and relations in the knowledge base, adding the secondly matched entities and relations to the initial schema graph to obtain the first schema graph, the first schema graph containing text nodes, image nodes and relations therebetween; from the knowledge base, retrieving neighbor nodes within two hops of nodes in the first schema graph; adding the retrieved neighbor nodes and relations therebetween to the first schema graph to obtain the second schema graph.

3. The system of claim 2, wherein, The knowledge base is ConceptNet knowledge graph or Freebase knowledge graph.

4. The system of claim 2, wherein, The node encoder is configured to: obtain a first embedding model to extract text features from the question text, image features from the image, and multi-modal context features from the text and the image; obtain a second embedding model to extract fusion text features from fusion text, the fusion text being obtained by splicing the question text and text corresponding to neighbor nodes expanded from the node; for each node, determine an embedding vector of the node according to the multi-modal context features, entity features corresponding to the node, a type of the node, and the fusion text features extracted from the fusion text by the second embedding model, using at least one perception machine.

5. The system of claim 4, wherein, The first embedding model is a ViLBERT model, a VLBERT model, a UNITER model, or a ViLT model.

6. The system of claim 4, wherein, The second embedding model is a BERT model, a TinyBERT model, a GPT2 model, or a GPT3 model.

7. The system of claim 1, wherein, The pruning layer is configured to determine the weighted sum in the following manner: wherein, represents a similarity of an embedding vector of a node and a multi-modal context feature corresponding to the image and question text, represents a distance score of the node and a key node, represents a corresponding weight.

8. The system of claim 1, wherein, Each of the plurality of candidate reasoning paths is obtained in the following manner: starting from a starting node in the third schema graph, performing a random walk of k steps based on the paths constructed by the relations, and obtaining a candidate reasoning path when a ending node is reached, wherein the starting node is randomly selected from all key nodes each time, and the ending node is the answer node each time.

9. The system of claim 8, wherein, The k is greater than or equal to 3.

Images