A knowledge-enhanced visual question answering method guided by multimodal information

Abstract

This invention discloses a knowledge-enhanced visual question answering method based on multimodal information guidance, characterized by the following steps: Step 1, constructing global visual modal features of an image and extracting visual modal information; Step 2, performing attention operations on question entities based on the visual modal information, extracting the question entities that require the introduction of external knowledge, and introducing knowledge into the extracted questions to obtain enhanced questions; Step 3, encoding the enhanced questions into inference instructions, and using graph convolutional networks to perform inference in the scene graph generated from the image, guided by the inference instructions, and finally obtaining the corresponding answer.

Description

Technical Field

[0001] This invention relates to the field of computer vision question answering technology, specifically to a knowledge-enhanced visual question answering method guided by multimodal information. Background Technology

[0002] Visual question answering is a multimodal information processing technology involving research in natural language processing and computer vision. It is a typical multimodal analysis and reasoning task with applications in various fields, including visual impairment assistance and assisted diagnosis. Most existing visual question answering models focus on analyzing the existing information contained in the dataset, including the question, image, and answer information. However, this information is insufficient to fully cover all the knowledge required to answer the question. When answering a question, people automatically recall related prior knowledge, and models, lacking this common-sense knowledge, cannot accurately answer such questions. Therefore, for these types of questions, it is necessary to introduce external knowledge to supplement the information, which is essential for obtaining a general-purpose visual question answering system.

[0003] Existing visual question answering work involving external knowledge can be mainly divided into the following types: The first type is to construct fact triples related to questions and answers as part of the dataset, and generate query statements by encoding question images. The fact triples are then retrieved from the fact dataset and used as candidate answers for screening. This research method has a relatively simple way of obtaining relevant external knowledge and cannot effectively cover external knowledge information under complex structures.

[0004] The second type is a knowledge-introduced visual question answering method based on large-scale external knowledge retrieval. This method extracts relevant concepts from images and questions, and then retrieves corresponding structured and unstructured knowledge from multiple external knowledge bases based on these concepts. This type of method incorporates some retrieved knowledge at different stages of visual question answering to answer questions that require external knowledge. This type of method mainly relies on services such as search engines to support external knowledge retrieval. The introduced knowledge is relatively coarse and has not undergone fine-grained selection, which will ultimately introduce noise.

[0005] The third approach involves structuring image concepts and constructing relevant attributes and relationships between visual concepts based on external knowledge. Then, reasoning is performed on the structured visual concept network using a question-guided method, focusing on the visual concept entities most relevant to the correct answer to obtain the final answer. This method introduces some external knowledge into the visual modality, and the reasoning process is explicit. However, it does not handle information interaction between modalities well, and the introduction of external knowledge does not take into account the role of the question in knowledge requirements, thus failing to fully understand the semantic information of the question.

[0006] In summary, existing methods have several problems: the knowledge content introduced is limited, generally introduced through triples or embedded features of corresponding factual entities, without considering complex external knowledge association methods; the introduction of non-fine-grained knowledge through large-scale retrieval introduces noise; the introduction of external knowledge in a single modality does not take into account the information complementarity and mutual guidance between modalities; and the model cannot comprehensively refer to multimodal information to introduce external knowledge. Summary of the Invention

[0007] This invention is made to solve the above-mentioned problems, and aims to provide a knowledge-enhanced visual question answering method based on multimodal information guidance.

[0008] This invention provides a knowledge-enhanced visual question answering method based on multimodal information guidance, characterized by the following steps: Step 1, constructing global visual modal features of an image and extracting visual modal information; Step 2, performing attention operations on the question based on the visual modal information, extracting question entities that require the introduction of external knowledge, and introducing knowledge into the extracted question entities to obtain an enhanced question; Step 3, encoding the enhanced question into inference instructions, and using a graph convolutional network method to perform inference in the scene graph generated from the image, guided by the inference instructions, and finally obtaining the corresponding answer.

[0009] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following features: Step 1 includes the following steps: Step 1-1, input the original color image I, and crop and scale the image I; Step 1-2, use the feature layer of the VGG16 pre-trained model to extract features from the image I, and obtain the extracted image features I. n Steps 1-3, using the extracted image features I n As visual modal information I v .

[0010] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following feature: wherein step 2 includes the following steps: step 2-1, extracting visual modality information I v Visual information I is obtained by performing feature space transformation. p Mapping to the space corresponding to problem q; Step 2-2, based on visual information I p Perform an attention operation on problem q to obtain entity E that needs supplementary external information; in steps 2-3, introduce external knowledge k into the entities in entity E. e The embedded knowledge feature space is transformed and then merged to obtain the augmented problem q. k .

[0011] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following feature: wherein step 3 includes the following steps: step 3-1, for visual modal information I v Step 3-2 involves generating a scene graph using DETR to obtain visual entities and related information between them; step 3-2 involves enhancing the question q. k Encode the necessary instructions for problem reasoning; in step 3-3, guided by the instructions, use a graph convolutional network to reason about the scene graph through a problem-guided approach to obtain the corresponding answer.

[0012] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following features: wherein, step 1-2 includes the following steps: Step 1-2-1, calculate the global information of image I extracted through the feature layer of the pre-trained model VGG16: on image I, for each region pixel L, use convolutional networks and pooling networks with different receptive fields to perform max pooling operations on image features I. n Extraction is performed to obtain 7*7*512 dimensional features; Step 1-2-2, average pooling layer and fully connected layer are used to process image features I. n Further extraction and compression are performed to obtain image features I, a one-dimensional vector of size 4096. v .

[0013] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following features: wherein step 2-2 includes the following steps: step 2-2-1, extracting image features I n Spatial transformation is performed using a multilayer perceptron network to map the information into visual information I. p This ensures consistency with the problem coding space; step 2-2-2, converting visual information I... p Connect to problem feature q I In the middle, t vi =[t i ,I p ], where t i For each entity feature in the problem, then for q based on visual information features v Perform attention operations to acquire entity E that requires supplemental external knowledge.

[0014] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following feature: wherein, the specific process of steps 2-3 is as follows: for entity e in entity E i Query the corresponding external knowledge and obtain the high-dimensional vector k of the corresponding external knowledge embedding. e And through a multilayer sensing network, k ePerform feature space transformation mapping to k ep Then k ep Merge into the corresponding e i Among the features.

[0015] The knowledge-enhanced visual question answering method based on multimodal information guidance provided by this invention may also have the following features: the specific process of step 3-1 is as follows: DETR is used to generate scene graphs. First, the image features are extracted through a graph convolutional network. Then, the position information of the image is encoded. Next, the two are serialized and added together, and then fed into a Transformer network for encoding and decoding. Finally, a feedforward network is used to predict the category and object bounding boxes.

[0016] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following feature: Specifically, step 3-2 involves: processing the question feature q, which incorporates external knowledge, in step 2-2-2. I =[q1,q2,…,q n The vector is encoded into M instruction vectors using a Transformer.

[0017] The knowledge-enhanced visual question-answering method based on multimodal information guidance provided by this invention may also have the following features: Step 3-3 includes the following steps: Step 3-3-1, for the scene graph generated in Step 3-1, inference is performed using a graph convolutional network based on an attention mechanism, guided by the question instructions generated in Step 3-2 during the inference process; Step 3-3-2, the object nodes, their attributes, and edges in the scene graph are encoded to obtain the corresponding feature vectors, and the feature vectors of the attributes are added to the node feature vectors, and the scene graph structure is constructed based on the node and edge feature vectors; Step 3-3-3, graph convolutional inference is performed on the attention graph convolutional neural network based on the question instructions, wherein the guidance method of the instructions is: first, by transforming the feature space matrix W i Perform spatial transformation to obtain I np Maintain the same dimensions as the feature vectors in the graph, and then I np The nodes and edges are concatenated to obtain nodes and edges incorporating instructions. Then, graph convolution is performed based on graph attention to obtain the output feature vectors of all nodes in the graph. Max pooling is performed on all nodes to obtain the final output features of the graph convolution operation. Finally, the features are mapped to the corresponding answer category through a fully connected layer to obtain the final predicted answer.

[0018] The role and effect of invention

[0019] According to the knowledge-enhanced visual question answering method based on multimodal information guidance of the present invention, the specific steps are as follows: Step 1, constructing global visual modal features of the image and extracting visual modal information; Step 2, performing attention operation on the question based on the visual modal information, extracting the question entity that needs to introduce external knowledge, and introducing knowledge into the extracted question entity to obtain the enhanced question; Step 3, encoding the enhanced question into reasoning instructions, and using graph convolutional networks to perform reasoning in the scene graph generated from the image guided by the reasoning instructions, and finally obtaining the corresponding answer.

[0020] Therefore, this invention constructs global features of an image and then introduces external knowledge guided by multimodal information. By performing attention operations on a question that incorporates visual modal features, it obtains entities requiring supplementary external knowledge such as facts or common sense. It then queries the corresponding entities to aggregate the external knowledge entity features from the subgraph information, integrates these external knowledge entity features into the question features, encodes the question features into corresponding instructions, and updates the visual entity nodes on the scene graph extracted from the image using an attention graph convolutional network based on the question instructions. Finally, it performs probability mapping of the final visual entity nodes into the answer space, selecting the answer with the highest probability as the question answer. This invention, through the introduction of external knowledge guided by multimodal information based on image and question information, improves the model's generalization ability and achieves higher answer accuracy by introducing external knowledge at a finer granular level. Attached Figure Description

[0021] Figure 1 This is a flowchart of a knowledge-enhanced visual question-answering method guided by multimodal information, as described in an embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating the overall process of the knowledge-enhanced visual question-answering method guided by multimodal information in an embodiment of the present invention.

[0023] Figure 3 This is a flowchart illustrating the introduction of external knowledge guided by multimodal information in an embodiment of the present invention.

[0024] Figure 4 This is a diagram of a knowledge-enhanced visual question-answering process in an embodiment of the present invention;

[0025] Figure 5 This is a flowchart illustrating the problem instruction encoding in an embodiment of the present invention;

[0026] Figure 6 This is a flowchart of scene graph extraction in an embodiment of the present invention. Detailed Implementation

[0027] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following embodiments, in conjunction with the accompanying drawings, provide a detailed description of a knowledge-enhanced visual question-answering method based on multimodal information guidance.

[0028] In this embodiment, a knowledge-enhanced visual question-answering method based on multimodal information guidance is provided. This method references information from images and questions to introduce external knowledge, providing a more accurate and fine-grained way to introduce external knowledge.

[0029] Figure 1 This is a flowchart of the knowledge-enhanced visual question-answering method based on multimodal information guidance in this embodiment.

[0030] Figure 1 This is the overall flowchart of the knowledge-enhanced visual question-answering method based on multimodal information guidance in this embodiment.

[0031] like Figures 1-2 As shown, the knowledge-enhanced visual question-answering method based on multimodal information guidance involved in this embodiment includes the following steps:

[0032] Step S1: Construct global visual modal features of the image and extract visual modal information.

[0033] Figure 3 The flowchart for introducing external knowledge guided by multimodal information in this embodiment is shown.

[0034] like Figure 3 As shown in part (a) of the document, step S1 in this embodiment includes the following steps:

[0035] Step S1-1: Input the original color image I, and crop and scale image I.

[0036] Step S1-2: Use the feature layer of the VGG16 pre-trained model to extract features from image I, and obtain the extracted image features I. n The specific implementation method is as follows:

[0037] Step S1-2-1: Calculate the global information of image I extracted through the feature layer of the pre-trained model VGG16: On image I, for each region pixel L, use convolutional networks and pooling networks with different receptive fields to perform max pooling operations on image features I. n Extract the features to obtain 7*7*512 dimensions.

[0038] Step S1-2-2: Use average pooling layers and fully connected layers to process image features I. n Further extraction and compression are performed to obtain image features I, a one-dimensional vector of size 4096. v .

[0039] Step S2, as follows Figure 3 As shown in parts (b) and (c), attention operations are performed on the question based on visual modality information to extract question entities that require the introduction of external knowledge, and knowledge is introduced into the extracted question entities to obtain an enhanced question. The specific implementation method is as follows:

[0040] Step S2-1, extract the visual modality information I v Visual information I is obtained by performing feature space transformation. p The specific steps for mapping this to the space corresponding to problem q are as follows:

[0041] Information compression is performed on the extracted image features: The image features, after processing through convolutional and pooling layers, are 14*14*512 in dimension. These features are then compressed into a one-dimensional vector using average pooling and a fully connected layer.

[0042] I 7*7*512 =AvgPooling 7*7 (I n )

[0043] I v =FC 4096 (I 7*7*512 )

[0044] Among them, I n The features extracted for the pre-trained model have a dimension of 14*14*512. To obtain the overall visual information of the image, I... v This is a one-dimensional feature vector mapped to 4096 through a fully connected layer.

[0045] Using MLP to map and transform visual features, I v Mapped to a feature space consistent with the problem encoding:

[0046] I p =MLP(I v )

[0047] Among them, I v For the features extracted by the pre-trained model, I p The features are transformed from visual features, with dimension d = 300.

[0048] Step S2-2, based on visual information I p An attention operation is performed on problem q to obtain entity E that requires supplemental external information. The specific implementation method is as follows:

[0049] In the comprehensive image global visual features I p Based on this, the embedded problem features q I =[t0,t1,…,t nThe specific steps for performing attention manipulation are as follows:

[0050] Fuse image features into problem features q I The operation is as follows;

[0051] t vi =concat(t i ,I p )

[0052] Where t i ∈q o The features are concatenated into 600 dimensions to obtain problem features q that incorporate global graph information. v , t vi For problem entities that incorporate visual modal features.

[0053] Problem features q that incorporate visual features v Perform attention operations as follows:

[0054]

[0055] The values ​​of Q and K are q. v After matrix W Q W K The transformed mapping vector.

[0056] Step S2-3: Introduce external knowledge k into the entities in entity E. e The embedded knowledge feature space is transformed and then merged to obtain the augmented problem q. k .

[0057] For q after attention calculation s Select the top m entities with the highest probability values ​​and introduce the corresponding knowledge, as follows:

[0058] Select the first m attention scores obtained after the attention operation, representing the probabilities of the corresponding problem entities:

[0059] t m =Top m (qs)

[0060] Where q s Let t be the attention probability of the entity in the problem. m These are the first m problem entities selected.

[0061] Perform external knowledge queries on the selected m problem entities to obtain the corresponding external knowledge feature vectors:

[0062] k em =ConceptNet GCN-embedding (t m)

[0063] External knowledge features are mapped and transformed in a feature space and then fused into the corresponding entities.

[0064]

[0065] Where k em The associated entity information embedded from an external knowledge base using graph convolution, t m For the selected corresponding entity, k represents vector addition. em The knowledge graph database aggregates information related to local subgraphs within the knowledge graph structure, providing richer knowledge information compared to introducing a single entity. Then, it maps the entity features of m relevant external knowledge bases to the question feature space and incorporates them additively into the question features to obtain a knowledge-enhanced fusion representation q. k .

[0066] Figure 4 This is a diagram of the knowledge-enhanced visual question-answering process in this embodiment.

[0067] Step S3, as follows Figure 4 As shown, the augmentation problem is encoded as inference instructions, and inference is performed in the scene graph generated by the image using graph convolutional networks, guided by the inference instructions, and finally the corresponding answer is obtained.

[0068] This includes the following steps:

[0069] Step S3-1, for visual modal information I v The DETR method generates a scene graph to obtain visual entities and related information between them. The specific process is as follows:

[0070] The ResNet network is used as the backbone network to extract the features of the original image. The features are then reduced in dimensionality and added to the spatial location feature encoding. The result is then sent to the DETR encoder for encoding to obtain the features after encoding N objects.

[0071] The encoded result is input into the DETR decoder, similar to the decoding structure of the Transformer. The difference is that each layer outputs the result and calculates the loss. Learnable position embeddings are added, and two feedforward networks are connected after the last decoder layer to predict the detection box and its category, respectively.

[0072] Figure 5 This is a flowchart of encoding problem instructions into reasoning instructions in this embodiment.

[0073] Step S3-2, as follows Figure 5 As shown, this will enhance the problem qk Encode the code to obtain the instructions needed for problem reasoning. The specific process is as follows:

[0074] Use a 3-layer Transformer encoder for problem q k Encoding was performed using a multi-head attention mechanism, with 8 heads and a hidden layer dimension of 4*512 in the feedforward layer, to obtain the encoded question q. encoder .

[0075] The decoder of the Transformer is used to decode the encoded result, keeping the number of layers consistent with the number of layers measured in the encoding module. A non-autoregressive decoding method is used to obtain M decoded instructions.

[0076] Step S3-3: Guided by instructions, a graph convolutional network is used to reason about the scene graph through a question-guided approach to obtain the corresponding answer. The specific implementation method is as follows:

[0077] Step S3-3-1: For the scene graph generated in step 3-1, use an attention-based graph convolutional network for inference, guided by the question instructions generated in step S3-2 during the inference process.

[0078] Figure 6 This is a flowchart of scene graph extraction in this embodiment.

[0079] Step S3-3-2, as follows Figure 6 As shown, the object nodes, their attributes, and edges in the scene graph are encoded to obtain the corresponding feature vectors. The feature vectors of the attributes are added to the node feature vectors, and the scene graph structure is constructed based on the node and edge feature vectors.

[0080] Step S3-3-3: Perform graph convolutional inference operations on the attention graph convolutional neural network based on question instructions, wherein the instruction guidance method is as follows:

[0081] First, transform the feature space matrix W i Perform spatial transformation to obtain I np Maintain the same dimensions as the feature vectors in the graph, and then I np The nodes and edges are concatenated to obtain nodes and edges incorporating instructions. Then, graph convolution is performed based on graph attention to obtain the output feature vectors of all nodes in the graph. Max pooling is performed on all nodes to obtain the final output features of the graph convolution operation. Finally, the features are mapped to the corresponding answer category through a fully connected layer to obtain the final predicted answer.

[0082] The role and effect of the embodiments

[0083] According to the knowledge-enhanced visual question answering method based on multimodal information guidance involved in this embodiment, the specific steps are as follows: Step 1, construct the global visual modal features of the image and extract the visual modal information; Step 2, perform attention operations on the question entities based on the visual modal information, extract the question entities that need to introduce external knowledge, and introduce knowledge into the extracted questions to obtain the enhanced questions; Step 3, encode the enhanced questions into inference instructions, and use the graph convolutional network method to perform inference in the scene graph generated by the image with the inference instructions as guidance, and finally obtain the corresponding answer.

[0084] Therefore, the above embodiments construct global features of the image, then utilize external knowledge introduction guided by multimodal information. By performing attention operations on the question that integrates visual modal features, entities requiring supplementary external knowledge such as facts or common sense are obtained. External knowledge entity features, aggregated from subgraph information, are then queried through corresponding entities. These external knowledge entity features are fused into the question features, encoded into corresponding instructions, and updated using an attention graph convolutional network on the scene graph extracted from the image based on these instructions. Finally, the visual entity nodes are probabilistically mapped to the answer space, and the answer with the highest probability is selected as the question answer. This invention, through the introduction of external knowledge guided by multimodal information based on image and question information, improves the model's generalization ability and achieves higher answer accuracy by introducing external knowledge at a finer granular level.

Claims

1. A knowledge enhanced visual question answering method based on multi-modal information guidance, characterized in that, Comprising the following steps: Step 1, constructing global visual modal features of the image and extracting visual modal information, comprising the following steps: Step 1-1, inputting an original color image I, and performing cropping and scaling on the image I, Step 1-2, performing feature extraction on the image I by using the feature layers of the VGG16 pre-training model to obtain the extracted image feature I n , comprising the following steps: step 1-2-1, calculating the global information of the image I extracted by the feature layers of the pre-training model VGG16: on the image I, for each region pixel L, using a convolution network with different receptive fields and a pooling network maximum pooling operation to extract the image feature I n , obtaining a feature with a dimension of 7*7*512, step 1-2-2, further extracting and compressing the image feature I n by using an average pooling layer and a full connection layer to obtain a one-dimensional vector of the image feature I v with a size of 4096, Step 1-3, to extract the image features I n as visual modality information I v ; Step 2, performing attention operation based on the problem of the visual modal information, extracting problem entities that need to introduce external knowledge, and performing knowledge introduction on the extracted problem entities to obtain an enhanced question, comprising the following steps: Step 2-1, extracting the visual modality information I v Step 2-2, performing feature space conversion to obtain visual information I p , mapping to the corresponding space of the question q, Step 2-2, based on the visual information I p Attention operation is performed on the question q to obtain the entity E that needs to be supplemented with external information, including the following steps: Step 2-2-1, the image features I n Spatial conversion mapping to visual information I is performed through a multi-layer perception network p , so that it is consistent with the question encoding space, Step 2-2-2, the visual information I p is connected to the question features q I , t vi = [t i , I p ], where t i is each entity feature in the question, and then attention operation is performed on q v based on the visual information features to obtain the entity E that needs to be supplemented with external knowledge, Steps 2-3: Introduce external knowledge k into the entities in entity E. e The embedded knowledge feature space is transformed and then merged to obtain the augmented problem q. k The specific process is as follows: for entity e in entity E i Query the corresponding external knowledge and obtain the high-dimensional vector k of the corresponding external knowledge embedding. e And through a multilayer sensing network, k e Perform feature space transformation mapping to k ep Then k ep Merge into the corresponding e i Among the features; Step 3, encoding the enhanced question as a reasoning instruction, and performing reasoning in a scene graph generated by the image in the method of graph convolution network guided by the reasoning instruction, and finally obtaining a corresponding answer, comprising the following steps: Step 3-1, for the visual modal information I v The scene graph is generated by the DETR to obtain visual entities and inter-entity correlation information in the graph, and the specific process is as follows: the DETR is used for scene graph generation, first, the picture is extracted by the graph convolution network, second, the position information of the image is encoded, third, the two are serialized and added, and then fed into the Transformer network for encoding and decoding, and finally, the feedforward network is used for class and object bounding box prediction. Step 3-2, the enhanced question q k Encoding, obtaining instructions required for question reasoning, the specific process is: the question characteristics q I = [q1, q2, …, q n ] vector is encoded by Transform, and is encoded into M instruction vectors, Step 3-3, under the guidance of the instruction, using the graph convolution network, reasoning the scene graph in a question-guided manner to obtain the corresponding answer, comprising the following steps: step 3-3-1, for the scene graph generated in step 3-1, using the graph convolution network based on attention mechanism to reason, and the reasoning process is guided according to the question instruction generated in step 3-2, step 3-3-2, encoding the object nodes and their attributes and edges in the scene graph to obtain the corresponding feature vectors, the feature vectors of the attributes are added to the node feature vectors, the scene graph structure is constructed based on the node and edge feature vectors, step 3-3-3, performing graph convolution reasoning operation on the attention graph convolution neural network based on the question instruction guidance, wherein the instruction guidance mode is: first converting the feature space matrix W i to obtain I np , maintaining the same dimension as the feature vector in the graph, then splicing I np with the nodes and edges to obtain the nodes and edges integrated with the instruction, then performing graph convolution operation based on graph attention, finally obtaining the output feature vector of all nodes in the graph, performing maximum pooling operation on all nodes to obtain the final output feature of the graph convolution operation, then mapping to the answer category through the full connection layer to obtain the final predicted answer.

Images