A scene graph question answering method based on keyword guidance and GATv2-LSTM network

Abstract

The application discloses a scene graph question and answer method based on keyword guidance and a GATv2-LSTM network, and steps include: text and scene graph preprocessing, converting the text form question into a word vector sequence, deleting nodes irrelevant to the question in the graph and encoding to generate a new scene graph representation; the word vector sequence of the question is encoded by using a Transformer, the question keywords are extracted, the scene graph features are learned by using the keyword guidance GATv2, and the graph features are summarized and reasoned by using an LSTM network; finally, the question features and the scene graph features are fused, input into a classifier, and the final answer is obtained. The method uses the scene graph and the keyword guidance, and uses the LSTM for summarization and reasoning, can effectively reduce the gap between the modes, avoid the model from directly learning the mapping between the image and the text, and improve the question and answer accuracy.

Description

TECHNICAL FIELD

[0001] The present application is suitable for the field of visual question answering (VQA), and particularly relates to a Transformer Encoder encoder, a self-attention mechanism, and a GATv2-LSTM joint perception reasoning mechanism. The method has the advantages of extracting key words in the question, guiding the GATv2 network to perceive scene graph features using multiple key words, and summarizing these features using an LSTM network. Since it is based on a scene graph, the gap between different modalities of text and images is narrowed, and the perception reasoning ability is enhanced under the guidance of key words, thereby improving the accuracy of question answering. BACKGROUND

[0002] Visual question answering is an important branch of natural language processing and computer vision, which combines two different modalities. Its main task is to input pictures and corresponding questions, associate processed image information and text information, and infer to predict, generate and output natural language answers related to the question. VQA is widely used in medical assistance, disaster assessment and human-computer interaction, and is a research hotspot in recent years.

[0003] The Transformer model is essentially an encoder and decoder structure, mainly based on multi-head self-attention mechanism. It uses attention mechanism to capture the context dependency of each position in the input sequence, fully considers the importance of each word in the question sequence to the whole question, can better represent the question, and reduces semantic ambiguity.

[0004] GAT (Graph attention networks) is a kind of graph neural network. The original GAT network can only calculate static attention, so it cannot handle simple graph problems. GATv2 improves this problem by modifying the order of some operations of GAT. The GATv2 network processes data in graph structure. Unlike previous methods, it can update nodes through the attention values of the neighborhood nodes of each node. At the same time, the network uses a multi-head attention mechanism, which can learn multiple different information and is suitable for graph-based inductive learning problems, so it is suitable for scene graph learning.

[0005] LSTM (Long short-term memory) is a special RNN (Recurrent Neural Networks). It mainly includes three special gate structures: forget gate, input gate and output gate. Compared with simple RNN units, LSTM units additionally increase a storage unit Through the training process, the LSTM network can learn to remember useful information and forget irrelevant information, thereby better capturing the semantic dependencies between sequences and significantly reducing the problems of gradient disappearance and gradient explosion. SUMMARY

[0006] For general visual question answering models, they tend to directly extract information from images and texts and use complex attention mechanisms for interaction. However, since the original image is directly used, the model tends to object recognition and learning of the correlation between image and text, while ignoring the reasoning and understanding of the scene according to the text. The present application proposes a method of image question answering based on keyword guidance and GATv2-LSTM network, which is a method of image question answering combining perception and reasoning. First, the scene graph about the original image is obtained, and based on the scene graph, the GATv2 network is used to learn the most important information in the scene graph based on keyword guidance, then the image information learned by multiple keyword guidance is regarded as a sequence and input into the LSTM network for reasoning to obtain the final representation, and finally the final answer is obtained by fusing the question features. Since the scene graph is used, the gap between the two modalities of image and text can be effectively reduced. Based on keyword guidance, the model can perceive more effective features, and based on the LSTM network, the temporal relationship between multiple features can be learned, which improves the question answering accuracy and improves the perception and reasoning ability of the model.

[0007] The specific application comprises:

[0008] A scene graph question answering method based on keyword guidance and GATv2-LSTM network, characterized by comprising the following steps:

[0009] Step 1) Preprocessing the text question and scene graph, mainly including dividing the question into character sequences, removing punctuation and converting to word vector sequences, removing nodes unrelated to the question from the scene graph and encoding, etc., including the following steps:

[0010] Step 1.1) input the text form question data into the model using the input layer, the input text data is:

[0011]

[0012] wherein represents the th word or punctuation in the question;

[0013] Step 1.2) remove punctuation to obtain a word sequence, then convert the word to a 300-dimensional vector representation using GloVe :

[0014]

[0015] wherein represents the i-th word vector in the question text;

[0016] Step 1.3) POS tagging of the text question is performed using spaCy, and based on the label, the nouns appearing in the text are extracted, the cosine similarity between the nouns and the nodes in the scene graph is calculated, and the unimportant nodes in the scene graph are filtered out based on the cosine similarity, and the most similar part nodes and their adjacent nodes are retained;

[0017] Step 1.4) Based on the remaining nodes in the above scene graph, the scene graph is encoded, and the specific operation is as follows: new nodes are generated using the scene graph node attributes and the original scene graph node encoding, and new node edges are formed using the edge attribute encoding, so that the objects in the scene graph are represented by new nodes, and the relationships between objects are represented by edges, and the encoding method is as follows:

[0018]

[0019] wherein represents the original node information, represents the original edge information, represents the new node representation generated by encoding;

[0020] Step 2) Encoding the text question and extracting keywords, including the following steps:

[0021] Step 2.1) The text question is encoded using the Transformer Encoder module, and the encoded question takes into account the position information of each word in the question and the contribution of each word to the entire text question, which can better represent the question and reduce semantic ambiguity, and the encoding method is as follows:

[0022]

[0023] wherein represents the vector form of the question representation, represents the encoded question;

[0024] Step 2.2) The self-attention scores of each word in the question are obtained using the self-attention mechanism , and based on the self-attention scores, the most important keywords for answering the question are extracted;

[0025]

[0026] wherein represents the vector form of the question representation, represents the attention score of each word in the question;

[0027] ​Step 3) Use the question keywords obtained in step 2.2) as instructions to guide the GATv2 graph neural network to learn the new scene graph encoded in step 1.4), and use LSTM to summarize the features, mainly including the following steps:

[0028] Step 3.1) First, convert the question keywords into 300-dimensional instruction vectors using GloVe, then concatenate the instruction vectors with the edge representation and node representation of the generated scene graph, the implementation is as follows:

[0029]

[0030] wherein represents the instruction vector generated by the th keyword, and represent the new node and edge information, represents the concatenation operation.

[0031] Step 3.2) Input the concatenated new node representation and edge representation into multiple GATv2 networks to learn the scene graph features, and concatenate the learned graph features guided by multiple keywords into a sequence;

[0032] The calculation formula of GATv2 is as follows:

[0033]

[0034] wherein represents the attention coefficient, represents the attention score between node and node , and represents the new node representation calculated based on the attention score;

[0035]

[0036] wherein represents the final output scene graph information, represents the scene graph features learned by the GATv2 network via the th instruction vector;

[0037] Step 3.3) Since there is a time sequence relationship between different keywords, the guided graph features also contain a time sequence relationship, based on the graph feature sequence of step 3.2), use the LSTM network to learn this time sequence relationship. The input of the LSTM network is , and the output information is , wherein is the feature vector of the last time hidden layer, i.e. the LSTM output graph features ;

[0038] Step 4) The question representation obtained in step 2) is fused with the graph feature learned in step 3) to obtain the final answer, mainly including the following steps:

[0039] Step 4.1) First, the graph feature output in step 3.3) is pooled to reduce parameters and expand the perception field, and then the image feature and the question representation obtained in step 2.1) are multiplied element by element to fuse the two features;

[0040] Step 4.2) The fused feature is processed by using a two-layer MLP network, and finally input into a classifier to obtain the final answer, the specific operation is as follows:

[0041]

[0042] wherein and are fully connected layers, represents an MLP with two layers, represents element-wise point multiplication, represents the final fused feature representation, represents the final output answer.

[0043] The present application has the following advantages and beneficial effects:

[0044] The scene graph corresponding to the original image is used for image question answering, which avoids the model from tending to learn image target recognition and the association between image regions and text. Moreover, since the scene graph is a text type node and edge representation of the original image, that is, the scene graph uses nodes to represent objects in the original image and uses edges to represent corresponding relationships in the original image, this form of text representing image can effectively reduce the modal gap between image and text.

[0045] The present application extracts key words in the question by using a self-attention mechanism, and uses the key words to guide the graph neural network GATv2 to learn and generate multiple key scene graph features, and the scene graph features are spliced into a sequence. Based on the sequence, the LSTM network is used to learn the time sequence relationship in the sequence, which is equivalent to performing reasoning operation on the scene graph to further extract effective features. At the same time, the encoder of the Transformer is used to encode the question to reduce semantic ambiguity. Finally, the image feature and the text feature are fused, and the classifier is input to obtain the final answer. The present method enhances the image perception and image reasoning ability, and improves the question and answer accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 ​This is a flowchart illustrating how the present invention processes input scene diagrams and related textual problems.

[0047] Figure 2 This is a flowchart of the question-answering method based on keyword guidance and GATv2-LSTM scene graph of the present invention. Detailed Implementation

[0048] The advantages and objectives of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the description herein is only for explaining the present invention and is not intended to limit the present invention.

[0049] This invention proposes a scene graph question answering method based on keyword guidance and a GATv2-LSTM network. First, the original image and text questions are preprocessed, primarily involving removing punctuation, converting the question into a sequence, and then using GloVe to convert it into a 300-dimensional word vector sequence. Nouns are extracted using SpaCy. Based on the nouns in the question, the cosine similarity between scene graph nodes and nouns is calculated, retaining relevant nodes and their neighbors while discarding irrelevant nodes. Finally, node and edge information is used to encode and generate a new scene graph representation. Based on the preprocessed scene graph and question, a Transformer encoder is used to encode the question and extract keywords. These keywords are converted into guiding vectors to guide the GATv2 network in learning the scene graph. An LSTM network is then used to summarize the scene graph features learned from multiple keyword guidance methods to generate the final representation. Finally, image features and question features are fused and input into a classifier to obtain the answer. Figure 1 The following is a preprocessing flow for the scene diagram and problem in this invention. Figure 2 This is a flowchart of the keyword-guided GATv2-LSTM graph-aware reasoning model method of the present invention, which includes the following steps:

[0050] Step 1) Preprocess the text question and scene graph, mainly including segmenting the question into a character sequence, removing punctuation marks and converting it into a word vector sequence, removing irrelevant nodes from the scene graph and encoding them, including the following steps:

[0051] Step 1.1) Input the question data in text form into the model using the input layer. The input text data is as follows:

[0052]

[0053] in This indicates the first [item] in the problem. One word or punctuation mark;

[0054] Step 1.2) Remove punctuation marks to obtain the word sequence, and then use GloVe to convert the words into 300-dimensional vector representations. :

[0055]

[0056] wherein represents the i-th word vector in the question text;

[0057] Step 1.3) POS tagging is performed on the text question using spaCy, and based on the tag, the nouns appearing in the text are extracted, the cosine similarity between the nouns and the nodes in the scene graph is calculated, and the unimportant nodes in the scene graph are filtered out based on the cosine similarity, and the most similar part nodes and their adjacent nodes are retained;

[0058] Step 1.4) Based on the remaining nodes in the above scene graph, the scene graph is encoded, and the specific operation is as follows: new nodes are generated using the scene graph node attributes and the original scene graph node encoding, and new node edges are formed using the edge attribute encoding, so that the objects in the scene graph are represented by new nodes, and the relationships between the objects are represented by edges, and the encoding method is as follows:

[0059]

[0060] wherein represents the original node information, represents the original edge information, represents the new node representation generated by encoding;

[0061] Step 2) Encode the text question and extract keywords, including the following steps:

[0062] Step 2.1) encode the text question using the Transformer Encoder module, the encoded question fully considers the position information of each word in the question and the contribution of each word to the entire text question, which can better represent the question and reduce semantic ambiguity, and the encoding method is as follows:

[0063]

[0064] wherein represents the vector form of the question representation, represents the encoded question;

[0065] Step 2.2) obtain the self-attention score of each word in the question using the self-attention mechanism , based on the self-attention score, extract the most important keywords to answer the question;

[0066]

[0067] wherein represents the vector form of the question representation, ​represent the attention scores of each word in the question;

[0068] Step 3) Use the question keywords obtained in step 2.2) as instructions to guide the GATv2 graph neural network to learn the new scene graph encoded in step 1.4), and use LSTM to summarize the features, mainly including the following steps:

[0069] Step 3.1) First, convert the question keywords into 300-dimensional instruction vectors using GloVe, then concatenate the instruction vectors with the edge and node representations of the encoded scene graph, which can be implemented as follows:

[0070]

[0071] where represents the instruction vector generated by the th keyword, and represent the new node and edge information, represents the concatenation operation;

[0072] Step 3.2) Input the concatenated new node and edge representations into multiple GATv2 networks to learn the scene graph features, and concatenate the learned graph features guided by multiple keywords into a sequence;

[0073] The calculation formula of GATv2 is as follows:

[0074]

[0075] where represents the attention coefficient, represents the attention score between node and node , represents the new node representation calculated based on the attention score;

[0076]

[0077] where represents the final output scene graph information, represents the scene graph features learned by the GATv2 network via the th instruction vector;

[0078] Step 3.3) Since there is a temporal relationship between different keywords, the guided graph features also contain temporal relationships. Based on the graph feature sequence of step 3.2), use the LSTM network to learn this temporal relationship. The input of the LSTM network is , and the output information is , where to hide the feature vector of the last hidden layer, i.e. the graph feature output by LSTM ;

[0079] Step 4) fuse the question representation obtained in step 2) and the graph feature learned in step 3) to obtain the final answer, mainly including the following steps:

[0080] Step 4.1) first pool the graph feature output in step 3.3) to reduce parameters and expand the perception field, and then multiply the image feature and the question representation obtained in step 2.1) element by element to fuse the two features;

[0081] Step 4.2) use a two-layer MLP network to process the fused features, and finally input the classifier to obtain the final answer, the specific operation is as follows:

[0082]

[0083] wherein and are fully connected layers, represents an MLP with two layers, represents element-wise point multiplication, represents the final fused feature representation, represents the final output answer.​

Claims

1.A scene graph question answering method based on keyword guidance and GATv2-LSTM network, characterized in that, Comprising the following steps: Step 1) preprocessing the text question and scene graph, splitting the question into character sequences, removing punctuation and converting to word vector sequences, and encoding the scene graph by removing nodes unrelated to the question, comprising the following steps: Step 1.1) input the question data in text form into the GATv2-LSTM graph-aware inference model using the input layer; Step 1.2) remove punctuation to obtain a word sequence, and then convert the words to 300-dimensional vector representations using GloVe; Step 1.3) use spaCy to tag the text question with POS, extract the nouns appearing in the text based on the tags, calculate the cosine similarity between the nouns and the nodes in the scene graph, and filter out unimportant nodes in the scene graph based on the cosine similarity, retaining the most similar part nodes and their neighborhood nodes; Step 1.4) encode the scene graph based on the remaining nodes in the scene graph, specifically: generate new nodes using the scene graph node attributes and the original scene graph node encodings, form new node edges using edge attribute encodings, thereby representing objects in the scene graph using new nodes and representing relationships between objects using edges; Step 2) encode the text question and extract keywords, comprising the following steps: Step 2.1) encode the text question using the Transformer Encoder module; Step 2.2) use the self-attention mechanism to obtain the self-attention scores of each word in the vector-form question representation, and based on the self-attention scores, extract the most important keywords for answering the question; Step 3) use the question keywords obtained in step 2.2) as instructions to guide the GATv2 graph neural network to learn the new scene graph encoded in step 1.4), and use LSTM to summarize the features, comprising the following steps: Step 3.1) first convert the question keywords to 300-dimensional instruction vectors using GloVe, then concatenate the instruction vectors with the edge representations and node representations generated by encoding the scene graph; Step 3.2) input the concatenated new node representations and edge representations into multiple GATv2 networks to learn the scene graph features, and concatenate the learned graph features guided by multiple keywords into a graph feature sequence; Step 3.3) since different keywords contain temporal relationships, the graph features guided by them also contain temporal relationships, based on the graph feature sequence of step 3.2), use the LSTM network to learn this temporal relationship, the input of the LSTM network is the graph feature sequence, and the output information is the feature vector of the last time hidden layer, i.e. the graph feature output by the LSTM; Step 4) fuse the question representation obtained in step 2) with the graph features learned in step 3) to obtain the final answer, comprising the following steps: Step 4.1) first pool the graph features output by step 3.3), then multiply the image features and the question representation obtained in step 2.1) element by element to fuse the two features; Step 4.2) use a two-layer MLP network to process the fused features, and finally input them into a classifier to obtain the final answer.

Images