Visual question answering method based on original feature injection network with selective graph structure attention

Abstract

The application discloses a visual question answering method based on selective graph structure attention, aiming at solving the problems that the existing method "ignores the relationship between targets in visual features" and "image edge information is forgotten". The selective graph structure attention module is constructed, the relationship between targets is modeled by generating a scene graph (containing two types of nodes, "objects associated with a bounding box" and "object attributes"), then the edges and nodes of the scene graph are selectively selected according to the problem characteristics, and the bilateral information features are generated in combination with the common attention; meanwhile, the original feature injection module is constructed to restore the edge information of the image that is not paid attention to, and the two modules form the progressive cooperation of "relationship modeling-feature enhancement". The experiment on the VQA2.0 dataset proves that the accuracy of the model is significantly improved in the complex scene (especially the relationship type question) question answering task, effectively verifying the effectiveness of the method.

Description

TECHNICAL FIELD

[0001] The present application belongs to the method of visual question answering, and relates to the technical field of computer vision and natural language processing. BACKGROUND

[0002] Visual question answering is formulated as a classification problem in most researches, where the image and question are the inputs and the answer is the output class (due to the limited number of possible answers). Since the visual question answering task was proposed after the wide popularity of deep learning methods, almost all visual question answering solutions are using CNNs to model the image input and RNNs to model the question. Attention mechanisms have been extensively studied in visual question answering. This includes visual attention, which focuses on the where-to-look problem, and question attention, which focuses on the where-to-read problem. Since the image and question are two different modalities, it is straightforward to jointly embed the two modalities together to uniformly describe the image / question pair.

[0003] A common practice of existing models is to extract visual and linguistic features separately and then merge them into a common space. Then the answer to the input question is predicted from these fused bi-modal features. In early researches, researchers adopted some relatively simple fusion methods, such as feature concatenation, multiplication and dot product of feature vectors. Fukui et al. proved that more complex fusion methods can indeed improve prediction accuracy, so they introduced the bilinear (merge) method. In their work, the outer product of the two vectors of visual and linguistic features is used for fusion. Since the outer output has very high dimensional features, they adopted the concept of Gao et al. Gao et al. compressed the fused features and named it as MCB merge method. However, to ensure stable performance, the compressed features of MCB still tend to be high-dimensional, Kim et al. used the Hadamard product of two feature vectors to propose a low-rank bilinear pool called multi-modal low-rank bilinear pool (MLB). Yu et al. proposed a multi-modal factorized bilinear pool (MFB), which uses matrix factorization techniques to compute the fused features, thereby reducing the number of parameters and improving the convergence speed.

[0004] Attention mechanisms are effective in many visual and linguistic processing tasks, such as caption generation, action recognition, natural language processing, etc. Without exception, it is introduced into visual question answering and proved to be very helpful for answer prediction. So far, many methods have been developed, among which the common one is to guide attention in image regions. According to the type of image features, the methods are divided into two categories. On the one hand, the visual features of region proposals are used to focus on objects, and these attention objects are generated by bounding box or region proposal networks. Another kind of visual features is extracted from convolutional features.

[0005] There are several methods to create and use attention maps. Yang et al. developed an attention network that generates multiple attention maps on an image in a sequential manner, aiming to perform multiple inference steps. Kim et al. extended this idea by incorporating it into the remaining architecture to produce better attention information. Chen et al. proposed a structured attention model that can encode relationships across regions, aiming to correctly answer questions involving relationships between complex regions. Duy-Kien Nguyen et al. proposed the famous co-attention mechanism to better fuse the representations of images and question words. However, existing attention models mainly consider the possible interaction between image regions and question words, ignoring the self-correlation information of image regions themselves. In addition, some network structures are multi-layer iterative, which usually makes some valuable but unattended original image edge information completely forgotten after multiple bilateral co-attention operations. SUMMARY

[0006] The purpose of the present application is to solve the problem that the current visual question answering system generally uses convolutional neural networks or attention mechanisms to process visual features, which ignores the relationships between objects and objects in visual features. And, too much attention is paid to the matching of question words and image regions, ignoring valuable but unattended edge information of images.

[0007] The technical scheme adopted by the present application to solve the above technical problems is:

[0008] S1. Construct a selective graph structure attention module, construct a graph structure for visual features, and then select edges and nodes according to the problem features, find the relationship between the objects and objects in the visual features according to the problem, and then combine with the co-attention to generate bilateral information, making the answer clearer.

[0009] S2. Construct an original information injection module to restore valuable but unattended edge information of images.

[0010] S3. Combine the network in S2 and the network in S3 to construct a selective graph structure attention-based original feature injection network architecture.

[0011] S4. Training of the selective graph structure attention-based original feature injection network and visual question answering.

[0012] The selective graph structure attention module of the present application generates a graph structure according to image features, then finds the relationship between the objects and objects in the visual features according to the problem, and then combines with the co-attention to generate bilateral features, making the answer clearer. We describe the detailed operation as follows:

[0013] The graph structure is a directed multigraph, where each node corresponds to a scene entity, which can be an object associated with a bounding box or an attribute of an object. Each scene entity has a type corresponding to the predicted object or attribute label. The typed edges specify how the scene entities are related to each other. More formally, let E denote the set of scene entities and consider the set of binary relations R. Then, the graph structure SG is the set of ordered triples (s, p, o) subject, predicate, and object. -1

[0014] The state of the mapping space S is given by E x Q, where E is the nodes of the scene graph SG and Q represents the set of all questions. The mapping space S at the t-th node represents the current entity e t and question Q, denoted as St. Thus, the state S t ∈ S at time t ∈ N is represented by St= (e t , Q). The set of available actions from the mapping space Stis denoted by A St . It contains all outgoing edges of the node e t and their corresponding object nodes, represented by the following equation:

[0015] A St = {(r, e) ∈ R x E : S t = (et, Q) ∧ (et, r, e) ∈ SG} (1)

[0016] The action performed by the mapping space at the t-th node is denoted by a t ∈ A St . The node embeddings are passed through a multi-layer graph attention network (GAT) when creating the graph structure. GAT extends from graph convolutional networks through a self-attention mechanism, mimicking the convolution operator on regular grids by forming entity embeddings by embedding the node features of its neighbors. Thus, the generated embeddings are context-aware, which makes nodes with the same type but different graph neighborhoods distinguishable.

[0017] Features are extracted from the input image and question separately, and the graph structure is generated from the image features. We denote the history of the mapping space by the tuple of sequences H t = (H t-1 , A t-1 ), where t ≥ 1, H0= hub, t=0. The features h are encoded through a multi-layer LSTM, as shown in the following equation:

[0018] h t = LSTM(a t-1 ​​(2)

[0019] Where a t-1 =[r t-1 e t ]∈R 2d Corresponding to the use of rt-1 embedding of the previous action, e t E and F represent embedding the edge and target node into R, respectively. d middle.

[0020] h and the problem feature Q are simultaneously fed into the ReLU activation function, and after the action set operation At(), the visual feature I is generated by the softmax function, as shown in the formula below:

[0021] I = softmax(A) t (W2ReLU(W1[h t Q]))) (3

[0022] in The rows contain the potential representations of all allowed actions. Furthermore, Q∈R d Encode question Q. Draw action A based on feature I. t = (r, e)∈A St Formulas (2) and (3) yield a random policy π θ , where θ represents the set of trainable parameters.

[0023] The generated visual feature I and the question feature T are then fed into a common attention mechanism to generate an attention matrix containing bilateral information. For a given image feature I and question feature T, an attention matrix A is obtained through a cross product operation. Then, A is processed through a two-layer softmax algorithm to generate an attention matrix A'' for the question. T And the attention matrix A about the image I Finally, the image features I containing bilateral information are multiplied by both the image features and the question features respectively. c and problem characteristics T c This process can be represented by the following five formulas:

[0024] A = T × W R ×I T (4)

[0026] A I =softmax(A) (5)

[0027] A T =softmax(A T (6)

[0028] T c =T×A T(7)

[0029] I c = I x A I (8)

[0030] wherein the represents a weight matrix, and the dimensions of the attention matrix are both dxd.

[0031] The purpose of the original information injection module is to restore the image edge information and constrain the semantic deviation of the original image information, so as to help the model correctly understand the image. Specifically, after S1, the visual feature I c and the language feature T c are combined with the original features Ir and Tr respectively, and then linear operations are performed to generate image feature information I c and T c with the same dimensions as I cr and T cr :

[0032] I cr = linear I (concat(I c , I r )) (9)

[0033] T cr = linear T (concat(T c , T r )) (10)

[0034] wherein linear is a Linear function with 1024 hidden units with ReLU nonlinearity and dropout.

[0035] The visual question answering method based on the original feature injection network of the selection graph structure attention comprises a selection graph structure attention module and an original feature injection module.

[0036] The training method of the original feature injection network based on the selection graph structure attention is as follows:

[0037] In our implementation, all experiments are implemented using the PyTorch framework of Python 3.6 version, and the experiments are performed on a computer with Nvidia Tesla P100 GPU.

[0038] All images are resized to 448*448 before feeding into the CNN. All questions are tokenized using Python Natural Language Toolkit (nltk). We use the vocabulary provided by the CommonCrawl-840B Glove model for English word vectors. We limit the maximum length of questions to 14 words and then dynamically pad each question to allow for questions of different lengths. We use an eight-layer network throughout the experiments, i.e., with eight layers of compound attention and visual feature augmentation mechanisms (L=8). This number of layers is chosen based on our preliminary experiments. During training, we use the ADAM optimizer to train our model on VQA 2.0 with 400 batches. The weight decay is 0.01. We use exponential decay to gradually reduce the learning rate:

[0039]

[0040] where the initial learning rate a is set to a=0.001, and the decay period of VQA2.0 is set to 7 periods in turn; we set the parameters to b1=0.9, b2=0.99. In order to prevent overfitting, dropout loss is used, and the loss rate of each fully connected layer is p=0.3, and the loss rate of LSTM is p=0.1.

[0041] Compared with the prior art, the beneficial effects of the present application are:

[0042] 1. The selection graph structure attention module is selected to construct the graph structure of the visual feature, and the edges and nodes of the graph structure are selected according to the question characteristics, the target and the relationship between the targets of the visual feature are found according to the question, and then the double-sided information is generated by combining the common attention, so that the answer is clearer.

[0043] 2. The original feature injection module is designed to retain the original edge information of the image, which is used to improve the overall semantics of the image and improve the accuracy of the answer prediction. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a structural schematic diagram of the visual question answering method based on the original feature injection network of the selection graph structure attention.

[0045] Figure 2 It is a model schematic diagram of the selection graph structure attention network.

[0046] Figure 3 It is a model schematic diagram of the original feature injection network.

[0047] Figure 4A comparison chart of the results of the visual question answering model based on the original feature injection network with graph structure attention and other visual question answering models on the dataset.

[0048] Figure 5 A visual result chart for visual question answering. DETAILED DESCRIPTION

[0049] The accompanying drawings are only intended to be illustrative and cannot be construed as limiting the patent.

[0050] The present application is further described below in conjunction with the accompanying drawings and examples.

[0051] Figure 1 A structural diagram of the original feature injection network based on the selected graph structure attention. As shown in Figure 1 , the entire visual question answering framework mainly consists of two parts: the selected graph structure attention (Selected Graph Attention Network) and the original feature injection module (Original Feature Injection Module).

[0052] Figure 2 A diagram of the selected graph structure module. As shown in Figure 2 , the graph structure is a directed multigraph, where each node corresponds to a scene entity, which can be an object associated with a bounding box or an attribute of the object. Each scene entity has a type corresponding to the predicted object or attribute label. The typed edges specify how the scene entities are related to each other. More formally, let E denote the set of scene entities and consider the set of binary relations R. Then, the graph structure is a set of ordered triples (s, p, o) subject, predicate and object. In the rest of this work, the integrity of inverse relations is imposed, as for each (s, p, o) ∈ SG it is implied that (o, p, s) ∈ SG. -1

[0053] The state of the mapping space S is given by E × Q, where E is the node of the scene graph SG and Q represents the set of all questions. The mapping space S represents the current entity e t and question Q at the tth node, denoted as St. Therefore, St = (e t , Q) represents the state of S t ∈ S at t ∈ N. The set of available actions from the mapping space St is denoted by A St . It contains all outgoing edges of the node e t and their corresponding object nodes, represented by the following formula:

[0054] A St = {(r, e) ∈ R × E: S​t = (et, Q) A (et, r, e) e SG} (1)

[0055] At the same time, A t e A St represents the action performed by the mapping space at the t-th node. When creating the graph structure, the node embedding is passed through a multi-layer graph attention network (GAT). GAT extends from the graph convolutional network through a self-attention mechanism, which imitates the convolution operator on the regular grid by embedding the node features of its neighbors to form the entity embedding. Therefore, the generated embedding is context-aware, which makes nodes with the same type but different graph neighborhoods distinguishable.

[0056] Features are extracted from the input image and question respectively, and a graph structure is generated from the image features. We use the number tuple H t = (H t-1 , A t-1 ) to represent the history record of the mapping space, where t > 1, H0= hub, t = 0. The feature h is encoded by a multi-layer LSTM, as shown in the following formula:

[0057] h t = LSTM(a t-1 ) (2)

[0058] where a t-1 = [r t-1 , e t ] e R 2d corresponds to embedding the previous action using rt-1embedding, e t , etc. represent embedding the edge and target node into R d .

[0059] h and the question feature Q are put into the ReLU activation function at the same time, and then through the action set operation At(), the visual feature I is generated by the soffmax function, as shown in the following formula:

[0060] I = soffmax(A t (W2ReLU(W1[h t Q]))) (3)

[0061] where the rows of A contain the potential representation of all allowable actions. In addition, Q e R d encodes the question Q. According to the feature I, the action A t = (r, e) e A St is drawn. Formulas (2) and (3) derive a random policy π θ , where θ represents a set of trainable parameters.

[0062] The generated visual feature I and the question feature T are then input into the common attention to generate attention containing bilateral information. For a given image feature I and question feature T, the attention matrix A is obtained by cross multiplication operation, and A is then passed through double-layer softmax to generate the attention matrix A about the question T and the attention matrix A about the image I Finally, the image feature I containing bilateral information is obtained by multiplying with the image feature and the question feature respectively c and the question feature T c This process can be represented by the following five formulas:

[0063] A = T x W R x I T (4)

[0065] A I = softmax(A) (5)

[0066] A T = softmax(A T ) (6)

[0067] T c = T x A T (7)

[0068] I c = I x A I (8)

[0069] wherein the of formula (1) represents a weight matrix, and the dimensions of the attention matrix are both d x d.

[0070] Figure 3 is a schematic diagram of the original feature injection module. As shown in Figure 3 The purpose of the original information injection module is to restore the edge information of the image and constrain the semantic deviation of the original information of the image, thereby helping the model to correctly understand the image. Specifically, after S1, the visual feature I c and the language feature T c are combined with the original features Ir and Tr respectively, and then linear operation is performed to generate image feature information I c and T c with the same dimensions as I cr and T cr :

[0071] I cr = linear I (concat(I c , I r )) (9)

[0072] T cr = linear T (concat(T c , T r )) (10)

[0073] where linear is a Linear function with 1024 hidden units with ReLU nonlinearity and dropout.

[0074] Figure 4 A comparison chart of the results of the visual question answering of the original feature injection network based on the selection graph structure attention and the visual question answering model of other networks on the VQA2.0 dataset. As shown in Figure 4 , the visual question answering result of the original feature injection network based on the selection graph structure attention is more accurate than other models.

[0075] Figure 5 A visual result chart of the visual question answering model. As shown in Figure 4 , given an image and a question, the original feature injection network model based on the selection graph structure attention can generate the corresponding answer.

[0076] The present application proposes a kind of original feature injection network with selection graph structure attention, for visual question answering is proposed in this paper.Introduce selection graph structure to find out the relationship between visual feature target and target according to question, make answer more clear.In addition, original feature injection module is also added to retain the original edge information of image.With the help of the proposed selection graph structure attention, graph structure is constructed to visual feature, then according to question feature, graph structure selects edge and node, finds out the relationship between visual feature target and target according to question, then combines with common attention, effectively fuses visual feature and question feature.A large number of experiments on VQA2.0 database show that the model has achieved good results in visual question answering.In future work, we will continue to explore how to better learn the semantics of image and question text, and effectively integrate it for answer reasoning.

[0077] Finally, the details of the above examples of the present application are only examples made for explaining the present application, and any modification, improvement and replacement of the above examples, etc. for those skilled in the art, should be included in the protection scope of the claims of the present application.

Claims

1. A visual question answering method based on a selection graph structure attention-based original feature injection network, characterized in that, The method The procedure includes the following steps: S1. Construct a graph structure attention module to build a graph structure for visual features, then select edges and nodes for the graph structure based on the problem features, find the relationship between visual feature targets and targets based on the problem, and then combine it with common attention to generate features containing bilateral information. S2. Construct a raw information injection module to recover valuable but overlooked edge information in the image; S3. Combine the modules in S1 and S2 to construct a raw feature injection network architecture based on selection graph structure attention; S4. Training and visual question answering of networks based on original feature injection into selection graph structure attention; The specific process of S1 is as follows: The graph structure attention module is selected to generate a graph structure based on image features. Then, based on the question, the relationship between visual feature targets and targets is identified. Finally, it is combined with common attention to generate bilateral features, making the answer clearer. The specific process is as follows: The graph structure is a directed multigraph, where each node corresponds to a scene entity. A scene entity refers to a recognizable visual element in the image, specifically including objects or object attributes associated with bounding boxes. Each scene entity has a type corresponding to a predicted object or attribute label, and the entered edges specify how scene entities are related to each other. Let E represent the set of scene entities and R represent the set of binary relations, then the graph structure... It is a set of ordered triples (s, p, o); integrity is imposed on the inverse relation, and for each (s, p, o) ∈ SG, it is implied that (s, p) ∈ SG. -1 ,o)∈SG; The state of the mapping space S is given by E×Q, where E is a node of the graph structure SG, and Q represents the set of all problems; the t-th node of the mapping space S represents the entity e in which it currently resides. t And question Q, denoted as S t That is, when t∈N, S t =(e t Q)∈S; from the mapping space S t The available action set is provided by A St This indicates that it contains node e. t The expression for all outgoing edges and their corresponding object nodes is: A St ={(r,e)∈R×E:S t =(e t ,Q)∧(e t ,r,e)∈SG} (1) Let A t ∈A St This represents the action performed at the t-th node in the mapping space; when creating the graph structure, node embeddings are passed through a multi-layer graph attention network (GAT); GAT is extended from graph convolutional networks through a self-attention mechanism, mimicking convolution operators on a regular grid, and forming entity embeddings by embedding features of neighboring nodes; Features are extracted from the input image and the question, and a graph structure is generated from the image features. The graph structure is then processed using a tuple H. t =(H t-1 A t-1 ) represents the history of the mapping space, where t≥1, H0=hub, t=0; the feature h is encoded using a multi-layer LSTM and expressed as follows: h t =LSTM(a t-1 ) (2) Where a t-1 =[r t-1 ,e t ]∈R 2d ;e t This indicates embedding the edge and target node into R. d middle; h and the problem feature Q are input together into the ReLU activation function, followed by the action set taking operation A. t After (), visual feature I is generated using the softmax function, with the expression: I=softmax(A t (W2ReLU(W1[h t Q]))) (3) in The rows contain the potential representations of all allowed actions, Q∈R d Encode problem Q; draw action A based on feature I. t = (r,e)∈A St Formulas (2) and (3) yield a random policy π θ θ represents the set of trainable parameters; The generated visual feature I and the question feature T are fed into a common attention function to generate an attention function containing bilateral information. For a given image feature I and question feature T, an attention matrix A is obtained through a cross product operation. A is then processed through a two-layer softmax to generate an attention matrix A' for the question. T And the attention matrix A about the image I Then multiply by the image features and the question features respectively to obtain the image feature I containing bilateral information. c and problem characteristics T c The process expression is: A=T×W R ×I T (4) A I =softmax(A) (5) A T =softmax(A T ) (6) T c =T×A T (7) I c =I×A I (8) Formula (4) Let represent the weight matrix and the attention matrix, both of which have a dimension of d×d.

2. The visual question answering method based on the original feature injection network with selection graph structure attention as described in claim 1, Its features are, The specific process of S3 is as follows: The visual question answering method based on selection graph structure attention includes a selection graph structure attention module and an original feature injection module.

Images