A Visual Question Answering Model Based on Dual-Enhanced Transformers

Abstract

The present invention discloses a visual question answering model based on a double-enhanced transformer. The visual question answering model includes a global enhancement module and a position enhancement module. The global enhancement module is used to fuse and represent visual features and text features in the visual question answering model. The position enhancement module is used to represent relative position information of visual features in the visual question answering model. The global enhancement model first calculates global features through local features, and then concatenates the global features and local features as overall features and sends them into the transformer. The position enhancement model uses the geometric structure of the bounding box, which includes two coordinates representing the upper left corner and the lower right corner of the visual feature respectively, to obtain the center coordinates and length and width of the bounding box, and then calculates the relative geometric relationship of the visual features and sends them into the neural network. This model overcomes the information loss caused by the flat operation of visual features and the problem that the current visual question answering models only focus on local feature interaction.

Description

Technical Field:

[0001] The present invention relates to computer vision and natural language processing. Specifically, the present invention proposes a visual question answering model based on a dual-enhanced transformer. Background Art:

[0002] The goal of a visual question answering [1] model is to answer a natural language question about an image given an image. Visual question answering is a challenging task that requires fine-grained semantic understanding of images and questions and predicts accurate answers through multimodal reasoning. With years of development, the great success of visual question answering has been supported by a series of methods.

[0003] Most early deep neural network-based visual question answering [2] models gave answers based on visual feature vectors and embedded questions. However, the feature representation of an image may lose key information for correctly answering questions about local regions of the image. To overcome this problem, many attention mechanism-based models [3] have been proposed, such as learning visual attention on image regions guided by the input question. In addition to visual attention, learning text attention in questions is also very important, so co-attention models [4] have been proposed to achieve image-guided question attention and question-guided visual attention. A bottom-up attention model [5] is used to obtain attention weights for different regions in an image trained by Faster R-CNN on Visual Genome data. Some recent work, the MCAN model [6], introduced the transformer model [7] to model intra-modal and inter-modal interactions between any image region and any question word.

[0004] However, due to the limitations of the transformer structure, the inevitable information loss caused by visual feature flattening operations has not been solved. In addition, existing models only model local interactions and do not consider global features, and this method fails to expand the ability of complex multimodal reasoning in visual question answering.

[0005] [References]

[0006] [1]Antol, Stanislaw, et al. "Vqa: Visual question answering." Proceedings of the IEEE international conference on computer vision. 2015.

[0007] [2] Zhou, Bolei, et al. "Simple baseline for visual question answering." arXiv preprint arXiv:1512.02167 (2015).

[0008] [3] Yang, Zichao, et al. "Stacked attention networks for image question answering." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[0009] [4] Lu, Jiasen, et al. "Hierarchical question-image co-attention for visual question answering." Advances in neural information processing systems 29 (2016).

[0010] [5] Anderson, Peter, et al. "Bottom-up and top-down attention for image captioning and visual question answering." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[0011] [6] Yu, Zhou, et al. "Deep modular co-attention networks for visual question answering." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019.

[0012] [7]Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30(2017). Summary of the Invention:

[0013] Aiming at the problems existing in the prior art, the present invention provides a visual question answering model based on a dual-enhanced transformer, which overcomes the information loss caused by the flat operation of visual features and the problem that the current visual question answering model only focuses on local feature interaction.

[0014] The present invention solves its practical problems by adopting the following technical solutions:

[0015] 1. A visual question answering model based on a dual-enhanced transformer, the visual question answering model includes a global enhancement module and a position enhancement module;

[0016] The global enhancement module is used to fuse and represent visual features and text features in the visual question answering model;

[0017] The position enhancement module is used to represent the relative position information of visual features in the visual question answering model;

[0018] Wherein: the global enhancement module fusing and representing visual features and text features in the visual question answering model includes the following steps:

[0019] The global enhancement module converts the regions of the extracted image into visual local features V through Faster R-CNN l ;

[0020] The global enhancement module converts word vectors into text local features Q through a two-layer bidirectional LSTM l ;

[0021] The global enhancement module extracts the visual local features V according to the following formula l to obtain visual global features V g ;

[0022]

[0023] The global enhancement module extracts the text local features Q according to the following formula l to obtain text global features Q g ;

[0024]

[0025] The global enhancement module extracts the visual local features V according to the following formulal Fused with the visual global feature V g to obtain the visual overall feature V;

[0026] V = [V l :V g

[0027] The global enhancement module fuses the text local feature Q l with the text global feature Q g to obtain the text overall feature Q;

[0028] Q = [Q l :Q g .

[0029] Furthermore, the position enhancement module is used to represent the relative position information of the visual features in the visual question answering model, including the following steps:

[0030] The position enhancement module performs position recognition of the bounding box on the visual local feature, that is:

[0031]

[0032] The position enhancement module calculates the center coordinates of the bounding box for the visual local feature according to the following formula;

[0033]

[0034] The position enhancement module calculates the relative position relationship of the local area for the visual local feature according to the following formula;

[0035]

[0036] The position enhancement module performs position recognition of the bounding box on the visual global feature, that is:

[0037] {(0,0),(1,1)}

[0038] The position enhancement module calculates the center coordinates of the global bounding box for the visual global feature;

[0039]

[0040] The position enhancement module calculates the relative position relationship of the whole local area for the visual all features;

[0041] The position enhancement module embeds the relative position relationship of the local area and the relative position relationship of the whole local area into a high-dimensional vector through a fully connected layer; when embedding the relevant position relationships into scalars, relative position information is obtained; that is:

[0042] ​Re ij = FFN(Re ij )

[0043] Re ij = ReLU(Re ij w Re )

[0044] Wherein, the dimension of Re is the number of image regions multiplied by the number of image regions.

[0045] Beneficial effects:

[0046] Compared with the prior art, the beneficial effects brought by the technical solution of the present invention are as follows: In the visual question answering transformer model, the present invention introduces two enhancement modules, namely the position enhancement module and the global enhancement module. The former solves the problem of information loss caused by the flat operation of visual features and enriches the expression ability of visual features. The latter solves the problem that the current visual question answering model only considers local feature interaction and does not consider global features. In addition, the present invention verifies the model of the present invention in the visual question answering task, proving the effectiveness and mutual complementarity of the two modules proposed by the present invention. In the VQA2.0 dataset task of visual question answering, compared with the naive visual question answering transformer model, the overall accuracy of the present invention is increased by 0.64%, the accuracy in YES / NO questions is increased by 0.26%, the accuracy in Num questions is increased by 2.46%, and the accuracy in Other questions is increased by 0.70%. Through the above experiments, the present invention finds that the model of the present invention effectively improves the performance of the naive visual question answering transformer model. Description of the drawings:

[0047] Figure 1 is a flowchart of a visual question answering model based on a dual-enhanced transformer;

[0048] Figure 2 is a structural diagram of the position enhancement module of a visual question answering model based on a dual-enhanced transformer;

[0049] Figure 3 is a structural diagram of the global enhancement module of a visual question answering model based on a dual-enhanced transformer. Detailed implementation manners:

[0050] The present invention proposes a visual question answering model based on a dual-enhanced transformer. The following further details the implementation process of this invention patent in conjunction with Figure 1 to do a further detailed description.

[0051] A visual question answering model based on a dual-enhanced transformer, the model includes a position enhancement module and a global enhancement module.

[0052] The position enhancement module is used for visual features. First, using the geometric structure of the bounding box, which contains two coordinates representing the upper left corner and the lower right corner of the visual feature respectively, the center coordinates, length, and width of the bounding box can be obtained. Then, the relative geometric relationship of the visual features is calculated, and finally, it is fed into the neural network to obtain the relative position information. In the Self-Attention of the transformer, the present invention adds the result obtained by multiplying it with the query matrix and the transposed key matrix to supplement the information loss caused by the flat operation of the visual features. Through this operation, the semantic information of the visual features becomes richer and contains rich position information.

[0053] The global enhancement model is used for visual features and text features. The visual features are extracted by Faster R-CNN. The question words are first converted into glove word vectors and then fed into a two-layer bidirectional LSTM to obtain the text feature representation. Then, average pooling, that is, taking the average values of the visual features and the text features, is used to obtain the global representation of the visual features and the global representation of the text features. Then, the local features and the global features are concatenated and fed into the transformer as the input vector.

[0054] The present invention discloses a visual question answering model based on a dual-enhanced transformer. The model includes a position enhancement module and a global enhancement module. The position enhancement model introduces the relative position information of the visual features, thereby improving the visual feature representation. The position enhancement model first uses the geometric structure of the bounding box, which contains two coordinates representing the upper left corner and the lower right corner of the visual feature respectively, to obtain the center coordinates, length, and width of the bounding box. Then, it calculates the relative geometric relationship of the visual features and finally feeds it into the neural network to obtain the relative position information. The global enhancement module jointly models the interactions inside and outside the module between the global features and the local features to obtain a comprehensive representation of the visual features and the text features. The global enhancement model first calculates the global features from the local features, and then concatenates the global features and the local features as the overall features and feeds them into the transformer. For the position enhancement module, the global enhancement module can also be used as a supplement that only considers the local position relationship and does not consider the interaction problem between the global position and the local position. For the global enhancement module, the position enhancement module is used as a supplement that only considers the interaction at the feature level and does not consider the interaction of the position information. The two modules influence and promote each other. The present invention overcomes the information loss caused by the flat operation of the visual features and the problem that the current visual question answering models only focus on the interaction of local features.

[0055] Figure 1 Shows the flowchart of the method; Figure 2 Shows the structural diagram of the position enhancement module designed by the present invention; Figure 3 Shows the structural diagram of the global enhancement module designed by the present invention. The specific steps of the present invention are as follows:

[0056] (1) For visual features, the regions of the image are extracted by means of Faster R-CNN to obtain the visual feature V l , and for text features, the question is first tokenized and converted into glove word vectors, and then fed into a two-layer bidirectional LSTM to obtain the text feature Q l .

[0057] (2) By means of average pooling, the visual global feature V g and the text global feature Q g are extracted, and the formula is as follows:

[0058]

[0059]

[0060] (3) The global features and local features are concatenated as the input vector of the transformer visual question answering model, and the formula is as follows:

[0061] V = [V l :V g

[0062] Q = [Q l :Q g

[0063] where [:] is the concatenation operation.

[0064] (4) By means of Faster R-CNN mentioned above to extract the image region, in addition to the features, the bounding box can also be obtained, which is the upper left corner and the lower right corner of the image region Then the center coordinates of the image region can be calculated, and the formula is as follows:

[0065]

[0066] By interacting the length, width and center coordinates of two image regions, the relative geometric relationship can be obtained, and the formula is as follows:

[0067]

[0068] In addition to extracting the local image region by means of Faster R-CNN, the present invention also considers the whole image, that is, the global bounding box. Specifically, the upper left corner and the lower right corner of the global bounding box are {(0, 0), (1, 1)} (after normalization), and then through the formula mentioned above, the relative geometric relationship between the global region and the local region can be obtained.

[0069] ​​(5) Embed the obtained relative geometric relationships into high-dimensional vectors through a fully connected layer with activation function. Finally, embed the relevant geometric relationships into a scalar to obtain relative position information.

[0070] Re ij = FFN(Re ij )

[0071] Re ij = ReLU(Re ij w Re )

[0072] where the dimension of Re is the number of image regions multiplied by the number of image regions.

[0073] (6) Through the above parts, global features and relative position information are obtained, corresponding to the global enhancement module and the position enhancement module respectively. The global enhancement module has been introduced, and the global features are concatenated with the local features and fed into the transformer. After obtaining the relative position information, the position enhancement module still needs to perform relevant operations in Self-Attention.

[0074] (7) The overall formula of Self-Attention in the transformer is as follows: 0

[0075]

[0076] (8) After introducing the relative position information Re, the overall formula is as follows:

[0077]

[0078] (9) Through the above operations, the position enhancement module and the global enhancement module are introduced into the naive visual question answering transformer model, solving the problem of information loss caused by flat operations in visual features and the problem that current visual question answering models only consider local feature interactions.

[0079] (10) The present invention is verified for effectiveness on the VQA-v2 dataset, which is the most commonly used dataset for visual question answering tasks. It contains question-answer pairs annotated by humans for images from the COCO dataset. Each picture has 3 questions, and each question has 10 answers. The whole dataset can be divided into 3 parts, namely the training set (80k pictures and 444k question-answer pairs), the validation set (40k pictures and 214k question-answer pairs), and the test set (80k pictures and 448k question-answer pairs). The answer with the highest frequency will be regarded as the correct answer. All question types can be divided into 3 types: Yes / No, Number, and Other.

[0080] In the experiments of this section, the evaluation metric of the VQA-v2 dataset is accuracy. The parameter settings for the experiments are as follows: the overall parameters adopt those of the naive Visual Question Answering Transformer model. The specific experimental results are as follows.

[0081] Table 1 Experimental results of each model on the visual question answering task

[0082] Model Y / N Num Other All Bottom-Up 81.82 44.21 56.05 65.32 MCAN 86.82 53.26 60.72 70.63 Base 88.87 58.18 63.78 73.48 Base+PA 88.90 59.93 63.92 73.87 Base+GA 88.91 59.22 64.12 73.70 Our Method 88.93 60.64 64.48 74.12

[0083] As shown in the above table, the models of the present invention have achieved the best results on the VQA-v2 dataset. Compared with the baseline model, the accuracy of the models of the present invention has increased by at least 0.06%, 2.46%, 0.70% and 0.64% in Y / N, Num, Other, and All. From the above experimental results, it can be seen that compared with the naive Visual Question Answering Transformer model, the models of the present invention still exhibit obvious performance advantages, demonstrating the effectiveness of the two modules.

[0084] Generally speaking, the present invention provides a visual question answering model based on a dual-enhanced transformer. The model proposes two modules, namely a position enhancement module and a global enhancement model. The position enhancement model solves the problem of information loss caused by flat visual features, and at the same time introduces position information to enrich the visual feature representation. The global enhancement module solves the problem that the model only considers local feature interactions and does not consider global features, enriching the representation of visual and text features. For the position enhancement module, the global enhancement module can also serve as a supplement for only considering local position relationships without considering global and local position interactions. For the global enhancement module, the position enhancement module is a supplement for only considering feature-level interactions without considering position information interactions. The two modules influence and promote each other.

[0085] Based on this, the present invention proposes a visual question answering model based on a dual-enhanced transformer. The model includes a position enhancement module and a global enhancement module. The position enhancement model introduces the relative position information of visual features, thereby improving the representation of visual features. The position enhancement model first utilizes the geometric structure of the bounding box, which contains two coordinates representing the upper left corner and the lower right corner of the visual feature respectively, to obtain the center coordinates and the length and width of the bounding box, then calculates the relative geometric relationship of the visual features, and finally sends it into the neural network to obtain the relative position information. The global enhancement module jointly models the interactions inside and outside the module by combining the global features and the local features to obtain a comprehensive representation of the visual features and the text features. The global enhancement model first calculates the global features through the local features, then concatenates the global features and the local features as the overall features, and sends them into the transformer. For the position enhancement module, the global enhancement module can also be used as a supplement that only considers the local position relationship without considering the interaction between the global position and the local position. For the global enhancement module, the position enhancement module is used as a supplement that only considers the interaction at the feature level without considering the interaction of position information. The two modules influence and promote each other.

[0086] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the spirit of the present invention and the scope protected by the claims, those of ordinary skill in the art can also make many specific transformations in various forms under the inspiration of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A visual question answering model based on a double-enhanced transformer, characterized in that: The visual question answering model includes a global enhancement module and a position enhancement module; The global enhancement module is used to fuse and represent visual features and text features in the visual question answering model; The position enhancement module is used to represent the relative position information of visual features in the visual question answering model; Among them: The global enhancement module fusing and representing visual features and text features in the visual question answering model includes the following steps: The global enhancement module converts the regions extracted from the image into visual local features V through Faster R-CNN l ; The global enhancement module converts the word vectors into text local features Q through a two-layer bidirectional LSTM l ; The global enhancement module extracts the visual global feature V according to the following formula for the visual local feature V l to obtain the visual global feature V g ; The global enhancement module extracts the local text feature Q according to the following formula l to obtain the global text feature Q g ; The global enhancement module fuses the visual local feature V l and the visual global feature V g to obtain the visual overall feature V; V = [V l :V g ​ The global enhancement module fuses the local text feature Q l with the global text feature Q g to obtain the overall text feature Q; Q = [Q l :Q g 。 2. The visual question answering model based on a dual-boost transformer according to claim 1, characterized in that: The position enhancement module representing the relative position information of visual features in the visual question answering model includes the following steps: The position enhancement module performs position recognition of the bounding box on the visual local features, that is: The position enhancement module calculates the center coordinates of the bounding box for the visual local features according to the following formula; The position enhancement module calculates the relative position relationship of the local area for the visual local features according to the following formula; The position enhancement module performs position recognition of the bounding box on the visual global features, that is: {(0,0),(1,1)} The position enhancement module calculates the center coordinates of the global bounding box for the visual global features; The position enhancement module calculates the relative position relationship of the whole local area for all visual features; The position enhancement module embeds the relative position relationship of the local area and the relative position relationship of the whole local area into a high-dimensional vector through a fully connected layer; when embedding the relevant position relationships into a scalar, relative position information is obtained; that is: Re ij = FFN(Re ij ) Re ij = ReLU(Re ij w Re ) Among them: The dimension of Re is the number of image regions multiplied by the number of image regions.

Images