A visual question answering method based on a double-feature collaborative relationship attention network

Abstract

The present application relates to the technical field of visual question answering, and particularly relates to a visual question answering method based on a double-feature collaborative relationship attention network, comprising: constructing a geometric correlation self-attention module; constructing a relationship-enhanced cross-attention module; constructing a double-feature collaborative relationship attention network based on the geometric correlation self-attention module and the relationship-enhanced cross-attention module; after preprocessing the question and the image, inputting the double-feature collaborative relationship attention network model for prediction to obtain a predicted answer; and on the basis of an encoder-decoder framework, the model uses a brand-new multi-level collaborative decoding layer stack, realizes the correlation of deep semantics between regional features and grid features, encodes the geometric information of the image through the geometric correlation self-attention unit, realizes the integration of features and spatial attributes, and thus solves the problem that the traditional VQA model is insufficient in exploring the relationship between objects and detailed features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual question answering, and particularly relates to a visual question answering method based on a double-feature collaborative relationship attention network. BACKGROUND

[0002] The region features and grid features extracted from the object detection network contain rich image semantic information and are widely used in visual question answering. The feature information focused by the two kinds of features is different: the region features focus on the content of the object level, and the grid features are better at expressing the context information and fine-grained feature attributes of the image.

[0003] However, the existing VQA models only use one of the two, lack understanding of the external relationship between objects and analysis of feature details, thereby causing the VQA model to be insufficient in exploring the relationship between objects and detailed features. SUMMARY

[0004] The present application aims to provide a visual question answering method based on a double-feature collaborative relationship attention network, and aims to solve the problem of insufficient exploration of the relationship between objects and detailed features by the traditional VQA model.

[0005] To achieve the above-mentioned purpose, the present application provides a visual question answering method based on a double-feature collaborative relationship attention network, comprising the following steps:

[0006] a geometric correlation self-attention module is constructed;

[0007] a relationship-enhanced cross-attention module is constructed;

[0008] a double-feature collaborative relationship attention network is constructed based on the geometric correlation self-attention module and the relationship-enhanced cross-attention module;

[0009] After the question and the image are preprocessed, they are input into the double-feature collaborative relationship attention network model for prediction to obtain a predicted answer.

[0010] The specific way of the geometric correlation self-attention module is:

[0011] a position-related attention module is constructed;

[0012] a position encoder is incorporated based on the position-related attention module to obtain the geometric correlation self-attention module.

[0013] The visual question answering method based on the double-feature collaborative relationship attention network comprises a geometric correlation self-attention module, a relationship-enhanced cross-attention module, a visual decoding module, a multi-level collaborative decoding module, a multi-modal fusion module and a question prediction module.

[0014] The multi-level collaborative decoding module comprises a GRSA-GA-FFN sub-module, a RACA-GA-FFN sub-module and a SA-CA-FFN sub-module.

[0015] The specific manner in which the question and the image are input into the double-feature collaborative relationship attention network model for prediction to obtain a predicted answer is as follows:

[0016] The question and the image are input into the geometry-related self-attention module for extraction to obtain feature information and a position vector.

[0017] The feature information and the position vector are input into the double-feature collaborative relationship attention network to obtain processed image information and image features.

[0018] The processed image information and the image features are fused by the multi-modal fusion module, and then decoded by the multi-level collaborative decoding module to obtain a fusion result.

[0019] The fusion result is predicted by the question prediction module to obtain a predicted answer.

[0020] The visual question answering method based on the double-feature collaborative relationship attention network comprises the following steps: a geometry-related self-attention module is constructed; a relationship-enhanced cross-attention module is constructed; a double-feature collaborative relationship attention network is constructed based on the geometry-related self-attention module and the relationship-enhanced cross-attention module; a question and an image are preprocessed and then input into the double-feature collaborative relationship attention network model for prediction to obtain a predicted answer; and on the basis of an Encoder-Decoder framework, the model uses a brand-new Multi-Level Collaborative Decoder (MLCD) Layer stack to realize the association of deep semantics between regional features and grid features. Specifically, the geometry information of the image is encoded by a Geometry-Related Self-Attention (GRSA) unit to realize the integration of features and spatial attributes, and a Relativity-Augmented Cross-Attention (RACA) unit is proposed to realize the complementary reinforcement of effective information between the two features, thereby solving the problem that the traditional VQA model does not sufficiently explore the relationship between objects and detailed features. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0022] Figure 1 are two features and corresponding QA instance graphs.

[0023] Figure 2 is a DFCRAN model architecture

[0024] Figure 3 is a Faster R-CNN framework graph.

[0025] Figure 4 is a structural diagram of a language encoder.

[0026] Figure 5 is a structural diagram of a geometry-related self-attention module.

[0027] Figure 6 is a structural diagram of a relationship-enhanced cross-attention module.

[0028] Figure 7 is a structural diagram of a multi-level collaborative decoding module.

[0029] Figure 8 is a multi-modal fusion structural diagram.

[0030] Figure 9 is an attention weight visualization diagram.

[0031] Figure 10 is a prediction result visualization diagram.

[0032] Figure 11 is a flowchart of a visual question answering method based on a double-feature collaborative relationship attention network provided by the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0034] Please refer to Figures 1 to 11 The present application provides a visual question answering method based on a double-feature collaborative relationship attention network, comprising the following steps:

[0035] S1 constructing a geometry-related self-attention module;

[0036] S11 constructing a position-related attention module;

[0037] S12 fusing a position encoder based on the position-related attention module to obtain the geometry-related self-attention module.

[0038] Specifically, the position encoder: the position encoding plays an important role in the original Transformer, replacing the traditional RNN (recurrent neural network) to obtain the word position. Similarly, image features also have position information, specifically the spatial coordinates of each feature. We use this position information to calculate the position encoding of image features.

[0039] Spatial Positional Encoding (SPE) refers to the absolute position of the feature in the image, which is independent information. For grid and region features, we use two different ways to calculate the output value of SPE. For the spatial position encoding of grid features, since the image is divided into uniform grids in space, the position of the feature in different images is fixed. We use two sets of 1-dimensional sine and cosine encodings with different frequencies to obtain the spatial position encoding of grid features ):

[0040]

[0041] where represents the corresponding row index and column index of each feature in the grid, and represents the spatial position encoding of the feature with position , which is obtained by the encoding of the corresponding row and column, calculated by the following formula:

[0042]

[0043] where represents the index of the corresponding row or column of each feature, represents the position relative to zero. The spatial position encoding of the region feature cannot be obtained according to the above formula, because the position of the region feature in different images is not fixed. We obtain the region spatial position encoding ) by encoding the position vector of the region feature :

[0044]

[0045] where is the feature number, ​to map the parameter matrix, for the spatial position encoding of the region features.

[0046] Relative Positional Encoding (RPE) refers to the reference relationship between each feature in the image. The positional vector is used to calculate the geometric information of each feature, including the center position coordinates , relative width and relative height :

[0047]

[0048] where is the geometric information of a single feature, which is a 4-d vector, from the position vectors of the region and grid features . Then we use the corresponding of each feature to calculate RPE. First, the relative relationship between the features and is represented as a 4-dimensional vector:

[0049]

[0050] Then, use the encoding method to map the 4-dimensional to a high-dimensional space of 64 dimensions, and the specific method is to calculate the sine and cosine functions of different frequencies. Since we use RPE in the multi-head mode, finally map the encoding result to group of 1-dimensional scalars:

[0051]

[0052] where is the RPE between the final obtained features , the feature serial number is represented by , and the learnable mapping parameter matrix is , which is mapped times.

[0053] S2 constructs a relationship-enhanced cross-attention module;

[0054] S3 constructs a double-feature collaborative relationship attention network based on the geometric-related self-attention module and the relationship-enhanced cross-attention module;

[0055] The visual question answering method based on the dual feature cooperation relationship attention network comprises a geometry-related self-attention module, a relationship enhancement cross-attention module, a visual decoding module, a multi-level cooperative decoding module, a multi-modal fusion module and a question prediction module.

[0056] Specifically, the geometry-related self-attention module is used for Self-Attention of an image. Since more attention is paid to the performance of the features themselves in the process of processing, the spatial information of the features is not fully utilized, which leads to that the model cannot accurately analyze the relationship between each feature. In order to make up for this loss, on the basis of Scaled Dot-Product Attention, position encoding is integrated, and geometry-related attention (GRA) is proposed to calculate a more effective attention weight matrix. Then, the GRA is used as the attention calculation method of Self-Attention, and geometry-related self-attention (GRSA) is proposed. For SPE, the input query and key are modified. For RPE, the updated query and key are used to calculate the weight of the attention layer, and the corresponding RPE is introduced to strengthen the mutual correlation between the features:

[0057]

[0058]

[0059] wherein represents a feature of single attention head operation, represents SPE corresponding to the query and the key respectively, is RPE corresponding to the feature when the single head attention calculation is performed, and finally, the obtained attention weight is multiplied by the input value to obtain the output of the GRA:

[0060]

[0061] Therefore, the calculation method of the multi-head GRA (MHGRA) can be represented as:

[0062]

[0063]

[0064] wherein represents complete RPE corresponding to the query and the key, represents each GRA head:

[0065]

[0066] where Finally, assuming the input features are , the GRSA unit can be formally expressed as:

[0067]

[0068]

[0069] The relationship-augmented cross-attention module: the region feature pays attention to the attributes of each object itself in the image, lacks the connection with the global context information, and the expression of some detailed information of the feature is also not accurate enough, but the grid feature is just the opposite. In the theoretical level, the two are complementary. In order to perfect the relationship between the region feature and the grid feature, we propose a relationship-augmented cross-attention (RACA) unit to realize the complementary reinforcement between the region feature and the grid feature. Different from the traditional method of cross-attention between two independent features, in order to pay attention to the overall geometric information of the image, we first concatenate the region feature and the grid feature in the quantity dimension:

[0070]

[0071] where is the feature after concatenation, which is taken as the key and value of RACA, and the query is .

[0072] A set of ( is a type of region and grid feature, ) is input into a single attention. First, the corresponding spatial position coding is added to by formula 16, and it is input into multi-head attention (formula 3):

[0073]

[0074] where is the feature after updating. By formula 22, the operation of the traditional Cross-Attention is completed, but the information between the two features has not been fully associated. In order to solve this problem, we design a relationship reinforcement module (RAM) to refine the attention weight obtained by multi-head attention again, and is again relationship reinforced. First, the weight obtained by multi-head attention is mapped into two different weights:

[0075]

[0076]

[0077]

[0078] in This represents the set of multi-head attention weights. These are global weights used to enhance contextual features. Weights are assigned to features themselves, emphasizing the information inherent in the features. To map the parameter matrix, we first process two weight matrices. For We first filter the feature using the corresponding mask, then add relative position information using the addition function in Equation 16, and finally recalculate the weight of each feature using the softmax function.

[0079]

[0080] in For the updated context weights, This is the corresponding RPE for the query and key. For We directly feed it into the mask and softmax for updating:

[0081]

[0082] in This is the processed self-weight. Then, we utilize... and To strengthen The correlation between features. This part also uses a multi-head mechanism. First, and We create point sets to refine the global impact on individual data. Then, we use residual joins to... Adding it to the value aims to preserve the original feature content.

[0083]

[0084]

[0085] in for Enhanced features This indicates the specific content calculated by each head. Indicates the first Size The parameters for the linear transformation. Finally, and Dot product, realize the association of geometric information and features:

[0086]

[0087]

[0088] where is the output of RAM, denotes the calculation of each head, is the transformation parameter of the head , is the linear transformation parameter of the result. Finally, is added to the original , and the result processed by the RACA unit is obtained:

[0089]

[0090] where is one of the region and grid features, is the splicing result of the two image features.

[0091] The visual decoding module: a Multi-Level Collaborative Decoder (MLCD) network layer containing GRSA and RACA units is designed to cross-calculate the information corresponding to each other in the region and grid features, and obtain image information related to the problem keywords;

[0092] The multi-level collaborative decoding module: the multi-level collaborative decoding module includes a GRSA-GA-FFN submodule, a RACA-GA-FFN submodule, and a SA-CA-FFN submodule;

[0093] It is composed of two parallel structures, as shown in Figure 7 , which are used to process region features and grid features respectively. In the MLCD layer, we use the Guided-Attention (GA) unit to realize the guidance of the problem to the image features, which uses multi-head attention. Assuming that the input image features and the problem features , the calculation method of GA is represented as:

[0094]

[0095] Unlike traditional decoders, we divide the MLCD into three sub-modules. First, we combine image features with geometric information through the GRSA-GA-FFN sub-module to obtain problem-related region and grid information. Second, we calculate the relationship between the object and the global through the RACA-GA-FFN sub-module. Finally, since the previous two GA units filter part of the image information according to the problem, actually weakening the influence of image features in the entire model, and in the feature fusion part, the model is more inclined to obtain the weight attribute of the feature itself, therefore we use the traditional Cross-Attention to establish the SA-CA-FFN sub-module, and pay attention to the feature information in the image again. The calculation method of the CA unit is:

[0096]

[0097] where represents one of the region or grid features. We designed a visual decoder stacked by layers of MLCD Layers (denoted by ):

[0098]

[0099] where . Finally, the region features and grid features obtained by processing through the visual decoder are ;

[0100] To verify the influence of the third sub-module SA-CA-FFN in the MLCD Layer on the performance of the model, we designed two variants for comparison with the complete model: (1) the MLCD Layer only contains the GRSA-GA-FFN and RACA-GA-FFN two sub-modules, denoted as "NULL". (2) On the basis of (1), add the SA-GA-FFN sub-module, more deeply use the problem features to guide the image features. By comparing the results of the two variants, we can see that due to the strong influence of the GA unit on the features, the first variant model has good results (overall accuracy 68.37%), and the addition of an SA-GA-FFN does not greatly improve the performance of the model (overall accuracy 68.39%). Considering the influence of the GA unit on the image features, we add the SA-CA-FFN sub-module on the basis of (1) to form the complete MLCD Layer, and obtain the best performance.

[0101] The multi-modal fusion module: after processing through the collaborative relationship attention module, we obtain refined problem features , region features and grid features Before predicting the final result, we need to map and fuse these three features into a one-dimensional vector. First, we use the attentional flatten module proposed in to eliminate the influence of different feature numbers. This module is similar to a gating mechanism: assuming the input feature sequence , we first calculate the weight of each feature using MLP and softmax function. Then we multiply the resulting weight with the corresponding feature item. Finally, we add the product results to get a one-dimensional vector, representing the result of eliminating the feature number:

[0102]

[0103] where the dimension of changes to - -1, is the flattened result, is a linear parameter. Since our model uses two image features, we need to calculate the weight of each feature corresponding to the feature set separately, and then add them up:

[0104]

[0105] The final answer predicted by the model is the best match result obtained from a large-scale answer vocabulary dictionary (containing candidate answers). In existing work, we input into a fully connected layer with sigmoid activation function to predict the score of each candidate answer in the vocabulary dictionary:

[0106]

[0107] where represents the set of candidate answer scores in the vocabulary dictionary, represents the sigmoid activation function, which limits the value to , is a linear parameter matrix. Finally, we choose the answer corresponding to the highest score in as the final result of the model.

[0108] We can regard the last stage of the model as a multi-classification task, so we use binary cross-entropy (BCE) as the loss function to train the model:

[0109]

[0110] where ​denotes the number of questions in a batch, , denote the score of the first question corresponding to the first true label and the prediction result in the vocabulary dictionary.

[0111] S4 inputs the preprocessed question and image into the dual-feature collaborative relation-attention network model for prediction to obtain a predicted answer.

[0112] S41 inputs the question and image into the geometry-related self-attention module for extraction to obtain feature information and a position vector;

[0113] S42 inputs the feature information and the position vector into the dual-feature collaborative relation-attention network to obtain processed image information and image features;

[0114] S43 fuses the processed image information and the image features through the multi-modal fusion module and then decodes the fused result using the multi-level collaborative decoding module to obtain a fusion result;

[0115] S44 predicts the fusion result through the question prediction module to obtain a predicted answer.

[0116] We propose Dual-Feature Collaborative Relation-Attention Network (DFCRAN) to solve the problem of insufficient exploration of object relationships and detailed features in traditional VQA models. Based on the Encoder-Decoder framework, our model uses a new Multi-Level Collaborative Decoder (MLCD) Layer stack to realize the association of deep semantic features between region features and grid features. Specifically, our model encodes the geometric information of the image through the Geometry-Related Self-Attention (GRSA) unit, realizing the integration of features and spatial attributes. We also propose the Relativity-Augmented Cross-Attention (RACA) unit to realize the complementary reinforcement of effective information between two kinds of features. Extensive experimental results prove the effectiveness of the proposed DFCRAN model, which performs very well on the VQA 2.0 dataset. In future work, we plan to study more efficient collaborative fusion methods and extend them to other multi-modal tasks.

[0117] The above only discloses a preferred embodiment of the visual question answering method based on the dual-feature cooperative relationship attention network of the present application, and of course cannot limit the scope of the present application. Those skilled in the art can understand that all or part of the processes of the above embodiment can be implemented, and equivalent changes made according to the claims of the present application still belong to the scope covered by the present application.

Claims

1. A visual question answering method based on a dual-feature collaborative attention network, characterized in that, Includes the following steps: Construct a geometry-related self-attention module, which is used for image self-attention. A relationship-enhancing cross-attention module is constructed, which is used to achieve inter-layer complementary enhancement of regional features and grid features; A dual-feature collaborative relation attention network is constructed based on the geometrically related self-attention module and the relation-enhanced cross-attention module. After preprocessing the question and image, they are input into the dual-feature collaborative relationship attention network model for prediction to obtain the predicted answer; The specific method of the geometry-related self-attention module is as follows: Construct a location-related attention module; Based on the location-related attention module integrated with the location encoder, a geometrically related self-attention module is obtained; Position encoder: Position encoding plays an important role in the original Transformer, replacing the traditional RNN (Recurrent Neural Network) to obtain word positions; similarly, image features also have position information, specifically the spatial coordinates of each feature; we use this position information to calculate the position encoding of image features; Spatial Positional Encoding (SPE) refers to the absolute position of a feature in an image; it is independent information. For grid and region features, we use two different methods to calculate the SPE output value. For the spatial positional encoding of grid features, since... Divide the image into spaces The grid is uniform, so the feature locations in different images are fixed; we use two sets of 1D sine and cosine codes of different frequencies to obtain the spatial location codes of the grid features. ): in This represents the row and column indices of each feature in the grid, and Indicates the position as The spatial location encoding of the features is determined by the encoding of the corresponding rows and columns. Obtained by calculating using the following formula: in This represents the row or column index corresponding to each feature. This represents the position relative to zero; the spatial location encoding of region features cannot be obtained using the above formula because the positions of region features in different images are not fixed; we encode the position vector of the region features. To obtain the regional spatial location code ( ): in For feature serial number, For the mapping parameter matrix, For the first Spatial location encoding of regional features; Relative Positional Encoding (RPE) refers to the reference relationship between each feature in an image; it uses position vectors. To calculate the geometric information of each feature, including the center position coordinates. relative width and relative height : in It is the geometric information of a single feature, which is a 4-d vector. Location vectors from region and grid features Then, we use the corresponding feature To calculate RPE; first, the features and The relative relationship between them can be represented as a 4-dimensional vector: Then, the 4D was encoded using an encoding method. Mapping to a 64-dimensional high-dimensional space involves calculating sine and cosine functions of different frequencies; since we use RPE in multi-head mode, the final encoded result is mapped to... Group 1 scalar: in For the final characteristics RPE between Indicates the feature number, For a learnable mapping parameter matrix, perform Sub-mapping; The relationship-enhancing cross-attention module: first, it processes the previously processed region features... and mesh features splicing in terms of quantity: in The concatenated features are used as the key and value in RACA, and the queries are respectively... ; Send a group to a single attention , As one of the regions and grid features, First, to Add the corresponding spatial location code to the input and feed it into the multi-head attention mechanism: in For the updated features; The design of the relationship reinforcement module RAM further refines the attention weights obtained from multi-head attention calculations. Further relationship reinforcement is performed; first, the weights obtained from multi-head attention are mapped to two different weights: in This represents the set of multi-head attention weights. These are global weights used to enhance contextual features. Weights are assigned to features themselves, emphasizing the information inherent in the features. To map the parameter matrix, we first process two weight matrices; for We first filter the features using the corresponding mask, adding relative position information, and finally recalculate the weights of each feature using the softmax function. in For the updated context weights, For the corresponding RPE of query and key; We directly feed it into the mask and softmax for updating: in The processed self-weights; then, we utilize and To strengthen Correlation between features; This part also uses a multi-head mechanism; firstly... and Create a point set to refine the global impact on individual points; then use residual joins to... Adding it to the value aims to preserve the original feature content. in for Enhanced features This indicates the specific content calculated by each head. Indicates the first Size The parameters for the linear transformation; finally, and Dot product enables the association between geometric information and features: in For RAM output, This indicates how each head is calculated. For the first Size Transformation parameters, The linear transformation parameters of the result; ultimately, With the original Adding them together, we get the result processed by the RACA unit: in As one of the region and grid features, This is the result of splicing two image features; The MLCD is subdivided into three sub-modules. First, we combine image features with geometric information through the GRSA-GA-FFN sub-module to obtain problem-related region and mesh information. Second, we calculate the relationship between objects and the global context through the RACA-GA-FFN sub-module. Finally, since the first two GA units filter some image information based on the problem, effectively weakening the influence of image features in the overall model, and in the feature fusion part, the model prefers to obtain the weight attributes of the features themselves, we use the traditional Cross-Attention to establish the SA-CA-FFN sub-module to focus on the feature information in the image again. The calculation method of the CA unit is as follows: in This represents one of the features of a region or a grid; we designed a system based on... MLCDLayers Represents a stacked visual decoder: in Finally, the region features and grid features obtained through the visual decoder are respectively .

2. The visual question answering method based on a dual-feature collaborative attention network as described in claim 1, characterized in that, The visual question answering method based on a dual-feature collaborative relationship attention network includes a geometrically related self-attention module, a relationship-enhancing cross-attention module, a visual decoding module, a multi-level collaborative decoding module, a multi-modal fusion module, and a question prediction module.

3. The visual question answering method based on a dual-feature collaborative attention network as described in claim 2, characterized in that, The multi-level collaborative decoding module includes a GRSA-GA-FFN submodule, a RACA-GA-FFN submodule, and an SA-CA-FFN submodule. The GRSA-GA-FFN submodule is used to combine image features with geometric information to obtain problem-related region and mesh information; The RACA-GA-FFN submodule is used to calculate the relationship between objects and the global context; The SA-CA-FFN submodule is used to focus on feature information in the image.

4. The visual question answering method based on a dual-feature collaborative attention network as described in claim 2, characterized in that, The specific method for obtaining the predicted answer by preprocessing the question and image and then inputting them into a dual-feature collaborative relationship attention network model is as follows: The problem and image are input into the geometrically correlated self-attention module for extraction, resulting in feature information and a position vector. The feature information and the position vector are input into the dual-feature collaborative attention network to obtain the processed image information and image features. The processed image information and the image features are fused by the multi-mode fusion module, and then decoded by the multi-level collaborative decoding module to obtain the fusion result; The problem prediction module predicts the fusion result to obtain the predicted answer.

Images