A visual question answering method based on image global feature injection

Abstract

The application belongs to the cross-modal method combined by computer vision and natural language processing. The purpose is to provide a visual question answering method based on image global feature injection; by taking the image global feature as a supplement to the regional feature, the visual reasoning ability of the model is promoted, so that the prediction result is more accurate. The technical scheme is: the image regional feature is extracted through the object detection module, and the question feature is extracted through the word vector model and the time sequence model; then the question feature and the image regional feature are sent into the attention network to extract the context content required for answering the question, and in the process of attention learning, the global feature of the image is generated through intra-layer aggregation and inter-layer aggregation; then the bilateral gate mechanism is used to fuse the image regional feature, the image global feature and the question feature; finally, the answer prediction is carried out through the classification network.

Description

TECHNICAL FIELD

[0001] The application belongs to the cross-modal method combined with computer vision and natural language processing, and particularly relates to a visual question answering method based on image global feature injection. BACKGROUND

[0002] Visual question answering (VQA) aims to predict the correct answer to the corresponding question according to the given image, and has become an important role connecting the two fields of natural language processing and computer vision, and has attracted extensive attention of researchers. In addition to answering the given question, VQA is mainly applied to answering inquiries of the blind, assisting doctors in clinical analysis and diagnosis, etc. At present, in order to be able to handle complex questions and obtain more accurate answers, the focus of VQA research is on comprehensive understanding of visual and language content, extraction and representation of two modal features, and fusion and reasoning of cross-modal information.

[0003] In the past few years, thanks to the proposal of the "bottom-up" mechanism, the use of a pre-trained object detector to extract the original significant region features relying only on the visual input itself gradually replaces the grid convolution features of ConvNet. The use of these region features based on the bounding box to represent the picture information, combined with the attention mechanism to effectively capture the correlation between the image regions and the words, greatly improves the performance of the VQA model, and becomes the design standard of the visual question answering model thereafter. However, although these region features of the image are obtained by calculating the target probability of all regions, they still ignore the image-level features, so the understanding of the information conveyed by the image is not comprehensive. If only the local information of the image is focused on and the global guidance is ignored, the prediction of the answer of the visual question answering model is more likely to be wrong, which is not only due to the limited number of classes in the object detector, but also the isolation of each object processing, which will lead to the deviation of the relationship between objects. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the above background art, and a visual question answering method based on image global feature injection is proposed; by taking the image global feature as a supplement to the region feature, the visual reasoning ability of the model is promoted, so that the prediction result is more accurate.

[0005] The technical scheme adopted by the present application is a visual question answering method based on image global feature injection, comprising the following steps:

[0006] Step 1, pre-processing the input image, extracting the image features acceptable to the model through the object detection module;

[0007] Step 2, pre-processing the input question, extracting the question features acceptable to the model through the pre-trained word vector model and the time series model;

[0008] Step 3, input the image features and question features obtained in steps 1 and 2 into an attention network containing layer aggregation to obtain image region features, question features and image global features containing more noise information;

[0009] Step 4, for the layer-aggregated image global features in step 3, filter the noise information through inter-layer aggregation;

[0010] Step 5, perform feature fusion on the image global features, image region features and question features generated in steps 3 and 4 through a bilateral gating mechanism to obtain fused features;

[0011] Step 6, input the fused features obtained in step 5 into a multi-classifier to select the answer with the highest probability as the prediction result of the model.

[0012] The features of the application are further characterized in that:

[0013] In step 1, the input image is preprocessed by using Faster RCNN as a target detector, first extracting image feature maps using VGG and ResNet basic networks, then obtaining fixed-size proposal frame feature maps according to the region proposal network and region proposal pooling, and then classifying and regressing the proposal frame feature maps to obtain a dynamic number of candidate object features as visual features;

[0014] For each input image, the extracted visual features are as follows:

[0015] X=[x1,x2,…,x M ],x i ∈R dim_X

[0016] In the formula: X is the image feature extracted from each image, x i represents the feature of any candidate object, R dim_X dim_X represents the dimension of each candidate object feature, and M is the number of image candidate features.

[0017] In step 2, the input question is preprocessed, which means that the input question is first divided into words according to spaces and punctuation marks, and the number of words contained in each question is padded or truncated to the same length; then a GloVe model with a dimension of 300 is used for word embedding to map the words to the same vector space; for words not in the GloVe model, a random vector is selected for initialization; finally, a long short-term memory network with a hidden layer dimension of dim_Y is used to extract the output of each time unit as a language feature; as follows:

[0018] Y=[y1,y2,…,y N ],yi ∈R dim_Y

[0019] where Y is the extracted question feature for each question, y i represents the output of any one time unit, dim_Y is the dimension of the language feature, and N is the same length of the question padding or truncation.

[0020] The method of layer-wise aggregation in step 3 is: for the input image X = [x1, x2, …, x M , first obtain the global image feature g with more noise through average pooling; then splice the image global feature and the image region feature to generate a new image feature C 0 ; The process of layer-wise aggregation is as follows:

[0021]

[0022] C 0 = [x1, x2, …, x M , g].

[0023] The attention network in step 3 refers to a deep joint attention learning structure following the encoding and decoding structure; the specific process is as follows:

[0024] 1) For the input of the question feature, an encoder learning containing L layers of self-attention modules will be performed; the execution process of the l+1 layer encoder includes:

[0025] (1) First, the encoder input Y l ∈R N×dim is mapped to h parallel heads through a weight matrix ; then the scaled dot product attention is performed on each parallel head; the specific process is shown in the following formula:

[0026]

[0027] where: is the attention result of the i-th parallel head, is the scaling factor; softmax is the normalization function; Y l is the output of the question feature after l-layer encoder;

[0028] (2) Then, the attention results generated by splicing the h parallel heads are obtained, and the multi-head attention containing subspace information is obtained; finally, the attention result is subjected to residual and normalization operation, and the input of the next layer encoder is obtained; the specific process is shown in the following formula:

[0029] Y l+1 = LayerNorm(Concat(head1, …, headh )W o +Y l )

[0030] wherein LayerNorm is a normalization operation, Concat is a concatenation operation, W o is a learning parameter, and h is the number of parallel heads of multi-head attention;

[0031] (3) Then, the question features containing the context required for predicting the answer output by the L-layer encoder are input into a multi-layer perceptron (MLP) containing two ReLU nonlinear activations, and then the feature weights a Y ∈R N are calculated by a softmax function. Y a L = softmax(MLP(Y L ));

[0032]

[0033] wherein MLP is a multi-layer perceptron, L is the number of encoder layers, Y l is the question feature output after the L-layer encoder, and Y (M+1)×dim is the output of the image feature after the l-layer decoder.

[0034] (4) Finally, the question features for feature fusion are generated in the form of weighted summation. The specific formula is as follows:

[0035]

[0036] wherein N is the same length after padding or truncation of the question, is the weight of each question feature vector, is the i-th question feature vector.

[0037] 2) For image feature input, an L-layer decoder containing a self-attention module is learned; wherein the execution process of the l+1-layer decoder includes:

[0038] (1) First, the C l ∈R l is mapped to h parallel heads through a weight matrix ; then, a scaled dot-product attention is performed on each parallel head; the specific process is shown in the following formula:

[0039]

[0040] wherein is the attention result of the i-th parallel head, is a scaling factor, and C l is the output of the image feature after the l-layer decoder.

[0041] (2) Then the attention results generated by splicing h parallel heads are obtained to get multi-head attention containing subspace information; finally, residual and normalization operations are performed on the attention results as the input of the next layer encoder; the specific process is shown in the following formula:

[0042]

[0043] In the formula: LayerNorm is a normalization operation, Concat is a splicing operation, W o is a learning parameter;

[0044] 3) In addition to self-attention learning, cross-modal attention learning of question-guided images is also needed; the specific formula is as follows:

[0045]

[0046]

[0047] In the formula: the image feature output at this time is

[0048] Extract the region features of the image First, input the image region features containing the context required for predicting the answer into a multi-layer perceptron (MLP) containing two ReLU nonlinear activations, and then calculate the feature weights a X ∈R M through the SoftMax function; the specific formula is as follows:

[0049] a X = softmax(MLP(X L ));

[0050] Finally, the image region features for feature fusion are generated in the form of weighted summation The specific formula is as follows:

[0051]

[0052] In the formula: is the weight of each image region feature vector, is the i-th image region feature vector.

[0053] The method of interlayer aggregation in step 4 is:

[0054] (1) First, extract the image global features in each layer decoder to obtain G = [g 0 , g 1 , …, g L ] ∈R L×dim; then it is sent into a single-layer long short-term memory network, and the output of each layer is extracted to obtain the image global feature H = [h 0 , h 1 , …, h L ] ∈ R l ×dim ;

[0055] The formula is as follows:

[0056] h i = LSTM (g i , h i-1 ) (i = 1, 2, …, L)

[0057] In the formula: g i is the image global feature of the i-th decoder, LSTM is a long short-term memory network, and h i is the output of the i-th LSTM unit;

[0058] (2) Then the correlation between the question feature and the image global feature h i ∈ R dim (i = 1, 2, …, L) is calculated as the feature weight, and the formula is as follows:

[0059]

[0060] In the formula: a h i is the correlation weight, is the question feature, and sigmoid is an activation function;

[0061] (3) Finally, the correlation weight is obtained, and the image global feature H is weighted and summed to obtain the image global feature used for feature fusion The formula is as follows:

[0062]

[0063] The method of the bilateral gating mechanism in step 5 is:

[0064] (1) First, the gating value G is calculated according to the image region feature obtained in step 3 and step 4, the question feature and the image global feature The formula is as follows:

[0065]

[0066] In the formula: G ∈ R 1

[0067] (2) Then use G to control the weight of image region features, use (1-G) to control the weight of visual global features, and obtain the final fusion features through feature addition To obtain the best performance; The formula is as follows:

[0068]

[0069] In the formula, All are learning parameters.

[0070] The working method of the multi-classifier in the step 6 is that the fusion features obtained in the step 5 are input into a network composed of multiple linear layers, and finally the result is controlled between 0-1 through a sigmoid function, as the probability of each candidate answer; The formula is as follows:

[0071] A' = sigmoid (Linear (f))

[0072] In the formula: A' is the model predicted answer, Limer is linear regression, and f is the fusion feature.

[0073] The beneficial effects of the present application are:

[0074] 1. The present application proposes an attention network for generating image global features, aggregates image local information in the attention network, and aggregates interlayer image global features through a long short-term memory network, so as to clear noise information. The image global features are injected into the visual question and answer model.

[0075] 2. The present application proposes a new feature fusion method, which adaptively allocates the weight between image global features and local features through a bilateral gating mechanism, so as to better fuse with the question features. BRIEF DESCRIPTION OF DRAWINGS

[0076] Figure 1 The network model structure diagram of the method described in the embodiment of the present application.

[0077] Figure 2 The structure diagram of the interlayer aggregation module in the method described in the embodiment of the present application.

[0078] Figure 3 The structure diagram of the feature fusion module of the bilateral gating mechanism in the method described in the embodiment of the present application. SPECIFIC IMPLEMENTATION METHOD

[0079] The following further illustrates the embodiments shown in the accompanying drawings.

[0080] The visual question and answer method based on image global feature injection of the present application comprises the following steps:

[0081] Step 1: pre-process the input image, extract image features that can be accepted by the model through the object detection module;

[0082] The pre-processing of the input image is to use Faster RCNN as the target detector. First, the VGG and ResNet basic network is used to extract the image feature map, and then the region proposal network and region proposal pooling are used to obtain the fixed size of the proposal frame feature map. Then, the proposal frame feature map is classified and regressed to obtain a dynamic number of candidate object features as visual features.

[0083] For each input image, the extracted visual features are as follows:

[0084] X = [x1, x2, …, x M ], x i ∈ R dim_X

[0085] In the formula: X is the extracted image feature of each image, x i represents the feature of any candidate object, R dim_X dim_X represents the dimension of each candidate object feature, and M is the number of image candidate features.

[0086] Step 2: pre-process the input question, extract question features that can be accepted by the model through the pre-trained word vector model and time sequence model;

[0087] The pre-processing of the input question means: first, according to the space and punctuation, the input question is divided into word form, and the number of words contained in each question is filled or truncated to the same length; then the GloVe model with a dimension of 300 is used for word embedding, and the words are mapped to the same vector space; for the words not in the GloVe model, a random vector is selected for initialization; finally, the long short-term memory network with a hidden layer dimension of dim_Y is used to extract the output of each time sequence unit as the language feature; as follows:

[0088] Y = [y1, y2, …, y N ], y i ∈ R dim_Y

[0089] In the formula: Y is the extracted question feature of each question, y i represents the output of any time sequence unit, dim_Y is the dimension of the language feature, and N is the same length after filling or truncation.

[0090] Step 3: input the image features and question features obtained in steps 1 and 2 into the attention network containing layer cohesion to obtain the context information and image global features required for answering the question.

[0091] 1. The method of intra-layer aggregation is: for the input X = [x1, x2, …, xN] of the image, first obtain the global feature g of the image with more noise by average pooling; then splice the global feature of the image and the regional feature of the image to generate a new image feature C M 0 ; The process of intra-layer aggregation is as follows:

[0092]

[0093] C 0 = [x1, x2, …, xN, g]; M

[0094] 2. The attention network refers to: the attention network in the application follows a deep joint attention learning structure of encoding and decoding structure; specifically:

[0095] 1) For the input of the question feature, an encoder learning containing L layers of self-attention modules is performed, wherein the execution process of the l+1 layer encoder includes:

[0096] (1) First, the encoder input Y l ∈R N×dim is mapped to h parallel heads through a weight matrix ; Then, a scaled dot-product attention is performed on each parallel head; The specific process is shown in the following formula:

[0097]

[0098] In the formula: is the attention result of the i-th parallel head, is a scaling factor; softmax is a normalized exponential function; Y l is the output of the question feature after l layer encoder;

[0099] (2) Then, the attention results generated by splicing the h parallel heads are obtained, and the multi-head attention containing subspace information is obtained; Finally, the attention result is subjected to residual and normalization operations, and the input of the next layer encoder is obtained; The specific process is shown in the following formula:

[0100] Y l+1 = LayerNorm (Concat (head1, …, head h ) W o + Y l )

[0101] In the formula: LayerNorm is a normalization operation, Concat is a splicing operation, W o ​​where h is the number of parallel heads of multi-head attention for learning parameters;

[0102] (3) Then the question features containing the context required for predicting the answer after the output of the L-layer encoder are learned are input into a multi-layer perceptron (MLP) containing two ReLU nonlinear activations, and then the individual feature weights a are calculated by a SoftMax function Y ∈R N The specific formula is as follows:

[0103] a Y = softmax(MLP(Y L ));

[0104] wherein: MLP is a multi-layer perceptron, L is the number of encoder layers, Y L is the question feature output after the L-layer encoder;

[0105] (4) Finally, the question features for feature fusion are generated in the form of weighted summation The specific formula is as follows:

[0106]

[0107] wherein: N is the same length after the question is filled or truncated, the weight of each question feature vector, is the i-th question feature vector.

[0108] 2) For image feature input, an L-layer decoder containing a self-attention module is learned; wherein the execution process of the l+1-layer decoder includes:

[0109] (1) First, the C l ∈R (M+1)×dim is mapped to h parallel heads through a weight matrix ; then a scaled dot-product attention is performed on each parallel head; the specific process is shown in the following formula:

[0110]

[0111] wherein: is the attention result of the i-th parallel head, is a scaling factor, C l is the output of the image feature after the l-layer decoder;

[0112] (2) Then, the multi-head attention containing subspace information is obtained by splicing the attention results generated by the h parallel heads; finally, the attention result is subjected to residual and normalization operations, serving as the input of the next layer encoder; the specific process is shown in the following formula:

[0113]

[0114] where LayerNorm is a normalization operation, Concat is a concatenation operation, W o is a learning parameter;

[0115] 3) In addition to self-attention learning, cross-modal attention learning of question-guided images is also needed; the specific formula is as follows:

[0116]

[0117]

[0118] where the image feature output is

[0119] The region features of the image are extracted First, the image region features containing the context required for predicting the answer are input into a multi-layer perceptron (MLP) containing two ReLU nonlinear activations, and then the feature weights a X ∈R M are calculated through the SoftMax function; the specific formula is as follows:

[0120] a X = softmax(MLP(X L ));

[0121] Finally, the image region features for feature fusion are generated in the form of weighted summation The specific formula is as follows:

[0122]

[0123] where: is the weight of each image region feature vector, is the i-th image region feature vector.

[0124] Step 4: For the image global features aggregated in the layer in step 3, the method of inter-layer aggregation (the structure of the inter-layer aggregation module is shown in Figure 2 ) is as follows:

[0125] (1) First, the image global features in each layer of the decoder are extracted to obtain G = [g 0 , g 1 , …, g L ] ∈R L×dim ; Then, it is sent into a single-layer long short-term memory network to extract the output of each layer to obtain the image global features H = [h 0 , h 1 , …, hL ]∈R L ×dim ;

[0126] The formula is shown below:

[0127] h i =LSTM(g i h i-1 (i = 1, 2, ..., L)

[0128] Where: g i The image global features are represented by the i-th layer decoder, LSTM is a Long Short Memory network, and h i The output of the i-th LSTM unit

[0129] (2) Next, calculate the problem characteristics. and global features of the image h i ∈R dim The correlation of (i = 1, 2, ..., L) is used as the feature weight, and the formula is as follows:

[0130]

[0131] In the formula: a h i For relevance weights, The problem features are defined by sigmoid, which is the activation function.

[0132] (3) The final relevance weights The global image features H are weighted and summed to obtain the global image features used for feature fusion. The formula is shown below:

[0133]

[0134] Step 5: Combine the global image features, image region features, and problem features generated in Steps 3 and 4 using a bilateral gating mechanism for feature fusion (see the feature fusion module structure of the bilateral gating mechanism for details). Figure 3 ), thereby obtaining fusion characteristics.

[0135] The bilateral gating method is as follows:

[0136] (1) First, based on the image region features obtained in steps 3 and 4, Problem characteristics and global features of the image Calculate the gate value G; the formula is shown below:

[0137]

[0138] In the formula: G∈R1

[0139] (2) Then use G to control the weight of image region features, use (1-G) to control the weight of visual global features, and obtain the final fusion features through feature addition To obtain the best performance; The formula is as follows:

[0140]

[0141] In the formula, All are learning parameters.

[0142] Step 6: input the fusion features obtained in step 4 into a multi-classifier, and select the answer with the highest probability as the prediction result of the answer through the multi-classifier

[0143] The working method of the multi-classifier is: input the fusion features obtained in step 5 into a network composed of multiple linear layers, and finally control the result between 0-1 through a sigmoid function as the probability of each candidate answer, and the formula is as follows:

[0144] A' = sigmoid (Linear (f))

[0145] In the formula: A' is the model prediction answer, Limer is linear regression, and f is the fusion feature.

[0146] In summary, based on the VQA2.0 dataset, the image region features are extracted by the Faster RCNN, and the question features are extracted by the Glove model and the LSTM network; then the question features and the image region features are input into the attention network to extract the context content required for answering the question, and in the process of attention learning, the global features of the image are generated through layer aggregation and interlayer aggregation; then the double gate mechanism is used to fuse the image region features, the image global features and the question features; finally, the answer is predicted through the classification network.

[0147] Specifically, for the image region features, the local information in the attention network is aggregated to form the interlayer image global representation. Then the long short-term memory network is used to sequentially aggregate the image global representation between layers, so as to discard the useless information in the previous layer. Compared with directly using the pooling technology to extract the global feature representation in the original image, this way greatly reduces the influence of noise information on the result prediction. Then, the invention fuses the image global and region features with the question features in a self-adaptive weight manner through the double gate mechanism.

[0148] The innovation points of the present application are: first, the image global features generated by intra-layer and inter-layer polymerization can reduce noise information compared with ordinary average pooling; second, the generated image global features are injected into the model, and the bilateral gating mechanism is used to participate in feature fusion, which can enhance the model's understanding ability of visual content, so that the model has strong generalization ability when facing more complex visual scenes, thereby improving the accuracy of the model's answer prediction.

[0149] Simulation experiment and experimental results:

[0150] 1. Dataset

[0151] The model is verified on the VQA-v2 dataset, which is mainly derived from the MS COCO dataset and adds artificial labeled questions and answers based on images, and is the most commonly used benchmark dataset in the VQA task. In order to solve the defects of the previous generation VQA-v1 dataset, each question in the VQA-v2 dataset corresponds to two different pictures and has two different answers, which expands the examples of the training set, the validation set and the test set. Effectively prevent the model from performing excellent performance through simple memory. The dataset contains about 1105K questions and 204K pictures, of which the training set, the validation set and the test set each account for 40%, 20% and 40%. 25% of the data in the test set is called test-dev. As shown in Table 1, the questions are divided into three types: Yes / No, Other and Num, and each question contains 10 corresponding answers. The experimental indicators are divided into the accuracy of each question type and the overall accuracy. The model proposed in the present application is trained on the training set and the validation set, and in order to ensure fair comparison with other works, the test results are reported on the test-dev set and the test-standard set.

[0152] 2. Experimental environment

[0153] The model proposed in the present application is experimented on the Nvidia GeForce RTX 3090 graphics card, Ubuntu 16.04 operating system, CUDA 11.1 supported PyTorch 1.9.0 framework. In the training process, in order to prevent overfitting, the Dropout layer of the model uses a probability of 0.5, and the training batch is 64. The Adam optimizer (β1=0.9, β2=0.98) is trained for a total of 16 cycles, and the learning rate is 0.0001 for the first 10 cycles. After 10 cycles, the learning rate of each cycle is reduced by 1 / 10.

[0154] 3. Analysis of experimental results

[0155] The present application uses the best single model, compared with the current advanced visual question answering model, the experimental results are shown in Table 1. Among them, top-down-bottom-up (BUTD) is the champion of VQA challenge in 2017, proposes to use the image object features detected by Faster RCNN in the attention module instead of the original grid features extracted by RestNet to answer the input question. MFH is the most advanced bilinear fusion method. BAN proposes a bilinear attention mechanism that uses information from each channel to generate multi-modal attention. DFAF is a multi-layer stacked network that dynamically learns the information flow within and between modalities to achieve feature fusion. MLIN achieves multi-modal interaction through a small amount of information, which helps to learn and encode between modalities from a global perspective, avoiding capturing too much noise information. MCAN proposes an Encoder-Decoder joint attention learning method, which improves the prediction accuracy of the model.

[0156] Table 1 compares the most advanced visual question answering model on the VQA v2 test set

[0157]

[0158] From the above table, it can be seen that the method of the present application has a significant performance improvement of 6.65% in the overall accuracy on test-dev compared with the attention-based method (such as the classic BUTD). Compared with the subsequent BUTD-based method for improving attention on the basis of image region features, such as DFAF, MLIN and MCAN, the performance is improved. Because the method of the present application generates image global features to participate in the feature fusion process, it can better understand the information conveyed by the image and generate satisfactory answers.

Claims

1. A visual question answering method based on image global feature injection, comprising the following steps: Step 1, pre-processing the input image, extracting image features acceptable to the model through an object detection module; Step 2, pre-processing the input question, extracting question features acceptable to the model through a pre-trained word vector model and a time series model; Step 3, inputting the image features and question features obtained in steps 1 and 2 into an attention network containing layer aggregation to obtain image region features, question features, and image global features containing more noise information; Step 4, filtering noise information for the layer-aggregated image global features in step 3 through inter-layer aggregation; Step 5, performing feature fusion on the image global features, image region features, and question features generated in steps 3 and 4 through a bilateral gating mechanism to obtain fused features; Step 6, inputting the fused features obtained in step 5 into a multi-classifier to select the answer with the highest probability as the model's prediction result through the multi-classifier; In step 1, the input image is pre-processed using Faster RCNN as the target detector, first extracting image feature maps using VGG and ResNet base networks, then obtaining fixed-size proposal frame feature maps according to the region proposal network and region proposal pooling, and finally classifying and regressing the proposal frame feature maps to obtain a dynamic number of candidate object features as visual features; For each input image, the extracted visual features are as follows: In the formula: image features extracted for each image, representing the features of any one candidate object, representing the dimension of each candidate object feature , M is the number of image candidate features; The preprocessing of the input question in step 2 means that firstly, the input question is divided into words according to spaces and punctuation marks, and the number of words contained in each question is filled or truncated to the same length; then word embedding is performed using a GloVe model with a dimension of 300 to map the words into the same vector space; for words not in the GloVe model, random vectors are selected for initialization; finally, a long short-term memory network with a hidden layer dimension of 128 is used to extract the output of each time sequence unit as a language feature; ​ As shown in the following formula: wherein: is the question feature extracted for each question, represents the output of any one time unit, is the dimension of the language feature, N is the same length of question padding or truncation; The method of layer-wise polymerization in step 3 is: for the input of the image , first obtain the global feature of the image with more noise by the way of average pooling g ; then splice the image global feature and the image regional feature to generate a new image feature C 0 ; the process of layer-wise polymerization is as follows: ； The attention network in step 3 refers to a deep joint attention learning structure following the encoding and decoding structure; specifically as follows: 1) for input of problem features, will be performed The layer contains encoder learning of self-attention modules; wherein The execution flow of the layer encoder includes: (1) First, the encoder input is mapped to individual parallel heads through a weight matrix ; then scaled dot-product attention is performed on each parallel head; the specific process is shown in the following formula: wherein: is the attention result for the th parallel head, is a scaling factor; softmax is a normalizing function; Y l is the output of the problem-specific feature l layer encoder. (2) Then through splicing The attention results generated by the parallel heads are used to obtain a multi-head attention that includes subspace information; finally, residual and normalization operations are performed on the attention results, which are then used as the input to the next layer encoder; the specific process is shown in the following equation: In the formulae: LayerNorm is a normalization operation, Concat is a concatenation operation, is a learning parameter, h is the number of parallel heads of multi-head attention; (3) Then the question features containing the required context for predicting the answer output by the learning of the layer encoder are input into a multi-layer perceptron (MLP) containing two ReLU nonlinear activations, and then the respective feature weights are calculated by a SoftMax function ; the specific formula is as shown below: ​ ； In the formula: MLP For multilayer perceptrons, L For encoder layers, Y L To pass L Problem feature output after layer encoder; (4) Finally, the problem features for feature fusion are generated in the form of weighted summation ; The specific formula is as follows: the number of layers of the encoder wherein: N the same length as the truncation of the problem, the weight of each problem feature vector, is the i-th problem feature vector; 2) for image feature input, will be The decoder learns from the self-attention module; wherein The execution flow of the layer decoder includes: (1) First, the is mapped to weight matrix parallel heads; then the scaled dot-product attention is performed for each parallel head; the specific process is shown in the following formula: wherein: is the attention result for the th parallel head, is a scaling factor, C l is the output of the image feature after passing through the l layer decoder; (2) Then through splicing The attention results generated by the parallel heads are used to obtain a multi-head attention that includes subspace information. Finally, residual and normalization operations are performed on the attention results, which are then used as the input to the next layer encoder. The specific process is shown in the following formula: In the formula: LayerNorm is a normalization operation, Concat is a concatenation operation, is a learning parameter; 3) In addition to self-attention learning, cross-modal attention learning of question-guided images is also needed; the specific formula is as follows: In this case, the image feature output is ; Extracting region features of an image First, the image region features containing the context required for predicting the answer are input into a multi-layer perceptron (MLP) containing two ReLU nonlinear activations, and then the respective feature weights are calculated by a SoftMax function ; the specific formula is as follows: ； Finally, the image region features for feature fusion are generated in the form of weighted summation ; the specific formula is as follows: wherein: is a weight for each image region feature vector, is the jth image region feature vector; and i is the jth image region feature vector. The method of inter-layer aggregation in step 4 is: (1) Firstly, the global image features in each layer decoder are extracted to obtain ; then, the global image features are obtained by sending them into a single-layer long short memory network and extracting the output of each layer ; The formula is as follows: In the formula: g i is the first i image global feature of the layer decoder, LSTM is a long short-term memory network, h i is the first i output of the LSTM unit (2) Then calculate the relevance of the problem features and the image global features as the feature weights, as shown in the following formula: wherein: a h i is a relevance weight, is a question feature, sigmoid is an activation function; (3) the correlation weight finally obtained The image global feature [mathematical formula] is weighted and summed to obtain an image global feature used for feature fusion The formula is as follows: 。 2. The image global feature injection based visual question answering method according to claim 1, characterized in that: The method of bilateral gating mechanism in step 5 is: (1) First, the image region features obtained in steps 3 and 4 , problem features and image global features are used to calculate the gating value G； as shown in the following formula: In the formulae: (2) Then use Control the weight of image region features, use Control the weight of visual global features; and obtain the final fusion features by feature addition To obtain the best performance; The formula is as follows: In the formulae; to learn parameters.

3. The image global feature injection based visual question answering method according to claim 2, characterized in that: The working method of the multi-classifier in step 6 is: input the fused features obtained in step 5 into a network composed of multiple linear layers, and finally control the result between 0-1 through the sigmoid function as the probability of each candidate answer; the formula is as follows: where: A' is the model predicting the answer, Limear is the linear regression, f is the fused feature.

Images