An image question answering method based on joint modality interaction

Abstract

The present invention discloses an image question-answering method based on modal joint interaction, which includes the following steps: preprocessing images and questions to obtain corresponding image feature vectors and question high-level feature vectors; constructing an image question-answering network and obtaining intra-modal unit attention features and inter-modal unit attention features; deep fusion of the intra-modal unit attention features and inter-modal interaction attention features through bimodal joint interaction and residual stacking to obtain output image features and output question features of the hidden layer; feature merging of the output image features and output question features of the last hidden layer to obtain final features; and mapping the final features to a multi-category vector through linear transformation for answer prediction. The present invention can achieve bidirectional guidance between image and question features, improve the model's interactive capabilities, enhance information sharing across modal semantic spaces, enhance the model's multimodal interactive capabilities, and improve the effect of answer classification.

Description

Technical Field

[0001] The invention relates to the field of artificial intelligence deep learning image question answering technology, and specifically to an image question answering method based on modal joint interaction. Background Art

[0002] Currently, the computing industry has seen significant improvements in both network speed and performance. Consequently, data is being transmitted across the internet in various modalities. From text to image transmission, and then to voice, video, and other modalities, all of this is inseparable from the advancement of computer software and hardware. Image question answering, a classic cross-modal task, requires computers to locate key information from a given image and question and infer the answer. Compared to single-modal tasks like object detection and question-answering systems, this requires computer models to have a more granular understanding of image and text information, as well as the ability to integrate and interact with both modalities to achieve model reasoning.

[0003] Image question answering requires understanding both the image and the question text separately. Images are stored in the form of high-dimensional data structures and contain a large amount of visual information such as color and shape. This information is very intuitive to humans but extremely challenging for computers.

[0004] With the development of computer vision, more and more image feature extraction methods can effectively extract and encode information in images, enabling computers to understand various types of information in images. However, question texts are natural languages, which contain a large amount of semantic and grammatical information, and are not intuitively understandable for computers. In addition, natural language can have problems such as polysemy and ambiguity, which also increases the difficulty for computers to understand natural language. Currently, word embedding methods have emerged in the field of natural language processing, which can enable computers to better understand natural language, which is critical to improving computer reasoning capabilities. However, due to the semantic differences between image and text features, it is difficult to effectively integrate them and enable the model to acquire reasoning capabilities. In addition, the model's interactive capabilities are low, which is also a major difficulty in image question answering tasks. Summary of the Invention

[0005] In response to the above-mentioned deficiencies in the prior art, the present invention provides an image question-answering method based on modal joint interaction, which solves the problem in the prior art of being unable to predict answers due to the difficulty in fusing image and text features.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0007] An image question answering method based on modality joint interaction is provided, which includes the following steps:

[0008] S1. Initialize the object detection network with convolutional neural network as the core through the pre-trained residual network to obtain the dynamic features of the input image; fill the dynamic features of the input image to obtain the image feature vector;

[0009] S2. Use the pre-trained global word vector model to vectorize the question text to obtain the question text feature vector; perform feature processing on the question text feature vector to obtain the question high-level feature vector;

[0010] S3. Construct an image question answering network with intra-modality unit attention and inter-modality interactive attention as the core; input the image feature vector and the question high-level feature vector into the image question answering network to obtain the intra-modality unit attention feature and the inter-modality unit attention feature;

[0011] S4. Deeply fuse the intra-modality unit attention features and inter-modality interaction attention features through bidirectional joint interaction of modalities and residual stacking to obtain the output image features and output question features of different hidden layers;

[0012] S5. Merge the output image features and the output question features of the last hidden layer through feature stacking to obtain the final features; map the final features to a multi-category vector through linear transformation for answer prediction.

[0013] Furthermore, the pre-trained residual network in step S1 adopts the ResNet-101 network structure, and the convolutional neural network adopts the Faster R-CNN network structure; the global word vector model in step S2 adopts the GloVe model.

[0014] Furthermore, the specific steps of step S2 are as follows:

[0015] S2-1. Segment and vectorize the m words in the question text to obtain the question text feature vector;

[0016] S2-2. Fill the question text feature vector with 0 to obtain a dimension of M QUES ×EMB_SIZE vector representation in, Represents the vector representation of the question text at time t, M QUES Indicates the number of question words, EMB_SIZE indicates the embedding dimension of the question;

[0017] S2-3, introduce a bidirectional gated recurrent unit, according to the formula:

[0018]

[0019]

[0020] Get the bidirectional hidden state at time t, that is, the forward hidden state and the reverse hidden state Among them, GRU(·) represents the recurrent neural network model, represents the forward hidden state at time t-1, Represents the reverse hidden state at time t+1;

[0021] S2-4. Concatenate the question text feature vectors at all times to obtain the final question high-level feature vector

[0022] Furthermore, the specific steps of step S3 are as follows:

[0023] S3-1. Initialize the model parameters of the image question answering network and input the image feature vector and the question high-level feature vector into the image question answering network;

[0024] S3-2. Train the image question answering network model, set the preset answer category as the training target, train the image question answering network through the back propagation algorithm and stochastic gradient descent, adjust the parameters of the image question answering network, and obtain the trained image question answering network; the parameters of the image question answering network include the learnable weight matrix W n and the bias term b n ;

[0025] S3-3. Construct the image modality intra-unit attention module and the question modality intra-unit attention module. Select one of the modality intra-unit attention modules and use the formula:

[0026] Q,K,V=trans(X)

[0027] Get the query vector Q, key vector K, and value vector V; where n represents the number of words in the question or the number of objects in the image, EMB_DIM represents the embedding dimension of the question, X represents the feature vector of a certain mode with dimension n×EMB_DIM, and trans(·) represents the conversion of the feature vector X into a multi-headed feature vector; the feature vector X includes the image feature vector and the question high-level feature vector;

[0028] S3-4, according to the formula:

[0029]

[0030] Get the attention score matrix S; where K T represents the transposed matrix of the key vector K, d k Indicates the EMB_DIM size of the query vector Q;

[0031] S3-5, according to the formula:

[0032] A=softmax(S), S∈R MH×n×n

[0033] Get the attention weight matrix A; where softmax(·) represents the normalized exponential function, R represents a real number, and MH represents the number of attention heads;

[0034] S3-6. Convert the multi-head feature vector to obtain a multi-head feature vector of the same dimension as the original input, according to the formula:

[0035] O=trans'(A·V)

[0036] O' = LayerNorm(O + Dropout(O))

[0037] FFN(O')=max(0,O'W1+b1)W2+b2

[0038] O I =LayerNorm(O'+Dropout(FFN(O')))

[0039] Get the intra-modal unit attention matrix O I , that is, the intra-modal unit attention feature of the modality; where trans'(·) represents the dimension conversion function, O represents the initial intra-modal unit attention matrix, O' represents the intermediate intra-modal unit attention matrix, Dropout(·) represents random inactivation, LayerNorm(·) represents the normalization function, FFN(·) represents the feedforward neural network, max(·) represents the implementation of the ReLU activation function, W1 represents the learnable weight matrix from the input layer to the hidden layer, W2 represents the learnable weight matrix from the hidden layer to the output layer, b1 represents the bias term from the input layer to the hidden layer, and b2 represents the bias term from the hidden layer to the output layer;

[0040] S3-7, repeat steps S3-3 to S3-6 to obtain the intra-modal unit attention matrix O of another modality i ;

[0041] S3-8. Construct inter-modal interactive attention, taking the features of one modality as the query and the features of the other modality as the key value, according to the formula:

[0042]

[0043] Get the attention weight matrix A1; where Q1 represents the query vector of one modality feature, K1 represents the key vector of another modality feature, and K1 T represents the transposed matrix of the key vector K1, d1 represents the EMB_DIM size of the query vector Q1, and V1 represents the value vector of another modal feature;

[0044] S3-9, according to the formula:

[0045] O1=LayerNorm(A1+Dropout(A1))

[0046] O A =LayerNorm(O1+Dropout(FFN(O1)))

[0047] Get the inter-modal unit attention matrix O A , that is, the inter-modal unit attention feature; where O1 represents the initial inter-modal unit attention matrix.

[0048] Furthermore, the loss function adopted by the back propagation algorithm in step S3-2 is a binary cross entropy loss function.

[0049] Furthermore, the specific steps of step S4 are as follows:

[0050] S4-1, according to the formula:

[0051]

[0052] The inter-modal interactive attention feature XO guided by image features and the inter-modal interactive attention feature YO guided by question features are obtained; where CR(·) represents the inter-modal interactive attention, X' represents the unit attention feature within the image modality, and Y' represents the unit attention feature within the question modality;

[0053] S4-2, according to the formula:

[0054]

[0055] Obtain the inter-modal interactive attention feature X[i] guided by image features after the depth stacking of the i-th hidden layer and the inter-modal interactive attention feature Y[i] guided by question features after the depth stacking of the i-th hidden layer; where i represents the i-th hidden layer, XO[i-1] represents the inter-modal interactive attention feature guided by image features of the i-1-th hidden layer, and YO[i-1] represents the inter-modal interactive attention feature guided by question features of the i-1-th hidden layer;

[0056] S4-3, according to the formula:

[0057]

[0058] The output image feature x[i] of the i-th hidden layer and the output question feature y[i] of the i-th hidden layer are obtained; where X[i-1] represents the inter-modal interactive attention feature guided by the image feature after the depth stacking of the i-1-th hidden layer, and Y[i-1] represents the inter-modal interactive attention feature guided by the question feature after the depth stacking of the i-1-th hidden layer. i represents the image trainable weight variable of the i-th hidden layer, β i represents the problem trainable weight variable of the i-th hidden layer.

[0059] The formula of the linear transformation mapping in step S5 is as follows:

[0060] O F =proj(concat(x[I],y[I]))

[0061] Among them, O F represents the answer, x[I] represents the output image feature x[i] of the last hidden layer, y[I] represents the output question feature of the last hidden layer, concat(·) represents the concatenation operator, and proj(·) represents the linear transformation mapping function.

[0062] The beneficial effects of the present invention are:

[0063] 1. The present invention can achieve two-way guidance between image and question features and improve the model's interaction capability by constructing a model for image-text feature extraction and deep fusion based on modal joint interaction and introducing a mechanism for modal joint interaction; by using a residual deep stacking fusion mechanism, it enhances information sharing in cross-modal semantic space.

[0064] 2. The bidirectional modality guidance mechanism in this paper takes into account the deep interaction between the two modalities and adopts a joint forward and reverse guidance of the attention features of units within the two modalities, which enhances the model's multimodal interaction capabilities and improves the answer classification effect.

[0065] 3. The residual stacking fusion mechanism in this invention adopts a deep stacking approach to allow the features after bidirectional guidance to further interact; the design of its residual dynamic mechanism improves the expressive ability and avoids the problem of vanishing gradients of deep neural networks during training, thereby improving the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a specific flow chart of the present invention;

[0067] Figure 2 This is a working diagram of the deep fusion of modal bidirectional joint interaction and residual stacking of the present invention. DETAILED DESCRIPTION

[0068] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0069] In one embodiment of the present invention, Figure 1 As shown in FIG, an image question answering method based on modality joint interaction includes the following steps:

[0070] S1. Initialize the object detection network with convolutional neural network as the core through the pre-trained residual network to obtain the dynamic features of the input image; fill the dynamic features of the input image to obtain the image feature vector;

[0071] S2. Use the pre-trained global word vector model to vectorize the question text to obtain the question text feature vector; perform feature processing on the question text feature vector to obtain the question high-level feature vector;

[0072] S3. Construct an image question answering network with intra-modality unit attention and inter-modality interactive attention as the core; input the image feature vector and the question high-level feature vector into the image question answering network to obtain the intra-modality unit attention feature and the inter-modality unit attention feature;

[0073] S4. Deeply fuse the intra-modality unit attention features and inter-modality interaction attention features through bidirectional joint interaction of modalities and residual stacking to obtain the output image features and output question features of different hidden layers;

[0074] S5. Merge the output image features and the output question features of the last hidden layer through feature stacking to obtain the final features; map the final features to a multi-category vector through linear transformation for answer prediction.

[0075] The pre-trained residual network in step S1 adopts the ResNet-101 network structure, and the convolutional neural network adopts the FasterR-CNN network structure; the global word vector model in step S2 adopts the GloVe model.

[0076] The specific steps of step S2 are as follows:

[0077] S2-1. Segment and vectorize the m words in the question text to obtain the question text feature vector;

[0078] S2-2. Fill the question text feature vector with 0 to obtain a dimension of M QUES×EMB_SIZE vector representation in, Represents the vector representation of the question text at time t, M QUES Indicates the number of question words, EMB_SIZE indicates the embedding dimension of the question;

[0079] S2-3, introduce a bidirectional gated recurrent unit, according to the formula:

[0080]

[0081]

[0082] Get the bidirectional hidden state at time t, that is, the forward hidden state and the reverse hidden state Among them, GRU(·) represents the recurrent neural network model, represents the forward hidden state at time t-1, Represents the reverse hidden state at time t+1;

[0083] S2-4. Concatenate the question text feature vectors at all times to obtain the final question high-level feature vector

[0084] like Figure 2 As shown, the specific steps of step S3 are as follows:

[0085] S3-1. Initialize the model parameters of the image question answering network and input the image feature vector and the question high-level feature vector into the image question answering network;

[0086] S3-2. Train the image question answering network model, set the preset answer category as the training target, train the image question answering network through the back propagation algorithm and stochastic gradient descent, adjust the parameters of the image question answering network, and obtain the trained image question answering network; the parameters of the image question answering network include the learnable weight matrix W n and the bias term b n ;

[0087] S3-3. Construct the image modality intra-unit attention module and the question modality intra-unit attention module. Select one of the modality intra-unit attention modules and use the formula:

[0088] Q,K,V=trans(X)

[0089] Get the query vector Q, key vector K, and value vector V; where n represents the number of words in the question or the number of objects in the image, EMB_DIM represents the embedding dimension of the question, X represents the feature vector of a certain mode with dimension n×EMB_DIM, and trans(·) represents the conversion of the feature vector X into a multi-headed feature vector; the feature vector X includes the image feature vector and the question high-level feature vector;

[0090] S3-4, according to the formula:

[0091]

[0092] Get the attention score matrix S; where K T represents the transposed matrix of the key vector K, d k Indicates the EMB_DIM size of the query vector Q;

[0093] S3-5, according to the formula:

[0094] A=softmax(S), S∈R MH×n×n

[0095] Get the attention weight matrix A; where softmax(·) represents the normalized exponential function, R represents a real number, and MH represents the number of attention heads;

[0096] S3-6. Convert the multi-head feature vector to obtain a multi-head feature vector of the same dimension as the original input, according to the formula:

[0097] O=trans'(A·V)

[0098] O' = LayerNorm(O + Dropout(O))

[0099] FFN(O')=max(0,O'W1+b1)W2+b2

[0100] O I =LayerNorm(O'+Dropout(FFN(O')))

[0101] Get the intra-modal unit attention matrix O I, that is, the intra-modal unit attention feature of the modality; where trans'(·) represents the dimension conversion function, O represents the initial intra-modal unit attention matrix, O' represents the intermediate intra-modal unit attention matrix, Dropout(·) represents random inactivation, LayerNorm(·) represents the normalization function, FFN(·) represents the feedforward neural network, max(·) represents the implementation of the ReLU activation function, W1 represents the learnable weight matrix from the input layer to the hidden layer, W2 represents the learnable weight matrix from the hidden layer to the output layer, b1 represents the bias term from the input layer to the hidden layer, and b2 represents the bias term from the hidden layer to the output layer;

[0102] S3-7, repeat steps S3-3 to S3-6 to obtain the intra-modal unit attention matrix O of another modality i ;

[0103] S3-8. Construct inter-modal interactive attention, taking the features of one modality as the query and the features of the other modality as the key value, according to the formula:

[0104]

[0105] Get the attention weight matrix A1; where Q1 represents the query vector of one modality feature, K1 represents the key vector of another modality feature, and K1 T represents the transposed matrix of the key vector K1, d1 represents the EMB_DIM size of the query vector Q1, and V1 represents the value vector of another modal feature;

[0106] S3-9, according to the formula:

[0107] O1=LayerNorm(A1+Dropout(A1))

[0108] O A =LayerNorm(O1+Dropout(FFN(O1)))

[0109] Get the inter-modal unit attention matrix O A , that is, the inter-modal unit attention feature; where O1 represents the initial inter-modal unit attention matrix.

[0110] The loss function used by the back propagation algorithm in step S3-2 is the binary cross entropy loss function.

[0111] like Figure 2 As shown, the specific steps of step S4 are as follows:

[0112] S4-1, according to the formula:

[0113]

[0114] The inter-modal interactive attention feature XO guided by image features and the inter-modal interactive attention feature YO guided by question features are obtained; where CR(·) represents the inter-modal interactive attention, X' represents the unit attention feature within the image modality, and Y' represents the unit attention feature within the question modality;

[0115] S4-2, according to the formula:

[0116]

[0117] Obtain the inter-modal interactive attention feature X[i] guided by image features after the depth stacking of the i-th hidden layer and the inter-modal interactive attention feature Y[i] guided by question features after the depth stacking of the i-th hidden layer; where i represents the i-th hidden layer, XO[i-1] represents the inter-modal interactive attention feature guided by image features of the i-1-th hidden layer, and YO[i-1] represents the inter-modal interactive attention feature guided by question features of the i-1-th hidden layer;

[0118] S4-3, according to the formula:

[0119]

[0120] The output image feature x[i] of the i-th hidden layer and the output question feature y[i] of the i-th hidden layer are obtained; where X[i-1] represents the inter-modal interactive attention feature guided by the image feature after the depth stacking of the i-1-th hidden layer, and Y[i-1] represents the inter-modal interactive attention feature guided by the question feature after the depth stacking of the i-1-th hidden layer. i represents the image trainable weight variable of the i-th hidden layer, β i represents the problem trainable weight variable of the i-th hidden layer.

[0121] The formula of the linear transformation mapping in step S5 is as follows:

[0122] O F =proj(concat(x[I],y[I]))

[0123] Among them, O F represents the answer, x[I] represents the output image feature x[i] of the last hidden layer, y[I] represents the output question feature of the last hidden layer, concat(·) represents the concatenation operator, and proj(·) represents the linear transformation mapping function.

[0124] In summary, the present invention can achieve two-way guidance between image and question features and improve the model's interactive ability by constructing a model for image-text feature extraction and deep fusion based on modal joint interaction and introducing a mechanism for modal joint interaction; and enhance information sharing in cross-modal semantic space by using a residual deep stacking fusion mechanism. The modal bidirectional guidance mechanism of the present invention takes into account the deep interaction between the two modalities, and adopts forward and reverse joint guidance of the unit attention features within the two modalities, thereby enhancing the model's multimodal interaction ability and improving the answer classification effect; the residual stacking fusion mechanism of the present invention adopts a deep stacking method to allow the features after bidirectional guidance to further interact; the design of its residual dynamic mechanism improves the expressive ability and avoids the problem of vanishing gradients of deep neural networks during training, thereby improving the generalization of the model.

Claims

1. An image question answering method based on joint modality interaction, characterized by: The following steps are involved: S1. Initialize the object detection network with convolutional neural network as the core through the pre-trained residual network to obtain the dynamic features of the input image; fill the dynamic features of the input image to obtain the image feature vector; S2. Use the pre-trained global word vector model to vectorize the question text to obtain the question text feature vector; perform feature processing on the question text feature vector to obtain the question high-level feature vector; S3. Construct an image question answering network with intra-modality unit attention and inter-modality interactive attention as the core; input the image feature vector and the question high-level feature vector into the image question answering network to obtain the intra-modality unit attention feature and the inter-modality unit attention feature; S4. Deeply fuse the intra-modality unit attention features and inter-modality interaction attention features through bidirectional joint interaction of modalities and residual stacking to obtain the output image features and output question features of different hidden layers; S5. Merge the output image features and the output question features of the last hidden layer through feature stacking to obtain the final features. Map the final features to a multi-category vector through linear transformation to predict the answer. The specific steps of step S3 are as follows: S3-1. Initialize the model parameters of the image question answering network and input the image feature vector and the question high-level feature vector into the image question answering network; S3-2. Train the image question answering network model, set the preset answer categories as training targets, train the image question answering network through back propagation algorithm and stochastic gradient descent, adjust the parameters of the image question answering network, and obtain the trained image question answering network; the parameters of the image question answering network include the learnable weight matrix and bias ; S3-3. Construct the image modality intra-unit attention module and the question modality intra-unit attention module. Select one of the modality intra-unit attention modules and use the formula: Get the query vector Q, key vector K and value vector V; where, The number of words representing the question or the number of objects in the image, represents the embedding dimension of the problem, and X represents the dimension The eigenvector of a certain mode of Indicates converting the feature vector X into a multi-head feature vector; the feature vector X includes the image feature vector and the problem high-level feature vector; S3-4, according to the formula: Get the attention score matrix ;in, represents the transposed matrix of the key vector K, Represents the query vector Q size; S3-5, according to the formula: Get the attention weight matrix ;in, represents the normalized exponential function, R represents a real number, and MH represents the number of attention heads; S3-6. Convert the multi-head feature vector to obtain a multi-head feature vector of the same dimension as the original input, according to the formula: Get the intra-modal unit attention matrix , that is, the intra-modal unit attention feature of the modality; among them, represents the dimension conversion function, represents the initial intra-modal unit attention matrix, represents the intermediate intra-modal unit attention matrix, represents random dropout, represents the normalization function, represents a feedforward neural network, Indicates the implementation of ReLU activation function, represents the learnable weight matrix from the input layer to the hidden layer, represents the learnable weight matrix from the hidden layer to the output layer, represents the bias from the input layer to the hidden layer, Represents the bias term from the hidden layer to the output layer; S3-7. Repeat steps S3-3 to S3-6 to obtain the intra-modal unit attention matrix of another modality ; S3-8. Construct inter-modal interactive attention, taking the features of one modality as the query and the features of the other modality as the key value, according to the formula: Get the attention weight matrix ;in, A query vector representing a modality feature, represents the key vector of another modality feature, Represents the key vector The transposed matrix of Represents the query vector of size, A vector of values ​​representing another modal feature; S3-9, according to the formula: Get the inter-modal unit attention matrix , that is, the inter-modal unit attention feature; among them, represents the initial inter-modal unit attention matrix.

2. The image question answering method based on modality joint interaction according to claim 1, characterized in that: The pre-trained residual network in step S1 adopts the ResNet-101 network structure, and the convolutional neural network adopts the Faster R-CNN network structure; the global word vector model in step S2 adopts the GloVe model.

3. The image question answering method based on modality joint interaction according to claim 1, characterized in that: The specific steps of step S2 are as follows: S2-1. Segment and vectorize the m words in the question text to obtain the question text feature vector; S2-2. Fill the question text feature vector with 0 to obtain a dimension of Vector representation of ;in, Represents the vector representation of the question text at time t, Indicates the number of question words, Represent the embedding dimension of the problem; S2-3, introduce a bidirectional gated recurrent unit, according to the formula: Get the bidirectional hidden state at time t, that is, the forward hidden state and the reverse hidden state ;in, represents a recurrent neural network model, represents the forward hidden state at time t-1, Represents the reverse hidden state at time t+1; S2-4. Concatenate the question text feature vectors at all times to obtain the final question high-level feature vector .

4. The image question answering method based on modality joint interaction according to claim 1, characterized in that: The loss function used by the back propagation algorithm in step S3-2 is a binary cross entropy loss function.

5. The image question answering method based on modality joint interaction according to claim 1, characterized in that: The specific steps of step S4 are as follows: S4-1, according to the formula: Obtain inter-modal interactive attention features guided by image features and inter-modal interactive attention features guided by question features ;in, represents the interactive attention between modalities, represents the unit attention feature within the image modality, Represents the attention characteristics of units within the question modality; S4-2, according to the formula: Get the first Intermodal interactive attention features guided by image features after deep stacking of hidden layers Hedi Intermodal interactive attention features guided by question features after deep stacking of hidden layers ;in, Indicates the hidden layers, Indicates the The inter-modal interactive attention features guided by image features in the hidden layer, Indicates the The inter-modal interactive attention features guided by question features in the hidden layer; S4-3, according to the formula: Get the first The output image features of the hidden layer Hedi The output problem features of the hidden layer ;in, Indicates the The inter-modal interactive attention features guided by image features after the depth stacking of hidden layers, Indicates the The inter-modal interactive attention features guided by question features after the deep stacking of hidden layers, Indicates the The image-trainable weight variables of the hidden layers are Indicates the The problem of hidden layers is to train the weight variables.

6. The image question answering method based on modality joint interaction according to claim 1, characterized in that: The formula of the linear transformation mapping in step S5 is as follows: in, Indicates the answer, Represents the output image features of the last hidden layer , represents the output problem feature of the last hidden layer, represents the concatenation operator, Represents a linear transformation mapping function.

Images