An image question answering method based on attention pre-training

Abstract

This invention discloses an image question answering method based on attention pre-training. The method comprises three modules: width attention, depth attention, and attention pre-training. The width attention module models the feature associations between the question sentence and image features at different levels, exploring cross-modal feature interactions from different perspectives. The depth attention module refines important image regions related to the question through repeated attention calculations to gain a deeper understanding of the association between the image and the question. The attention pre-training module uses annotated attention distribution maps as prior knowledge to supervise the learning of width and depth attention, achieving weight initialization for both. This invention uses manually annotated attention distribution maps as prior knowledge to guide the learning of width and depth attention, and initializes the corresponding attention models through pre-training, significantly improving the model's convergence speed and the accuracy of answer prediction.

Description

Technical Field

[0001] This invention relates to an image question answering method based on attention pre-training, belonging to the technical fields of multimodal data analysis and intelligent question answering. Background Technology

[0002] The rapid development of the internet and the widespread adoption of mobile smart devices have greatly facilitated people's ability to take photos and record their lives anytime, anywhere. Currently, hundreds of millions of images are taken and uploaded to various social networking platforms every day to showcase personal experiences or lifestyles. Against this backdrop, understanding the massive amounts of image content uploaded online and, in turn, providing smarter and more intelligent life services has attracted widespread attention. Image question answering, as an important branch of intelligent question answering and a crucial method for understanding image content, has sparked a tremendous research boom in both academia and industry since its inception. However, because images and questions belong to different modalities, a significant semantic gap exists between cross-modal data, making image question answering a challenging but promising task. Breakthroughs in image question answering technology can not only promote the development of cross-modal data analysis but also have wide applications in human-computer interaction, self-service customer service, and other scenarios, holding significant importance for the development of related fields.

[0003] Current image question answering methods primarily employ attention mechanisms to model the association between images and text, learning an effective image-question joint embedding for answer prediction. However, existing attention mechanisms suffer from several shortcomings. Firstly, cross-modal feature association learning only models the image and question sentence features from a single dimension and perspective, failing to conduct fine-grained interactions between the two types of features from multiple angles and depths, thus hindering the full exploitation of cross-modal feature associations. Secondly, current attention mechanisms rely on implicit, unsupervised cross-modal feature interactions, learning feature associations solely based on information fed back from the answer classifier. This approach lacks strong prior knowledge guidance, making it difficult to define the learning objective of attention. Therefore, an effective image question answering method is urgently needed to capture fine-grained interactions between images and question sentences from multiple perspectives, while fully utilizing human prior knowledge for supervised attention guidance, thereby improving the accuracy of the model's question answers. Summary of the Invention

[0004] The technical problem this invention aims to solve is to overcome the shortcomings of existing image question answering technologies and provide an image question answering method based on attention pre-training. This method first utilizes breadth attention to explore the interaction between the question sentence and image features at different levels; then, it uses depth attention to progressively refine the image regions related to the question; next, it initializes the weights of breadth and depth attention through an attention pre-training model; finally, it establishes a classifier to predict the answer.

[0005] The technical solution adopted in this invention is as follows: an image question answering method based on attention pre-training, which mainly includes three parts: width attention, depth attention, and attention pre-training. Width attention extracts complex visual images into three levels of features and uses a visual attention mechanism to model the relationship between the question sentence and the three types of image features, thereby exploring cross-modal feature interactions from different perspectives. Depth attention refines the image region features related to the question through repeated attention calculations to gain a deeper understanding of the fine-grained feature relationships between the question sentence and the image. The attention pre-training part uses manually labeled attention distribution maps as prior knowledge and extracts them as prior attention vectors to supervise the learning of the attention distribution jointly learned by width attention and depth attention. Supervised attention learning achieves attention pre-training, and the trained model weights are used to initialize the weights of width attention and depth attention, thereby improving the model's convergence speed and the accuracy of answer prediction.

[0006] The present invention provides an image question answering method based on attention pre-training, the specific steps of which are as follows:

[0007] Step 1: Use ResNet network to extract the region feature matrix 1 from the image. On this basis, use stacked self-attention network to extract image features 2 and image features 3. At the same time, use GloVe word vectors and LSTM network to encode the question sentence into question feature vector. Design a visual attention network to model the feature association between question features and the three types of image features respectively, and generate joint features of image width.

[0008] Step 2: Utilize a memory network to calculate the deep interaction between the joint features of the question sentence and the image width, thereby gradually refining the image regions related to the question. The memory network repeatedly calculates the attention distribution of the question sentence on the joint features of the image width and updates the attention memory units to achieve a fine-grained understanding of the association between the question and the image.

[0009] Step 3: Use the manually labeled attention distribution map as prior knowledge and extract it as attention distribution vector. Use this vector to perform supervised pre-training on the attention distribution jointly learned from width attention and depth attention. Use the width and depth attention weights obtained from the pre-training to initialize the corresponding attention model, thereby improving the convergence speed of the model and the accuracy of answer prediction.

[0010] Step 4: Construct a multi-class answer classifier using softmax, and use the question-image joint embedding generated by deep attention as the input of the classifier to predict the question answer.

[0011] In step one, image feature 2 is obtained by modeling image feature 1 using a self-attention network, and image feature 3 is obtained by modeling image feature 2 using a self-attention network. Image feature 1, image feature 2 and image feature 3 represent image features at different levels, and they reflect complex visual image information from multiple perspectives.

[0012] In step two, the memory network is a neural network-based attention mechanism. It maps the joint features of the image width to memory cells and updates the values ​​in the memory cells by calculating the attention weight distribution of the problem feature vector on the memory cells. The attention calculation and memory update are repeated multiple times to achieve fine-grained interaction between the problem and the image, so as to gain a deeper understanding of the problem and image content.

[0013] In step three, the manually annotated attention distribution map is a layer obtained by the annotator manually sharpening the image areas that help answer the questions, given an image and image-related questions. The manually annotated attention distribution is the target of width attention and depth attention learning, and it is used as prior knowledge to guide the learning of attention mechanisms.

[0014] In step four, the answer classifier is a multi-class classifier constructed by combining a multilayer perceptron with a softmax activation function. Each classifier item corresponds to a candidate answer. In addition, the classifier uses cross-entropy as the loss function and takes the image-question joint embedding learned by deep attention as input for answer prediction.

[0015] The beneficial effects of this invention are as follows: Current image question answering methods mainly model the feature interactions between questions and images from a single dimension and perspective, making it difficult to fully capture cross-modal information associations. Furthermore, existing attention mechanisms lack strong prior knowledge guidance, making it difficult to clearly define the learning objectives of attention, often leading the model to focus on image regions unrelated to the question. In contrast, the image question answering method based on attention pre-training proposed in this invention has the following two advantages: First, this invention explores multi-level, fine-grained interactions between questions and images from both breadth and depth dimensions. By representing image features as multiple levels and performing in-depth iterative cross-modal feature association learning, the model learns the joint embedding of image and question to predict answers, based on a full understanding of the content information of both the image and the question. Second, this invention introduces an attention pre-training method, using manually labeled attention distribution maps as prior knowledge to guide breadth and depth attention learning. Pre-training initializes the corresponding attention model, significantly improving the model's convergence speed and the accuracy of answer prediction. Attached Figure Description

[0016] Figure 1This is a flowchart illustrating an image question-answering method based on attention pre-training. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0018] Example 1: As Figure 1 As shown, an image question answering method based on attention pre-training aims to predict the answer to a question given an image and a question related to the image content. The specific implementation steps are as follows:

[0019] Step 1: Broad Attention

[0020] For a given image, this invention first uses a ResNet network to encode the image information into an image region feature matrix 1, denoted as R1; then, R1 is used as input to a self-attention network to model its internal dependencies and output an image region feature matrix 2, denoted as R2; next, R2 is used as input to the self-attention network to output an image region feature matrix 3, denoted as R3, where R... k ={r k1 ,...,r ki ,...,r k(m×m)}, 1≤k≤3, and r ki R represents k The feature vector corresponding to the i-th image region is given by m×m, where m×m is the number of image regions. For the question sentence related to the image, each word in the question sentence is encoded using GloVe word vectors, and the word sequence is modeled using LSTM. The last output vector q of LSTM is used to represent the feature vector of the question.

[0021] This invention utilizes a visual attention mechanism to model the feature associations between three types of image region features represented by width and question features. For ease of computation, the image region feature matrix and question vector are first mapped to a c-dimensional target space. This process can be described as follows: q (c) =σ(W q q+b q ), where W r and W q It is a learnable weight matrix, b r and b q This is the learnable bias term, and σ is the non-linear activation function ReLU; then, q (c) Perform spatial copying to transform it into matrix Q (c) And the size of the spatial structure of the matrix is ​​the same as that of the matrix. Consistent, all being c×(m×m); then, the present invention will With Q (c)fused by element-wise multiplication Based on this, a 1×1 convolution kernel and a softmax function are used to calculate the attention distribution of the problem on the image region. This process can be represented as: in The symbol represents element-wise multiplication, * represents convolution, and W represents element-wise multiplication. α It is a learnable convolutional kernel, b α This is a learnable bias term; subsequently, the attention distribution is used to weight and model the attention intensity in different image regions: Where α ki Represents the attention distribution α k The i-th element, Finally, the computation results of the three visual attention mechanisms corresponding to the three image features are fused element-wise to generate joint image width features.

[0022] Step Two: Deep Attention

[0023] Building upon width attention, this invention performs in-depth problem and image feature association mining to model fine-grained cross-modal feature interactions. First, a fully connected neural network is used to map the joint features of image width. Shoot as memory unit M R Next, calculate the problem feature q. (c) With memory unit M R The similarity is used, and the softmax function is used to calculate the problem for memory unit M. R Attention distribution α m Then, the present invention further incorporates α m With memory unit M R Multiply the components and map the result through a fully connected neural network to the memory unit M. R The update process can be represented as follows: α m =softmax(q) (c) M R M R ←(α m ⊙M R W m +b m Among them, W m and b m... Where 1≤t≤T, the superscript t represents the value of the t-th calculation, after T rounds of calculation. This can be viewed as calculating the attention distribution of an image using depth attention, building upon width attention. This can be viewed as image features related to the question after deep interaction between the question and the image; finally, this invention will use the question feature q (c) The matrix Q obtained by performing spatial copying (c) and The process involves fusion and mapping to generate a joint embedding vector v between the problem and the image. This process can be represented as follows: Among them, W v and b v Let represent the learnable weight matrix and bias term, respectively, and σ be the non-linear activation function ReLU.

[0024] Step 3: Attention Pre-training

[0025] To fully leverage prior human attention knowledge to guide the learning of breadth and depth attention, this invention first obtains an attention-annotated image-question dataset through crowdsourcing. During this process, annotators are instructed to sharpen image regions that aid in answering questions. The manually sharpened layers can be viewed as the attention distribution during human image-based question answering, and this attention distribution serves as the learning objective for the breadth and depth attention techniques designed in this invention. Therefore, this invention extracts the manually annotated attention distribution from the labeled dataset, using it as prior knowledge, and then utilizes supervised machine learning to guide the learning of breadth and depth attention. Specifically, this invention employs a ResNet network to extract an attention distribution vector α from the manually annotated attention map. h The length of this vector is equal to the number of image regions, and each element represents the human attention value for the corresponding image region. Because this invention learns a problem-oriented attention distribution vector for the image through joint learning of width attention and depth attention, it also acquires this vector. Therefore, supervised learning can be used to train both breadth and depth attention simultaneously, in order to force... The value is infinitely close to α hThe supervised pre-training process uses mean squared error as the loss function. Through attention pre-training, the breadth and depth attention are guided by prior knowledge of human attention to learn a more effective attention distribution. Finally, this invention uses the model weights obtained from attention pre-training to initialize the weights of depth and breadth attention, thereby improving the model's convergence speed and the accuracy of answering questions.

[0026] Step 4, Answer Prediction:

[0027] This invention employs a multilayer perceptron to construct a softmax multi-class classifier, using the question-image joint embedding vector v obtained in step two as the classifier's input. Each classification item in the multi-class classifier corresponds to an answer in the candidate answer set. During training, this invention uses cross-entropy as the loss function for model training. In actual prediction, the candidate option with the highest classification probability is selected as the answer to the question in the image-based question, thereby achieving answer prediction.

[0028] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection of the claims.

Claims

1. An attention-based pre-training picture question and answer method, characterized in that: The method comprises the following steps: A. width attention: encode the visual picture into hierarchical picture features 1, picture features 2 and picture features 3 by using a residual neural network and a stacked self-attention network to represent the picture information in the width, meanwhile, encode the text question into question features by using a Glove word vector and a long short-term memory network, and calculate the feature correlation between the question features and the picture features 1, picture features 2 and picture features 3 respectively by using a visual attention mechanism to generate picture width joint features; B. depth attention: perform attention calculation on the picture width joint features generated by the width attention and the question features by using a memory memory network, and the attention distribution calculation and memory update are repeated multiple times in the process; C. attention pre-training: use the manually annotated attention distribution map as prior knowledge, and extract it into an attention vector by using a residual neural network; use the prior attention vector to supervise the learning of the attention distribution vector learned by the width attention and the depth attention; and initialize the weights of the width attention and the depth attention by attention pre-training; D. Answer Prediction: A classifier is constructed using a multilayer perceptron and cross-entropy is used as the classification loss function. The question-image joint embedding learned through deep attention is used as the classifier input to predict the question answer. In step A, image feature 1 is the image region feature matrix extracted using a ResNet network; image feature 2 is the feature matrix obtained by modeling image feature 1 using a self-attention mechanism; and image feature 3 is the image feature matrix generated by modeling image feature 2 using a self-attention mechanism. The specific method is as follows: First, the image information is encoded into image region feature matrix 1 using a ResNet network, denoted as R1; then, R1 is used as the input of a self-attention network to model its internal dependencies and output image region feature matrix 2, denoted as R2; then, R2 is used as the input of a self-attention network to output image region feature matrix 3, denoted as R3, where R... k ={r k1 ,...,r ki ,...,r k(m×m) }, 1≤k≤3, and r ki R represents k The feature vector corresponding to the i-th image region is m×m, where m×m is the number of image regions. For the image-related question sentence, each word in the question sentence is encoded using GloVe word vectors, and the word sequence is modeled using LSTM. The last output vector q of LSTM is used to represent the feature vector of the question. In step A, the width joint feature is a weighted fusion of three image feature matrices. The weight of each image feature matrix is ​​an attention vector, which is obtained by visual attention calculation between the corresponding image feature and the question feature. The method of modeling the feature association between the three image region features represented by the width and the question feature using the visual attention mechanism is as follows: First, the image region feature matrix and the question vector are mapped to a c-dimensional target space. This process can be described as follows: q (c) =σ(W q q+b q ), where W r and W q It is a learnable weight matrix, b r and b q This is the learnable bias term, and σ is the non-linear activation function ReLU; then, q (c) Perform spatial copying to transform it into matrix Q (c) And the size of the spatial structure of the matrix is ​​the same as that of the matrix. Consistent, all being c×(m×M); then, the present invention will With Q (c) fused by element-wise multiplication Based on this, a 1×1 convolution kernel and a softmax function are used to calculate the attention distribution of the problem on the image region. This process can be represented as: in Indicates element-wise multiplication, * indicates convolution operation, W α It is a learnable convolutional kernel, b α This is a learnable bias term; subsequently, the attention distribution is used to weight and model the attention intensity in different image regions: Where α ki Represents the attention distribution α k The i-th element, Finally, the computation results of the three visual attention mechanisms corresponding to the three image features are fused element-wise to generate joint image width features. In step B, deep attention involves performing deep cross-modal feature interaction between the joint features of image width and the problem features. This process is implemented using an iterative memory network. By repeatedly calculating the attention distribution of the problem on the image and updating the memory units, the model gradually refines the features of important image regions, capturing the deep correlation between the problem and the image. Specifically, the method is as follows: First, a fully connected neural network is used to map the joint features of image width... Shoot as memory unit M R Next, calculate the problem characteristic q. (c) With memory unit M R The similarity is used, and the softmax function is used to calculate the problem for memory unit M. R Attention distribution α m Then, the present invention further incorporates α m With memory unit M R Multiply the components and map the result through a fully connected neural network to the memory unit M. R The update process can be represented as follows: α m =softmax(q) (c) M R M R ←(α m ⊙M R W m +b m Among them, W m and n m ... Where 1≤t≤T, the superscript t represents the value of the t-th calculation, after T rounds of calculation. This can be viewed as calculating the attention distribution of an image using depth attention, building upon width attention. This can be viewed as image features related to the question after deep interaction between the question and the image; finally, this invention will use the question feature q (c) The matrix Q obtained by performing spatial copying (c) and The process involves fusion and mapping to generate a joint embedding vector v between the problem and the image; this process can be represented as follows: Among them, W v and b v Let represent the learnable weight matrix and bias term, respectively, and σ be the non-linear activation function ReLU; The weight initialization in step C is to initialize the weights of the corresponding attention after the supervised pre-training of the width and depth attention by using the manually annotated attention prior knowledge, and then fine-tune the weights again in the whole model training process; the answer prediction in step D is realized by a softmax multi-class classifier, the classes of the classifier correspond to the answers in the candidate answer set, the classifier takes the picture-question joint features as input, and predicts the answer distribution probability by minimizing the cross-entropy loss function.

2. The attention-based pre-training picture question answering method according to claim 1, characterized in that: The manually annotated attention in step C is to sharpen the picture area that helps to correctly answer the question under the given question sentence and corresponding picture, and the layer annotated by the annotator is regarded as the prior knowledge of human visual question answering and the learning goal of the attention model.

3. The attention-based pre-training picture question answering method according to claim 1, characterized in that: The supervised learning in step C is between two attention distribution vectors, the supervised vector in the supervised learning is the attention distribution vector learned by the depth attention on the basis of the width attention, and the prior attention vector used for supervised learning is extracted from the manually annotated attention distribution map.

Images