Visual question-answering method and system adopting cascade question-answering model
By using a cascaded question-answering model that combines image and text information, and employing Faster R-CNN and attention mechanisms, the problem of insufficient information fusion in existing visual question-answering models is solved, resulting in a highly accurate visual question-answering system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 石帅
- Filing Date
- 2023-11-16
- Publication Date
- 2026-04-17
AI Technical Summary
Existing visual question answering models typically perform answer reasoning in a single-stage manner, neglecting the rich semantic information of the predicted answer and struggling to effectively integrate key information from the image and the question.
A cascaded question-answering model is adopted, which extracts image features through a convolutional neural network, uses Faster RCNN to detect salient target regions, and combines question self-attention and layer-by-layer feedforward attention mechanisms to focus on key information in the image and question, generate fused features of image and text, and finally predict the answer through a probability distribution model.
It improves the accuracy and robustness of visual question answering models, enabling them to better understand the semantic information of questions and images and achieve highly accurate answer prediction.
Smart Images
Figure CN121880490A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence networks, and specifically relates to a visual question answering method and system that employs a cascaded question answering model. Background Technology
[0002] Visual content description and visual question answering are popular research areas in computer vision and natural language processing. Visual content description aims to generate sentences using natural language to describe the content of visual images, while visual question answering aims to answer natural language questions related to the content of images. In visual question answering tasks, classic models typically involve two steps: processing the image and question using convolutional neural networks and recurrent neural networks to extract relevant features; and inferring the answer by fusing the features of the question and the image. To better understand the content of the question and the image, many methods utilize attention mechanisms to focus on important regions in the image and keywords in the question. Some methods use co-attention mechanisms to simultaneously focus on key information in both the question and the image. Additionally, some methods propose bilinear networks to effectively fuse image and question features. Another method uses Tucker tensor decomposition, which reduces model parameters and effectively represents the bilinear relationship between vision and language, achieving good performance. Early visual question answering methods generated the probability of each answer based on given knowledge and selected the answer with the highest probability as the predicted answer. These methods typically perform answer reasoning in a single-stage manner, making only one answer prediction and thus ignoring the information in the predicted answer. However, these answers often contain rich semantic information, which can improve the prediction accuracy of visual question answering models. Furthermore, predicted answers with high probability are highly correlated with the question-image pair and may even contain the correct answer. Therefore, utilizing predicted candidate answers is crucial for visual question answering models. The above summarizes the relevant methods and techniques for visual content description and visual question answering. Summary of the Invention
[0003] The purpose of this invention is to provide a visual question answering method that employs a cascaded question answering model.
[0004] A visual question answering method employing a cascaded question answering model includes the following steps:
[0005] (1) Input the original image and the natural language question. The preprocessing module performs cropping and scaling preprocessing on the original image and word segmentation and part-of-speech tagging preprocessing on the natural language question. The preprocessed image and question are output.
[0006] (2) Input the preprocessed image into the visual model, use a convolutional neural network to extract features from the image, and generate the feature representation of the image;
[0007] (3) Input the preprocessed question, the question model transforms the question into a word sequence representation, and further generates a global representation of the question;
[0008] (4) The answer model assigns attention weights to each word to focus on relevant regions in the image. These weights are determined by two attention mechanisms: layer-wise feedforward attention to extract key words in the problem and self-attention to generate word vector representations. Then, the word vector representations and image feature representations are fused to obtain the association between each word and the relevant target region in the image.
[0009] (5) The global representation of the input question, the association from words to region attention network output, combined with visual and textual information, and the probability distribution model are used to predict the final answer to the question.
[0010] The convolutional neural network described uses a pre-trained Faster R-CNN network to detect salient target regions in an image and extract visual features from each region; for each image, the Faster R-CNN network selects X target regions, and the image features of each region are... The set is:
[0011] The problem model extracts word sequence representations and a global representation of the problem, encoding the words in the problem into one-hot word vectors:
[0012] K = {k1,k2,…,k} a ,…,k A};
[0013] A is the length of the problem, k a It is the one-hot word vector of the a-th word. Each word is mapped to a word embedding matrix Z with dimension q. j vector
[0014]
[0015] The problem representation is extracted using a GRU network, and the output of the last hidden layer is selected as the global representation of the problem.
[0016] d = GRU(I j ).
[0017] The process of obtaining the association between each word and the relevant target region in the image includes:
[0018] Determine the first learning parameter J in the word attention network ju ;
[0019] Identify the intermediate representations of the problem words:
[0020] uj =J ju I j ;
[0021] Determine the second learning parameter J in the word attention network jc ;
[0022] Generate word attention maps:
[0023] c j =softmax(J jc u j );
[0024] Get the complete word representation:
[0025]
[0026] in yes The v-th element in It is C j The v-th element in the expression represents the weight of the v-th word;
[0027] Determine the first learned parameter J in the image attention network bu The second learning parameter J du and the third learning parameter J bc ;
[0028] Determine the intermediate representation of the problem image:
[0029] u β =η(J bu I 0 )⊙η(J du μ)I T ;
[0030] η is the ReLU activation function, ⊙ represents bitwise multiplication, and I represents a vector with all elements being 1;
[0031] Generate image attention map:
[0032] c β =softmax(J bc u β );
[0033] The representation vector of the generated image:
[0034]
[0035] It is C β The vth element;
[0036] Determine the first learning parameter J in the answer model. bsSecond learning parameter J ds ;
[0037] By fusing image representations with language representations, fused features are generated:
[0038] s=(J bs Ω)⊙η(J ds d)I T .
[0039] Step (5) includes fusing features through a fully connected layer and a softmax operation to predict the probability that each candidate answer is the correct answer:
[0040] c n =softmax(J n s);
[0041] J n These are the first learned parameters for the fully connected layer;
[0042] Finally, the answer with the highest predicted probability is selected as the predicted answer.
[0043] The present invention also aims to provide a visual question answering system employing a cascaded question answering model, comprising:
[0044] (1) Preprocessing module: receives the input original image and natural language question, performs cropping and scaling preprocessing on the original image, performs word segmentation and part-of-speech tagging preprocessing on the natural language question, and outputs the preprocessed image and question;
[0045] (2) The vision module receives the preprocessed image, uses a convolutional neural network to extract features from the image, and generates a feature representation of the image. The convolutional neural network uses a pre-trained Faster R-CNN network to detect salient target regions in the image and extract visual features from each region. For each image, the Faster R-CNN network selects X target regions, and the image features of each region are... The set is:
[0046] (3) The question module receives the preprocessed question, transforms the question into a word sequence representation, and further generates a global representation of the question; the question module extracts the word sequence representation and the global representation of the question, and encodes the words in the question into one-hot word vectors:
[0047] K = {k1,k2,…,k} a ,…,k A};
[0048] A is the length of the problem, k aIt is the one-hot word vector of the a-th word. Each word is mapped to a word embedding matrix Z with dimension q. j vector
[0049]
[0050] The problem representation is extracted using a GRU network, and the output of the last hidden layer is selected as the global representation of the problem.
[0051] d = GRU(I j );
[0052] (4) The answer module assigns attention weights to each word to focus on relevant regions in the image. These weights are determined by two attention mechanisms: layer-wise feedforward attention to extract key words in the question and self-attention to generate word vector representations. Then, the word vector representations and image feature representations are fused to obtain the association between each word and the relevant target region in the image.
[0053] The association between each word and the relevant target region in the image includes:
[0054] Determine the first learning parameter J in the word attention network ju ;
[0055] Identify the intermediate representations of the problem words:
[0056] u j =J ju I j ;
[0057] Determine the second learning parameter J in the word attention network jc ;
[0058] Generate word attention maps:
[0059] c j =softmax(J jc u j );
[0060] Get the complete word representation:
[0061]
[0062] in yes The v-th element in It is C j The v-th element in the expression represents the weight of the v-th word;
[0063] Determine the first learned parameter J in the image attention network bu The second learning parameter J duand the third learning parameter J bc ;
[0064] Determine the intermediate representation of the problem image:
[0065] u β =η(J bu I 0 )⊙η(J du μ)I T ;
[0066] η is the ReLU activation function, ⊙ represents bitwise multiplication, and I represents a vector with all elements being 1;
[0067] Generate image attention map:
[0068] c β =softmax(J bc u β );
[0069] The representation vector of the generated image:
[0070]
[0071] It is C β The vth element;
[0072] Determine the first learning parameter J in the answer module bs Second learning parameter J ds ;
[0073] By fusing image representations with language representations, fused features are generated:
[0074] s=(J bs Ω)vη(J ds d)I T ;
[0075] (5) Prediction module: The system uses the probability distribution module to predict the final answer to the question based on the global representation of the input question, the correlation between words and the output of the region attention network, and visual and textual information.
[0076] The fused features, through fully connected layers and softmax operations, predict the probability that each candidate answer is the correct answer:
[0077] c n =softmax(J n s);
[0078] J n These are the first learned parameters for the fully connected layer;
[0079] Finally, the answer with the highest predicted probability is selected as the predicted answer.
[0080] The beneficial effects of this invention are as follows:
[0081] This invention proposes a visual question answering method and system employing a cascaded question answering model. It effectively integrates visual attention and linguistic attention to fully extract important semantic information from the target region of the image and the question. Furthermore, a simple yet effective question self-attention mechanism allows for focused attention on important words in the question, acquiring relevant semantic information. Then, the representation obtained using the question self-attention mechanism can locate question-related regions in the image. Simultaneously, Faster R-CNN is used to extract target region features from the image, and an image attention network is used to establish a finer-grained association between the target region and the question words. Finally, the question representation, which integrates image representation and GRU network-extracted representations, is input into an answer classifier for answer prediction, improving reasoning ability and better understanding the semantic information in the question text and visual image. Therefore, this invention employs a cascaded question answering model, combining image and text information, and through the collaborative processing of multiple modules, achieves a highly accurate and robust visual question answering system. Attached Figure Description
[0082] Figure 1 This is a schematic diagram of the method of the present invention.
[0083] Figure 2 This is a schematic diagram of the system of the present invention.
[0084] Figure 3 This is the visualization result of the present invention. Detailed Implementation
[0085] The present invention will now be further described with reference to the accompanying drawings.
[0086] like Figure 1 As shown, a visual question answering method employing a cascaded question answering model includes the following steps:
[0087] (1) Input the original image and the natural language question. The preprocessing module performs cropping and scaling preprocessing on the original image and word segmentation and part-of-speech tagging preprocessing on the natural language question. The preprocessed image and question are output.
[0088] (2) Input the preprocessed image into the visual model, and use a convolutional neural network to extract features from the image to generate a feature representation of the image; the convolutional neural network uses a pre-trained Faster R-CNN network to detect significant target regions in the image and extract visual features from each region; for each image, the Faster R-CNN network is used to select X target regions, and the image features of each region are... The set is:
[0089] (3) Input the preprocessed question, the question model transforms the question into a word sequence representation, and further generates a global representation of the question; the question model extracts the word sequence representation and the global representation of the question, and encodes the words in the question into one-hot word vectors:
[0090] K = {k1,k2,…,k} a ,…,k A};
[0091] A is the length of the problem, k a It is the one-hot word vector of the a-th word. Each word is mapped to a word embedding matrix Z with dimension q. j vector
[0092]
[0093] The problem representation is extracted using a GRU network, and the output of the last hidden layer is selected as the global representation of the problem.
[0094] d = GRU(I j ).
[0095] (4) The answer model assigns attention weights to each word to focus on relevant regions in the image. These weights are determined by two attention mechanisms: layer-wise feedforward attention to extract key words from the problem and self-attention to generate word vector representations. Then, the word vector representations and image feature representations are fused to obtain the association between each word and the relevant target region in the image. The obtained association between each word and the relevant target region in the image includes:
[0096] Determine the first learning parameter J in the word attention network ju ;
[0097] Identify the intermediate representations of the problem words:
[0098] u j =J ju I j ;
[0099] Determine the second learning parameter J in the word attention network jc ;
[0100] Generate word attention maps:
[0101] c j =softmax(J jc u j );
[0102] Get the complete word representation:
[0103]
[0104] in yes The v-th element in It is C j The v-th element in the expression represents the weight of the v-th word;
[0105] Determine the first learned parameter J in the image attention network bu The second learning parameter J du and the third learning parameter J bc ;
[0106] Determine the intermediate representation of the problem image:
[0107] u β =η(J bu I 0 )⊙η(J du μ)I T ;
[0108] η is the ReLU activation function, ⊙ represents bitwise multiplication, and I represents a vector with all elements being 1;
[0109] Generate image attention map:
[0110] c β =softmax(J bc u β );
[0111] The representation vector of the generated image:
[0112]
[0113] It is C β The vth element;
[0114] Determine the first learning parameter J in the answer model. bs Second learning parameter J ds ;
[0115] By fusing image representations with language representations, fused features are generated:
[0116] s=(J bs Ω)⊙η(J ds d)I T .
[0117] (5) The global representation of the input question, the correlation between words and region attention network outputs, and the combination of visual and textual information are used to predict the final answer to the question using a probability distribution model. Specifically, this includes fusing features through fully connected layers and softmax operations to predict the probability that each candidate answer is the correct answer:
[0118] c n =softmax(J n s);
[0119] J n These are the first learned parameters for the fully connected layer;
[0120] Finally, the answer with the highest predicted probability is selected as the predicted answer.
[0121] like Figure 2 As shown, a visual question-answering system employing a cascaded question-answering model includes:
[0122] (1) Preprocessing module: receives the input original image and natural language question, performs cropping and scaling preprocessing on the original image, performs word segmentation and part-of-speech tagging preprocessing on the natural language question, and outputs the preprocessed image and question;
[0123] (2) The vision module receives the preprocessed image, uses a convolutional neural network to extract features from the image, and generates a feature representation of the image. The convolutional neural network uses a pre-trained Faster R-CNN network to detect salient target regions in the image and extract visual features from each region. For each image, the Faster R-CNN network selects X target regions, and the image features of each region are... The set is:
[0124] (3) The question module receives the preprocessed question, transforms the question into a word sequence representation, and further generates a global representation of the question; the question module extracts the word sequence representation and the global representation of the question, and encodes the words in the question into one-hot word vectors:
[0125] K = {k1,k2,…,k} a ,…,k A};
[0126] A is the length of the problem, k a It is the one-hot word vector of the a-th word. Each word is mapped to a word embedding matrix Z with dimension q. j vector
[0127]
[0128] The problem representation is extracted using a GRU network, and the output of the last hidden layer is selected as the global representation of the problem.
[0129] d = GRU(I j );
[0130] (4) The answer module assigns attention weights to each word to focus on relevant regions in the image. These weights are determined by two attention mechanisms: layer-wise feedforward attention to extract key words in the question and self-attention to generate word vector representations. Then, the word vector representations and image feature representations are fused to obtain the association between each word and the relevant target region in the image.
[0131] The association between each word and the relevant target region in the image includes:
[0132] Determine the first learning parameter J in the word attention network ju ;
[0133] Identify the intermediate representations of the problem words:
[0134] u j =J ju I j ;
[0135] Determine the second learning parameter J in the word attention network jc ;
[0136] Generate word attention maps:
[0137] c j =softmax(J jc u j );
[0138] Get the complete word representation:
[0139]
[0140] in yes The v-th element in It is C j The v-th element in the expression represents the weight of the v-th word;
[0141] Determine the first learned parameter J in the image attention network bu The second learning parameter J du and the third learning parameter J bc ;
[0142] Determine the intermediate representation of the problem image:
[0143] u β =η(J bu I0 )⊙η(J du μ)I T ;
[0144] η is the ReLU activation function, ⊙ represents bitwise multiplication, and I represents a vector with all elements being 1;
[0145] Generate image attention map:
[0146] c β =softmax(J bc u β );
[0147] The representation vector of the generated image:
[0148]
[0149] It is C β The vth element;
[0150] Determine the first learning parameter J in the answer module bs Second learning parameter J ds ;
[0151] By fusing image representations with language representations, fused features are generated:
[0152] s=(J bs Ω)⊙η(J ds d)I T ;
[0153] (5) Prediction module: The system predicts the final answer to the question by combining visual and textual information, based on the global representation of the input question, the correlation between words and the output of the region attention network, and a probability distribution model.
[0154] The fused features, through fully connected layers and softmax operations, predict the probability that each candidate answer is the correct answer:
[0155] c n =softmax(J n s);
[0156] J n These are the first learned parameters for the fully connected layer;
[0157] Finally, the answer with the highest predicted probability is selected as the predicted answer.
[0158] Furthermore, this invention can use evaluation algorithms to compare predicted answers with actual answers, generating evaluation metrics such as accuracy and recall.
[0159] This invention proposes a visual question answering method and system employing a cascaded question answering model. The invention uses a cascaded question answering model, taking images and questions as input, and processing them collaboratively through multiple modules to ultimately obtain a predicted answer. This workflow design enables the model to better understand and answer visual questions, improving the accuracy and robustness of the question answering system. The invention uses convolutional neural networks to extract features from images and a pre-trained Faster R-CNN network to select salient target regions and extract visual features. This design better captures important information in images, improving the accuracy of image feature extraction. The invention uses a question model to transform questions into word sequence representations and global representations, uses a GRU network to extract question representations, and employs an attention mechanism to focus on relevant regions in the image. This design enables the model to better understand the question and focus on image regions related to the question. The invention uses two attention mechanisms (layer-wise feedforward attention and self-attention) to focus on relevant regions in the image and fuses word vector representations and image feature representations. This design improves the model's ability to fuse image and text information, thereby improving the accuracy of the predicted answer. The invention uses a probability distribution model to predict the final answer to the question and obtains the answer with the highest predicted probability through fully connected layers and softmax operations. This design enables the model to generate reasonable answers based on image and text information, improving prediction accuracy. In summary, this invention employs a cascaded question-answering model, combining image and text information, and through the collaborative processing of multiple modules, achieves a highly accurate and robust visual question-answering system. Compared to existing technologies, this invention offers better performance and practicality. Figure 3 As shown in the figure, the image regions and keywords of most interest in this invention are marked with red boxes and red text, respectively. The visualization results in the figure demonstrate that the proposed model's advantage lies in its ability to accurately locate key target regions and keywords in the question, thereby inferring the correct answer. For example, for the question "Where is the sign pointing?", the proposed model first focuses on the keyword "sign", then accurately locates the region of "sign" in the image, and finally infers the correct answer "right". Two other examples further demonstrate the effectiveness of this invention.
Claims
1. A visual question answering method employing a cascaded question answering model, comprising the following steps: (1) Input the original image and the natural language question. The preprocessing module performs cropping and scaling preprocessing on the original image and word segmentation and part-of-speech tagging preprocessing on the natural language question. The preprocessed image and question are output. (2) Input the preprocessed image into the visual model, use a convolutional neural network to extract features from the image, and generate the feature representation of the image; (3) Input the preprocessed question, the question model transforms the question into a word sequence representation, and further generates a global representation of the question; (4) The answer model assigns attention weights to each word to focus on relevant regions in the image. These weights are determined by two attention mechanisms: layer-wise feedforward attention to extract key words in the question and self-attention to generate word vector representations. Then, the word vector representations and image feature representations are fused to obtain the association between each word and the relevant target region in the image. (5) The global representation of the input question, the association from words to region attention network output, combined with visual and textual information, and the probability distribution model are used to predict the final answer to the question.
2. The visual question answering method using a cascaded question answering model according to claim 1, characterized in that: The convolutional neural network described uses a pre-trained Faster R-CNN network to detect salient target regions in an image and extract visual features from each region; for each image, the Faster R-CNN network selects X target regions, and the image features of each region are... The set is:
3. The visual question answering method using a cascaded question answering model according to claim 1, characterized in that: The problem model extracts word sequence representations and a global representation of the problem, encoding the words in the problem into one-hot word vectors: K={k1,k2,...,k a ,...,k A }; A is the length of the problem, k a It is the one-hot word vector of the a-th word. Each word is mapped to a word embedding matrix Z with dimension q. j vector The problem representation is extracted using a GRU network, and the output of the last hidden layer is selected as the global representation of the problem. d=GRU(I j )。 4. A visual question answering method employing a cascaded question answering model according to claim 1, characterized in that: The process of obtaining the association between each word and the relevant target region in the image includes: Determine the first learning parameter J in the word attention network ju ; Identify the intermediate representations of the problem words: in j =J ju AND j ; Determine the second learning parameter J in the word attention network jc ; Generate word attention maps: c j =softmax(J jc u j ): Get the complete word representation: in yes The v-th element in It is C j The v-th element in the expression represents the weight of the v-th word; Determine the first learned parameter J in the image attention network bu The second learning parameter J du and the third learning parameter J bc ; Determine the intermediate representation of the problem image: in β =η(J bu AND O )⊙η((J du μ)I T ; η is the ReLU activation function, ⊙ represents bitwise multiplication, and I represents a vector with all elements being 1; Generate image attention map: c β =softmax(J bc u β ): The representation vector of the generated image: It is C β The vth element; Determine the first learning parameter J in the answer model bs Second learning parameter J ds ; By fusing image representations with language representations, fused features are generated: s=(J bs Ω)⊙η(J ds d)I T 。 5. A visual question answering method employing a cascaded question answering model according to claim 1, characterized in that: Step (5) includes fusing features through a fully connected layer and a softmax operation to predict the probability that each candidate answer is the correct answer: c n =softmax(J n s); J n These are the first learned parameters for the fully connected layer; Finally, the answer with the highest predicted probability is selected as the predicted answer.
6. A visual question-answering system employing a cascaded question-answering model, characterized in that, include: (1) Preprocessing module: receives the input original image and natural language question, performs cropping and scaling preprocessing on the original image, performs word segmentation and part-of-speech tagging preprocessing on the natural language question, and outputs the preprocessed image and question; (2) The vision module receives the preprocessed image, uses a convolutional neural network to extract features from the image, and generates a feature representation of the image. The convolutional neural network uses a pre-trained Faster R-CNN network to detect salient target regions in the image and extract visual features from each region. For each image, the Faster R-CNN network selects X target regions, and the image features of each region are... The set is: (3) The question module receives the preprocessed question, transforms the question into a word sequence representation, and further generates a global representation of the question; the question module extracts the word sequence representation and the global representation of the question, and encodes the words in the question into one-hot word vectors: K={k1,k2,...,k a ,...,k A }; A is the length of the problem, k a It is the one-hot word vector of the a-th word. Each word is mapped to a word embedding matrix Z with dimension q. j vector The problem representation is extracted using a GRU network, and the output of the last hidden layer is selected as the global representation of the problem. d=GRU(I j ); (4) The answer module assigns attention weights to each word to focus on relevant regions in the image. These weights are determined by two attention mechanisms: layer-wise feedforward attention to extract key words in the question and self-attention to generate word vector representations. Then, the word vector representations and image feature representations are fused to obtain the association between each word and the relevant target region in the image. The association between each word and the relevant target region in the image includes: Determine the first learning parameter J in the word attention network ju ; Identify the intermediate representations of the problem words: in j =J ju AND j ; Determine the second learning parameter J in the word attention network jc ; Generate word attention maps: c j =softmax(J jc u j ): Get the complete word representation: in yes The v-th element in It is C j The v-th element in the expression represents the weight of the v-th word; Determine the first learned parameter J in the image attention network bu The second learning parameter J du and the third learning parameter J bc ; Determine the intermediate representation of the problem image: in β =η(J bu AND 0 )⊙η(J du μ)I T ; η is the ReLU activation function, ⊙ represents bitwise multiplication, and I represents a vector with all elements being 1; Generate image attention map: c β =softmax(J bc u β ): The representation vector of the generated image: It is C β The vth element; Determine the first learning parameter J in the answer module bs Second learning parameter J ds ; By fusing image representations with language representations, fused features are generated: s=(J bs Ω)⊙η(J ds d)I T ; (5) Prediction module: The system uses the probability distribution module to predict the final answer to the question based on the global representation of the input question, the correlation between words and the output of the region attention network, and visual and textual information. The fused features, through fully connected layers and softmax operations, predict the probability that each candidate answer is the correct answer: c n =softmax(J n s); J n These are the first learned parameters for the fully connected layer; Finally, the answer with the highest predicted probability is selected as the predicted answer.