A method to overcome language priors in visual question answering based on self-contrastive learning
Through self-contrast learning methods, the image-related areas are identified using attention and anti-attention mechanisms, and the problem of visual question-answer model being affected by language priors is solved, the robustness and performance of the model is improved, and the use of additional information is achieved without using Performance improvements in the case of annotation.
Patent Information
- Application Number
- CN202111557673.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-12-20
AI Technical Summary
Existing visual question-and-answer models are affected by language priors when processing images and questions, resulting in significant performance declines in tests after reorganizing the datasets and difficulty in reducing language priors without introducing additional annotations.
Using a self-contrast learning-based method, the image is identified and unrelated areas related to the problem through attention and counterattention mechanisms, and a joint representation is generated, and the joint optimization of self-contrast loss and basic VQA classification loss is focused on the relevant area prediction answers. .
Significantly improves the robustness of the visual Q&A model and image semantic compliance, improves performance on VQA-CP v1 and VQA-CP v2, especially achieves 59.00% state-of-the-art performance on VQA-CP v2, absolutely Performance improvements by 6.51%.
Smart Images

Figure CN114973041B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a language priori method for overcoming visual question-answering based on self-contrastive learning, and belongs to the technical field of computer natural language processing and computer vision. Background Art
[0002] Visual Question Answering (VQA) aims to automatically answer natural language questions based on visual content, which is one of the benchmark tasks of multimodality (e.g., language and image). It requires visual analysis, language understanding, multimodal information fusion, and reasoning. In recent years, the research on VQA has attracted great interest, and various benchmark datasets have been released. With the introduction of a large number of works, the VQA task has made significant progress. A large number of works have attempted to understand images and questions, but recent studies have found that these works are largely driven by the surface language correlation (i.e., language prior) in training QA pairs, while ignoring the image content. For example, these models tend to use "2" to answer the question "How many...?" and "tennis" to answer the question "What sport...?", while ignoring the combination of image content to reason about the question. To help address these bias factors, Agrawal et al. created the diagnostic benchmark VQA-CP in 2018 (VQA changes the prior by reorganizing the training and validation splits of various VQA datasets). Most existing works design various attention mechanisms to learn the relationship between modalities, which can achieve good results on VQA benchmarks such as VQA v2. However, the performance of these works drops significantly on VQA-CP due to the language prior. To mitigate the language prior, existing works focus on reducing the statistical prior of the problem and increasing the dependency and interpretability of the image, which can be roughly divided into learning with auxiliary tasks and learning without auxiliary tasks. Learning without auxiliary tasks uses an auxiliary QA branch to standardize the training of the target VQA model or a specific complex learning strategy. These methods attempt to add an auxiliary branch to capture the language prior to weaken its effect. Auxiliary task learning introduces additional manual supervision and auxiliary tasks (visual basis, image captioning, etc.) to increase image dependency and interpretability). These methods can better understand the image content under the guidance of auxiliary tasks, thereby achieving better performance. But the inherent data bias seriously leads to surface language correlation. Therefore, it is crucial to reduce the inherent language prior without introducing additional annotations while relying on relevant visual areas to make decisions. Existing work on reducing language prior can be roughly divided into learning with auxiliary tasks and learning without auxiliary tasks. Learning with auxiliary tasks. These works introduce additional manual supervision and auxiliary tasks to increase the dependency and interpretability of images. Summary of the invention
[0003] The present invention provides a method for overcoming the language prior of visual question answering based on self-contrastive learning. It utilizes novel self-contrastive learning to overcome the language prior problem of VQA, and can focus on relevant areas to predict the correct answer to a given question about an input image, thereby improving the reasoning ability and robustness of the VQA model.
[0004] The technical solution of the present invention is: a language priori method for overcoming visual question answering based on self-contrastive learning, and the specific steps of the method are as follows:
[0005] Step 1: First, the question, image and answer options are used as experimental data. Then, the image is preprocessed to extract the feature map, and the question is preprocessed to generate the question feature vector.
[0006] Step 2: Use the attention layer to learn to identify the image and question-related areas. After the preprocessing operation in Step 1, the attention mechanism uses the question to calculate the attention weight on the image area to locate the image area related to the question. The obtained question feature q and the weighted image feature Fusion into a joint representation r;
[0007] Step 3: Use the anti-attention layer to identify the currently irrelevant or less relevant image regions; use the attention weight value obtained in Step 2 to combine the problem feature q and the weighted anti-image feature Fusion is a joint representation r0, which focuses on irrelevant areas and ignores relevant areas on the image for contrast;
[0008] Step 4, post-processing: Through the joint representation r0 obtained in Step 3 and the joint representation r obtained in Step 1, the proposed network is trained to optimize the joint loss of self-contrastive loss Lscl and the basic VQA classification loss Lvqa. We can focus on the relevant areas to predict the correct answer to a given question about the input image.
[0009] Furthermore, the step Step 1 includes the following:
[0010] Step 1.1, first use the pre-trained model Faster-R-CNN to extract a series of visual target features from the image;
[0011] Step 1.2, embed the question into words and pass it to a single-layer GRU to generate question features;
[0012] Furthermore, the specific steps of Step 2 are as follows:
[0013] Step 2.1, after extracting the image and question features, pass them to the attention layer to convert the image features and question features into a space of the same dimension;
[0014] Step 2.2, calculate the attention weight, and then generate a normalized attention weight for each feature map. The final image feature is the weighted sum of all input features;
[0015] Step 2.3: Combine the weighted image features with the question features obtained in Step 1.2 into a joint feature representation r, and then calculate the probability distribution of each answer a in the candidate answer set A.
[0016] Furthermore, the specific steps of Step 3 are as follows:
[0017] Step 3.1, the attention mechanism uses the question to calculate the attention weight on the image region to locate the image region related to the question. However, the anti-attention mechanism is the opposite of the attention mechanism. It helps the VQA model overcome the language prior by focusing the question on irrelevant areas and ignoring relevant areas on the image for contrast.
[0018] Step 3.2, starting from the attention weight value obtained in Step 2, use the attention weight a obtained by the attention layer and the normalized inverse attention weight α′, which performs a negative operation, Opponent (a) = -a or Opponent (a) = e -a Make larger weights smaller and smaller weights larger, so that the attention weights output by the softmax function are concentrated on irrelevant areas;
[0019] Step 3.3: After learning the anti-attention weights, generate weighted anti-image features Then, similar to the attention layer, we will weight the image features Combined with the question feature q obtained in Step 1.2 to form a joint feature representation r0, and then calculate the probability distribution of each answer a in the candidate answer set A
[0020] Furthermore, the specific steps of Step 4 are as follows:
[0021] Step 4.1. First, the loss layer contains two branches. The first loss layer aims to utilize the basic VQA model probability distribution It is optimized by minimizing the binary cross entropy loss, and the loss function is defined as Lvqa;
[0022] Step 4.2, the other branch is the self-contrastive layer, which aims to use the distribution of answers predicted by the self-contrastive layer to increase the correlation and dependency between questions and images. We first considered an objective function similar to QICE
[36] . Thanks to this, we believe that there is a certain correlation between the answers predicted based on the relevant and irrelevant regions of the same image, that is, the predicted answers are mutually exclusive. Therefore, some answers defined by the self-contrastive layer are excluded, and then a self-contrastive learning loss Lscl is proposed to increase the correlation between questions and images;
[0023] Step 4.3. The proposed network is trained to optimize the joint loss of the self-contrastive loss Lscl in Step 4.2 and the base VQA classification loss Lvqa in Step 4.1. Through this scheme, we can focus on the relevant regions to predict the correct answer to a given question about the input image.
[0024] Furthermore, the model of the anti-attention layer is similar to the attention layer. First, using the attention weight a obtained by the attention layer, the normalized anti-attention weight α′ can be calculated as: α′=softmax(opponent(a)). After learning the anti-attention weight, we generate the anti-image feature As shown below: We will weight the inverse image features Combined with the question feature q obtained in Step 1.2 to form a joint feature representation r0, and then calculate the probability distribution of each answer a in the candidate answer set A The formula is as follows: q′0=f″ q (q), v′0=f″ v (v) Among them, f v , f q , f0 is the transformation function and w0 represents the weight matrix to be learned.
[0025] The beneficial effects of the present invention are:
[0026] 1. This paper solves the VQA problem through a novel self-contrastive learning method, overcoming language priors by contrasting the answers generated from question-related and question-irrelevant regions in the image.
[0027] 2. After self-contrastive learning training, the model is forced to learn more information based on relevant image regions. It effectively increases the semantic compliance and interpretability of the image. In this way, image features and problem context no longer exist in isolation during the modeling process.
[0028] 3. Extensive experiments are conducted on popular benchmarks VQA-CP v1 and VQA-CP v2. Experimental results show that our method can significantly improve the performance of benchmark datasets without using additional annotations. In particular, by building on top of the LMH model, we achieve 59.00% state-of-the-art performance on VQA-CP v2, with an absolute performance improvement of 6.51%. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A diagram of the architecture of a method for overcoming language priors in visual question answering based on self-contrastive learning.
[0030] Figure 2 A comparison between the present invention and several VQA model variants that overcome language priors;
[0031] Figure 3 This is an example of self-contrastive learning of the present invention;
[0032] Figure 4 It is a flow chart of the present invention. DETAILED DESCRIPTION
[0033] Example 1: Figure 1-Figure 4 As shown, a method for overcoming language priors in visual question answering based on self-contrastive learning is characterized in that the specific steps of the method are as follows:
[0034] Step 1: First, the question, image and answer options are used as experimental data. Then, the image is preprocessed to extract the feature map, and the question is preprocessed to generate the question feature vector.
[0035] Step 2: Use the attention layer to learn to identify the image and question-related areas. After the preprocessing operation in Step 1, the attention mechanism uses the question to calculate the attention weight on the image area to locate the image area related to the question. The obtained question feature q and the weighted image feature Fusion into a joint representation r;
[0036] Step 3: Use the anti-attention layer to identify the currently irrelevant or less relevant image regions; use the attention weight value obtained in Step 2 to combine the problem feature q and the weighted anti-image feature Fusion is a joint representation r0, which focuses on irrelevant areas and ignores relevant areas on the image for contrast;
[0037] Step 4, post-processing: Through the joint representation r0 obtained in Step 3 and the joint representation r obtained in Step 1, the proposed network is trained to optimize the joint loss of self-contrastive loss Lscl and the basic VQA classification loss Lvqa. We can focus on the relevant areas to predict the correct answer to a given question about the input image.
[0038] Furthermore, the step Step 1 includes the following:
[0039] Step 1.1, first use the pre-trained model Faster-R-CNN to extract a series of visual target features from the image;
[0040] Step 1.2, embed the question into words and pass it to a single-layer GRU to generate question features;
[0041] Furthermore, the specific steps of Step 2 are as follows:
[0042] Step 2.1, after extracting the image and question features, pass them to the attention layer to convert the image features and question features into a space of the same dimension;
[0043] Step 2.2, calculate the attention weight, and then generate a normalized attention weight for each feature map. The final image feature is the weighted sum of all input features;
[0044] Step 2.3: We merge the weighted image features with the question features obtained in Step 1.2 into a joint feature representation r, and then calculate the probability distribution of each answer a in the candidate answer set A.
[0045] Furthermore, the specific steps of Step 3 are as follows:
[0046] Step 3.1, the attention mechanism uses the question to calculate the attention weight on the image region to locate the image region related to the question. However, the anti-attention mechanism is the opposite of the attention mechanism. It helps the VQA model overcome the language prior by focusing the question on irrelevant areas and ignoring relevant areas on the image for contrast.
[0047] Step 3.2, starting from the attention weight value obtained in Step 2, use the attention weight a obtained by the attention layer and the normalized inverse attention weight α′, which performs a negative operation, Opponent (a) = -a or Opponent (a) = e -a Make larger weights smaller and smaller weights larger, so that the attention weights output by the softmax function are concentrated on irrelevant areas;
[0048] Step 3.3: After learning the anti-attention weights, generate weighted anti-image features Then, similar to the attention layer, we will weight the image features Combined with the question feature q obtained in Step 1.2 to form a joint feature representation r0, and then calculate the probability distribution of each answer a in the candidate answer set A
[0049] Furthermore, the specific steps of Step 4 are as follows:
[0050] Step 4.1. First, the loss layer contains two branches. The first loss layer aims to utilize the basic VQA model probability distribution It is optimized by minimizing the binary cross entropy loss, and the loss function is defined as Lvqa;
[0051] Step 4.2. Another branch is the self-contrastive layer, which aims to increase the correlation and dependency between the question and the image by using the distribution of answers predicted by the self-contrastive layer. We first considered an objective function similar to QICE
[36] . Thanks to this, we believe that there is a certain correlation between the answers predicted based on the relevant and irrelevant regions of the same image, that is, the predicted answers are mutually exclusive. Therefore, some answers defined by the self-contrastive layer are excluded. Then, a self-contrastive learning loss Lscl is proposed to increase the correlation between the question and the image.
[0052] Step 4.3. The proposed network is trained to optimize the joint loss of the self-contrastive loss Lscl in Step 4.2 and the base VQA classification loss Lvqa in Step 4.1. Through this scheme, we can focus on the relevant regions to predict the correct answer to a given question about the input image.
[0053] Furthermore, the model of the anti-attention layer is similar to the attention layer. First, using the attention weight a obtained by the attention layer, the normalized anti-attention weight α′ can be calculated as: α′=softmax(opponent(a)). After learning the anti-attention weight, we generate the anti-image feature As shown below: We will weight the inverse image features Combined with the question feature q obtained in Step 1.2 to form a joint feature representation r0, and then calculate the probability distribution of each answer a in the candidate answer set A The formula is as follows: q′0=f″ q (q), v′0=f″ v (v) Among them, f v , f q, f0 is the transformation function and w0 represents the weight matrix to be learned.
[0054] The present invention conducts a large number of experiments on the popular benchmarks VQA-CP v1 and VQA-CP v2. The experimental results show that the method of the present invention can significantly improve the performance of the benchmark datasets without using additional annotations. In particular, by building on top of the LMH model, 59.00% of the state-of-the-art performance is achieved on VQA-CP v2, and the absolute performance is improved by 6.51%. The results are shown in Table 1 and Table 2.
[0055] Table 1 shows the experimental results of the present invention on VQA-CP v2
[0056]
[0057] Table 2 shows the experimental results of the present invention on VQA-CP v1
[0058]
[0059] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A method for overcoming language priors in visual question answering based on self-contrastive learning, characterized by: The specific steps of the method are as follows: Step 1: First, the question, image and answer options are used as experimental data. Then, the image is preprocessed to extract the feature map, and the question is preprocessed to generate the question feature vector. Step 2: Use the attention layer to learn to identify the image and the area related to the question; After the preprocessing operation in Step 1, the attention mechanism uses the question to calculate the attention weight on the image area to locate the image area related to the question, and the obtained question feature q and the weighted image feature Fusion into a joint representation r; Step 3: Use the anti-attention layer to identify the currently irrelevant or less relevant image areas; use the attention weight value obtained in Step 2 to combine the problem feature q and the weighted anti-image feature Fusion is a joint representation r0, which focuses on irrelevant areas and ignores relevant areas on the image for contrast; Step 4, post-processing: Through the joint representation r0 obtained in Step 3 and the joint representation r obtained in Step 1, the network is trained to optimize the joint loss of self-contrast loss Lscl and basic VQA classification loss Lvqa, which can focus on relevant areas to predict the correct answer to a given question about the input image.
2. The method for overcoming language priors in visual question answering based on self-contrastive learning according to claim 1, characterized in that: The step Step 1 includes the following: Step 1.1, first use the pre-trained model Faster-R-CNN to extract a series of visual target features from the image; Step 1.2: embed the question into words and pass it to a single-layer GRU to generate question features.
3. The method for overcoming language priors of visual question answering based on self-contrastive learning according to claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.1, after extracting the image and question features, pass them to the attention layer to convert the image features and question features into a space of the same dimension; Step 2.2, calculate the attention weight, and then generate a normalized attention weight for each feature map. The final image feature is the weighted sum of all input features; Step 2.3: Combine the weighted image features and question features into a joint feature representation r, and then calculate the probability distribution of each answer a in the candidate answer set A.
4. According to claim 1, the method for overcoming language priors in visual question answering based on self-contrastive learning: the specific steps of step 3 are as follows: Step 3.1, the attention mechanism uses the question to calculate the attention weight on the image area to locate the image area related to the question; however, the anti-attention mechanism is the opposite of the attention mechanism, which helps the VQA model overcome the language prior by focusing the question on irrelevant areas and ignoring relevant areas on the image for contrast; Step 3.2, starting from the attention weight value obtained in Step 2, use the attention weight a obtained by the attention layer and the normalized inverse attention weight α′, which performs a negative operation, Opponent (a) = -a or Opponent (a) = e -a Make the large weights smaller and the small weights larger, so that the attention weights output by the softmax function are concentrated on irrelevant areas; Step 3.3: After learning the anti-attention weights, generate weighted anti-image features Then, similar to the attention layer, the weighted inverse image features Combined with the question feature q obtained in Step 1 to form a joint feature representation r0, and then calculate the probability distribution of each answer a in the candidate answer set A 5. According to the method for overcoming language priors in visual question answering based on self-contrastive learning according to claim 1: the specific steps of step 4 are as follows: Step 4.
1. First, the loss layer contains two branches. The first loss layer aims to utilize the basic VQA model probability distribution It is optimized by minimizing the binary cross entropy loss, and the loss function is defined as Lvqa; Step 4.
2. Another branch is the self-contrastive layer, which aims to use the distribution of answers predicted by the self-contrastive layer to increase the correlation and dependency between questions and images. First, an objective function similar to QICE[36] is considered. Thanks to this, it is believed that there is a certain correlation between the answers predicted based on the relevant and irrelevant regions of the same image, that is, the predicted answers are mutually exclusive. Therefore, some answers defined by the self-contrastive layer are excluded. Then, a self-contrastive learning loss Lscl is proposed to increase the correlation between questions and images. Step 4.
3. The proposed network is trained to optimize the joint loss of the self-contrast loss Lscl in Step 4.2 and the basic VQA classification loss Lvqa in Step 4.
1. Through this scheme, it can focus on the relevant areas to predict the correct answer to a given question about the input image.
6. The method for overcoming language priors in visual question answering based on self-contrastive learning according to claim 1, characterized in that: The model of the anti-attention layer is similar to the attention layer. First, the attention weight a obtained by the attention layer is used, and the normalized anti-attention weight α′ is calculated as: α′=softmax(opponent(a)). After learning the anti-attention weight, the anti-image feature is generated. As shown below: The weighted inverse image features It is fused with the obtained question feature q into a joint feature representation r0, and then the probability distribution of each answer a in the candidate answer set A is calculated. The formula is as follows: q′0=f″ q (q), v′0=f″ v (v) Among them, f v , f q , f0 is the transformation function and w0 represents the weight matrix to be learned.
