A visual question answering method based on contrastive learning and multimodal alignment

Abstract

The invention discloses a visual question answering method based on contrastive learning and multimodal alignment, which belongs to the field of visual question answering. The method realizes robust visual question answering under an imbalanced visual question answering dataset. Existing robust visual question answering methods based on data enhancement are often based on counterfactual sample enhancement, and counterfactual samples are added to the training as data augmentation, but the process of distinguishing counterfactual samples is not constructed in the feature and prediction levels, and the relationship between samples is not deeply explored. The present invention proposes a visual question answering method based on contrastive learning and multimodal alignment to solve the problem of language bias in visual question answering, and reduces the complexity of the model by performing contrastive learning at the feature and prediction levels, thereby improving the generalization ability of the visual question answering model, thereby realizing robust visual question answering in language bias scenarios.

Description

Technical Field

[0001] The invention belongs to the field of visual question answering, and the method can realize robust visual question answering under an imbalanced visual question answering data set. Background Art

[0002] In the most common form of visual question answering (VQA), the data includes a picture and a question, which requires the machine to give a corresponding answer to its question. The questions that the model needs to answer change in real time, while other computer vision tasks, such as object detection and object segmentation, are given in advance. VQA tasks are more in line with the true form of artificial intelligence and can help models understand vision and language more deeply. In the field of visual question answering, VQA models often rely on the surface correlation between questions and answers when answering questions, while ignoring image information, that is, the problem of language bias occurs. The core reason is the imbalanced distribution of training data. Language bias brings great difficulties to the application of VQA, resulting in poor robustness of the model in application and low interpretability of the model. This bias is usually widely present in multimodal data, which seriously affects the practical application of multimodal machine learning. At the same time, it may also lead to discrimination. For example, deploying relevant VQA models in public places such as social platforms may cause some discrimination and misleading due to bias, affecting the reasonable use of technology.

[0003] Existing robust visual question answering methods based on data augmentation are often based on counterfactual sample augmentation, and counterfactual samples are added to the training as data augmentation, but the process of distinguishing counterfactual samples is not constructed in the feature and prediction levels, and the relationship between samples is not deeply explored. However, since the augmented counterfactual samples are all synthetic and have similar appearance characteristics (such as the local mask of the counterfactual samples), and the factual samples are different from the natural image samples of the same type, this will cause positive and negative samples to be confused in the feature space after augmentation. Existing methods for distinguishing augmented samples do not establish a pairwise contrast relationship between the three samples, which is important for de-obfuscation and reducing complexity. Moreover, the contrast relationship is only considered from the feature level, and only the contrast relationship of the original samples is analyzed. The contrast relationship is incomplete, and the feature mapping relationship at the prediction level is not considered, which is prone to prediction confusion. Summary of the invention

[0004] In response to the problems in the background technology, the solution proposed in the present invention can establish three pairs of comparative relationships between each two at the feature and prediction levels, which can effectively reduce the confusion between augmented samples and improve the generalization ability of the model.

[0005] The technical solution of the present invention is: a visual question answering method based on contrastive learning and multimodal alignment, the method comprising:

[0006] Step 1: The form of visual question answering problem is: given the original feature D = {I i ,Q i ,a i} N , which contains N pairs of images I i , Question Q i , answer a i ;

[0007] Step 2: Use the CSS (Counterfactual Sample Synthesis) algorithm to augment the samples and use them as training samples, where CSS is the method function;

[0008] (I + ,I - ,Q + ,Q - )=CSS(f vqa ,(I,Q,a)) (1)

[0009] Among them, f vqa is the target mapping function for the visual question answering task, (I + ,I - ,Q + ,Q - ) are respectively the positive fact sample of the image, the counterfactual sample of the image, the positive fact sample of the problem, and the counterfactual sample of the problem; CSS() is the calculation method function;

[0010] Step 3: Use Faster-rcnn to extract the region of interest in the image, use a question encoder eq to extract a set of word embeddings Q of the question, and use an object detector iq to extract a set of visual embeddings V of the region of interest; use the attention mechanism to fuse the features of V and Q to obtain the mixed features F(V,Q); use the following method to bring the original features and the positive fact features closer, and pull the positive fact features and the counterfactual features further apart;

[0011]

[0012]

[0013] L c1 =f c (F(V e ,Q e ),F(V e + ,Q e ),F(V e - ,Q e ))

[0014] L c2 =f c (F(Ve ,Q e ),F(V e ,Q e + ),F(V e - ,Q e - ))

[0015] L c3 =f c (F(V e ,Q e ),F(V e + ,Q e + ),F(V e - ,Q e - ))

[0016] L c =L c1 +L c2 +L c3

[0017] Among them, cos(·) represents the cosine function, x represents the input feature, that is, the original feature, the positive fact feature or the counterfactual feature, + represents the positive fact feature, - represents the counterfactual feature, and x T represents transpose, L c In order to input the three pairs of features into the contrast loss function to obtain the counterfactual contrast loss result and accumulate the losses after these three items, by making L c Minimize to bring the original feature and the positive fact feature closer, and distance the positive fact feature from the counterfactual feature;

[0018] Step 4: Input the original features, positive fact features, and counterfactual features obtained in step 3 into the classifier to obtain a classification vector, and use the following method to perform zooming and zooming on the classification vector;

[0019]

[0020]

[0021]

[0022] Among them, p, q are the classification vectors output by the classifier, It means to calculate p and The KL divergence between - ) represents the output classification vector of the classifier input image as counterfactual feature, p + represents the corresponding positive sample combination, is the contrast loss for the original sample, is the contrast loss for counterfactual samples;

[0023] Step 5: The final loss function is as follows:

[0024]

[0025] Among them, L vqa is the classification loss of the model, λ vqa , c , λ represent the corresponding weight coefficients respectively; by optimizing this function, the final loss function L CFT The smallest classification vector is obtained, and the classification result is judged by the vector.

[0026] This paper proposes a visual question answering method based on contrastive learning and multimodal alignment to solve the language bias problem in visual question answering. By performing contrastive learning at the feature and prediction levels, the complexity of the model is reduced and the generalization ability of the visual question answering model is improved, thereby achieving robust visual question answering in language bias scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Visual question answering method based on contrastive learning and multimodal alignment;

[0028] Figure 2 Schematic diagram of feature-level contrastive learning;

[0029] Figure 3 Schematic diagram of comparative learning at the prediction level, showing the feature contours between samples, where A, B, and C are the original sample, the factual sample, and the counterfactual sample, respectively. DETAILED DESCRIPTION

[0030] In order to measure language bias, the prior art proposes the VQA-CP v2 dataset, which is obtained by repartitioning the VQA v2 dataset. This dataset and its original version are currently the only open source datasets for evaluating language bias. In addition, the distribution of questions and answers in the training set and test set of this dataset is very different, that is, for the same type of questions, the distribution of answers in the training set and test set is quite different. Therefore, this dataset is very suitable for measuring whether the model has language bias. The training set of the VQA-CP v2 dataset contains 121K images, 438K questions, and 440M answers; the test set contains 98K images, 220K questions, and 220M answers.

[0031] When evaluating the correctness of a sentence, it is necessary to consider the correctness of the syntax and whether the semantics of the sentence are correct. To simplify the problem, most datasets for visual question answering limit the generated answers to words or phrases with a length of 1 to 3 words. The general evaluation method currently used is shown in the following formula. The dataset collects 10 different real answers to the question. When evaluating, the generated sentence (answer) needs to be compared with the answers of the 10 manual answers to obtain the accuracy

[0032] Table 1 compares the performance of the proposed method with other recent methods on VQA-CP v2. The selected comparison methods are as follows: UpDn, RUBi, LMH, RMFE, RandIm ] , CSSVQA, CFVQA, Greed ] and CLVQA. For fair comparison, the UpDn model is uniformly selected as the backbone network. Compared with the best method, the overall accuracy of the proposed method is improved by 0.19%, the average accuracy is improved by 0.89% (average of various indicators), and especially the Num accuracy is improved by 2.6%. The results show that the proposed method is superior to the current comparison method on VQA-CP v2.

[0033] For individual indicators, such as Num, yes / no, etc., the methods proposed in this paper are not all superior to other methods. This is because: 1) CFVQA is slightly higher than our method in terms of numerical indicators (Num). CFVQA is an integrated method based on causal reasoning. Similar to Boosting, CFVQA uses more networks as additional information than our method. Therefore, it is inappropriate to directly compare small indicators. 2) Greedy's method is slightly higher than our method in other indicators (Others). Greedy is also a Boosting method, so it is slightly higher than the method in this paper in other indicators.

[0034] Table 1. Performance comparison with the state-of-the-art accuracy of VQA-CP v2.

[0035]

[0036] Table 2 shows the performance comparison of this method and other recent models on VQA v2. The selected comparison methods are as follows: UpDn, AReg, SCR, GRL, RUBi, LMH, CSSVQA, CFVQA, Greedy, and CLVQA. The metric used for comparison is Gap, which represents the performance gap between the model on VQA-CP v2 and the model on VQA v2. It is an auxiliary verification metric and is widely used in the robustness research of visual question answering. The smaller the Gap, the more robust the model. The results show that the model has strong robustness. Compared with the CSSVQA method, the Gap metric is improved by more than 50%.

[0037] Table 2. Performance comparison on VQA v2. The gap represents the overall accuracy difference between VQA-CP v2 and VQA v2.

[0038]

[0039]

[0040] For individual indicators, the performance of this paper on VQA v2 is slightly lower than that of CSSVQA. This is because the method proposed in this paper will weaken the language prior more strongly to reduce language bias. Therefore, on the VQA v2 dataset with a relatively balanced data distribution, the accuracy will drop more drastically for samples that rely on language bias, because the model can sometimes only get the correct answer by relying on language prior. Looking at the dataset, it can be found that the samples in the numerical indicators are more dependent on language bias. Therefore, after better weakening the language prior, the quantitative indicators of the proposed method are slightly larger than CSSVQA. But the overall Gap indicator is smaller than CSSVQA. In addition, Figure 2 This illustrates the situation very well.

Claims

1. A visual question answering method based on contrastive learning and multimodal alignment, the method comprising: Step 1: The form of visual question answering problem is: given the original feature D = {I i ,Q i ,a i } N It contains N pairs of images I i , Question Q i , answer a i ; Step 2: Use the counterfactual sample synthesis method to augment the samples and use them as training samples, with CSS as the method function; (I + ,I - ,Q + ,Q - )=CSS(f vqa ,(I,Q,a)) (1) Among them, f vqa is the target mapping function for the visual question answering task, (I + ,I - ,Q + ,Q - ) are respectively the positive fact sample of the image, the counterfactual sample of the image, the positive fact sample of the problem, and the counterfactual sample of the problem; CSS() is the counterfactual sample synthesis method function; Step 3: Use Faster-rcnn to extract the region of interest in the image, use a question encoder eq to extract a set of word embeddings Q of the question, and use an object detector iq to extract a set of visual embeddings V of the region of interest; use the attention mechanism to fuse the features of V and Q to obtain the mixed features F(V,Q); use the following method to bring the original features and the positive fact features closer, and pull the positive fact features and the counterfactual features further apart; L c1 =f c (F(V e ,Q e ),F(V e + ,Q e ),F(V e - ,Q e )) L c2 =f c (F(V e ,Q e ),F(V e ,Q e + ),F(V e - ,Q e - )) L c3 =f c (F(V e ,Q e ),F(V e + ,Q e + ),F(V e - ,Q e - )) L c =L c1 +L c2 +L c3 Among them, cos(·) represents the cosine function, x represents the input feature, that is, the original feature, the positive fact feature or the counterfactual feature, + represents the positive fact feature, - represents the counterfactual feature, and x T represents transpose, L c In order to input the three pairs of features into the contrast loss function to obtain the counterfactual contrast loss result and accumulate the losses after these three items, by making L c Minimize to bring the original feature and the positive fact feature closer, and distance the positive fact feature from the counterfactual feature; Step 4: Input the original features, positive fact features, and counterfactual features obtained in step 3 into the classifier to obtain a classification vector, and use the following method to perform zooming and zooming on the classification vector; Among them, p, q are the classification vectors output by the classifier, It means to calculate p and The KL divergence between - ) represents the output classification vector of the classifier input image as counterfactual feature, p + represents the corresponding positive sample combination, is the contrast loss for the original sample, is the contrastive loss for counterfactual samples; Step 5: The final loss function is as follows: Among them, L vqa is the classification loss of the model, λ vqa , c , λ represent the corresponding weight coefficients respectively; by optimizing this function, the final loss function L CFT The smallest classification vector is obtained, and the classification result is judged by the vector.

Images