A debiasing approach for visual question answering based on dual counterfactuals

Abstract

The present invention discloses a visual question answering debiasing method based on dual counterfactuals, which relates to the field of computer vision. Visual counterfactual samples covering key visual areas and question counterfactual samples covering key words are respectively used as inputs of a bias model sharing the same set of parameters, and language bias is captured from two perspectives at the same time. The present invention generates pseudo labels using bias predictions of the bias model, and combines marginal losses to separate feature spaces of different answer samples, thereby minimizing the impact of language bias on a target visual question answering model.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a visual question answering debiasing method based on dual counterfactuals. Background Art

[0002] Visual Question Answering (VQA), as a popular research direction in the field of artificial intelligence and computer vision, has made great progress in recent years. However, the problem of language bias in visual question answering models still exists. Aishwarya Agrawal et al. first proposed the problem of language bias in 2016, pointing out that the uneven distribution of training data causes the model to rely too much on the language clues in the question when answering questions, ignoring the importance of image content. This bias will cause the model to perform poorly when dealing with new scenarios that are inconsistent with the distribution of training data. For example, for the question "What color is a banana?", 80% of the answers in the training samples are "yellow" and the remaining 20% ​​are other colors. At this time, if the test sample is given a picture of a banana of another color, the model will still answer "yellow". When the distribution of answers for each question type in the training set and the test set is different, the negative impact caused by language bias will be exposed, which makes the model unable to be applied to real-world scenarios.

[0003] In order to evaluate the degree to which the model is affected by language bias, researchers proposed the VQA-CP v2 dataset, which restructured the VQA v2 dataset so that the questions in the test set are significantly different from those in the training set in terms of type and distribution. Therefore, the evaluation results on the VQA-CP v2 dataset can intuitively demonstrate the model's robustness, reasoning ability, and ability to overcome language bias.

[0004] Various visual question answering debiasing methods proposed in recent years can be roughly divided into the following two categories, namely, ensemble-based methods and data augmentation-based methods.

[0005] (1) Methods based on data enhancement

[0006] The data augmentation method increases the diversity and complexity of the data to prevent the model from learning biased language patterns during training. However, this method uses the special prior information in the VQA-CP v2 dataset that the distribution of answers in the training set and the validation set are significantly different, which goes against the original intention of improving the robustness of the model and also leads to a significant increase in computational costs.

[0007] (2) Ensemble-based approach

[0008] The ensemble-based approach aims to design a bias model to debias the target visual question answering model. The bias model should capture the language bias in the training samples as much as possible, so that the bias model predicts the answer based only on the language bias. The prediction results of the bias model are used to guide the reasoning process of the target model, effectively avoiding language bias. However, unimodal input limits the expressiveness of the model and makes it difficult to capture complete language bias. Summary of the invention

[0009] In order to capture language bias completely and accurately without changing the distribution of training samples, the present invention proposes a visual question answering debiasing method based on dual counterfactuals. This method uses visual counterfactual samples covering key visual areas and question counterfactual samples covering keywords as inputs of a bias model that shares the same set of parameters, thereby capturing language bias from two perspectives at the same time.

[0010] Specifically, visual counterfactual samples can enable the visual question answering model to predict the answer only based on the question under the influence of language bias; question counterfactual samples can enable the model to answer the question only based on the question words in the question. By using the two types of counterfactual samples to train the bias model separately, the reasoning behavior of the visual question answering model under the influence of language bias can be fully simulated, thereby more completely modeling language bias. Finally, the present invention uses the biased prediction of the bias model to generate pseudo labels, and combines marginal loss to separate the feature space of different answer samples to minimize the impact of language bias on the target visual question answering model.

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] (1) Step 1: Counterfactual Sample Synthesis

[0013] Assume VQA samples are , while synthesizing visual counterfactual samples and the problem counterfactual sample . For visual counterfactual samples, we first use spaCy to extract question keywords and encode them using GloVe word embeddings, calculate their similarity scores with 36 visual target features in the image extracted by Faster R-CNN, and select high-scoring visual targets as candidate key targets; then use Grad-CAM gradients to measure the contribution of visual targets to the correct answer. Similarly, for question counterfactual samples, we use Grad-CAM gradients to track the answer classification layer and calculate the contribution of each object word (except question words) to the predicted probability of the true answer:

[0014] (1)

[0015] (2)

[0016] in, Indicates The image corresponding to the sample A visual target, Indicates The problem corresponding to the sample words. , Respectively represent the importance scores obtained by calculating the gradient of Grad-Cam (). Retain the K with the highest importance scores v and K q visual objects and words, and replace their original features with uniformly distributed random noise.

[0017] (2) Step 2: Training the bias model

[0018] Under the influence of language bias, visual question answering models usually directly rely on text information to answer questions without paying attention to visual information. In order to allow the bias model to better capture language bias and simulate the model's reasoning process under the influence of language bias as much as possible, two counterfactual samples are used as input to the bias model, namely:

[0019]

[0020] (3)

[0021] in, represents the bias model, , They represent the distribution of answers predicted by the bias model after inputting visual counterfactual samples and question counterfactual samples. The biased answers of the two counterfactual samples are adjusted by hyperparameters ∈ (0, 1) for weighted fusion, and the biased answer distribution modeled from two counterfactual perspectives is obtained as follows:

[0022] (4)

[0023] Get biased answers Y i b Finally, the bias model is trained using traditional cross entropy loss and KL divergence (Kullback-LeiblerDivergence). Cross entropy loss is a common method for training classification models, which measures the difference between the model's predicted probability distribution and the true label. For the training of bias models, cross entropy loss pushes the model to learn the statistical laws in the data set, that is, the correlation between questions and answers, so it automatically learns the bias in the data set and backpropagates to the network through this bias. The cross entropy loss function is as follows:

[0024] (5)

[0025] In addition, in order to make the bias model further capture the complex biases of the target model, the present invention uses KL divergence as the loss function to make the bias model follow the behavior of the target model:

[0026] (6)

[0027] KL divergence is a method to measure the difference between two probability distributions. Here, KL divergence is used to make the output of the bias model closer to the answer distribution of the target model, thereby simulating the complex bias that may exist in the target model.

[0028] In summary, the final loss function for training the bias model is:

[0029] (7)

[0030] By taking two counterfactual samples as input and combining two loss functions, the bias model is able to capture the language bias in the dataset more accurately.

[0031] (3) Step 3: Training the target model

[0032] The target model is trained using the biased model, aiming to enable the target model to answer questions using complete visual and textual information without being affected by language bias.

[0033] The biased model predictions depend on the language bias in the dataset, so its predictions Combined with real answers Synthetic Pseudo Labels ,

[0034] (8)

[0035] in, , , Respectively represent Pseudo labels, true labels, and bias model prediction results for samples. Represents the sigmoid function.

[0036] In order to further remove the language bias in the target model, the marginal loss is introduced to separate the feature space of different answers under different question types, thereby reducing the impact of language bias. Specifically, after the visual and text features are fused to obtain a joint representation Then, the classification process is expressed as With the weight vector Angle between The cosine function of , and then calculate the corresponding marginals for different answers :

[0037] (9)

[0038] (10)

[0039] The margin corresponding to each answer is determined by the specific question type. After obtaining the marginal value of each answer category, the marginal loss It can be expressed as:

[0040] (11)

[0041] By adding a corresponding marginal value to each sample, different feature spaces can be divided into different answer categories, effectively removing the bias in the target model.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention uses two counterfactual samples to respectively train the bias model, which can comprehensively simulate the reasoning behavior of the visual question answering model under the influence of language bias, and thus more completely model the language bias; the present invention uses the bias prediction of the bias model to generate pseudo labels, and combines the marginal loss to separate the feature space of different answer samples, thereby minimizing the impact of language bias on the target visual question answering model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a framework diagram of the visual question answering debiasing model based on dual counterfactuals in the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In an embodiment of the present invention, a visual question answering debiasing method based on dual counterfactuals includes the following steps:

[0046] (1) Step 1: Counterfactual Sample Synthesis

[0047] Assume VQA samples are , while synthesizing visual counterfactual samples and the problem counterfactual sample . For visual counterfactual samples, we first use spaCy to extract question keywords and encode them using GloVe word embeddings, calculate their similarity scores with 36 visual target features in the image extracted by Faster R-CNN, and select high-scoring visual targets as candidate key targets; then use Grad-CAM gradients to measure the contribution of visual targets to the correct answer. Similarly, for question counterfactual samples, we use Grad-CAM gradients to track the answer classification layer and calculate the contribution of each object word (except question words) to the predicted probability of the true answer:

[0048] (1)

[0049] (2)

[0050] in, Indicates The image corresponding to the sample A visual target, Indicates The problem corresponding to the sample words. , Respectively represent the importance scores obtained by calculating the gradient of Grad-Cam (). Retain the K with the highest importance scores v and K q visual objects and words, and replace their original features with uniformly distributed random noise.

[0051] (2) Step 2: Training the bias model

[0052] Under the influence of language bias, visual question answering models usually directly rely on text information to answer questions without paying attention to visual information. In order to allow the bias model to better capture language bias and simulate the model's reasoning process under the influence of language bias as much as possible, two counterfactual samples are used as input to the bias model, namely:

[0053]

[0054] (3)

[0055] in, represents the bias model, , They represent the distribution of answers predicted by the bias model after inputting visual counterfactual samples and question counterfactual samples. The biased answers of the two counterfactual samples are adjusted by hyperparameters ∈ (0, 1) for weighted fusion, and the biased answer distribution modeled from two counterfactual perspectives is obtained as follows:

[0056] (4)

[0057] Get biased answers Y i b Finally, the bias model is trained using traditional cross entropy loss and KL divergence (Kullback-LeiblerDivergence). Cross entropy loss is a common method for training classification models, which measures the difference between the model's predicted probability distribution and the true label. For the training of bias models, cross entropy loss pushes the model to learn the statistical laws in the data set, that is, the correlation between questions and answers, so it automatically learns the bias in the data set and backpropagates to the network through this bias. The cross entropy loss function is as follows:

[0058] (5)

[0059] In addition, in order to make the bias model further capture the complex biases of the target model, the present invention uses KL divergence as the loss function to make the bias model follow the behavior of the target model:

[0060] (6)

[0061] KL divergence is a method to measure the difference between two probability distributions. Here, KL divergence is used to make the output of the bias model closer to the answer distribution of the target model, thereby simulating the complex bias that may exist in the target model.

[0062] In summary, the final loss function for training the bias model is:

[0063] (7)

[0064] By taking two counterfactual samples as input and combining two loss functions, the bias model is able to capture the language bias in the dataset more accurately.

[0065] (3) Step 3: Training the target model

[0066] The target model is trained using the biased model, aiming to enable the target model to answer questions using complete visual and textual information without being affected by language bias.

[0067] The biased model predictions depend on the language bias in the dataset, so its predictions Combined with real answers Synthetic Pseudo Labels ,

[0068] (8)

[0069] in, , , Respectively represent Pseudo labels, true labels, and bias model prediction results for samples. Represents the sigmoid function.

[0070] In order to further remove the language bias in the target model, the marginal loss is introduced to separate the feature space of different answers under different question types, thereby reducing the impact of language bias. Specifically, after the visual and text features are fused to obtain a joint representation Then, the classification process is expressed as With the weight vector Angle between The cosine function of , and then calculate the corresponding marginals for different answers :

[0071] (9)

[0072] (10)

[0073] The margin corresponding to each answer is determined by the specific question type. After obtaining the marginal value of each answer category, the marginal loss It can be expressed as:

[0074] (11)

[0075] By adding a corresponding marginal value to each sample, different feature spaces can be divided into different answer categories, effectively removing the bias in the target model.

[0076] As a further embodiment of the present invention, the present invention experimentally verifies the above method and selects three datasets, VQA-CPv2, VQA-CP v1, and VQA v2, which include images from the COCO image dataset and related questions and answers.

[0077] In order to fairly evaluate the model performance, a common metric, consensus metric, is used. For each question-image pair, the accuracy of the predicted answer is calculated as:

[0078] (12)

[0079] in, Indicate answer If at least three subjects provided this answer, then the answer was considered 100% correct.

[0080] In order to verify the performance of the proposed visual question answering debiasing method, UpDn and LXMERT are used as baseline models. The comparison methods include data enhancement-based debiasing methods: SSL, CSS, CSST, MUTANT, MMBS, and ensemble-based debiasing methods: LMH, LPF, CF-VQA, GenB, RMLVQA, etc.

[0081] Table 1 compares the results of the dual counterfactual-based visual question answering debiasing model (MCD) proposed in this paper and other advanced debiasing methods on VQA-CP v2 and VQA-CP v1. The effect of MCD on the two baseline models UpDn and LXMERT is significantly improved. The overall accuracy of UpDn is improved by 29.28% and 21.64%, and the overall accuracy of LXMERT is improved by 20.67% and 27.51%, respectively. At the same time, the accuracy of the MCD method is better than all the comparison methods. For questions such as Num and Other that need to be answered with the help of image information, MCD achieves the best effect, indicating that the MCD method has achieved a good debiasing effect.

[0082] Table 2 shows the accuracy of the proposed MCD on the VQA V2 dataset. Since the answer distributions of the VQA V2 training set and the test set are very similar, the model that relies on language bias can achieve better accuracy on this dataset. Therefore, the accuracy of MCD on UpDn shows a 4.25% decrease, and when the LXMERT model is used as the baseline, it still achieves a 2.91% improvement.

[0083] Table 3 shows the results of the ablation experiment of the visual question answering debiasing model based on dual counterfactuals. Bias Model, V-Counterfactual, Q-Counterfactual, and Dual-Counterfactual represent the bias model, visual counterfactual sample input, question counterfactual sample input, and two counterfactual samples input simultaneously. It can be seen from the table that both counterfactual samples can have a debiasing effect, and when two counterfactual samples are input simultaneously, language bias can be captured more comprehensively.

[0084]

[0085]

[0086] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0087] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A visual question answering debiasing method based on dual counterfactuals, characterized in that: The following steps are involved: (1) Step 1: Counterfactual Sample Synthesis Assume VQA samples are , while synthesizing visual counterfactual samples and the problem counterfactual sample ; For visual counterfactual samples, we first extract question keywords using spaCy and encode them using GloVe word embedding, calculate their similarity scores with the visual target features in the image extracted by Faster R-CNN, and select high-scoring visual targets as candidate key targets. We then use Grad-CAM gradients to measure the contribution of the visual targets to the correct answer. For the counterfactual sample of the question, the Grad-CAM gradient is used to track the answer classification layer and calculate the contribution of each object word except the question word to the predicted probability of the true answer: （1） （2） in, Indicates The image corresponding to the sample A visual target, Indicates The problem corresponding to the sample words, , They represent the importance scores obtained by calculating the gradients of Grad-Cam (), and the K with the highest importance scores are retained. v and K q visual targets and words, and replace their original features with uniformly distributed random noise; (2) Step 2: Training the bias model Two counterfactual samples are used as input to the bias model, namely: （3） in, represents the bias model, , They represent the distribution of answers predicted by the bias model after inputting visual counterfactual samples and question counterfactual samples. The biased answers of the two counterfactual samples are adjusted by hyperparameters ∈ (0, 1) for weighted fusion, and the biased answer distribution modeled from two counterfactual perspectives is obtained as follows: （4） Get biased answers Finally, the bias model is trained using the traditional cross entropy loss and KL divergence (Kullback-Leibler Divergence). The cross entropy loss function is as follows: （5） Using KL divergence as the loss function, the bias model is made to follow the behavior of the target model: （6） KL divergence is used to make the output of the bias model closer to the answer distribution of the target model, thereby simulating the complex bias that may exist in the target model; The final loss function for training the bias model is: （7） (3) Step 3: Training the target model The prediction results of the biased model depend on the language bias in the dataset, which Combined with real answers Synthetic Pseudo Labels , (8) in, , , Respectively represent Pseudo labels, true labels, and bias model prediction results for samples. Represents the sigmoid function; Margin loss is introduced to separate the feature space of different answers under different question types and reduce the influence of language bias; visual and text features are fused to obtain a joint representation Then, the classification process is expressed as With the weight vector Angle between The cosine function of , and then calculate the corresponding marginals for different answers : （9） （10） The margin corresponding to each answer is determined by the specific question type. The marginal loss is calculated based on the proportion of the following answers. After obtaining the marginal value of each answer category, the marginal loss It can be expressed as: （11） By adding a corresponding marginal value to each sample, different answer categories are divided into different feature spaces to remove bias in the target model.

Images