A visual question answering debiasing method based on a jumbled problem type

Abstract

The application discloses a visual question answering debiasing method based on a disorder problem type, and has the characteristics that the method comprises the following steps: a) splitting a question in each image-question pair into a question type and other parts; b) extracting the question type of each question in a data set to form a question type set; c) randomly selecting a question type different from the question type before splitting in the question type set, and connecting the question type with other parts of the question; and d) defining a loss function, taking the constructed negative sample as a penalty term, so that the visual question answering model answers the question according to the image instead of language priori. Compared with the prior art, the application has the remarkable effect of eliminating language priori, constructs a negative sample by using self-supervised learning, does not introduce additional supervision information, can improve the accuracy of the visual question answering model to a certain extent, effectively reduces the language priori problem of the visual question answering, has simple method, good effect and good application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual question answering based on multi-modal feature fusion, in particular to a visual question answering debiasing method based on disordering question types. BACKGROUND

[0002] The visual question answering (VQA) task is a challenging and practical task in the field of artificial intelligence, and is a multi-modal task, which is a cross-research direction involving computer vision and natural language processing, and requires visual analysis, language understanding and multi-modal reasoning. Visual question answering refers to: given an image and a question related to the image, the goal of visual question answering is to combine the visual information of the image and the text content of the question, and obtain the answer to the question by cross-feature fusion and reasoning on the image and the text.

[0003] Existing VQA models mostly rely on spurious linguistic correlations, i.e. the surface correlation between question types and their corresponding frequent answers, rather than multi-modal reasoning based on vision and language. For example, when a VQA model answers a question related to numbers, it outputs the most common answer "1" to the question related to numbers in the training data, without looking at the input image part. Therefore, if the VQA model memorizes strong linguistic priors during training, its performance on out-of-distribution (OOD) datasets (e.g. VQA-CP v2) very different from the training set and test set will be poor.

[0004] A mainstream method to mitigate linguistic bias is to augment training data by using additional annotations or data augmentation. Counterfactual training sample generation helps to balance the training data and greatly outperforms other debiasing methods on VQA-CP v2. Only when a question corresponds to an image can it be answered correctly. The method verifies the effectiveness of debiasing training in improving the generalization ability of VQA models. Annotation-based methods try to directly increase image dependency by introducing external visual supervision. In general, annotation-based methods have better performance than non-annotation-based methods because they try to keep the visual attention of the VQA model consistent with human attention. However, these methods require large-scale visual annotations, which are not easy to obtain and require a lot of manpower and material resources. SUMMARY

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a debiased visual question answering method based on shuffling question types. This method employs a debiasing operation on the main source of bias—the question modality—by splitting the question into question type and other parts. Self-supervised learning is used to construct negative samples, and a loss function is used as a penalty term. This allows the VQA model to overcome linguistic priors and arrive at the correct answer based on a given image, improving the model's robustness. It effectively solves or mitigates the linguistic prior problem in visual question answering. The method is simple, effective, and has promising application prospects.

[0006] The objective of this invention is achieved as follows: A visual question answering (VQA) bias removal method based on shuffling question types. Its key feature is that, to mitigate the linguistic prior problem in VQA, it employs bias removal operations targeting the main source of bias: question modalities. To reduce linguistic priors generated by questions, the question is decomposed into question type and other components. Self-supervised learning is used for negative sample construction without introducing additional supervisory information. A loss function is designed as a penalty term, enabling the VQA model to overcome linguistic priors and obtain the correct answer based on a given image, thus improving the model's robustness. The VQA bias removal method based on shuffling question types includes the following steps:

[0007] 1) Break down the question in each image-question pair into question type and other parts of the question, specifically including:

[0008] 1.1: Analysis of the sources of language priors in visual question answering. Through experiments and verification, biases mainly originate from text modalities.

[0009] 1.2: In the text modality, that is, in the question, the main sources of language priors are the question type and keywords. Since keywords are difficult to locate, keywords and other words are classified as other parts of the question.

[0010] 1.3: Set the question type length to 2, split a question into a question type and other parts of the question, and store them in two lists respectively.

[0011] 2) Extract the question type of each question in the dataset to form a question type set.

[0012] 3) Randomly select a different question type from the set of question types than before splitting, and perform a connection operation with the other parts of that question, specifically including:

[0013] 3.1: Iterate through the list storing other parts of the problem, and randomly select a new problem type from the problem type set that is different from the original problem type of the current list item.

[0014] 3.2: Concatenate the other part of each question with the randomly selected question type to form the constructed negative sample and put it into the dataset.

[0015] 4) Define the loss function according to the above operation, make the constructed negative sample as a penalty term, so that the VQA model answers the question according to the image, not according to the language prior, which includes:

[0016] 4.1: Use Faster-RCNN and LSTM respectively to extract the features of the sample visual and text modalities, and perform multi-modal feature fusion to obtain the cross-modal feature F(Q i ,I i ).

[0017] 4.2: VQA can be regarded as a multi-label classification task, and the cross-modal feature F(Q i ,I i ) is sent to the classifier cls for classification, and the answer space prediction is performed by the following formula (a):

[0018]

[0019] Where Q i is the feature of the i-th sample text modality, and I i is the feature of the i-th sample visual modality.

[0020] 4.3: The VQA model needs to minimize the multi-label loss function L vqa , which can be represented by the following formula (b):

[0021]

[0022] Where ξ is the sigmoid function; l i is the corresponding target score of each label of the i-th sample.

[0023] 4.4: After the operation of step 3, the question is represented as Q′ i After repeating step 4.2 to obtain the answer space prediction probability of the negative sample, the loss function L shu represented by the following formula (c) can be constructed:

[0024]

[0025] Where P(A i |Q′ i ,I i ) is the answer space probability distribution of the i-th negative sample. Because the answer is derived from the negative sample, P(A i |Q′ i ,I iThe smaller the value of P(A), the better. Mathematically, minimizing P(A) is the ideal outcome. i |Q′ i ,I i The gradient of ) is greater than that of minimizing 1-log[P(A)]. i |Q′ i ,I i The gradient of [] is stable, so the loss function L can be […]. shu The optimized form is shown in equation (d) below:

[0026]

[0027] 4.5: Using the two loss functions mentioned above, we can obtain the following equation (e) for optimizing the final loss function L of the VQA model. sqt :

[0028] L sqt =L vqa +αL shu (e).

[0029] Here, α represents the weight parameter.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1) This invention splits the questions in the dataset into question types and other parts, randomly selects one question type from the set of question types, and recombines it with the other parts. This allows for the construction of negative samples using self-supervised learning without incorporating additional supervisory information. This method overcomes the language prior problem in the VQA domain, enabling VQA models to answer questions based on key regions of a given image.

[0032] 2) The loss function L proposed in this invention sqt It can be used as a penalty term to mitigate bias from language modalities, improve the robustness of the VQA model, and significantly improve the model's accuracy on the VQA-CP v2 dataset. Attached Figure Description

[0033] Figure 1 This is a flowchart of the present invention;

[0034] Figure 2 This is a schematic diagram of the operation process for an example. Detailed Implementation

[0035] See Figure 1 The depolarization removal method of the present invention includes the following steps:

[0036] 1) Break down the question in each image-question pair into question type and other parts of the question, specifically including:

[0037] 1.1: Analyze the source of language prior in visual question answering, through experiments and verification, bias mainly comes from text modality.

[0038] 1.2: In the text modality, that is, the question, the main source of language prior is the question type and the keyword. Since the keyword is difficult to locate, the keyword and other words are divided into the question other part.

[0039] 1.3: Set the question type length to 2, split a question into question type and question other part, and store them in two lists respectively.

[0040] 2) Extract the question type of each question in the data set to form a question type set.

[0041] 3) Randomly select a question type different from the question type before splitting from the question type set, and connect it with the other part of the question, which includes:

[0042] 3.1: Traverse the list storing the question other part in turn, and randomly extract a new question type different from the original question type of the current list item in the question type set.

[0043] 3.2: Splice the question other part of each item with the randomly extracted question type to form a constructed negative sample and put it into the data set.

[0044] 4) Define the loss function according to the above operation, make the constructed negative sample as a penalty term, so that the VQA model answers the question according to the image, not according to the language prior, which includes:

[0045] 4.1: Use Faster-RCNN and LSTM to extract the features of the sample visual modality and text modality respectively, and perform multi-modal feature fusion to obtain cross-modal feature F(Q i ,I i ).

[0046] 4.2: VQA can be regarded as a multi-label classification task, and the cross-modal feature F(Q i ,I i ) is sent to the classifier cls for classification, and the answer space is predicted by the following formula (a):

[0047]

[0048] Where Q i is the feature of the i-th sample text modality, and I i is the feature of the i-th sample visual modality.

[0049] 4.3: The VQA model needs to minimize the multi-label loss function L vqawhich can be represented by the following (b) formula:

[0050]

[0051] wherein ξ is a sigmoid function; l i is the corresponding target score of each label of the i th sample.

[0052] 4.4: After the operation of step 3, the question representation is Q i After the answer space prediction probability of the negative sample is obtained by repeating step 4.2, the loss function L shu :

[0053]

[0054] wherein P(A i |Q i ′,I i ) is the answer space probability distribution of the i th negative sample. Since the answer is derived from the negative sample, it is required that the value of P(A i |Q i ′,I i ) is as small as possible. In mathematics, the gradient of minimizing P(A i |Q i ′,I i ) is more stable than the gradient of minimizing 1-log[P(A i |Q i ′,I i )], so the loss function L shu can be optimized as the following (d) formula:

[0055]

[0056] 4.5: Using the above two loss functions, the following (e) formula can be used to obtain the final loss function L sqt for optimizing the VQA model:

[0057] L sqt =L vqa +αL shu (e)。

[0058] wherein α represents a weight parameter.

[0059] The application focuses on eliminating language priori in the text mode, constructing negative samples by using self-supervised learning, not introducing additional supervision information, and improving the accuracy of the visual question and answer model to a certain extent. Visual question and answer aims to make the model answer the question according to the given image, and eliminate language priori (de-bias) to improve the robustness of the model.

[0060] The application will be further described in connection with the specific embodiments and the accompanying drawings.

[0061] Embodiment 1

[0062] Referring to Figure 2 , the question in one image-question pair in the VQA-CP v2 dataset is extracted, and the question is divided into the question type "What color" and the other part "is the banana". In the question type set QTSet, a question type "Is there" different from the question type "What color" is randomly sampled and combined with the other part "is the banana" to obtain a negative sample "Is there is the banana". The original sample and the constructed negative sample are respectively sent into the VQA model for training to obtain different answer distribution predictions, and the loss function L is calculated. vqa and L shu , and the final loss function L sqt is obtained by weighted summation.

[0063] The above only describes the preferred embodiments of the application, and within the scope defined by the claims of the application, certain modifications can be made, but all will fall within the protection scope of the application.

Claims

1. A debiased visual question answering method based on a disordered problem type, characterized by, The method comprises the following specific steps: Step 1: split the question in each image-question pair into question type and question other part; Step 2: extract the question type of each question in the data set to form a question type set; Step 3: randomly select a question type different from the question type before splitting from the question type set, and connect it with the other part of the question; Step 4: define a loss function according to the above operation, make the constructed negative sample as a penalty item, so that the VQA model answers the question according to the image; The step 1 specifically comprises: 1.1: analyze the source of language prior in visual question answering, and verify that the bias comes from the text modality through experiments; 1.2: in the text modality, the source of language prior is the question type and the keyword, and the keyword and other words are divided into question other part; 1.3: set the question type length to 2, split a question into question type and question other part, and store them in two lists respectively; The step 3 specifically comprises: 3.1: traverse the list storing the question other part in turn, and randomly extract a new question type different from the original question type of the current list item in the question type set; 3.2: splice each item of the question other part with the randomly extracted question type to form a constructed negative sample, and put it into the data set; The step 4 specifically comprises: 4.1: Extracting features of visual modalities from samples using Faster-RCNN ; Use LSTM to extract features from text modalities , will feature and Multimodal feature fusion is performed to obtain cross-modal features. ; 4.2: The cross-modal features are fed into a classifier cls for classification, and the prediction of the answer space is made by the following (a) formula: (a). (a);​ wherein Q i is a feature of the i-th sample text modality, I i is a feature of the i-th sample visual modality; 4.3: According to the answer space The loss function is calculated from the ground truth by the following (b) formula; (b)； wherein ξ is a sigmoid function; l i is the corresponding target score for each label of the i-th sample; 4.4: Express the question obtained as The answer space prediction probability of the negative sample is obtained by repeating the above step 4.2, and the loss function L is constructed by the following (c) formula shu : (c)； wherein, is the answer space probability distribution for the ith negative sample; 4.5: Weighted sum of the two loss functions of step 4.3 and step 4.4 to obtain the final loss function L for optimizing the VQA model sqt which is represented by the following (e) formula: (e)； wherein is a weight parameter.

Images