Language modal debiased visual question answering method based on knowledge distillation
By employing knowledge distillation techniques and adaptive sample-level reweighted learning, the dependence of visual question answering models on language modalities is resolved, improving the model's generalization ability and learning efficiency, reducing overfitting, and demonstrating excellent performance, especially in handling visual question answering tasks with long-tailed distributions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAZHONG UNIV OF SCI & TECH
- Filing Date
- 2024-07-10
- Publication Date
- 2026-05-29
AI Technical Summary
Existing visual question answering models tend to rely on prior knowledge of language modalities during the learning process, resulting in poor generalization ability. Existing solutions also suffer from additional computational overhead or high annotation costs.
We employ a knowledge distillation-based approach, using a teacher model as supervision to train a student model. We utilize soft labeling and adaptive sample-level reweighting learning strategies to mitigate language modality bias and prevent overfitting.
This approach reduces reliance on prior language knowledge in visual question answering tasks, improves the model's generalization ability, reduces additional computational and annotation costs, and enhances the learning performance on tail samples.
Smart Images
Figure CN118885586B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to artificial intelligence technology, and more particularly to a visual question answering method based on knowledge distillation and language modality debiasing. Background Technology
[0002] Traditional Visual Question Answering (VQA) models tend to rely on prior knowledge from the linguistic modality for answer prediction during the learning and reasoning process. In this case, answer prediction is based on a shortcut within the linguistic modality (i.e., the statistical correlation between the question and the answer) rather than a comprehensive understanding of multimodal knowledge, resulting in poor generalization ability of VQA models. To address this issue, previous solutions have primarily focused on two approaches. One approach is based on data augmentation, such as CSS and CSST, which construct counterfactual samples by masking key objects in the image, guiding the model to focus on all important objects in the image and keywords in the question, thus mitigating the influence of linguistic priors. However, this approach incurs significant additional annotation costs. The other approach is based on the model design itself. Its typical strategy is to introduce an additional branch that processes only linguistic modality information, capturing linguistic modality bias information, and then subtracting the influence of the branch model during inference. While this approach aligns with human intuition, it also introduces additional computational overhead during the inference phase. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide a language modality debiasing visual question answering method based on knowledge distillation, which addresses the shortcomings of the prior art.
[0004] The technical solution adopted by this invention to solve its technical problem is: a visual question answering method based on knowledge distillation and language modality debiasing, comprising the following steps:
[0005] 1) Obtain a given image v∈I and a question q∈Q about the image;
[0006] 2) The student model is used to process the question and the image to obtain the answer to the question; the process of obtaining the student model is as follows:
[0007] 2.1) Construct the teacher model (T-VQA) and student model (S-VQA);
[0008] 2.2) The teacher model (T-VQA) is trained using a training set, which includes images, question texts, and answer texts;
[0009] 2.3) Train the student model (S-VQA);
[0010] 2.4) Using the teacher model and based on the set loss function, knowledge distillation is performed on the student model. The parameters of the teacher model are fixed and not updated. The entire learning process is optimized by updating the parameters of the student model to obtain a trained student model.
[0011] According to the above scheme, when training the teacher model in step 2.2), the input includes visual modal features and linguistic modal features extracted from images, question text, and answer text.
[0012] The output is the probability distribution of the answers predicted by the teacher model.
[0013] According to the above scheme, in step 2.2), visual modal features are extracted using a convolutional neural network (CNN) and a target detection network Faster R-CNN; linguistic modal features are extracted using a word embedding model and a text encoder.
[0014] According to the above scheme, the loss function used for training in step 2.2) is the binary cross-entropy (BCE) loss function:
[0015]
[0016] Where, p i y represents the model's prediction. i represents the true label, and N represents the number of samples in the dataset.
[0017] According to the above scheme, in step 2.4), the loss function is set as follows:
[0018]
[0019] Where α is the smoothing factor, express The entropy is a constant; Output probabilities for the teacher model And student model output probability The KL divergence between them, where y is the one-hot label. This is the cross-entropy.
[0020] According to the above scheme, in step 2.4), the loss function is set as follows:
[0021]
[0022] in,
[0023]
[0024]
[0025] Here, exp() is an exponential function. Output probability distribution for the teacher model. Output probability distribution for the student model. The binary cross-entropy (BCE) loss function is used.
[0026] α is the smoothing factor, and H is the information entropy function. express The entropy is a constant;
[0027] Output probabilities for the teacher model And student model output probability The KL divergence between them, where y is the one-hot label. This is the cross-entropy.
[0028] The beneficial effects of this invention are:
[0029] 1. The solution of the present invention avoids additional data labeling by introducing a teacher model and using its output as a soft label for supervising the student model;
[0030] 2. A new learning strategy is introduced, using soft labels as a regularizer to address the overfitting problem of common samples and reduce the dependence on language priors.
[0031] 3. This invention utilizes the relative relationship between the teacher model output and the student model output to dynamically adjust the weight of each sample, thereby mitigating the overfitting problem of head samples in previous schemes, promoting a more balanced learning process, and preventing the most common answer from incorrectly dominating the correct answer. Attached Figure Description
[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:
[0033] Figure 1 This is a structural schematic diagram of an embodiment of the present invention;
[0034] Figure 2 This is a schematic diagram illustrating the influence of the temperature hyperparameter τ and the weight scaling factor β on the model performance in an embodiment of the present invention. Detailed Implementation
[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0036] like Figure 1 As shown, a language modality-based debiased visual question answering method based on knowledge distillation includes the following steps:
[0037] 1) Obtain a given image v∈I and a question q∈Q about the image;
[0038] 2) The student model is used to process the question and the image to obtain the answer to the question; the process of obtaining the student model is as follows:
[0039] 2.1) Construct the teacher model (T-VQA) and student model (S-VQA);
[0040] 2.2) The teacher model (T-VQA) is trained using a training set, which includes images, question texts, and answer texts;
[0041] When training the teacher model, the input includes visual modal features and linguistic modal features extracted from images, question text, and answer text, and the output is the probability distribution of the answers predicted by the teacher model.
[0042] Visual modal features were extracted using a convolutional neural network (CNN) and a Faster R-CNN object detection network; linguistic modal features were extracted using a word embedding model and a text encoder.
[0043] The loss function used for training is the binary cross-entropy (BCE) loss function:
[0044]
[0045] Where, p i y represents the model's prediction. i represents the true label, and N represents the number of samples in the dataset.
[0046] 2.3) Train the student model (S-VQA);
[0047] 2.4) Using the teacher model and based on the set loss function, knowledge distillation is performed on the student model. The parameters of the teacher model are fixed and not updated. The entire learning process is optimized by updating the parameters of the student model to obtain a trained student model.
[0048] In step 2.4), the loss function is set as follows:
[0049]
[0050] Where α is the smoothing factor, express The entropy is a constant; Output probabilities for the teacher model And student model output probability The KL divergence between them, where y is the one-hot label. Cross-entropy;
[0051] The loss function described above reveals that knowledge distillation essentially involves the student model simultaneously minimizing the cross-entropy relative to the ground truth labels and the KL divergence relative to the teacher model. The second term in this equation, namely... This can be viewed as a regularization term, designed to adjust the parameters of the student model to prevent overfitting. Therefore, the soft labels in knowledge distillation act as a regularizer. Specifically, in visual question answering tasks, the VQA dataset exhibits a long-tailed distribution; for example, yellow bananas constitute the vast majority. Therefore, when the model faces a question like "What color is a banana?", the true label is usually 'yellow'. During learning, the model tends to remember, often overfitting these head samples while ignoring tail samples (e.g., green bananas). As a result, when faced with the question "What color is a banana?", the model typically predicts 'yellow' with high confidence. However, this characteristic undoubtedly hinders the model from acquiring more general knowledge during training. Therefore, this invention utilizes the regularization effect of soft labels in knowledge distillation to alleviate the overfitting problem in VQA tasks and reduce the model's overemphasis on head samples. In addition to the regularization effect, the soft labels in the knowledge distillation paradigm can also provide additional supervisory signals with semantically similar meanings, further aiding model learning. For example, when answering the question "...what color is it?", color-related supervision signals ('red', 'blue', 'yellow', etc.) from the teacher model have high values. These can help the student model narrow down the range of candidate answers, tell the model that the question is color-related, and prevent it from predicting other non-color-related answers.
[0052] Given the long-tail distribution phenomenon in the VQA dataset, the model tends to focus more on the head samples, resulting in insufficient learning of the tail samples. To alleviate this problem, this invention further optimizes the model by proposing an adaptive sample-level reweighted learning strategy to reduce overfitting of the head samples while increasing the model's attention to the tail samples.
[0053] In step 2.4), the loss function is set as follows:
[0054]
[0055] in,
[0056]
[0057] Here, exp() is an exponential function. Output probability distribution for the teacher model. Output probability distribution for the student model. The binary cross-entropy (BCE) loss function is used.
[0058] α is the smoothing factor, and H is the information entropy function. express The entropy is a constant; Output probabilities for the teacher model And student model output probability The KL divergence between them, where y is the one-hot label. This is the cross-entropy.
[0059] Finally, during training, the learning objective is to minimize the overall loss, i.e. and The weighted sum.
[0060]
[0061] Here, β is a scalar weight hyperparameter used to balance the adaptive loss and the knowledge distillation loss.
[0062] To evaluate the effectiveness of the proposed solution, this invention is primarily validated through experiments on real-world datasets.
[0063] Dataset and Baseline Scheme Introduction: This invention evaluates the performance of its proposed scheme on the VQAv2 and VQA-CPv2 datasets. Statistical information for the two datasets is shown in Tables 1 and 2, respectively. The scheme is compared with benchmark methods SAN, S-MRL, UpDn, GVQA, and previous state-of-the-art methods, including data augmentation-based methods such as HINT, SCR, CL, SSL, CSS, AEMP, CSST, KDDAug, SQT, D-VQA, and DDG, as well as non-data augmentation methods such as AReg, RUBi, LMH, DLR, AdaVQA, CCB, LP-Focal, Re-scaling, CF-VQA, IntroD, GGE, GGD, MDDC, RMLVQA, and GenB.
[0064] Table 1. VQAv2 Dataset Information
[0065] Dataset partitioning Number of images Number of questions Number of answers training set 82k 248k 2.4M Validation set 40k 121k 1.2M total 123k 369k 3.6M
[0066] Table 2 VQA-CPv2 Dataset Information
[0067] Dataset partitioning Number of images Number of questions Number of answers training set 121k 438k 4.3M test set 98k 219k 2.1M total 123,287 658k 6.5M
[0068] Evaluation metrics: In the experiment, this invention uses accuracy to measure the model's performance.
[0069]
[0070] Here, #humans provided answers indicates that each question has 10 manually annotated real answers. All questions are divided into three types: Yes / No questions, Number questions, and Other questions, with All representing all types of questions. During the evaluation process, Acc. is used to evaluate the accuracy of each type of question.
[0071] Implementation Details: The method of this invention is plug-and-play, and any other VQA model can be used as its teacher model. To better demonstrate the effectiveness of the KDAR scheme of this invention, a biased VQA model is selected as the teacher model. In the experiments, D-VQA is selected as the teacher model. The student model uses UpDn, combining only the loss term of this invention with the sample loss of the teacher. During the training phase, the temperature constant τ is set to 2.5. Other parameters follow the experimental settings of the teacher model and are not changed, such as learning rate, batch size, or optimizer. In the implementation of this invention, the hyperparameter β is set to 3.
[0072] Table 3 Comparison of experimental results on the VQA-CPv2 dataset
[0073]
[0074]
[0075]
[0076] Experimental comparison with state-of-the-art models: The experimental results on the VQA-CPv2 dataset are shown in Table 3. The present invention yielded the following observations: (1) The method of the present invention achieved the best overall accuracy among all comparison methods, including both augmented and non-augmented methods; (2) The method of the present invention performed best when answering more challenging questions, namely “Num.” and “Other” questions, which indicates that the method of the present invention can truly improve performance by removing linguistic bias.
[0077] Combining with different backbone models: To further demonstrate the scalability of the proposed method, additional experiments were conducted based on different backbone networks, including SAN, UpDn, and LXMERT. LXMERT is a multimodal pre-trained model that has demonstrated strong performance in many downstream multimodal tasks. From the experimental results shown in Table 4, it was observed that the performance of all backbone networks was significantly improved when combined with the proposed method. In particular, when combined with LXMERT, the proposed method achieved a new state-of-the-art overall accuracy of 71.33% on the VQA-CPv2 dataset. These improvements are significant and consistent, further validating the effectiveness of the proposed method.
[0078] Table 4 shows the experimental results of combining KDAR with different backbone models.
[0079]
[0080] Ablation experiment: Two modules in the method proposed in this invention and The effectiveness is shown in Table 5. From the experimental results, it can be seen that the present invention, when used alone... When the module is used, the accuracy exceeds the baseline UpDn model by 0.92%, which means that the proposed reweighting method can help reduce distribution bias, and when The model achieved an absolute improvement of 20.95% when used for evaluation. The results indicate that soft labeling of the teacher model can mitigate language bias in the VQA task. Furthermore, this invention shows that when… and When used together, the best overall accuracy of 62.86% was achieved.
[0081] Table 5. Effects of different components on ablation experiments.
[0082]
[0083] Furthermore, this invention evaluated the effects of different combinations of hyperparameters, including the influence of the weighting hyperparameter β and the temperature hyperparameter τ on VQA-CPv2. The results are as follows: Figure 2 As shown in the figure, the accuracy of different values of β increases and then decreases with increasing temperature τ. When β = 3 and τ = 2.5, the solution of the present invention achieves the best overall accuracy.
[0084] Experimental Results on VQAv2 Dataset: To verify the performance of the proposed solution on the Independent and Identically Distributed (IID) data scenario, additional experiments were conducted on the VQAv2 dataset. The corresponding experimental results are shown in Table 6. The experimental results show that the proposed method achieved an overall accuracy of 65.54%, which is the best among all compared methods. Especially on the most challenging "Other" type problems, the proposed method significantly outperforms other solutions.
[0085] Table 6 shows the experimental results on the VQAv2 dataset.
[0086]
[0087]
[0088] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A language modality-based debiased visual question answering method based on knowledge distillation, characterized in that, Includes the following steps: 1) Obtain a given image And a question about images ; 2) Use the student model to process the question and the image to obtain the answer to the question; The process of obtaining the student model is as follows: 2.1) Construct teacher and student models; 2.2) Train the teacher model using the training set, which includes images, question text, and answer text; Visual modal features were extracted using a convolutional neural network (CNN) and a faster R-CNN object detection network; linguistic modal features were extracted using a word embedding model and a text encoder. The loss function used for training is the binary cross-entropy loss function: in, This represents the model's prediction. Indicates the true label, Indicates the number of samples in the dataset; 2.3) Train the student model; 2.4) Using the teacher model and based on the set loss function, knowledge distillation is performed on the student model. The parameters of the teacher model are fixed and not updated. The parameters of the student model are updated to optimize the entire learning process and obtain a trained student model. The loss function is defined as follows: ； in, ； ； in, It is an exponential function. Output probability distribution for the teacher model. Output probability distribution for the student model. The binary cross-entropy loss function; It is a smoothing factor; Output probabilities for the teacher model And student model output probability KL divergence between them; y is the one-hot label; .
2. The language modality debiasing visual question answering method based on knowledge distillation according to claim 1, characterized in that, In step 2.2), when the teacher model is trained, the input includes visual modal features and linguistic modal features extracted from images, question text, and answer text. The output is the probability distribution of the answers predicted by the teacher model.
3. An electronic device, characterized in that, include: One or more processors; as well as Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 2.
4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 2.