Question type perception based test time vqa model debiasing method and system

Abstract

The application belongs to the technical field of human-computer interaction, and provides a test-time VQA model debiasing method and system based on question type perception. The test-time VQA model debiasing method based on question type perception comprises sequentially selecting a group of data from a test set as positive samples and establishing negative samples; the question type corresponding to each sample is adaptively matched with different answer entropy value change rate thresholds; a VQA model trained by a training set is used to predict the answers of the positive samples and the negative samples; the answer entropy values of the predicted positive samples and negative samples are calculated, and an adaptive entropy value minimization method is used to judge the reliability of each sample; the answer entropy value change rate of the reliable samples is calculated, and then compared with the answer entropy value change rate threshold of the corresponding type question to judge whether the sample is biased, and the unbiased sample is used to optimize the pre-trained VQA model until all the test set data is predicted. It can reduce the influence of language bias samples in the test set on model adaptation.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, and in particular relates to a method and system for removing biases from a VQA model during testing based on problem type awareness. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] As a primary channel for human-computer interaction in the era of artificial intelligence, visual dialogue robots can be categorized into three types: task-oriented dialogue, question-and-answer dialogue, and chat-based dialogue. Among these, visual question-answering (VQA) systems differ from traditional text-based question-and-answer systems. Given an image, the visual question-answering model needs to analyze the content of the textual question and correctly answer the questions described in the text based on the image.

[0004] Existing VQA models perform well when they have access to large-scale training datasets and when the training and test sets share similar data distributions. However, current processes for adapting pre-trained visual question answering models at test time suffer from the following problems:

[0005] (1) Current visual question answering models rely too much on prior language distribution in the training data. Once there is a significant difference in language distribution between the training set and the test set, the model will perform poorly on the test set, which is the language bias problem.

[0006] (2) In the VQA task, current research on test-time adaptive methods in the context of visual question answering introduces a strategy to minimize the entropy of unreliable test samples. However, it ignores the fact that different types of questions face different answer spaces, and using a fixed entropy threshold cannot make good use of the constantly changing data distribution. Summary of the Invention

[0007] To address the technical problems existing in the background art, the present invention provides a test-time VQA model bias removal method and system based on question type awareness, which can adapt to the process of pre-training the visual question answering model during test time and improve the prediction performance of the model in different application scenarios.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides a method for debiasing a test-time VQA model based on problem type awareness.

[0010] A bias removal method for test-time VQA models based on problem type awareness, comprising:

[0011] A set of data is selected sequentially from the test set as positive samples, and negative samples are established; among them, the question type corresponding to each sample is adaptively matched with different answer entropy change rate thresholds;

[0012] Use a VQA model trained on a training set to predict the answers for positive and negative samples;

[0013] Calculate the entropy values ​​of the predicted positive and negative samples, and use an adaptive entropy minimization method to determine the reliability of each sample;

[0014] Calculate the rate of change of answer entropy values ​​for reliable samples, and then compare it with the threshold for the rate of change of answer entropy values ​​for the corresponding type of question to determine whether the sample is biased. Use unbiased samples to optimize the pre-trained VQA model until all test set data has been predicted.

[0015] As one implementation method, once all test samples have been read, the parameters of the VQA model are updated, and all test samples are predicted. The criterion for judging the prediction result of each sample is: the maximum value in the positive sample prediction probability distribution is selected as the final result.

[0016] As one implementation method, in the process of determining whether a sample is biased, if the rate of change of the entropy value of the answer is lower than the threshold of the rate of change of the entropy value of the answer to the corresponding type of question, the sample will be considered biased; otherwise, it will be unbiased.

[0017] As one implementation method, the overall optimization loss of the pre-trained VQA model during testing consists of four parts: test-time adaptive optimization loss, debiased entropy minimization, direct optimization loss for positive samples, and direct optimization loss for negative samples.

[0018] As one implementation method, the process of using a VQA model trained on a training set to predict the answer for the corresponding data set is as follows:

[0019] Collect question types and directly collect answers from the dataset to obtain the answer space for the entire dataset;

[0020] A pre-trained BERT model is used to determine the correlation between a specific question type and different answers; based on the correlation values ​​from largest to smallest, the top number of answers are selected to define a fine-grained answer space for a specific question type.

[0021] Human expert knowledge is incorporated to update and refine the answer space, thereby generating a new answer space to represent candidate answers.

[0022] As one implementation method, the process of creating negative samples is as follows:

[0023] To construct negative samples, randomly select an irrelevant image from a small batch of known problems.

[0024] A second aspect of the present invention provides a test-time VQA model bias removal system based on problem type awareness.

[0025] A test-time VQA model bias removal system based on problem type awareness, comprising:

[0026] The negative sample construction module is used to sequentially select a set of data from the test set as positive samples and build negative samples; wherein, the question type corresponding to each sample adaptively matches different answer entropy change rate thresholds;

[0027] The answer prediction module is used to predict the answers for positive and negative samples using a VQA model trained on the training set.

[0028] The reliability assessment module is used to calculate the entropy values ​​of the predicted positive and negative samples, and uses an adaptive entropy minimization method to determine the reliability of each sample.

[0029] The bias detection module calculates the rate of change of answer entropy values ​​for reliable samples and compares it with the threshold for the rate of change of answer entropy values ​​for the corresponding type of question to determine whether the sample is biased. Unbiased samples are then used to optimize the pre-trained VQA model until all test set data has been predicted.

[0030] As one implementation method, the test-time VQA model bias removal system based on problem type awareness further includes:

[0031] The prediction module updates the parameters of the VQA model after all test samples have been read, and the prediction of all test samples is completed. The criterion for judging the prediction result of each sample is: the maximum value in the prediction probability distribution of positive samples is selected as the final result.

[0032] As one implementation method, in the deviation judgment module, during the process of judging whether a sample is biased, if the rate of change of the answer entropy value is lower than the threshold of the rate of change of the answer entropy value of the corresponding type of question, the sample will be considered biased; otherwise, it will be unbiased.

[0033] As one implementation method, the overall optimization loss of the pre-trained VQA model during testing consists of four parts: test-time adaptive optimization loss, debiased entropy minimization, direct optimization loss for positive samples, and direct optimization loss for negative samples.

[0034] As one implementation method, in the answer prediction module, the process of using a VQA model trained on the training set to predict the answer for the corresponding data set is as follows:

[0035] Collect question types and directly collect answers from the dataset to obtain the answer space for the entire dataset;

[0036] A pre-trained BERT model is used to determine the correlation between a specific question type and different answers; based on the correlation values ​​from largest to smallest, the top number of answers are selected to define a fine-grained answer space for a specific question type.

[0037] Human expert knowledge is incorporated to update and refine the answer space, thereby generating a new answer space to represent candidate answers.

[0038] As one implementation method, the process of building negative samples in the negative sample construction module is as follows:

[0039] To construct negative samples, randomly select an irrelevant image from a small batch of known problems.

[0040] A third aspect of the present invention provides a computer-readable storage medium.

[0041] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the problem-type-aware test-time VQA model debiasing method described above.

[0042] A fourth aspect of the present invention provides a computer device.

[0043] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the problem type-aware test-time VQA model debiasing method described above.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] This invention introduces adaptive entropy minimization based on problem type to identify fine-grained and unreliable samples, controlling the interference of samples on model adaptation during testing. In response to the language bias problem in existing VQA models, this invention uses the rate of change to determine whether there is bias in the samples, and uses unbiased samples to optimize the model, thereby reducing the impact of language biased samples in the test set on model adaptation and improving the prediction performance of the model in different application scenarios.

[0046] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0047] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0048] Figure 1This is an example of language bias in the VQA model;

[0049] Figure 2 These are schematic diagrams of different types of answer spaces;

[0050] Figure 3 This is a flowchart of the test-time VQA model bias removal method based on problem type awareness according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the adaptive entropy threshold according to an embodiment of the present invention;

[0052] Figure 5 These are some examples of results from embodiments of the present invention. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0054] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0055] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0056] Terminology Explanation:

[0057] Visual Question Answering (VQA) is a multimodal artificial intelligence task that requires robots to simultaneously understand images and text, answering natural language questions about the content of images. Specifically, VQA tasks require models to answer natural language questions related to a given image. These questions can relate to objects, scenes, relationships, or other image features. Typically, questions are provided in text form, such as "How many cats are in this image?" or "What is the subject of this picture?" The model needs to analyze the image content and the question, then generate or select an appropriate text answer. Visual question answering technology has high application value and potential. For example, in the medical field, VQA can be used for medical image analysis and disease diagnosis, improving doctors' work efficiency. In the education field, VQA can be used for educational assistance and intelligent education, helping students better understand teaching materials, etc.

[0058] Language bias refers to the VQA model's over-reliance on linguistic priors when answering questions or its being influenced by linguistic expressions when processing question and image information, rather than truly understanding the meaning of the question or the content of the image.

[0059] "Out-of-distribution" (OOD): A term commonly used in machine learning and statistics, referring to a situation where a sample or data point belongs to a distribution of data that the model has not seen or included in during training.

[0060] SGD (Stochastic Gradient Descent): Gradient descent is a common algorithm used to optimize machine learning models. It is an iterative optimization algorithm, typically used to train neural networks and other machine learning models.

[0061] Entropy minimization: an optimization method used in machine learning and statistical modeling, typically to reduce the uncertainty or disorder of a model, thereby improving its performance and robustness. Entropy is a concept in information theory that represents the uncertainty of a random variable.

[0062] SOTA is an abbreviation for "state-of-the-art," referring to the most advanced technology or method in a particular field. In machine learning and artificial intelligence, SOTA typically refers to a model or algorithm that performs best on a specific task or dataset.

[0063] Transformer is a neural network architecture used for Natural Language Processing (NLP) tasks. It consists of an encoder and a decoder. The encoder converts the input text into a series of vector representations, while the decoder decodes these vectors into the corresponding output text. In general, Transformer is a neural network architecture based on a self-attention mechanism, which can better handle sequential data and is widely used in natural language processing tasks.

[0064] In this invention, visual dialogue robots represent the most representative entry point into the era of artificial intelligence. With the rapid development of deep learning technology, different types of dialogue robots (such as Baidu Xiaodu and Xiaomi speakers) have gradually entered people's lives. As the main channel for human-computer interaction in the era of artificial intelligence, dialogue can be divided into three categories: task-oriented dialogue, question-and-answer dialogue, and chat-based dialogue. Among them, visual question answering (VQA) systems belong to a branch of question-and-answer systems in the field of natural language processing. Unlike traditional text-based question answering, given an image, the visual question answering model needs to analyze the content of the textual question and correctly answer the questions described in the text based on the image. Multimodal interaction is a further development direction for dialogue robots, and as the most representative downstream task of multimodal interaction, visual question answering has received widespread attention from the academic community since its inception.

[0065] With the rapid development of deep learning, breakthroughs in numerous association techniques have provided fertile ground for the development of visual question answering (VQA). Models such as ResNet, Faster R-CNN, and Clip in the field of vision, and the Transformer architecture in the field of natural language processing, have benefited from the rapid advancements in these NLP and computer vision (CV) methods. Existing VQA models perform well when they have access to large-scale training datasets and when the training and test sets share similar data distributions. However, these well-trained VQA models often encounter difficulties when facing out-of-distribution scenarios in real-world applications. In these cases, the answer distributions between the training and test datasets may differ. For example, ... Figure 1 As shown, when we query a trained VQA model with the question "What color is her shirt?", it might provide a biased answer like "black," reflecting the answer distribution in the training dataset while ignoring the real visual environment. To address this problem, many researchers are exploring methods such as Overcoming Language Bias in Visual Question Answering (VQA) and Test-time Adaptation. These two methods will be briefly introduced below.

[0066] Language debiasing in visual question answering (VQA) is crucial because current VQA models heavily rely on prior language distributions when generating answers. Once the language distributions of the training and test sets differ significantly, current VQA models suffer a substantial performance degradation due to language bias. To enhance the robustness of existing VQA models, researchers have focused on three main approaches: data construction, data augmentation, and model design. To mitigate the reliance on additional annotations, recent methods have proposed constructing negative samples using self-supervised learning or adversarial learning paradigms. These methods typically introduce debiasing techniques during training and select the best model based on performance on the test dataset. However, in practical applications, the model should not have access to information about distribution mismatches before evaluation. Therefore, this invention follows this novel and intuitive setting, using only test data with a different distribution than the training data to optimize the VQA model through a test-time adaptive approach.

[0067] Test-time adaptation, a technique designed to adapt pre-trained models to potentially changing target test domains without requiring test label information, is widely used in image classification. Researchers have proposed numerous test-time adaptation methods; however, to date, only one study has focused on test-time adaptation in the context of Visual Question Answering (VQA). This work proposes a method to minimize the entropy of unreliable test samples. However, this work overlooks the fact that different types of questions may have diverse answer spaces (e.g., ...). Figure 2 As shown, simply applying a fixed entropy threshold may not be suitable for constantly changing data distributions. Accordingly, this invention proposes an entropy minimization and language debiasing method based on problem type, making full use of adaptive entropy threshold constraints to adapt the VQA model during testing.

[0068] Example 1

[0069] Reference Figure 3 This embodiment provides a method for removing biases from a test-time VQA model based on problem type awareness, which specifically includes the following steps:

[0070] Step 1: Select a set of data as positive samples from the test set in sequence, and establish negative samples; wherein, the question type corresponding to each sample is adaptively matched with different answer entropy change rate thresholds.

[0071] In practice, the process of establishing negative samples is as follows:

[0072] To construct negative samples, randomly select an irrelevant image from a small batch of known problems.

[0073] For example: Select a pair of test samples (v j ,q j ) as positive samples, and then from question qj A non-relevant image is randomly selected from a small batch to construct a negative sample.

[0074] Step 2: Use the VQA model trained on the training set to predict the answers for positive and negative samples.

[0075] In practice, the process of using a VQA model trained on the training set to predict the answer for the corresponding data set is as follows:

[0076] Step 2.1: Collect question types and directly collect answers from the dataset to obtain the answer space for the entire dataset;

[0077] For example, question types (QTs) are first collected from the VQA-v2 dataset to facilitate efficient computation during test-time adaptation. Notably, no additional data is used to pre-train the VQA model. Answers are then collected directly without any additional question information, yielding the answer space AS over the entire dataset.

[0078] Step 2.2: Use the pre-trained BERT model to determine the correlation between a specific question type and different answers; select the top number of answers from largest to smallest based on the correlation value, and define a fine-grained answer space for the specific question type;

[0079] Using pre-trained BERT, the correlation between a specific question type qt∈QT and different answers can be determined by the following equation:

[0080]

[0081] In the above formula, corr qt The relevance score is represented by , softmax represents the softmax function, and the specific calculation formula is shown in Equation 2. |AS| represents the size of the answer space AS.

[0082]

[0083] Based on the relevant values, select the top few answers (e.g., 1000) from largest to smallest, and define a fine-grained answer space for a specific question type.

[0084] Step 2.3: Introduce human expert knowledge to update the refined answer space, thereby generating a new answer space to represent candidate answers.

[0085] To enhance the reliability of this answer space, human expert knowledge is introduced to generate a new answer space AS. qt A threshold is used to represent candidate answers. Subsequently, an adaptive threshold is introduced based on the question type to enable fine-grained and reliable sample selection.

[0086] Step 3: Calculate the entropy values ​​of the predicted positive and negative samples, and use the adaptive entropy minimization method to determine the reliability of each sample.

[0087]

[0088] The above formula Represents sample (v) j ,q j Whether the sample (v) is valid data, II represents the judgment function, when the sample (v) is valid data. j ,q j The entropy of ) is less than the threshold Ent(|AS) qt When |), the value is 1 (i.e., a reliable sample); otherwise, it is 0. |AS qt | represents the number of candidate answers for question type qt. α is a hyperparameter for entropy. Entropy() is a typical entropy calculation equation, specifically calculated using Equation 4 below.

[0089]

[0090] In the above formula, N te The value represents the number of samples, θ is a hyperparameter, and log_softmax is the value calculated by the logsoftmax function as shown in Equation 5.

[0091]

[0092] Based on the adaptive entropy minimization method, unreliable samples can be identified, and reliable samples can be used to update the parameters of the pre-trained VQA model for adaptive testing. The formula for optimizing the loss is as follows:

[0093]

[0094] Step 4: Calculate the rate of change of answer entropy values ​​for reliable samples, and then compare it with the threshold for the rate of change of answer entropy values ​​for the corresponding type of question to determine whether there is any bias in the samples. Use unbiased samples to optimize the pre-trained VQA model until all test set data has been predicted.

[0095] The inherent linguistic bias in datasets often causes VQA models to rely on the superficial relevance between questions and answers rather than effectively incorporating visual information for reasoning. This problem can be exacerbated in scenarios involving test-time adaptation. Typically, an intrinsic approach to mitigating bias without utilizing additional information involves introducing a self-supervised learning objective.

[0096] In the specific implementation process, when judging whether a sample is biased, if the rate of change of the entropy value of the answer is lower than the threshold of the rate of change of the entropy value of the answer to the corresponding type of question, the sample will be considered biased; otherwise, it will be unbiased.

[0097] Predictions obtained by comparing negative sampling and Given these conditions, we first calculate their entropy using Formula 4 to obtain the entropy value for positive prediction. and the entropy value of negative prediction

[0098]

[0099] Then, the entropy values ​​within the small batch size B are normalized to ensure that the data is not too discrete.

[0100]

[0101] To more accurately determine whether a sample is biased, the concept of the rate of change of answer entropy Δ is introduced. If the rate of change of answer entropy Δ is lower than the threshold δ for the corresponding question type, the sample is considered biased; otherwise, it is unbiased. This process can be described as follows:

[0102]

[0103] Using selected biased yet reliable samples, we can eliminate the negative impacts and biases of these test samples during the test-time adaptation process. Therefore, the debiased entropy minimization based on Equations 3 and 9 can be expressed as follows:

[0104]

[0105] In addition to minimizing entropy for reliable biased samples using Equation 10, this paper also directly eliminates the linguistic bias of the VQA model during testing, i.e., minimizing entropy for unbiased and reliable samples as well. This process is described as follows:

[0106]

[0107] It is worth noting that the true answers to the test samples are not available during the test-time adaptation period. Therefore, the prediction with the highest score is selected as the pseudo-label, i.e. Therefore, the direct optimization loss for positive and negative samples can be expressed as:

[0108]

[0109] In summary, the goal is to optimize the VQA model during testing. The overall optimization loss of the pre-trained VQA model during testing consists of four parts: test-time adaptive optimization loss, debiased entropy minimization, direct optimization loss for positive samples, and direct optimization loss for negative samples, as shown in Equation 13.

[0110]

[0111] In one or more embodiments, once all test samples have been read, the parameters of the VQA model are updated, and prediction for all test samples is complete. The criterion for judging the prediction result of each sample is: selecting... Figure 3 The maximum value in the positive sample prediction probability distribution shown is the final result.

[0112] To verify the effectiveness of the method, tests were conducted on the public datasets VQA-CP-v1 and VQA-CP-v2. The data distribution of the two datasets is shown in Table 1. The training set of the former contains approximately 118K photos and 244K questions, while the test set contains approximately 87K photos and 125K questions. The training set of the latter contains approximately 121K photos and 438K questions, while the test set contains 98K photos and 220K questions.

[0113] Table 1. Distribution of VQA-CP-v1 and VQA-CP-v2 datasets

[0114]

[0115] The model is primarily implemented using Python and PyTorch. The experimental environment is Ubuntu 20.04 with an RTX 3090 GPU. Since our proposed method, QED, is model-independent, we incorporate QED into the UpDn baseline model to improve its generalization ability on VQA-CP V1 and VQA-CP V2. During the adaptation period at test time, we utilize the SGD optimizer to update the proposed adapted model, with the SGD momentum set to 0.9. The learning rate is initially initialized to 0.01, and the weight decay is 0.0001. The batch size B is set to 512. The hyperparameter α used to adjust the entropy is set to 0.1. The rate of change threshold δ is set to 2.0.

[0116] The main evaluation metric used below is accuracy. The method proposed in this embodiment is compared with the existing state-of-the-art methods TENT, TPT, CoTTA, and ETA on VQA-CP-*, as well as the existing VQA method TDS that adopts a test-time adaptive strategy. The comparison results on the above two datasets are shown in Table 2.

[0117] Table 2. Experimental results on the VQA-CP-v1 and VQA-CP-v2 test sets.

[0118]

[0119]

[0120] As can be seen from the table above, the method in this embodiment achieves a higher overall accuracy on both datasets than other methods, which sufficiently demonstrates the effectiveness of the method presented in this invention. Furthermore, Figure 5 The results show some predictions. Here, we mainly compare the method of this invention, the baseline method UpDn method, and the TDS method. It can be seen that the method of this invention has higher prediction accuracy.

[0121] Example 2

[0122] This embodiment provides a test-time VQA model bias removal system based on problem type awareness, which specifically includes the following modules:

[0123] (1) Negative sample construction module, which is used to select a set of data as positive samples from the test set in sequence and build negative samples; wherein, the question type corresponding to each sample is adaptively matched with different answer entropy change rate thresholds.

[0124] In the specific implementation process, the process of creating negative samples in the negative sample construction module is as follows:

[0125] To construct negative samples, randomly select an irrelevant image from a small batch of known problems.

[0126] (2) Answer prediction module, which is used to predict the answers for positive and negative samples using a VQA model trained on the training set.

[0127] Specifically, in the answer prediction module, the process of using a VQA model trained on the training set to predict the answer for the corresponding data set is as follows:

[0128] Collect question types and directly collect answers from the dataset to obtain the answer space for the entire dataset;

[0129] A pre-trained BERT model is used to determine the correlation between a specific question type and different answers; based on the correlation values ​​from largest to smallest, the top number of answers are selected to define a fine-grained answer space for a specific question type.

[0130] Human expert knowledge is incorporated to update and refine the answer space, thereby generating a new answer space to represent candidate answers.

[0131] (3) Reliability judgment module, which is used to calculate the answer entropy value of the predicted positive and negative samples, and uses the adaptive entropy minimization method to judge the reliability of each sample.

[0132] (4) Deviation judgment module, which is used to calculate the change rate of answer entropy value of reliable sample, and then compare it with the threshold of the change rate of answer entropy value of corresponding type of question to determine whether the sample has deviation. Unbiased samples are used to optimize the pre-trained VQA model until all test set data are predicted.

[0133] In the specific implementation process, in the deviation judgment module, when judging whether a sample is biased, if the rate of change of the answer entropy value is lower than the threshold of the rate of change of the answer entropy value of the corresponding type of question, the sample will be considered biased; otherwise, it will be unbiased.

[0134] In practice, the overall optimization loss of the pre-trained VQA model during testing consists of four parts: test-time adaptive optimization loss, debiased entropy minimization, direct optimization loss for positive samples, and direct optimization loss for negative samples.

[0135] In one or more embodiments, the problem type-aware test-time VQA model bias removal system further includes:

[0136] The prediction module updates the parameters of the VQA model after all test samples have been read, and the prediction of all test samples is completed. The criterion for judging the prediction result of each sample is: the maximum value in the prediction probability distribution of positive samples is selected as the final result.

[0137] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0138] Example 3

[0139] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the problem type-aware test-time VQA model bias removal method described above.

[0140] Example 4

[0141] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the problem type-aware test-time VQA model bias removal method described above.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0143] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0145] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for bias removal in a test-time VQA model based on problem type awareness, characterized in that, include: A set of data is selected sequentially from the test set as positive samples, and negative samples are established; among them, the question type corresponding to each sample is adaptively matched with different answer entropy change rate thresholds; Use a VQA model trained on a training set to predict the answers for positive and negative samples; Calculate the entropy values ​​of the predicted positive and negative samples, and use an adaptive entropy minimization method to determine the reliability of each sample; Calculate the rate of change of answer entropy values ​​of reliable samples, and then compare it with the threshold of the rate of change of answer entropy values ​​of corresponding types of questions to determine whether there is any bias in the samples. Use unbiased samples to optimize the pre-trained VQA model until all test set data has been predicted. In determining whether a sample is biased, if the rate of change of the entropy value of the answer is lower than the threshold of the rate of change of the entropy value of the answer to the corresponding type of question, the sample will be considered biased; otherwise, it will be unbiased. The process of using a VQA model trained on the training set to predict the answer for the corresponding data set is as follows: Collect question types and directly collect answers from the dataset to obtain the answer space for the entire dataset; A pre-trained BERT model is used to determine the correlation between question types and different answers; based on the correlation values, the top number of answers are selected from largest to smallest, and a fine-grained answer space is defined for each question type. Human expert knowledge is incorporated to update and refine the answer space, thereby generating a new answer space to represent candidate answers.

2. The test-time VQA model bias removal method based on problem type awareness as described in claim 1, characterized in that, Once all test samples have been read, the parameters of the VQA model are updated, and the prediction of all test samples is completed. The criterion for judging the prediction result of each sample is: the maximum value in the positive sample prediction probability distribution is selected as the final result.

3. The test-time VQA model bias removal method based on problem type awareness as described in claim 1, characterized in that, The overall optimization loss of the pre-trained VQA model during testing consists of four parts: test-time adaptive optimization loss, debiased entropy minimization, direct optimization loss for positive samples, and direct optimization loss for negative samples.

4. The test-time VQA model bias removal method based on problem type awareness as described in claim 1, characterized in that, The process of creating negative samples is as follows: To construct negative samples, randomly select an irrelevant image from a small batch of known problems.

5. A test-time VQA model bias removal system based on problem type awareness, using the method described in claim 1, characterized in that, include: The negative sample construction module is used to sequentially select a set of data from the test set as positive samples and build negative samples; wherein, the question type corresponding to each sample adaptively matches different answer entropy change rate thresholds; The answer prediction module is used to predict the answers for positive and negative samples using a VQA model trained on the training set. The reliability assessment module is used to calculate the entropy values ​​of the predicted positive and negative samples, and uses an adaptive entropy minimization method to determine the reliability of each sample. The bias detection module calculates the rate of change of answer entropy values ​​for reliable samples and compares it with the threshold for the rate of change of answer entropy values ​​for the corresponding type of question to determine whether the sample is biased. Unbiased samples are then used to optimize the pre-trained VQA model until all test set data has been predicted.

6. The test-time VQA model bias removal system based on problem type awareness as described in claim 5, characterized in that, The test-time VQA model bias removal system based on problem type awareness also includes: The prediction module updates the parameters of the VQA model after all test samples have been read, and the prediction of all test samples is completed. The criterion for judging the prediction result of each sample is: the maximum value in the prediction probability distribution of positive samples is selected as the final result. or In the deviation judgment module, during the process of judging whether a sample is biased, if the rate of change of the entropy value of the answer is lower than the threshold of the rate of change of the entropy value of the answer of the corresponding type of question, the sample will be considered biased; otherwise, it will be unbiased. or The overall optimization loss of the pre-trained VQA model during testing consists of four parts: test-time adaptive optimization loss, debiased entropy minimization, direct optimization loss for positive samples, and direct optimization loss for negative samples. or In the answer prediction module, the process of using a VQA model trained on the training set to predict the answer for the corresponding data set is as follows: Collect question types and directly collect answers from the dataset to obtain the answer space for the entire dataset; A pre-trained BERT model is used to determine the correlation between question types and different answers; based on the correlation values, the top number of answers are selected from largest to smallest, and a fine-grained answer space is defined for each question type. Incorporate human expert knowledge and update the refined answer space to generate a new answer space to represent candidate answers; or In the negative sample construction module, the process of creating negative samples is as follows: To construct negative samples, randomly select an irrelevant image from a small batch of known problems.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the problem type-aware test-time VQA model debiasing method as described in any one of claims 1-4.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the problem type-aware test-time VQA model debiasing method as described in any one of claims 1-4.

Images