Visual Question Answering with Unlabeled Image Augmentation

Abstract

A system and method for training a visual question-answering model includes training (702) a teacher model by performing image-conditioned visual question generation on a visual language model (VLM) and a target visual question-answering dataset using images to generate question and answer pairs. Unlabeled images are pseudo-labeled (708) using the teacher model to decode synthetic question-answer pairs for the unlabeled images. The synthetic question-answer pairs for the unlabeled images are merged (712) with actual data from the target visual question-answering dataset to generate a self-augmented training set. A student model is trained (714) using the VLM and the self-augmented training set to return visual answers to text queries.

Description

[Technical Field]

[0001] Related Application Information This application claims priority to U.S. Provisional Application No. 63 / 422,629, filed November 4, 2022, U.S. Provisional Application No. 63 / 423,945, filed November 9, 2022, and U.S. Application No. 18 / 497,079, filed October 30, 2023, all of which are incorporated herein by reference in their entireties.

[0002] The present invention relates to visual question answering (VQA), and more particularly to a system and method for training a visual question answering model using unlabeled images. [Background technology]

[0003] 2. Description of Related Art Visual question answering (VQA) is a multimodal task that requires a model to answer questions based on input images. VQA can be used for a wide range of applications, including object, scene, action, and attribute recognition, counting, spatial reasoning, knowledge-based reasoning, and common sense reasoning. The primary paradigm for training a VQA model is to fine-tune a pre-trained base model (a visual language model) on a target VQA dataset. While annotated datasets of natural images are moderately large and contain diverse question-answer pairs, datasets for specialized VQA tasks, such as knowledge-based VQA or VQA in other domains (e.g., medicine, art), are often small with fewer question-answer pairs. Training a VQA model on a small target dataset can lead to overfitting, resulting in reduced robustness and generalization performance. Expanding the dataset by collecting additional annotations for knowledge-intensive tasks or specialized domains is often prohibitively expensive. Summary of the Invention

[0004] According to one aspect of the present invention, a method includes training a teacher model to generate question and answer pairs by performing image-conditioned visual question generation on a visual language model (VLM) and a target visual question-answering dataset using images. Unlabeled images are pseudo-labeled using the teacher model to decode synthetic question-answer pairs of unlabeled images. The synthetic question-answer pairs of unlabeled images are merged with actual data from the target visual question-answering dataset to generate a self-augmented training set. A student model is trained using the VLM and the self-augmented training set to return visual answers to text queries.

[0005] According to another aspect of the present invention, a system for training a visual question-answering model includes a hardware processor and a memory that stores a computer program, which, when executed by the hardware processor, causes the hardware processor to train a teacher model by performing image-conditional visual question generation on a visual language model (VLM) and a target visual question-answering dataset using images to generate question and answer pairs, pseudo-label unlabeled images using the teacher model to decode synthetic question-answer pairs of the unlabeled images, integrate the synthetic question-answer pairs of the unlabeled images with actual data from the target visual question-answering dataset to generate a self-augmented training set, and train a student model using the VLM and the self-augmented training set to return visual answers to text queries.

[0006] According to another aspect of the present invention, there is provided a computer program product for training a visual question-answering model, the computer program product including a non-transitory computer-readable storage medium having embedded program instructions executable by a computer to cause the computer to perform a method including training a teacher model to generate question and answer pairs by performing image-conditioned visual question generation on a visual language model (VLM) and a target visual question-answering dataset using images, pseudo-labeling unlabeled images using the teacher model to decode synthetic question-answer pairs of the unlabeled images, integrating the synthetic question-answer pairs of the unlabeled images with actual data from the target visual question-answering dataset to generate a self-augmented training set, and training a student model using the VLM and the self-augmented training set to return visual answers to text queries.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0008] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0009] [Figure 1] FIG. 1 is a block / flow diagram illustrating a high-level system / method for training a visual question-answering model according to one embodiment of the present invention.

[0010] [Figure 2] FIG. 1 illustrates question-answer and image associations that can be used in training, according to one embodiment of the present invention.

[0011] [Figure 3] FIG. 1 is a block / flow diagram illustrating a system / method for training a visual question answering model according to one embodiment of the present invention.

[0012] [Figure 3] FIG. 1 is a block / flow diagram illustrating a system / method for training a visual question-answering model according to one embodiment of the present invention.

[0013] [Figure 4] FIG. 1 is a block diagram illustrating an exemplary processing system used in accordance with one embodiment of the present invention.

[0014] [Figure 5] FIG. 2 is a generalized example of a neural network according to an embodiment of the present invention.

[0015] [Figure 6] FIG. 1 is a block diagram illustrating a medical system using a visual question-answering model, according to one embodiment of the present invention.

[0016] [Figure 7] FIG. 1 is a flow diagram illustrating a method for training a visual question-answering model according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] According to an embodiment of the present invention, a system and method are provided that introduces a data augmentation technique for visual question answering (VQA). The data augmentation technique generates additional training data in the form of synthetic question-answer pairs for images from a target VQA dataset and pseudo-labels (synthetic question-answer pairs) for unlabeled images (images without associated question-answer pairs) from the target VQA dataset. The generated data is combined with data (image + question-answer pairs) from the target VQA dataset to form a larger training set. A VQA model fine-tuned based on this combined data can improve the model's robustness and generalization performance.

[0018] In a useful embodiment, a data augmentation pipeline for VQA training generates additional training data in the form of additional synthetic question-answer pairs for images from the target dataset and new synthetic question-answer pairs for unlabeled images from the target dataset. Techniques are also provided to remove noisy synthetic question-answer pairs to improve the training set.

[0019] In one embodiment, synthetic question-answer pairs can be generated. A visual question generation (VQG) module is trained using images as input and question-answer pairs as output. Once the VQG is trained, it is used to generate pseudo-labels for unlabeled images from a target dataset.

[0020] Traditional visual question generation methods rely on available ground truth answers or annotations such as bounding boxes. Therefore, existing visual question generation methods cannot easily utilize unlabeled images. Furthermore, visual question generation methods are limited by the types of annotations available. For example, bounding box-based methods may not be applicable in situations where there are very few object-centric questions, and are further limited by the closed set assumption.

[0021] In traditional self-training, such as object detection, the task of generating pseudo labels is identical to the task of prediction. According to this embodiment, generating pseudo labels (question and answer pairs based on images) is a different task from prediction (generating answers based on question and image pairs). In self-training, labeled data is used to train a teacher model. The teacher model labels unlabeled auxiliary data. Then, labeled data augmented with newly labeled (pseudo-labeled) data is used to train a student model. In this embodiment, the teacher's task (generating questions and answers for images) is different from the student's task (generating answers for images). Therefore, the student and the teacher are trained to optimize different goals.

[0022] In contrast to semi-supervised learning, methods according to embodiments of the present invention do not require unlabeled images. The pseudo-labels generated by this method are valid even when added to a fully annotated image set, such as a complete VQA dataset. Like visual question generation, natural language augmentation relies on the existence of labels (questions) for images and often relies on a limited set of handcrafted rules, making it impossible to use unlabeled images. Furthermore, natural language augmentation is limited in the variety of questions it can generate, since all augmented questions are semantically identical variations of existing questions.

[0023] In a useful embodiment, unlabeled images are utilized by generating new questions and answers. Domain generalization in VQA remains an unexplored topic, but it can be used to improve the speed and accuracy of VQA tasks.

[0024] Unlabeled images are inexpensive and often available. They can be used to generate new question-and-answer pairs for unlabeled images, which can then be used during training to fine-tune a large-scale autoregressive visual language model on the target VQA task. The model itself can be used to generate synthetic training data by directly labeling raw images with new questions and answers that are used to augment the existing training data. In contrast to existing approaches, no pre-trained object detectors, handcrafted augmentation rules, bounding boxes, guidance, or captions for unlabeled images are required. Our system learns to generate question-and-answer pairs that match the type and distribution of the target VQA task.

[0025] Large-scale visual language models can be used for self-training. A three-stage framework is presented. In the first stage, a teacher model is trained by updating the model weights to generate questions and answers drawn from the same approximate distribution as the target VQA task. In the second stage, unlabeled images are provided to the teacher, and question-and-answer pairs are probabilistically generated for the unlabeled images. In the third stage, a student model is trained by restoring the model weights to the pre-trained weights and fine-tuning them based on a concatenation of synthetic and real question-and-answer pairs. This three-stage framework is based on self-training and pseudo-labeling to leverage unlabeled images when fine-tuning large-scale visual language models on the target VQA task. This framework improves VQA task performance in two different image domains and significantly improves robustness as measured by at least three challenging VQA tests for robustness. This framework further improves domain generalization from natural images to other domains. It also achieves improved zero-shot transfer during transfer learning and preserves numerical inference.

[0026] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0027] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer-readable medium may include any apparatus that stores, communicates, propagates, or transfers a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0028] Each computer program can be tangibly stored on a machine-readable storage medium or device (e.g., program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention can also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0029] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0030] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0031] Referring now in detail to the drawings, numerals within the figures represent identical or similar elements. Referring initially to FIG. 1, an exemplary high-level system / method for training a visual question answering (VQA) model according to one embodiment of the present invention is shown. Web-scale pre-training of image-text pairs 102 provides a large-scale visual language model (VLM) 104 with important knowledge that may not be fully utilized in small VQA datasets, even with transfer learning, due to limited training data. The VLM 104 is used to generate and fine-tune image, question, and answer associations in block 106. The updated VQA model 108 undergoes transfer learning to deploy small-scale VQA tasks 110. The VLM 104 then uses a teacher model (VQG) initialized from the large-scale VLM 104, which generates questions and answers conditional only on images. IC) 114, which is then self-improved using unlabeled images 116. IC Self-training on a small dataset 110 augmented with pseudo-labeled images creating synthetic pairs 118 from the ensemble of images 114 improves over fine-tuning on a purely small dataset 110. A VQA model 122 is trained using the augmented dataset. Throughout the description and figures, the following abbreviations are used: Q = question, A = answer, I = image. P(A|Q,I) is the conditional probability that A is in Q,I.

[0032] Referring to FIG. 2, in one embodiment, self-training is used to train the VQA 122 (FIG. 1). A large VLM 214 contains dark knowledge that can be extracted by probabilistic decoding (e.g., using kernel sampling). In block 202, captions are derived from a pre-trained base VLM. A sample of, for example, 1,000 images 212 from the VLM 214 can caption images using correct knowledge that cannot be verified when posed as a VQA task. Even when decoding deterministically (e.g., beam search), the VLM 214 will self-contradict in, for example, 5% of images. In block 206, where prompts 222 associated with images 224 and captions 226 written by the VLM are provided, another dataset (e.g., VQAv2) can be used for fine-tuning. Here, the captions 216 can be converted to Boolean MC QA (question + modified Boolean answer) and compared to the VLM 214 for fine-tuning. Block 210 determines whether the VLM is self-consistent. Inset panel 230 shows how self-consistency decreases as the diversity of captions (the top-p parameter used in kernel sampling) increases.

[0033] 3, a schematic diagram illustrates a framework according to one embodiment of the present invention. In block 306, a teacher model VQG IC308 is trained using images and question-answer pairs from the target dataset 305 and the VLM 302. The teacher model 308 is image-conditioned to associate question-answer pairs with images (I) by optimizing a loss function (LVQG). The updated teacher model 308 is then used with unlabeled images 310 for pseudo-labeling in block 312 for only the unlabeled images (I), and pseudo-labels for the unlabeled images 310 are generated (by decoding the questions and answers from the teacher model 308).

[0034] By associating unlabeled images 310 with pseudo-labeled questions and answers (Q', A'), synthetic pairs 314 are created. The synthetic pairs 314 are used as additional training data to improve accuracy. In block 316, the target dataset 305 and the synthetic pairs 314 are merged to provide a self-augmented training set 318. A student VQA model 324 is then trained in block 320 using real training samples from the VLM 302 augmented with the pseudo-labeled images in the self-augmented training set 318. Training involves inputting images with questions and minimizing a loss function for the answers (LVQA).

[0035] In one embodiment, the goal is to pseudo-label unlabeled images with generated questions and answers using a teacher model 308 in block 312, and then train a student model 324 using actual VQA pairs augmented with the generated VQA pairs in a self-augmented training set 318. To generate pseudo-labels, a visual question generation (VQGIC) model 308 is trained with actual question-answer pairs and images as a teacher. This teacher model VQGIC 308 generates both questions and answers conditional only on the images, emphasizing the image-conditional nature of the model. Next, the teacher model 308 is given unlabeled images 310, which are probabilistically decoded by the teacher model 308 to generate pseudo-labels. The pseudo-labels are parsed into question-answer pairs in block 312. After the actual samples in the dataset have been augmented with self-generated samples, VQA training can continue. The approach employed is preferably compatible with any encoder-decoder multimodal architecture. This is possible because the approach can rely on direct image-to-text generation, and because the autoregressive decoders of large-scale visual language models (VLMs) are designed to be conditional on images.

[0036] In block 306, direct image-conditioned VQG training can include self-training. Self-training requires that a teacher model creates pseudo-labels and a student model learns to mimic them. To use unlabeled data for VQA 324, the teacher model 308 must be able to pose questions and provide answers given unlabeled images. This is a different task from VQA processing. Given an image I, a question Q, and an answer A, the VQA student 324 must approximate P(A|Q,I), and the teacher model 308 must approximate P(Q,A|I). Traditional visual question generation (VQG) approaches do not work with unlabeled data because they approximate P(Q|I,A), i.e., generate questions and potential answers conditional on the image. In contrast, according to embodiments of the present invention, an image-conditioned (IC) approach (VQG) is used. IC ) has been developed and adopted by the teacher model 308.

[0037] VQG approximating P(Q,A|I) IC To create a teacher model 308, the learning problem for such a model is treated as a text generation problem, and the autoregressive decoder of the visual-language model is trained to approximate P(T|I), where T = (Q, A). Let DQA be the question-answering dataset for creating the teacher model. sample

number

[0038] question: <question>?,answer: <anser>(1), where: <question>and <anser>are replaced by the contexts Q and A, respectively.

[0039] T(y1, y2, … y n ), a teacher model (VQG) 308 is trained by optimizing

[0040]

number

[0041] Teacher Model VQG IC Once we have 308, we can proceed to self-training using unlabeled data 310. u To generate pseudo-labels (Q',A') for

number

number

number

number

number

[0042] We can start pseudo-labeling as many images as we want. Target dataset D QA Self-generated training data D' in the form QA Denotes the form (Q',A',I u ) can be obtained. Next, the real data set D QA and the unlabeled image D' QA We extend the self-generated question-answer pairs above and use them to create a self-augmented training dataset318

number

[0043]

number

number

number

[0044] We experimentally validated our embodiment. Self-learning data augmentation improves performance. This performance improvement is maintained even when using 447,000 real-number pairs from VQAv2 for transfer learning, demonstrating that self-learning data augmentation provides substantial improvements over manual annotation. In fine art VQA, self-learning data augmentation improves overall performance, with particularly significant improvements observed for visually grounded questions. For example, self-learning data augmentation improves performance by at least 2.1% compared to the baseline model. In all domains, self-learning data augmentation improves domain generalization compared to the baseline model. This improvement is most pronounced in fine art images, because the fine art domain is closest to the natural image domain in terms of images, questions, and answers.

[0045] A self-training framework for fine-tuning large-scale visual language models for small-scale visual question answering tasks includes a teacher model. The teacher model is a visual question generation (VQG) model that can generate questions and answers from unlabeled images using knowledge from a large-scale visual language model, in contrast to existing VQG approaches that require ground truth annotations to generate questions and answers from images. This allows us to extend the self-training paradigm with unlabeled images to visual question answering. By augmenting manually annotated pairs in a small dataset with self-generated pairs obtained from unlabeled images, we train a student model that can be used in many applications where visual information is useful for answering text questions. These applications can include educational, medical, and browsing environments.

[0046] Referring to FIG. 4, a block diagram illustrates an exemplary processing system 400 for use in accordance with one embodiment of the present invention. The processing system 400 may include one or more computer processing units (e.g., CPUs) 401, one or more graphics processing units (GPUs) 402, one or more memory units 403, communication units 404, and peripherals 405. The GPU 402 may be a single-core or multi-core GPU. The CPU and / or GPU may be, in whole or in part, hardware processing subsystems. The one or more memory units 403 may include cache, RAM, ROM, and other memory (e.g., flash memory, optical memory, magnetic memory, etc.). The communication units 404 may include wireless and / or wired communication units (e.g., network (e.g., WiFi, etc.) adapters, etc.). The peripherals 405 may include display devices, user input devices, printers, imaging devices, etc. The elements of the processing system 400 are connected by one or more buses or networks (collectively designated 410).

[0047] In one embodiment, memory device 403 may store specially programmed software modules for converting a computer processing system into a special purpose computer configured to implement various aspects of the present invention. In one embodiment, special purpose hardware (e.g., application specific integrated circuits, field programmable gate arrays (FPGAs), etc.) may be used to implement various aspects of the present invention.

[0048] In one embodiment, memory device 403 stores program code for performing visual question and answer queries using deep learning. A VGA model 720 can be stored in memory 703 along with program code 722 for generating a user interface and responding to queries with visual and textual information.

[0049] Processing system 700 may also include other elements (not shown). For example, depending on the particular implementation, various other input and / or output devices may be included in processing system 700. Wireless and / or wired input and / or output devices may be used. Furthermore, various configurations of additional processors, controllers, memory, etc. may also be utilized. These and other variations of processing system 700 may also be provided.

[0050] Additionally, it should be understood that the various figures described below with respect to various elements and steps associated with the present invention may be implemented in whole or in part by all or portions of system 700 .

[0051] A VQA model is trained to handle inference in an information processing system. A VQA model includes an information processing structure with many highly interconnected processing elements (called "neurons" or "nodes") that operate in parallel to solve a specific problem. The VQA model is trained using a set of training data through a learning process that involves adjusting the weights between neurons. Through this learning process, the VQA model is configured for a specific application, such as responding to visual image and / or text queries.

[0052] 5, an exemplary diagram of a neural network 500 is shown. While a particular structure with three layers and a fixed number of fully connected neurons is shown, it should be understood that this is for illustrative purposes only. In practice, this embodiment may take any suitable form, including any number of layers and any pattern or connection pattern between them.

[0053] VQA models demonstrate the ability to derive meaning from complex or imprecise data and can be used to extract patterns and detect trends that are too complex for humans or other computer-based systems to detect. A neural network structure is generally known to have input neurons 502 that provide information to one or more "hidden" neurons 504. The connections 508 between the input neurons 502 and the hidden neurons 504 are weighted, and these weighted inputs are processed by the hidden neurons 504 according to some function within the hidden neurons 504. There can be any number of layers of hidden neurons 504, and the neurons can perform different functions. There are also various neural network structures, such as convolutional neural networks and max-out networks, which can vary depending on the structure and function of the hidden layers and the pattern of weights between layers. Each layer can perform a specific function and may include a convolutional layer, a pooling layer, a fully connected layer, a softmax layer, or any other suitable type of neural network layer. A set of output neurons 506 receives and processes the weighted inputs from the final set of hidden neurons 504.

[0054] This represents a "feedforward" computation in which information is propagated from input neurons 502 to output neurons 506. Once the feedforward computation is complete, the output is compared to the desired output obtained from the training data. The error relative to the training data is then processed in a "backpropagation" computation in which hidden neurons 504 and input neurons 502 receive information about the error propagating backward from output neurons 506. Once the backward error propagation is complete, a weight update occurs in which weighted connections 508 are updated to account for the received error. Note that the three modes of operation - feedforward, backpropagation, and weight update - do not overlap with one another. This is merely one type of computation; any suitable form of computation could be used instead.

[0055] To train a VQA model, training data can be divided into a training set and a test set. The training data includes pairs of inputs and known outputs (images and questions and answers). During training, inputs from the training set are fed into the VQA model using feedforward propagation. After each input, the output of the VQA model is compared to the respective known output. The discrepancy between the output and the known output associated with that particular input is used to generate an error value, which may be backpropagated through the VQA model, and then the weight values ​​of the VQA model may be updated. This process continues until the training set pairs are exhausted.

[0056] After training is complete, the VQA model can be tested against a test set to ensure that training has not led to overfitting. If the VQA model can generalize to new inputs other than those it has already been trained on, then the model is ready to use. If the VQA model does not accurately reproduce the known outputs of the test set, then additional training data is needed or the hyperparameters of the VQA model need to be adjusted.

[0057] The VQA model can be implemented in software, hardware, or a combination of both. For example, each weight 508 can be characterized as a weight value stored in computer memory, and the activation function of each neuron can be implemented by a computer processor. The weight value can store any suitable data value, such as a real number, a binary value, or a value selected from a fixed number of possibilities, that is multiplied by the associated neuron output. The weight value can store any suitable data value, such as a real number, a binary value, or a value selected from a fixed number of possibilities, that is multiplied by the associated neuron output.

[0058] Referring now to FIG. 6, an exemplary system / method 600 for responding to medical queries using visual and textual information is shown, according to one embodiment of the present invention. Medical professionals often require information in real time during patient consultations and procedures. Textual responses are often insufficient. To improve decision-making and provide more accurate information, visual responses including images, videos, and textual information provide a more appropriate and complete response. According to an embodiment of the present invention, information available to medical / healthcare professionals can include visual images and videos provided using an artificial intelligence system.

[0059] In one embodiment, a healthcare professional 610 can generate a VQA query 602 using natural language or text and images. The query 602 can include, for example, questions about images of wounds or lesions, images of rashes, or images of MRIs, CT scans, or X-rays. The query 602 can be forwarded to a VQA query processing system 604 either directly or over a network 608.

[0060] The VQA query processing system 604 can access the VQA model 606 directly or via a network 608. The VQA model 606 includes a student model trained using self-augmenting training data as described in accordance with embodiments of the present invention. The VQA model 606 uses a neural network in conjunction with the VQA processing system 604 to predict the best answer to a query using visual question answering (VQA) information. Using the training methods applied herein, the VQA model 606 can provide more accurate responses than traditional models. The generated responses can then be forwarded to a healthcare practitioner 610 and rendered on a peripheral device 612, such as a display and / or speaker. For example, the healthcare practitioner 610 can be presented with text, images, or video as needed. The healthcare practitioner 610 can also use this information to update patient data or assist medical personnel in their decision-making. For example, the system 600 can answer and respond to image queries by providing graphical data or images in the response to assist in diagnosing a condition.

[0061] The network 608 can interact with any part of the system and communicate the information and resources necessary to provide a VGA response. Because the information is communicated over the network 608, it is available to all users. The functionality provided to determine the VGA response can be offered as a service for medical staff or programmers to update patient profiles or to provide real-time information to healthcare practitioners 610 in distributed network settings, hospital settings, clinic settings, etc. The healthcare practitioners 610 can use the VGA response to make better-informed decisions, refresh their memories regarding procedures, and educate patients.

[0062] In other embodiments, the system / method 600 may be adapted for use in an educational or browsing environment. The VQA student model or models 606 may be trained on a particular domain or subject to, for example, help students answer questions with visual answers.

[0063] Referring to FIG. 7, a computer-implemented method for training a visual question-answering model, according to one embodiment, is described. In block 702, a teacher model is trained by performing image-conditioned visual question generation on a visual language model (VLM) that generates question and answer pairs using images and a targeted visual question-answering dataset. In block 704, the teacher model can be trained using deep learning to maximize the conditional likelihood of a question-answer pair given an image. In block 706, the targeted visual question-answering dataset is trained by inputting (Q, A) into a structured template to convert data samples into tokens T(y l ,y2,···y n ), where Q is the question and A is the answer, and optimize the loss over all question-image-answer pairs.

[0064] In block 708, the teacher model is used to pseudo-label the unlabeled image and decode the synthesized question-answer pair corresponding to the unlabeled image. In block 710, the unlabeled image I is decoded by obtaining the decoder logits. u The logits define the distribution of tokens in the natural language vocabulary of the teacher model.

[0065] In block 712, the synthetic question-and-answer pairs of unlabeled images are merged with real data from the targeted visual question-answering dataset to generate a self-augmented training set. In block 714, a student model is trained using the VLM and the self-augmented training set to return visual answers to text queries. The student model is trained to approximate P(T|I), where T = (Q,A), where Q is the question, A is the answer, and P(T|I) is the conditional probability of T on image I. Given image I, question Q, and answer A, the student model approximates P(A|Q,I), and the teacher model approximates P(Q,A|I), where P(A|Q,I) is the conditional probability of A on Q,I, and P(A,Q|I) is the conditional probability of A and Q on I, allowing unlabeled data to be used for training.

[0066] At block 716, the student model is trained with specific images and information. At block 718, the student model is used to respond to queries or inferences with visual answers within a specific subject. In one embodiment, the student model is trained with medical images and information and responds to medical queries with visual answers to assist medical personnel in decision making. In another embodiment, the student model is trained with educational subjects including images and information and responds to queries about those subjects with visual answers.

[0067] As employed herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that may be on-board or off-board or dedicated for use by the hardware processor subsystem.

[0068] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0069] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs). These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0070] References herein to "one embodiment" or "an embodiment" of the present invention, as well as other variations, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment," as well as any other variations, in various places throughout this specification do not necessarily all refer to the same embodiment. However, it should be understood that features of one or more embodiments may be combined given the teachings of the present invention provided herein.

[0071] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one," such as "A and / or B" or "at least one of A and B" will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0072] The foregoing is understood in all respects to be illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Various other feature combinations could be implemented by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.< / anser> < / question> < / anser> < / question>

Claims

1. 1. A computer-implemented method for training a visual question-answering model, comprising: training (702) a teacher model by performing image-conditioned visual question generation on a visual language model (VLM) and a target visual question-answering dataset using images to generate question and answer pairs; pseudo-labeling (708) unlabeled images using the teacher model and decoding synthesized question-answer pairs of the unlabeled images; merging (712) the synthetic question-answer pairs of the unlabeled images with real data from the target visual question-answering dataset to generate a self-augmenting training set; and training (714) a student model using the VLM and the self-augmented training set to return visual answers to text queries.

2. 2. The method of claim 1 , wherein training the student model comprises training the student model to approximate P(T|I), where T=(Q, A), Q is the question, A is the answer, and P(T|I) is the conditional probability of T given image I.

3. The target visual question-answering dataset is constructed by inputting (Q, A) into a structured template (where Q is the question and A is the answer) and dividing the data samples into tokens T(y 1 , y 2 , ...y 2 ) and optimizing the loss over all question-image-answer pairs.

4. The method of claim 1 , wherein training the teacher model comprises deep learning to maximize the conditional likelihood of a question-answer pair given an image.

5. Pseudo-labeling an unlabeled image is done by taking the logits of the decoder to generate the unlabeled image I u 10. The method of claim 1, further comprising generating pseudo-labels of logits, where the logits define a distribution over tokens of a natural language vocabulary of the teacher model.

6. 10. The method of claim 1, wherein given an image I, a question Q, and an answer A, the student model approximates P(A|Q,I) and the teacher model approximates P(Q,A|I), where P(A|Q,I) is the conditional probability of A given Q,I, and P(A,Q|I) is the conditional probability of A and Q given I, and wherein unlabeled data is available for training.

7. The method of claim 1 , wherein the student model is trained with medical images and information and further comprises responding to medical queries with visual answers to assist medical personnel in decision making.

8. 1. A system for training a visual question answering model, comprising: Hardware processors (401, 402); a memory (403) for storing a computer program, the computer program, when executed by the hardware processor, causing the hardware processor to: training (702) a teacher model by performing image-conditioned visual question generation on a visual language model (VLM) and a target visual question-answering dataset using the images to generate question and answer pairs; pseudo-labeling (708) unlabeled images using the teacher model and decoding synthetic question-answer pairs for the unlabeled images; merging (712) the synthetic question-answer pairs of the unlabeled images with real data from the target visual question-answering dataset to generate a self-augmenting training set; The system uses the VLM and the self-augmented training set to train (714) a student model to perform returning visual answers to text queries.

9. 9. The system of claim 8, wherein the computer program causes the hardware processor to train the student model to approximate P(T|I), where T=(Q, A), Q is the question, A is the answer, and P(T|I) is the conditional probability of T given image I.

10. The computer program instructs the hardware processor to input (Q, A) into a structured template (where Q is the question and A is the answer) and convert the data samples into tokens T(y 1 , y 2 , ...y 2 10. The system of claim 9, wherein the target visual question-answering dataset is generated by transforming the target sequence of images into a target sequence of images, and optimizing a loss over all question-image-answer pairs.

11. 9. The system of claim 8, wherein the computer program causes the hardware processor to train the teacher model by deep learning to maximize the conditional likelihood of a question-answer pair given an image.

12. The computer program causes the hardware processor to generate an unlabeled image I by obtaining the decoder logits. u 9. The system of claim 8, wherein the logits define a distribution over tokens of a natural language vocabulary of the teacher model.

13. 9. The system of claim 8, wherein given an image I, a question Q, and an answer A, the student model approximates P(A|Q,I) and the teacher model approximates P(Q,A|I), where P(A|Q,I) is the conditional probability of A given Q,I, and P(A,Q|I) is the conditional probability of A and Q given I, and wherein unlabeled data is available for training.

14. 10. The system of claim 8, wherein the student model is trained with medical images and information, and the computer program causes the hardware processor to respond with visual answers to medical queries to assist medical personnel in decision making.

15. 15. The system of claim 14, wherein the computer program causes the hardware processor to display a visual response on a display device.

16. 1. A computer program product for training a visual question answering model, the computer program product comprising: a non-transitory computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a computer, the computer comprising: training (702) a teacher model by performing image-conditioned visual question generation on a visual language model (VLM) and a target visual question-answering dataset using images to generate question and answer pairs; pseudo-labeling (708) unlabeled images using the teacher model and decoding synthesized question-answer pairs of the unlabeled images; merging (712) the synthetic question-answer pairs of the unlabeled images with real data from the target visual question-answering dataset to generate a self-augmenting training set; and training (714) a student model using the VLM and the self-augmenting training set to return visual answers to text queries.

17. 17. The computer program product of claim 16, wherein training the student model comprises training the student model to approximate P(T|I), where T=(Q, A), Q is the question, A is the answer, and P(T|I) is the conditional probability of T given image I.

18. 17. The computer program product of claim 16, wherein training the teacher model comprises training using deep learning to maximize a conditional likelihood of a question-answer pair given an image.

19. 17. The computer program product of claim 16, wherein given an image I, a question Q, and an answer A, the student model approximates P(A|Q,I) and the teacher model approximates P(Q,A|I), where P(A|Q,I) is the conditional probability of A given Q,I, and P(A,Q|I) is the conditional probability of A and Q given I, and wherein unlabeled data can be used in training.

20. 17. The computer program product of claim 16, wherein the student model is trained with medical images and information and further comprises responding to medical queries with visual answers to assist medical personnel in decision making.

Images