A visual question answering method and apparatus based on regularization and dual learning

Abstract

This invention discloses a visual question answering method and apparatus based on regularization and dual learning. The method specifically includes the following sub-steps: extracting keywords from questions in a VQA dataset; preprocessing image data and filtering key regions in the images; inputting the keywords and key regions in the images into VQA and VQG models for pre-training; constructing a visual entailment model and pre-training it; using the visual entailment model to determine relevance, constructing training samples, and co-training the pre-trained VQA and VQG models based on regularization and dual learning; and using the trained VQA and VQG models for visual question answering.

Description

Technical Field

[0001] This invention belongs to the field of visual question answering, and particularly relates to a visual question answering method and apparatus based on regularization and dual learning. Background Technology

[0002] Visual question answering is a crucial task in visual-language multimodal tasks. The performance metrics of visual question answering models measure their ability to understand visual and linguistic information, and advancements in this field contribute to the development of question answering systems in multimodal contexts. However, visual question answering datasets often exhibit linguistic bias, causing models to easily overlook visual information and directly answer questions based on statistical patterns. This significantly hinders model development and affects how people evaluate a model's visual and linguistic comprehension abilities. Traditional regularization methods include ensemble models that separate the influence of visual and linguistic information on the final answer. Data augmentation methods based on contrastive learning and causal models, or the use of diverse attention mechanisms, can correct data bias or perform causal analysis on the model. However, these methods fail to consider the information content relationships among visual, question, and answer information. While they may improve model performance on relevant "linguistic bias" diagnostic datasets, they compromise performance on standard datasets, failing to truly enhance the model's ability to recognize image information. Summary of the Invention

[0003] The purpose of this invention is to address the problem that existing VQA models tend to learn language biases while neglecting image information. It provides a visual question answering method and apparatus based on regularization and dual learning of VQA and VQG. This method fully utilizes the duality and imbalance of the two models in the direction of text information transmission, enabling them to form a dual relationship in text data flow. It also requires both models to maintain consistency in visual attention distribution, preventing model overconfidence, reducing the language bias learned by the VQA model, and increasing the model's perception and understanding of image information.

[0004] To achieve the above-mentioned objectives, the technical solution of the present invention is as follows: A first aspect of the present invention provides a visual question answering method based on regularization and dual learning, the method specifically including the following sub-steps:

[0005] Extract keywords from questions in the VQA dataset;

[0006] Image data preprocessing to filter key regions in the image;

[0007] Keywords and key regions of images are input into VQA and VQG models for pre-training;

[0008] Construct a visual entailment model and pre-train it;

[0009] The visual entailment model is used to determine relevance, and training samples are constructed. The pre-trained VQA and VQG models are then co-trained based on regularization and dual learning.

[0010] Visual question answering is performed using pre-trained VQA and VQG models.

[0011] A second aspect of the present invention provides an electronic device including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described visual question answering method based on regularization and dual learning.

[0012] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described visual question answering method based on regularization and dual learning.

[0013] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention proposes a visual question answering method based on regularization and dual learning. By utilizing the dual characteristics of VQA and VQG tasks, and considering the imbalance in the direction of text information transmission between the two—that is, the VQA model, when given a question with "high cue value," obtains a answer with "low cue value," easily learning linguistic priors; while the VQG model, conversely, is less likely to learn linguistic priors—this invention requires the VQA and VQG models to maintain consistency in visual attention, thereby making the VQA model focus more on learning image information; at the same time, it avoids the model becoming overconfident, which would further weaken the model's learning of question priors. Finally, it fully utilizes pre-trained VQA and VQG models for co-training based on regularization and dual learning. This enables the successful visual question answering using the trained VQA and VQG models. Attached Figure Description

[0014] Figure 1 A flowchart of the regularization learning method based on the task duality of VQA and VQG provided in this embodiment;

[0015] Figure 2 This is a diagram of the training framework structure provided in this embodiment;

[0016] Figure 3 This is a schematic diagram of data preprocessing provided in this embodiment;

[0017] Figure 4 An electronic device provided in this embodiment. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0021] The goal of Visual Question Answering (VQA) model M1 is to provide the correct answer given a question and an image, while the goal of Visual Question Generation (VQG) model M2 is to generate a corresponding question given an image and text prompts. Generally, VQA questions have short answers and are modeled as classification problems, while VQG problems are multimodal text generation problems.

[0022] Let the dataset for the VQA task be D = {I} i Q i A i} N That is, there are a total of N samples D i Each sample contains a (image, question, answer) triple, where Q i Let I represent the i-th question in dataset D. i and A i Represents the i-th image and question Q. i The answer.

[0023] like Figure 1 As shown, this invention proposes a visual question answering method based on regularization and dual learning, which specifically includes the following steps:

[0024] 1) Extract keywords from the questions in the VQA dataset.

[0025] Specifically include:

[0026] Part-of-speech tagging (POS tagging) is used to tag each question Q in the VQA dataset. i The nouns and verbs were labeled and filtered out as the first set of keywords.

[0027] The TF-IDF method (term frequency-inverse document frequency) is used to identify keywords in various question types in the VQA dataset. In this example, each question's Q... i At least the top 4 words with high TF-IDF values ​​were selected as the second keyword set.

[0028] Take the union of the keywords obtained in the above two steps to obtain each question Q. i The set of keywords, i.e., the question Q i The set of keywords is

[0029] 2) Preprocess the image data and filter out key regions in the image.

[0030] 2.1) Using the Faster-RCNN object detection model pre-trained on the Visual Genome dataset, object recognition is performed on all images to obtain object detection box coordinates, object category, and object color label.

[0031] For example, for image I i n were detected. i The set of coordinates S′ of an object and its corresponding detection box i ={S′ i1 S′ i2 , ..., S′ ini}, and the corresponding annotation information Cls i ={Cls i1 Cls i2 ,..,Cls ini}, Col i ={Col i1 Col i2 Col ini}, where Cls i For the set of object categories, Col i Let S' be the set of object color labels. Then (S') ij Clsi j Col ij ) represents the detection bounding box information, category information, and color information of the detected object with index j.

[0032] 2.2) Perform similarity matching between the object category and object color obtained in step 2.1) and the keyword set obtained in step 1) to obtain the key regions (Regions of Interest) in the image.

[0033] For example, for the question Q i The annotation information {Cls} obtained in step 2.1) i Col i}, using each (Cls ij Col ij The keyword set H obtained in step 1 i Perform similarity matching to obtain p i A collection of key areas and the p i For each key region, a backbone network is extracted using visual features to obtain the feature vectors of that region. As visual hints.

[0034] 3) Pre-train the VQA model M1 and the VQG model M2;

[0035] Using dataset D = {I i Q i A i} N Train VQA model M1, using {F i , H i Q i} N Train the VQG model M2. In this step, the VQA model M1 and the VQG model M2 are trained separately, and early stopping is used to control the model's ability to avoid convergence.

[0036] 4) Construct a visual entailment model, pre-train the visual entailment model M3, and construct training samples.

[0037] In this example, the visual entailment task is modeled as a regression task in the interval [0, 1], where label 1 indicates that the image is relevant to the question, and label 0 indicates that the image is irrelevant to the question. Specifically, image I... i And its corresponding question Q i Pair up, set label to 1; Image I i And other questions Q j Pair (i≠j) and set the label to 0. Pre-train the visual entailment model using cross-entropy loss until the model converges.

[0038] 5) Use the visual entailment model to determine relevance, construct training samples, and perform co-training on the pre-trained VQA model M1 and VQG model M based on regularization and dual learning until the VQA model M1 and VQG model M2 converge.

[0039] 5.1) Using the visual entailment model to determine relevance, construct positive and negative samples for training the VQA model M1:

[0040] Using the visual entailment model to determine relevance, image I... i And its corresponding question Q i Pairing, then Q i Positive Sample Problem Q + , View (I) i A i Q i ( ) represents a positive sample, and the correct answer is set to A. i The label is 1; the image I i And other questions Q j If (i≠j) are paired, then Q j For the negative sample problem Q - , View (I) i A - Q j ) represents a negative sample, where A - For the unsure tag, set the tag value to 1.

[0041] 5.2) The problem of generating M2 in the VQG model;

[0042] The VQG model M2 generates questions based on visual cues and text hints. Visual cues are the feature vectors F corresponding to key regions in the image. i Text prompt T i ={A j Q j -H j}, where A j For the answer, Q j -H j Question Q after word segmentation i With keyword H i The difference set of Q. Here, we require Q to be... j With F i They share a common original image. Let the problem generated by the greedy search method using the VQG model M2 be denoted as... The problem obtained by probability sampling is denoted as Since the problems generated by the VQG model M2 have a certain degree of realism, therefore... For weakly supervised samples,

[0043] 5.3) Construct pseudo-labels for the questions generated by the VQG model M2;

[0044] Problems in generating M2 models of VQG models The Visual Entailment Model (M3) is used for scoring, yielding a score r. This score r is used both as part of the reinforcement learning in the VQG model and as a label for weakly supervised samples in the VQA model. A custom lower threshold β and an upper threshold α are set; when r < β, the generated question is considered of poor quality, and the weakly supervised samples are... Let A j =A - Set the corresponding label to 1. When r > α, the generated question is considered to be of high quality, which is beneficial for weakly supervised samples. Based on answer A j The corresponding label is set to 1. However, when β < r < α, for weakly supervised samples... Based on answer A j The tag is set to r.

[0045] 5.4) Input the questions from positive samples, negative samples, and weakly supervised samples into the VQA model M1 to generate answers;

[0046] 5.5) Optimize the VQA model M1 and VQG model M2 using the loss function, and apply regularization constraints to the attention parts of the VQA model M1 and VQG model M2 to make the attention distribution of the VQA model M1 and VQG model M2 as consistent as possible.

[0047] In this example, the loss function is mainly divided into three parts: the loss function of VQA model M1, the loss function of VQG model M2, and the regularization loss function.

[0048] (A) Optimization of VQA Model M1

[0049] For the VQA model M1, the following weighted BCE (Batch Cross Entropy) loss function is used for optimization:

[0050] L BCE =log P vqa (a + |q + ,I))+log(1-P vqa (a - |q - ,I))+log(P vqa (a=A - |q - ,I))+λ vqa *P vqa (a g |q g ,I)

[0051] Among them, (I, a + q + ) is a positive sample, (I, a) - q - ) is a negative sample, (I, a) g q g A is a weakly supervised sample. - For the unsure tag, λ vqa This is a hyperparameter.

[0052] (B) M2 optimization of VQG model

[0053] For the VQG model M2, which uses the original data prompts to generate the problem, the following loss function is used to optimize the model:

[0054]

[0055] in, This is a problem generated using a greedy search algorithm in the VQG model M2. The problem is obtained by probability sampling from the VQG model M2; the R{·} function is a heuristic reinforcement learning score. r represents the visual entailment model for the generation problem. The rating, Let M1 be the probability given by the VQA model for answer A. Using the generation problem for VQG model M2 The gradient of the log-likelihood, λ vqg This is a hyperparameter.

[0056] (C) Regularization constraints

[0057] Regularization constraints are applied to the attention components of VQA model M1 and VQG model M2, requiring that the attention distributions of the visual attention components of VQA model M1 and VQG model M2 be as consistent as possible. The loss function is as follows:

[0058]

[0059] Among them, V vqa For the visual attention encoder of VQA model M1, V vqg This is the visual attention encoder for the VQG model M2. The corresponding encoding can be obtained using the encoder. `sim` is the similarity metric; in this example, the L2 norm is used. `sg[]` is the gradient stopping operation, which keeps the gradient unchanged for this part. Visual cue F is used... i Text prompt T i ={A j Q j -H jGenerating weakly supervised samples with VQG model M2 When i = j, let T be the time interval. i for In this example, the attention of VQA model M1 is required to shift to the attention of VQG model M2; when i≠j, let T be... i for In this example, the attention of VQG model M2 is required to be shifted to VQA model M1.

[0060] 5.6) Train the model according to 5.2)-5.5) until the VQA model M1 and the VQG model M2 converge.

[0061] like Figure 4 As shown, this application provides an electronic device including a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, they implement the method as described in any of the first aspects above.

[0062] The system also includes a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to each other to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0063] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0064] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor 102, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0065] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0066] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0067] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by processor 102, the computer program implements the methods described in any of the first aspects above. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory 101 (ROM), random access memory 101 (RAM), magnetic disks, or optical disks.

[0068] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A visual question answering method based on regularization and dual learning, characterized in that, The method specifically includes the following sub-steps: Extract keywords from questions in the VQA dataset; Image data preprocessing to filter key regions in the image; Keywords and key regions in the image are input into the VQA and VQG models for pre-training; Construct a visual entailment model and pre-train it; The visual entailment model is used to determine relevance, and training samples are constructed. The pre-trained VQA and VQG models are then co-trained based on regularization and dual learning. Visual question answering is performed using pre-trained VQA and VQG models. The construction and pre-training of the visual entailment model includes: The visual entailment task is a regression task within the range [0,1], where label 1 indicates that the image and the question are related, and label 0 indicates that the image and the question are not related. The visual entailment model is used to determine the relevance between a question and an image; where label 1 indicates that the image and the question are relevant, and label 0 indicates that the image and the question are unrelated. The visual entailment model was pre-trained using cross-entropy loss until the model converged. The process of using a visual entailment model to determine relevance, constructing training samples, and co-training pre-trained VQA and VQG models based on regularization and dual learning includes: The visual entailment model M3 is used to determine the relevance, and positive and negative samples are constructed for training the VQA model M1. Problem generation using the VQG model M2; The questions generated by the VQG model M2 are scored using the visual entailment model, and pseudo-labels are constructed based on the scores. Input the positive samples, negative samples, and questions generated by the VQG model M2 into the VQA model M1 to obtain the corresponding answers; The loss function is used to optimize the VQA model M1 and the VQG model M2, and the attention part of the VQA model M1 and the VQG model M2 is subject to regularization constraint to make the attention distribution of the VQA model M1 and the VQG model M2 as consistent as possible.

2. The visual question answering method based on regularization and dual learning according to claim 1, characterized in that, The process of extracting keywords from questions in a VQA dataset includes: A part-of-speech tagging algorithm was used to select the nouns and verbs from the questions in the VQA dataset, which were then used as the first set of prompt words. The TF-IDF method is used to identify keywords in various types of questions in the VQA dataset to obtain a set of second hint words. The keywords for each question in the VQA dataset are obtained by taking the union of the first set of prompt words and the second set of prompt words.

3. The visual question answering method based on regularization and dual learning according to claim 1, characterized in that, The process of constructing positive and negative samples for training the VQA model M1 to determine relevance using the visual entailment model includes: Using the visual entailment model to determine relevance, image I... i And its corresponding question Q i If paired, then Q i Positive Sample Problem Q + , View (I) i A i Q i ( ) represents a positive sample, and the correct answer is set to A. i The label is 1; the image I i And other questions Q j Pairing, i≠j, then Q j For the negative sample problem Q - ,View(I) i A - Q j ) represents a negative sample, where A - This is the unsure tag.

4. The visual question answering method based on regularization and dual learning according to claim 1, characterized in that, The process of generating problems using the VQG model M2 includes: The VQG model M2 generates questions based on visual and textual cues; the visual cues are the feature vectors F corresponding to key regions in the image. i Text prompt T i ={A j Q j -H j }, where A j For the answer, Q j -H j Question Q after keyword extraction i With keyword H i The difference set; the problem generated by the greedy search of the VQG model M2 is denoted as... The problem obtained by probability sampling is denoted as 5. The visual question answering method based on regularization and dual learning according to claim 1 or 4, characterized in that, The process of scoring questions generated by the VQG model M2 using a visual entailment model and constructing pseudo-labels based on the scores includes: The problem of generating VQG model M2 using visual entailment model M3. Scoring is performed to obtain a score r; a lower threshold β and an upper threshold α are set customwise. When r < β, weakly supervised samples are scored... Let A j =A - Set the corresponding label to 1; when r > α, for weakly supervised samples Based on answer A j Set the corresponding label to 1; when β < r < α, for weakly supervised samples Based on answer A j The tag is set to r.

6. The visual question answering method based on regularization and dual learning according to claim 1 or 4, characterized in that, The VQA model M1 and the VQG model M2 are optimized using a loss function, and regularization constraints are applied to the attention components of the VQA model M1 and the VQG model M2 as follows: The VQA model M1 is optimized using a weighted BCE loss function, as shown in the following formula: L BCE =log P vqa (a + |q + ,I))+log(1-P vqa (a - |q - ,I))+ log(P vqa (a=A - |q - ,I))+λ vqa *P vqa (a g |q g ,I) In the formula, (I,a) + ,q + (I,a) represents a positive sample. - ,q - (I,a) represents a negative sample. g ,q g A is a weakly supervised sample. - For the unsure tag, λ vqa For hyperparameters; The VQG model M2 is optimized using the loss function, as shown in the following formula: In the formula, This is a problem generated using a greedy search algorithm in the VQG model M2. The problem is obtained by probability sampling from the VQG model M2; the R{·} function is a heuristic reinforcement learning score. r represents the visual entailment model for the generation problem. The rating, Let M1 be the probability given by the VQA model for answer A; Using the generation problem for VQG model M2 The gradient of the log-likelihood, λ vqg For hyperparameters; Regularization constraints are applied to the attention components of VQA model M1 and VQG model M2, and the loss functions are as follows: Among them, V vqa For the visual attention encoder of VQA model M1, V vqg Here is the visual attention encoder for the VQG model M2; sim is the similarity metric; sg[] is the gradient stopping operation; and visual cues F are used. i Text prompt T i ={A j Q j -H j Generating weakly supervised samples with VQG and M2 models When i = j, let T be the time interval. i for Shift the attention of VQA model M1 to the attention of VQG model M2; when i≠j, denote T. i for This shifts the focus of the VQG model towards the VQA model.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the visual question answering method based on regularization and dual learning as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the visual question answering method based on regularization and dual learning as described in any one of claims 1-6.

Images