Visual question and answer method and system based on generative adversarial and sample optimization

Abstract

The invention provides a visual question and answer method and system based on generative adversarial and sample optimization, and relates to the technical field of visual question and answer, and the method comprises the steps: obtaining a question and an image in a visual question and answer task; inputting the question and the image into the trained visual question and answer model to obtain a visual question and answer result; wherein the visual question and answer model constructs a deviation model based on a generator, converts a random noise vector into a deviation feature representing the same dimension as the image feature, and generates a deviation answer vector according to an input question and the deviation feature; distinguishing whether the answer is true or false based on a discriminator; enabling the answer distribution of the deviation model to approach the answer distribution of the target model through the alternate training of the generator and the discriminator; and with the goal of maximizing the prediction probability of the visual question and answer model to correct answers, jointly training the generative adversarial network and the visual question and answer model. According to the invention, the multi-modal reasoning capability of the model can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual question answering, and in particular to a visual question answering method and system based on generative adversarial and sample optimization. Background Art

[0002] The visual question answer (VQA) task requires the model to integrate image and text information to answer questions, and is an important evaluation of multimodal understanding. However, existing models often rely on high-frequency priors in the training set and ignore the image content, making it difficult to avoid language bias problems. When the question involves a specific object, the model may make predictions based on high-frequency prior knowledge in the training set instead of using the image content. For example, when answering "What color is this object", the model may choose to associate with common colors in the training data and ignore the colors actually presented in the image. This phenomenon reflects the shortcomings of traditional models in the coordinated use of multimodal data, making it difficult to avoid the impact of bias problems.

[0003] In order to reduce the language bias problem, a variety of unbiased VQA methods have been proposed in related technologies to weaken surface correlation by introducing external supervision mechanisms. These external supervision mechanisms mainly include three categories: (1) The bias branch network weakens the bias effect by explicitly modeling the bias distribution between questions and answers and generating bias-related prediction branches. However, this method mainly relies on a single modality (such as language) for bias modeling and does not fully consider the complementary information between multimodal data (such as images or videos), which limits its performance in multimodal reasoning tasks. (2) Contrastive learning methods improve the robustness of the model by constructing positive and negative sample pairs to distinguish between true semantic associations and false correlations. However, the construction of positive and negative sample pairs is highly dependent on high-quality sample annotations. If the annotations are inaccurate or the sample selection is inappropriate, the model may learn misleading information, thereby reducing the performance of the model. (3) Debiased training strategies reduce the impact of bias by resampling or weighting data samples and adjusting the data distribution. However, this method relies too much on data preprocessing, especially when there are deep biases in the data itself. It may not be possible to completely eliminate the bias simply by adjusting the data distribution, and it may also lead to the loss of some valid information.

[0004] The above-mentioned methods in related technologies have achieved certain results in weakening the false correlation between questions and answers. However, most of these methods focus on the bias problem of language modality, and pay less attention to the modeling ability of deep interactions between visual and language modalities, which limits the comprehensive utilization of multimodal information by the model. Especially in scenarios where it is necessary to fully explore the complementary information between visual and language data, these methods are difficult to play an effective role, thus limiting the performance of the model. Summary of the invention

[0005] In order to address the shortcomings of the prior art, the purpose of the present invention is to provide a visual question answering method and system based on generative adversarial and sample optimization, which generates multimodal samples with complementary information to make up for the shortcomings of existing visual question answering tasks that only focus on single modal branches, thereby improving the multimodal reasoning ability of the model.

[0006] In order to achieve the above object, according to some embodiments, a first aspect of the present invention provides a visual question answering method based on generative adversarial and sample optimization, comprising: Get questions and images for visual question answering tasks; Input the question and image into the trained visual question answering model to obtain the visual question answering result; Among them, the visual question answering model is trained using a multimodal generation strategy and a counterfactual sample auxiliary strategy; The multimodal generation strategy includes: constructing a bias model based on a generator of a generative adversarial network, converting a random noise vector into a bias feature of the same dimension as the image feature representation, and generating a bias answer vector based on the input question and the bias feature; distinguishing whether the answer vector generated by the visual question answering model and the bias answer vector generated by the bias model are consistent with the facts based on a discriminator of the generative adversarial network; making the answer distribution of the bias model close to the answer distribution of the target model through alternating training of the generator and the discriminator; and jointly training the generative adversarial network and the visual question answering model with the goal of maximizing the prediction probability of the visual question answering model for the correct answer; The counterfactual sample auxiliary strategy includes: constructing counterfactual samples, training the visual question answering model with the goal of minimizing the prediction probability of the visual question answering model for the correct answer on the counterfactual samples.

[0007] A second aspect of the present invention provides a visual question answering system based on generative adversarial and sample optimization, comprising: A visual question answering task acquisition module, configured to acquire questions and images in a visual question answering task; A visual question answering result acquisition module is configured to input questions and images into a trained visual question answering model to obtain a visual question answering result; Among them, the visual question answering model is trained using a multimodal generation strategy and a counterfactual sample auxiliary strategy; The multimodal generation strategy includes: constructing a bias model based on a generator of a generative adversarial network, converting a random noise vector into a bias feature of the same dimension as the image feature representation, and generating a bias answer vector based on the input question and the bias feature; distinguishing whether the answer vector generated by the visual question answering model and the bias answer vector generated by the bias model are consistent with the facts based on a discriminator of the generative adversarial network; making the answer distribution of the bias model close to the answer distribution of the target model through alternating training of the generator and the discriminator; and jointly training the generative adversarial network and the visual question answering model with the goal of maximizing the prediction probability of the visual question answering model for the correct answer; The counterfactual sample auxiliary strategy includes: constructing counterfactual samples, training the visual question answering model with the goal of minimizing the prediction probability of the visual question answering model for the correct answer on the counterfactual samples.

[0008] According to a third aspect of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the above-mentioned visual question answering method based on generative adversarial and sample optimization.

[0009] According to a fourth aspect of the present invention, a computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, complete the steps of the above-mentioned visual question answering method based on generative adversarial and sample optimization.

[0010] In a fifth aspect of the present invention, a computer program product is provided, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned visual question answering method based on generative adversarial and sample optimization.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a visual question answering method and system based on generative adversarial and sample optimization. Multimodal samples with complementary information are generated based on a generative adversarial network, thereby effectively making up for the shortcomings of single-modal branches and improving the multimodal reasoning ability of the model. Counterfactual samples are generated to break the shortcut association between questions and answers, and the model's attention to image information is enhanced, thereby significantly improving the generalization performance of the model. The multimodal generation strategy and the counterfactual sample auxiliary strategy play a key role in improving the model's ability to capture visual-language interactive information and breaking the false association between questions and answers. The combination of the two alleviates the bias problem in multimodal reasoning and improves the accuracy of visual question answering and its robustness in complex scenarios.

[0012] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0014] Figure 1 Schematic diagram of the method in Embodiment 1 of the present invention; Figure 2 Schematic diagram of the multimodal generation module, where (a) is a schematic diagram of the adversarial training bias model, and (b) is a schematic diagram of the training target model; Figure 3 Schematic diagram of the counterfactual auxiliary module. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0016] Embodiment 1 Embodiment 1 of the present invention provides a visual question answering method based on generative adversarial and sample optimization, such as Figure 1-Figure 3 As shown, including: Get questions and images for visual question answering tasks; Input the question and image into the trained visual question answering model to obtain the visual question answering result; Among them, the visual question answering model is trained using a multimodal generation strategy and a counterfactual sample auxiliary strategy; The multimodal generation strategy includes: constructing a bias model based on a generator of a generative adversarial network, converting a random noise vector into a bias feature of the same dimension as the image feature representation, and generating a bias answer vector based on the input question and the bias feature; distinguishing whether the answer vector generated by the visual question answering model and the bias answer vector generated by the bias model are consistent with the facts based on a discriminator of the generative adversarial network; making the answer distribution of the bias model close to the answer distribution of the target model through alternating training of the generator and the discriminator; and jointly training the generative adversarial network and the visual question answering model with the goal of maximizing the prediction probability of the visual question answering model for the correct answer; The counterfactual sample auxiliary strategy includes: constructing counterfactual samples, training the visual question answering model with the goal of minimizing the prediction probability of the visual question answering model for the correct answer on the counterfactual samples.

[0017] The method provided in this embodiment has no restrictions on the specific visual question answering (VQA) model used, and can be flexibly applied to different VQA model backbones. In order to solve the problem of insufficient comprehensive utilization of multimodal information by the existing models in the prior art, this embodiment further explores the interaction and coordination mechanism between multimodal data to improve the overall performance of the VQA model, and proposes a new multimodal generation and sample auxiliary training strategy MGNA (Multimodal Generation with Negative-sample Assistance) to better utilize the complementary information of multimodal data, aiming to break through the limitations of existing unbiased methods, not only reducing the bias in the language modality, but also significantly improving the collaborative utilization of multimodal features.

[0018] In order to address the problem of insufficient information of single-modal branches in the multimodal reasoning process, this embodiment introduces a multimodal generation module based on a generative adversarial network to generate multimodal samples with complementary information, effectively making up for the shortcomings of single-modal branches and thus improving the multimodal reasoning capability of the model.

[0019] To address the problem that the shortcut association between questions and answers causes the model to ignore image information, this embodiment designs a counterfactual auxiliary module, which breaks this shortcut association by generating counterfactual samples, enhances the model's attention to image information, and thus significantly improves the generalization performance of the model.

[0020] like Figure 2 As shown in Figure 1, the multimodal generation strategy is implemented based on the multimodal generation module. The multimodal generation module generates a bias model that can simulate bias behavior by combining random noise vectors with given modalities through a generative adversarial network (GAN). The module also uses joint training with the target model (i.e., the visual question answering model) to enable the target model to better capture complementary information between modalities. The following is an introduction to the three main steps of the multimodal generation module in the execution path: random noise sampling, adversarial training bias model, and robust optimization of the target model.

[0021] Random noise sampling. The combination of a random noise vector and a given modality is used to learn the dataset bias and the possible biased behavior of the target model. Since the problem modality is usually more susceptible to dataset bias, the MGNA framework retains the problem modality and treats it as the base modality of the bias model. By fixing the input of the problem modality, the model can focus on simulating the image modality-related biased behavior that the target model may exhibit. However, relying solely on the problem modality is not enough to fully capture the bias information between modalities. To this end, the MGNA framework introduces a random noise vector in the image input. , as a source of bias, where m represents the number of objects in the image and 128 represents the dimension of the feature. The noise vector is drawn from a Gaussian distribution And transformed into a vector with the same dimension as the image feature representation through the generator G, that is, ,in represents the generated deviation feature, This design allows the model to simulate real image features in the bias model through the generated bias features, and combines the problem modality to model the potential bias information in the dataset, further enhancing the learning of modality complementarity.

[0022] Adversarial training bias model. In order to capture the complementary information in multimodal data and simulate the bias behavior of the target model, the core idea is to use the adversarial training strategy to learn the performance differences between the target model and the bias model in different modalities through the coordinated optimization of the generator and the discriminator. Specifically, using the generated noise features Constructing the bias model with the question modality Q, the obtained bias answer vector is expressed as , thus complementing the inadequacy of the single-modal branch in terms of information utilization. Since the generated features have the same spatial dimension as the target feature space, the model can directly use these noise features as input to the target modality, further deepening the learning and simulation of potential biases in the dataset.

[0023] In order to make the bias model learn the bias of a given question, the loss function of the bias model uses the traditional VQA loss, namely the binary cross entropy loss: ; in, represents the binary cross entropy loss function, represents the Sigmoid function, represents the probability of the true answer, and A represents the entire answer set.

[0024] In order to enable the deviation model to capture the performance differences of the target model between different modalities, this embodiment introduces a discriminator D, which aims to simulate and combat the random distribution deviations brought by different modal data by distinguishing whether the generated results of the target model and the deviation model are "real" or "false" answers. Specifically, the goal of the generator G is to minimize the deviation between the generated deviation features and the image features obtained by the target model, and to obtain the answer through the deviation model based on the joint input of the deviation features and the question modality, thereby simulating the deviation behavior of the target model. The task of the discriminator D is to distinguish the deviation answer vector generated by the generator by maximizing the adversarial target. and the answer vector generated by the target model Is it "true" or "false". The discriminator distinguishes the answer vectors predicted by the two models, and the results are expressed as and The goal of the adversarial network can be expressed as: ; Among them, the loss function of the discriminator is Defined as: ; in, Represents the visual modality features, represents the modal characteristics of the problem, is the average value calculation, Represents a logarithmic function.

[0025] Total loss function for the adversarial training bias model It can be expressed as follows: .

[0026] Robust optimization of the target model. During the robust optimization of the target model, the bias answer vector of the bias model It is generated based on the real image input into the bias model. On this basis, in order to further optimize the model performance, a debiasing loss function is further introduced to reduce the impact of the bias model on the target model and enhance the debiasing ability of the target model. In order to fully explore the complementary information between multimodal data, pseudo labels are used. To assist in training.

[0027] The design of pseudo labels can effectively enhance the debiasing effect. Close to the real answer When the pseudo label will be close to the true answer, thus guiding the bias model to simulate the output of the target model more accurately during the training process. On the other hand, when the difference between the output of the bias model and the true answer is large, the pseudo label amplifies the difference by using the Sigmoid function, thereby reducing the interference of the bias model on the training process. Specifically, the i-th element of the pseudo label is defined as: ; in, represents the true answer of the i-th sample, represents the bias model prediction answer of the i-th sample, 1 is used to limit the upper limit of the pseudo label value, 2 is the amplification factor, Represents the Sigmoid function. In this way, pseudo-labeling can not only enhance the debiasing effect, but also effectively utilize the complementary information between multimodal data in the debiasing process, further improving the robustness of the target model in complex tasks.

[0028] The bias model generates an answer distribution with bias information. , providing a variety of training signals for the target model. The target model uses pseudo labels To correct the impact of the bias model and improve its robustness. In this process, the target model and the bias model interact, learn and optimize together, so that the bias model can generate answers that are more in line with the expectations of the target model, and the target model can effectively remove the bias in the training data.

[0029] Finally, the debiasing loss function of the target model is It is defined as a binary cross entropy loss function, which has the following form: ; in, represents the binary cross entropy loss function, represents the answer distribution generated by the target model.

[0030] like Figure 3 As shown in the figure, the counterfactual sample auxiliary strategy is implemented based on the counterfactual auxiliary module. The counterfactual auxiliary module encourages the model to focus on the question-image pair by generating counterfactual samples, thereby avoiding over-reliance on a single modality (such as the question modality) and strengthening the interaction between multiple modalities. This module enables the target model to more effectively capture the key connection between the image and question modalities by generating counterfactual samples and optimizing the auxiliary training of the model. The execution steps of the counterfactual auxiliary module include counterfactual sample generation and auxiliary training optimization.

[0031] Counterfactual sample generation. For each sample , by randomly shuffling the problem and images Sampling is performed to construct two counterfactual samples, namely and ,in, represents a randomly selected image, represents a random selection problem. It should be noted that the image and Problems The counterfactual samples are independently randomly sampled, so the counterfactual samples break the semantic correspondence between the image and the question in the original sample, making the model face the perturbation challenge of multimodal information during training.

[0032] In VQA tasks, question and image modality information complement each other and are indispensable. and Problems When the matches are correct, the model can accurately predict the correct answer. On the contrary, if the counterfactual sample is input , because the modality correspondence is broken, the model cannot rely on the information of a single modality to infer the answer. This mechanism effectively prevents the model from over-relying on a single modality prior, emphasizes the complementarity of image and question modalities, and thus prompts the model to learn to make full use of the joint semantics of the two modalities to answer questions.

[0033] Assisted training optimization. To achieve this goal, the MGNA framework strengthens the joint reasoning ability of image and question modalities by minimizing the model's probability of predicting the correct answer on counterfactual samples, forcing the model to focus on the interaction of multimodal information during training. Counterfactual samples are constructed through human intervention, usually by modifying the key information that supports the correct answer in the question or image modality so that it can no longer directly support the original answer. In this way, when the model faces counterfactual samples, it needs to rely on the remaining modal information for reasoning, rather than relying solely on explicit clues in one modality.

[0034] Specifically, the core goal of optimizing on counterfactual samples is to reduce the model’s over-reliance on the positive sample answer. Suppose a sample consists of an image ,question and the correct answer Composition, by generating a counterfactual image sample , constructing a new input. In this case, the model calculates the conditional probability To predict the answer distribution. In order to weaken the model’s reliance on the lack of supporting information in the image modality, the counterfactual image sample loss function is introduced. : ; In the formula, k is the answer in answer set a Minimize This will cause the VQA model to predict the true answer away from the positive sample due to the lack of supporting visual information. Similarly, for counterfactual question samples , we can define the corresponding counterfactual problem sample loss function To achieve similar optimization goals: .

[0035] In addition, the VQA model needs to be optimized using binary cross entropy loss: ; Among them, N is the total samples of the data set, is the Sigmod function, represents the logarithmic function, represents the answer distribution generated by the original sample, Represents the true answer of the i-th sample.

[0036] Loss function of the counterfactual auxiliary module It can be defined as: ; It should be noted that the above method does not introduce additional annotation information, but makes full use of existing training sample resources.

[0037] We evaluate the performance of MGNA on a challenging VQA benchmark: (1) VQA-CP v2. It is a repartition of the training and test sets of the widely used VQA dataset VQA v2, ensuring that there is a clear difference in the distribution of answers for each question type in the training and test sets. VQA-CP v2 focuses on evaluating the generalization ability of the model in the presence of bias. We follow the official division of the VQA-CP v2 dataset.

[0038] (2) VQA v2. It is one of the most widely used datasets for the VQA task. The distribution of the training set and the validation set is the same. We mainly report the results on the validation set for a comprehensive comparison with other methods. To evaluate the model performance on VQA-CP v2, we use standard metrics to report the overall accuracy of all test samples (All) and the accuracy of three different question categories: Yes / No, Num, and Other. Among them, "Yes / No" represents questions that are answered with "Yes / No", "Num" represents questions that require numeric answers, and "Other" represents other types of questions (neither "Yes / No" nor "Num"). On VQA v2, we also report the accuracy of all test samples (All) and these three question categories (Yes / No, Num, Other) for a comprehensive comparison.

[0039] The method in this embodiment is independent of the specific model and can be flexibly applied to different VQA model backbones. To verify the effectiveness of MGNA, we conducted experiments on a variety of backbone models, including UpDn (Bottom-up and top-down attention). To ensure fair comparison and consistency with previous work, we use pre-trained Faster R-CNN to extract object features of images. Specifically, we extract the first 36 object features for each image, each with a dimension of 2048 and corresponding bounding box information. This setting applies to all datasets, including VQA-CP v2 and VQA v2, because these datasets are essentially built on VQA v2.

[0040] In terms of question feature extraction, we uniformly handle all questions. The maximum length of a question is set to 14, and the part exceeding the length will be truncated, and the part that is insufficient will be filled with filler words. The words in each question are first encoded by GloVe to obtain a word representation with an embedding dimension of 300. Subsequently, these word embeddings are sequence modeled through a single-layer GRU to generate question features with a dimension of 1280.

[0041] In order to evaluate the effectiveness of the model in mitigating language bias, we comprehensively compared the performance of multiple methods on the VQA-CP v2 and VQA v2 datasets. These methods include traditional baseline methods, data augmentation methods, answer re-ranking methods, and models based on biased branch networks or debiased training strategies. The experimental results show that different methods show significant differences in alleviating bias and improving model robustness, especially in low-frequency question types (such as numerical questions). Overall, MGNA shows a strong performance advantage and achieved excellent results on both VQA-CP v2 and VQA v2, as shown in Table 1.

[0042] Table 1 Comparison of the accuracy (%) of different methods on the VQA-CP v2 test set and VQA v2 validation set

[0043] In the table, † indicates that we reproduced. On the VQA-CP v2 test set, the MGNA method performed well, achieving an overall accuracy of 61.07%, surpassing most of the comparison methods, especially on the numerical question (Num), achieving an accuracy of 48.36%. Among the traditional baseline methods, SAN and GVQA performed poorly, with overall accuracy of 24.96% and 31.30%, respectively, while UpDn, although improved, still had an overall accuracy of only 39.74%. Data augmentation methods such as CSS+CL performed well on VQA-CP v2, with an overall accuracy of 59.18%, but they rely on changing the distribution of the training set, which may violate the original design of the VQA-CP dataset. Answer re-ranking methods such as DLR had an overall accuracy of 48.87%, showing some improvement, but lacked generalization ability. Among the methods based on biased branch networks or debiased training strategies, GenB had an overall accuracy of 59.76%, while RUBi performed slightly weaker at 47.11%. In contrast, MGNA performs more comprehensively in multimodal interaction and bias mitigation, significantly outperforming these methods.

[0044] On the VQA v2 validation set, MGNA also performed well, with an overall accuracy of 64.41%. Non-augmented methods solve the data imbalance problem without changing the data distribution, but their performance on IID data is slightly worse than that of augmented methods. For example, RMLVQA has an overall accuracy of 62.57% on VQA v2, while MGNA improves by 1.84%. In addition, compared with other debiasing training strategies, MGNA shows better overall performance on Yes / No and Other question types, with an overall accuracy 2.95% higher than GenB. It is worth mentioning that MGNA has a smaller generalization ability gap (1.44%) between VQA-CP v2 and VQA v2, indicating that it has stronger stability when facing data with different distributions.

[0045] In order to verify the specific contribution of the multimodal generation module and counterfactual auxiliary module in the MGNA framework to the model performance, we conducted an ablation experiment and the results are shown in Table 2.

[0046] Table 2 Ablation study results of VQA-CP v2 dataset

[0047] First, the MGNA model achieved an overall accuracy of 61.07% on the VQA-CP v2 test set, with accuracies of 89.07%, 48.36%, and 49.89% on Yes / No, Num, and Other question types, respectively. This shows that MGNA has comprehensive performance improvements in multimodal reasoning and bias mitigation.

[0048] Generative adversarial networks significantly improve the model's ability to capture multimodal complementary information by introducing the generation and interaction of visual and language modalities. After removing the adversarial network, the model's performance declined in all indicators, especially in numerical problems (Num). This is because numerical problems usually require complex inter-modal reasoning, and the generative nature of the adversarial network module helps simulate this deep interactive information and improve the model's reasoning ability. However, the performance of Yes / No questions improved slightly (from 89.07% to 89.19%) because such questions are less dependent on visual information.

[0049] After removing the counterfactual auxiliary module (NA), the overall accuracy of the model dropped more significantly, to only 56.92%. In particular, on numerical questions (Num), the accuracy dropped sharply from 48.36% to 14.55%, indicating that the NA module plays a key role in breaking the false association between questions and answers. In addition, on Yes / No questions, although the accuracy increased slightly (from 89.07% to 90.04%), this was because the intervention of counterfactual samples weakened the model's reliance on language priors.

[0050] When both the adversarial training bias model (MG) and the counterfactual auxiliary module (NA) are removed at the same time, the performance of the model drops to 39.74, which is almost the same as the performance of the baseline model. This shows that the synergy of MG and NA is the fundamental source of the performance improvement of the MGNA framework. MG provides information on deep interactions between modalities, while NA helps the model get rid of the constraints of false associations through counterfactual samples. This shows that the MG module and the NA module have a synergistic effect in improving the model's ability to utilize multimodal information and breaking the question-answer shortcut association. The combination of the two enables MGNA to not only capture richer multimodal information, but also achieve strong robustness in more complex scenarios.

[0051] The multimodal generation module (MG) and the counterfactual auxiliary module (NA) play a key role in improving the model's ability to capture visual-language interaction information and breaking the false association between questions and answers: the MG module significantly enhances the model's performance in complex reasoning tasks by generating multimodal samples with complementary information, especially in numerical problems (Num) that require deep interaction between vision and language. The NA module breaks the shortcut association between questions and answers by generating counterfactual samples, forcing the model to pay more attention to the role of image information. This improvement is particularly significant in Other and numerical problems, indicating that it is of great significance in alleviating language prior bias.

[0052] In general, the combination of the two modules enables MGNA to better alleviate the bias problem in multimodal reasoning and demonstrates comprehensive performance advantages in various types of problems. Future research can further optimize these two modules to better adapt to different types of problems and data distributions.

[0053] Embodiment 2 This embodiment provides a visual question answering system based on generative adversarial and sample optimization, including: A visual question answering task acquisition module, configured to acquire questions and images in a visual question answering task; A visual question answering result acquisition module is configured to input questions and images into a trained visual question answering model to obtain a visual question answering result; Among them, the visual question answering model is trained using a multimodal generation strategy and a counterfactual sample auxiliary strategy; The multimodal generation strategy includes: constructing a bias model based on a generator of a generative adversarial network, converting a random noise vector into a bias feature of the same dimension as the image feature representation, and generating a bias answer vector based on the input question and the bias feature; distinguishing whether the answer vector generated by the visual question answering model and the bias answer vector generated by the bias model are consistent with the facts based on a discriminator of the generative adversarial network; making the answer distribution of the bias model close to the answer distribution of the target model through alternating training of the generator and the discriminator; and jointly training the generative adversarial network and the visual question answering model with the goal of maximizing the prediction probability of the visual question answering model for the correct answer; The counterfactual sample auxiliary strategy includes: constructing counterfactual samples, training the visual question answering model with the goal of minimizing the prediction probability of the visual question answering model for the correct answer on the counterfactual samples.

[0054] It should be noted here that the various modules in this embodiment correspond one-to-one to the steps of the method in Example 1, and the specific implementation process is the same, which will not be repeated here.

[0055] Embodiment 3 This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method in the first embodiment.

[0056] The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and LA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0057] The memory may include one or more computer-readable media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable medium in the memory is used to store at least one computer program, which is used to be executed by the processor to implement a visual question answering method based on generative adversarial and sample optimization provided in an embodiment of the present disclosure.

[0058] Those skilled in the art can understand that the electronic device provided in this embodiment can include more or fewer components, or combine certain components, or adopt different component arrangements.

[0059] Embodiment 4 This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the method in the first embodiment are completed.

[0060] Embodiment 5 This embodiment provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method in Embodiment 1. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A visual question answering method based on generative adversarial and sample optimization, characterized in that: include: Get questions and images for visual question answering tasks; Input the question and image into the trained visual question answering model to obtain the visual question answering result; Among them, the visual question answering model is trained using a multimodal generation strategy and a counterfactual sample auxiliary strategy; The multimodal generation strategy includes: constructing a bias model based on a generator of a generative adversarial network, converting a random noise vector into a bias feature of the same dimension as the image feature representation, and generating a bias answer vector based on the input question and the bias feature; distinguishing whether the answer vector generated by the visual question answering model and the bias answer vector generated by the bias model are consistent with the facts based on a discriminator of the generative adversarial network; making the answer distribution of the bias model close to the answer distribution of the target model through alternating training of the generator and the discriminator; and jointly training the generative adversarial network and the visual question answering model with the goal of maximizing the prediction probability of the visual question answering model for the correct answer; The counterfactual sample auxiliary strategy includes: constructing counterfactual samples, training the visual question answering model with the goal of minimizing the prediction probability of the visual question answering model for the correct answer on the counterfactual samples.

2. A visual question answering method based on generative adversarial and sample optimization as claimed in claim 1, characterized in that: The random noise vector is sampled from a Gaussian distribution.

3. The visual question answering method based on generative adversarial and sample optimization according to claim 1, characterized in that: The goal of the generator is to minimize the deviation between the generated deviation features and the image features, and to obtain the answer through the deviation model based on the joint input of the deviation features and the question modality, so that the deviation model can simulate the deviation behavior of the target model.

4. The visual question answering method based on generative adversarial and sample optimization according to claim 1, characterized in that: The method aims to maximize the prediction probability of the visual question answering model for the correct answer and jointly trains the generative adversarial network and the visual question answering model, and also includes using pseudo-labels to assist in the training. When the output of the deviation model is close to the true answer, the pseudo-label is also close to the true answer, so as to guide the deviation model to accurately simulate the output of the target model during the training process. When the difference between the output of the deviation model and the true answer is large, the pseudo-label amplifies the difference and reduces the interference of the deviation model on the training process.

5. The visual question answering method based on generative adversarial and sample optimization according to claim 1, characterized in that: The counterfactual samples include image counterfactual samples and question counterfactual samples. In the image counterfactual samples, the images are randomly selected, and in the question counterfactual samples, the questions are randomly selected.

6. The visual question answering method based on generative adversarial and sample optimization according to claim 1, characterized in that: The loss functions in the counterfactual sample auxiliary strategy include the counterfactual question sample loss function, the counterfactual question sample loss function, and the cross entropy loss function.

7. A visual question answering system based on generative adversarial and sample optimization, characterized in that: include: A visual question answering task acquisition module, configured to acquire questions and images in a visual question answering task; A visual question answering result acquisition module is configured to input questions and images into a trained visual question answering model to obtain a visual question answering result; Among them, the visual question answering model is trained using a multimodal generation strategy and a counterfactual sample auxiliary strategy; The multimodal generation strategy includes: constructing a bias model based on a generator of a generative adversarial network, converting a random noise vector into a bias feature of the same dimension as the image feature representation, and generating a bias answer vector based on the input question and the bias feature; distinguishing whether the answer vector generated by the visual question answering model and the bias answer vector generated by the bias model are consistent with the facts based on a discriminator of the generative adversarial network; making the answer distribution of the bias model close to the answer distribution of the target model through alternating training of the generator and the discriminator; and jointly training the generative adversarial network and the visual question answering model with the goal of maximizing the prediction probability of the visual question answering model for the correct answer; The counterfactual sample auxiliary strategy includes: constructing counterfactual samples, training the visual question answering model with the goal of minimizing the prediction probability of the visual question answering model for the correct answer on the counterfactual samples.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to complete the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images