A multi-expert visual question answering debiasing method based on bias prototype routing

Abstract

The application relates to the technical field of visual question answering and multi-modal learning, and discloses a multi-expert visual question answering debiasing method based on bias prototype routing. The method extracts fusion features of images and questions, calculates multi-source bias strength by using a question bias branch, an image bias branch and a question type bias branch, and generates bias representation; high bias samples are screened according to the bias strength, the bias representation of the high bias samples is clustered, a plurality of bias prototypes are obtained, and corresponding bias models, debiasing experts and low bias experts are constructed; during training, a bias suppression signal is provided by a frozen bias model, and a gating network and an expert are optimized by combining answer supervision loss and expert differentiation loss; during testing, the weight of an expert judgment result is generated according to the similarity between a sample and each bias prototype, the expert output is weighted and fused to obtain an answer prediction. The application can adaptively select a debiasing expert for bias of different sources and strengths, and improve the prediction stability in a distribution change scene.

Description

Technical Field

[0001] This invention relates to the fields of visual question answering and multimodal learning technology, specifically to a method for constructing a biased prototype, training with multiple experts, and fusing biased routed answers for visual question answering tasks. Background Technology

[0002] Visual question answering tasks require models to output corresponding answers based on input images and natural language questions. This task not only requires recognizing targets, attributes, relationships, and scenes in images, but also understanding the semantics of the questions and establishing a correspondence between image evidence and question semantics. With the development of multimodal representation learning, visual question answering models have been applied to scenarios such as intelligent interaction, decision support, image retrieval, and robot perception.

[0003] Existing visual question answering models typically employ an image encoder to extract image features, a question encoder to extract semantic features of the question, and a multimodal fusion module to obtain a joint representation of the image and the question. Finally, a classifier or generator outputs the probability of candidate answers. However, when the training data distribution exhibits statistical correlation, the model is prone to learning shortcut features unrelated to real visual evidence, thus forming a dependence on language priors, visual co-occurrence priors, or question type priors.

[0004] For example, a model might predict answers based solely on high-frequency words in the question text, or solely on common object combinations in the image, or it might prioritize outputting high-frequency answers from the training set when encountering a specific question template. These biases can cause the model to perform well within the training distribution, but its generalization ability will significantly decrease when the answer distribution, question template, or image scene changes.

[0005] To mitigate the bias problem in visual question answering models, existing techniques have proposed solutions such as question branch-based bias removal, counterfactual sample construction, data augmentation, answer re-ranking, causal intervention, and contrastive learning. These solutions can reduce the model's dependence on a single source of bias to some extent, but they still have the following shortcomings.

[0006] First, existing solutions often treat bias as a uniform object, making it difficult to simultaneously distinguish between multiple sources of bias, such as question text bias, image bias, and question type bias. Second, existing solutions typically employ the same bias removal strategy for samples with different biases, lacking a mechanism for fine-grained divide-and-conquer based on bias patterns. Third, existing solutions may sacrifice answer accuracy on low-biased samples or in-distribution samples when improving out-of-distribution performance. Fourth, while hybrid expert structures can improve model capacity and inference capabilities, they typically lack interpretable semantic routing basis for bias in visual question answering scenarios.

[0007] Therefore, there is an urgent need for a robust visual question answering method that can identify the sources and intensity of sample bias, discover interpretable bias prototypes, train corresponding debiasing experts for different bias prototypes, and adaptively assign expert weights based on the bias representation of the input samples during the testing phase. Summary of the Invention

[0008] The purpose of this invention is to provide a multi-expert visual question answering debiasing method based on deviation prototype routing, which solves the technical problems of existing visual question answering debiasing methods, such as coarse modeling of deviation types, lack of semantic basis for expert division of labor, and difficulty in taking into account both low-biased and high-biased samples.

[0009] To achieve the above objectives, this invention provides a multi-expert visual question answering debiasing method based on deviation prototype routing. The method includes: acquiring images, natural language questions, and answer labels from training samples; extracting image features, question features, and image-question fusion features; training a basic visual question answering model and question deviation branches, image deviation branches, and question type deviation branches; calculating multi-source deviation intensity scores based on the outputs of the three types of deviation branches and constructing deviation representations; clustering multiple deviation prototypes based on the deviation representations and constructing low-deviation baseline prototypes based on low-deviation samples; constructing a deviation model and debiasing experts for each deviation prototype, and constructing low-deviation experts; and generating expert routing weights based on the similarity between the input samples and each prototype during the testing phase, and weighted fusing the answer probability distributions output by each expert.

[0010] The distribution of multimodal feature extraction and basic answer prediction is as follows: in, The sequence represents the training sample index, input image, natural language question, and candidate answer, respectively. The components are represented in sequence: image encoder, question encoder, multimodal fusion module, and answer classifier. These are represented, in order, image features, question features, and image-question fusion features; This represents the probability distribution of candidate answers in a basic visual question answering model.

[0011] The three types of bias branches output the following candidate answer probability distributions: in, These represent the probabilities of candidate answers output by the question bias branch, image bias branch, and question type bias branch, respectively. These are represented, in order, the problem bias branch, the image bias branch, and the problem type bias branch; These represent the problem type mapping function and the problem type characteristics, respectively.

[0012] The three types of bias scores and the multi-source bias intensity scores are calculated as follows: in, The two symbols represent the set of candidate answers and the answer label score, respectively. These represent, in order, the problem bias score, the image bias score, and the problem type bias score; Indicates the multi-source bias intensity fraction; This represents the non-negative weights corresponding to the three types of deviation scores.

[0013] The bias characterization and sample set division are as follows: in, The bias representation of the training samples; the feature concatenation operation in the formula represents vector concatenation; These represent the high-bias sample set, the low-bias sample set, and the bias intensity threshold, respectively.

[0014] The deviation prototype and the low-deviation baseline prototype are obtained as follows: in, The values ​​represent the deviation clusters, the number of deviation clusters, and the deviation cluster index, respectively. These represent the deviation prototype and the low-deviation baseline prototype, respectively. These represent the number of samples in the corresponding biased cluster and the number of low-biased samples, respectively.

[0015] For each deviation prototype, a corresponding deviation model and a debiasing expert are constructed, and the frozen deviation model is used to provide a deviation suppression signal for the corresponding debiasing expert. The deviation suppression loss is expressed as: in, This indicates that the deviation suppresses the loss; The bias model and the debiasing expert are represented in sequence. These represent the probabilities of candidate answers output by the bias model and the debiased expert, respectively.

[0016] Joint training employs answer supervision loss, bias suppression loss, and expert differentiation loss. The expert differentiation loss and total loss are expressed as follows: in, This represents the expert differentiation loss; the similarity calculation function in the formula is used to measure how close the probability distributions of the two expert answers are. These represent the probability distributions of the answers output by the two biased experts, respectively. The weights are represented in order: total loss, answer supervision loss, bias suppression loss weight, and expert differentiation loss weight.

[0017] During the testing phase, expert route weights are generated and expert outputs are merged using the following formula: in, The bias representation, cosine similarity, and gating network of the test samples are represented in that order. These represent the expert route weight vector and the individual expert route weight, respectively. The numbers represent the probability distribution of the fused answer and the final output answer, respectively.

[0018] In one implementation, the multimodal feature extraction module performs feature extraction and fusion processing for image and natural language problems; the multi-source bias intensity assessment module performs multi-source bias intensity calculation and bias representation generation processing; the bias prototype construction module performs sample partitioning, bias representation processing, and bias prototype construction; the multi-expert training module performs training of the bias model, debiased experts, low-bias experts, and gated networks; and the gated routing and answer fusion module performs expert routing weight generation and answer probability distribution fusion.

[0019] Compared with existing technologies, the present invention has at least the following beneficial effects: First, by constructing a multi-source bias intensity score through question bias, image bias, and question type bias, it can more comprehensively identify visual question answering bias samples; Second, by obtaining interpretable bias prototypes through bias representation clustering, it can distinguish different types and intensities of bias patterns; Third, by applying bias suppression signals to debiasing experts through bias models, it reduces the dependence of debiasing experts on high-confidence incorrect answers; Fourth, by fusing soft routing between low-bias experts and multiple debiasing experts, it simultaneously considers the prediction effects of low-bias samples and multiple types of high-bias samples; Fifth, the routing weights based on bias prototypes have clear bias semantic basis, improving the interpretability of the multi-expert visual question answering debiasing process. Attached Figure Description

[0020] Figure 1 This is an overall framework diagram of a multi-expert visual question answering debiasing method based on deviation prototype routing provided in an embodiment of the present invention.

[0021] Figure 2 This is a structural diagram of the gated routing and answer fusion module provided in an embodiment of the present invention.

[0022] Figure 3 This is a visual comparison diagram of the embodiments of the present invention with other methods. Detailed Implementation

[0023] The specific embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the following embodiments are used to explain the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention; without departing from the concept of the present invention, those skilled in the art can make equivalent substitutions or adjustments to the specific network structure, training data, parameter values ​​and deployment methods.

[0024] like Figure 1 As shown, this embodiment uses a visual question-answering training set as input. Each sample in the training set includes an image, a natural language question, and an answer label, which can be represented as: in, The numbers represent the training set, training sample index, total number of samples, input image, natural language question, and answer label distribution, respectively.

[0025] First, multimodal feature extraction is performed. The image encoder extracts image features, the question encoder extracts question semantic features, the multimodal fusion module generates fused features based on the image features and question features, and the answer classifier outputs the candidate answer probability distribution of the basic visual question answering model, specifically: in, The components are represented in sequence: image encoder, question encoder, multimodal fusion module, and answer classifier. The following are represented in sequence: image features, question features, image-question fusion features, candidate answers, and the prediction probability of the candidate answers by the basic visual question answering model.

[0026] Image encoders can be implemented using convolutional neural networks, visual transformation networks, or target region feature extraction networks; problem encoders can be implemented using recurrent neural networks, attention encoders, or pre-trained language encoders; and multimodal fusion modules can be implemented using bilinear fusion, attention fusion, or mapping-based concatenation fusion.

[0027] like Figure 1 As shown, three types of bias branches are constructed simultaneously during the training of the basic visual question answering model. The question bias branch predicts the answer based solely on question features, the image bias branch predicts the answer based solely on image features, and the question type bias branch predicts the answer based solely on question type features. The outputs of the three types of bias branches are represented as follows: in, These represent, in sequence, the problem deviation branch, the image deviation branch, the problem type deviation branch, and the problem type mapping function; The three types of bias branches represent the probability of candidate answers and the characteristics of question types, respectively.

[0028] The three types of bias scores are calculated based on the predicted probability of the true answer by the corresponding bias branch, and then the multi-source bias intensity score is obtained: in, The numbers represent, in order, the candidate answer set, answer label score, question bias score, image bias score, question type bias score, multi-source bias intensity score, and the non-negative weights corresponding to the three types of bias scores.

[0029] Training samples are partitioned based on multi-source bias intensity scores. When the multi-source bias intensity score is not less than a bias intensity threshold, the sample is classified as a high-biased sample; when the multi-source bias intensity score is less than the bias intensity threshold, the sample is classified as a low-biased sample. Specifically: in, These represent the high-bias sample set, the low-bias sample set, and the bias intensity threshold, respectively. The bias intensity threshold can be predetermined based on the performance of the validation set, or a fixed empirical value can be used.

[0030] A bias representation is constructed for each training sample. This bias representation includes the output of the question bias branch, the image bias branch, the question type bias branch, and the multi-source bias intensity score, expressed as follows: in, The bias representation of the training samples and the vector concatenation operation are represented sequentially.

[0031] The deviation representations in the high-deviation sample set are clustered to obtain multiple deviation clusters, and the center of each deviation cluster is used as the deviation prototype. The mean of the deviation representations in the low-deviation sample set is taken to obtain the low-deviation baseline prototype, specifically: in, These represent, in order: deviation cluster, number of deviation clusters, deviation cluster index, deviation prototype, low-deviation baseline prototype, number of samples in the corresponding deviation cluster, and number of low-deviation samples. Each deviation prototype corresponds to a sample pattern with similar deviation sources and deviation intensities.

[0032] like Figure 1 As shown, a bias model and a debiasing expert are constructed for each bias prototype, and an additional low-bias expert is also constructed. The bias model is pre-trained on high-biased samples within the corresponding bias cluster, and the weights are frozen after training. The debiasing expert uses the bias response provided by the corresponding bias model as a negative suppression signal during the joint training phase. The low-bias expert is used to maintain the answer prediction ability on low-biased samples and in-distribution samples. The above model and experts can be represented as: in, These represent the nth bias model, the nth debiasing expert, and the low bias expert, respectively.

[0033] For the nth debiasing expert, bias suppression loss is used to reduce the overlap between their predictions and the corresponding biased model on non-true answers: in, Let $\mathbf{x}$ represent the bias suppression loss, the probability of the candidate answer output by the $\mathbf{x} bias model, and the probability of the candidate answer output by the $\mathbf{x} debiasing expert, respectively. By minimizing this loss, when the bias model gives a high probability to a non-true candidate answer, the debiasing expert will reduce the prediction probability of that candidate answer.

[0034] In supervised training, the supervised loss is calculated based on the fused answer probability distribution and answer label: in, The numbers represent the supervised loss and the probability distribution of the fused answer, respectively.

[0035] To avoid convergence in the output distribution of multiple debiasing experts, an expert differentiation loss is set: in, The following are represented, in order: expert differentiation loss, similarity calculation function, probability distribution of answers output by two different biased experts, and index of another biased expert.

[0036] The total loss during the joint training phase is: in, The weights represent the overall loss, the bias suppression loss weight, and the expert differentiation loss weight, respectively. During training, the parameters of the gating network and each bias-reducing expert are jointly updated using the overall loss. The low-bias experts are trained using the answer-supervised loss, and the frozen bias model does not participate in parameter updates.

[0037] like Figure 2 As shown, during the testing phase, the input samples are first processed to extract bias representations in the same way as during the training phase. Then, the similarity between these bias representations and the low-bias baseline prototypes, as well as between each bias prototype, is calculated. Finally, the gating network generates expert routing weights. in, The following are represented in sequence: bias representation of test sample, similarity between test sample and prototype, cosine similarity function, gating network, expert routing weight vector, and individual expert routing weight.

[0038] The probability distributions of the candidate answers output by each expert are weighted and fused according to the expert routing weights, and the candidate answer with the highest probability is output: in, The test image, test question, probability of candidate answer output by the nth expert, and final output answer are represented in sequence.

[0039] During the training phase, this embodiment first trains the basic visual question answering model and three types of bias branches; then freezes the three types of bias branches, calculates the multi-source bias intensity scores and bias representations of the entire training set samples; then filters high-bias samples and clusters them to generate bias prototypes; next, it pre-trains the bias model based on the corresponding bias clusters and freezes its weights; finally, it jointly trains low-bias experts, multiple debiasing experts, and a gating network.

[0040] During the testing phase, when the input sample is closer to a high-bias prototype, the gating network increases the routing weight of the corresponding debiasing expert; when the input sample is closer to the low-bias benchmark prototype, the gating network increases the routing weight of the low-bias expert; when the input sample has multiple bias characteristics, the gating network outputs soft routing weights, enabling multiple experts to participate in answer prediction.

[0041] Figure 3 The visualization comparison results shown illustrate that, when the question text or image content easily induces the base model to give high-frequency incorrect answers, on the VQA-CP v2 dataset, this invention, through biased prototyping, bias suppression, and gated fusion, makes the final answer more dependent on the correspondence between real image evidence and question semantics compared to other traditional and advanced visual question answering methods. The dashed boxes represent key target areas that the model focuses on and that are related to question semantics.

[0042] Each module in this embodiment can be implemented by a computer program executed by one or more processors, or it can be deployed in a server, workstation, edge computing device, or cloud platform. The division between the modules described above is only for illustrating functional relationships and does not constitute a limitation on the number of software modules, hardware deployment location, or network layer implementation method.

Claims

1. A multi-expert visual question answering method for bias removal based on biased prototype routing, characterized in that, Includes the following steps: Step 1: Obtain images, natural language questions, and answer labels from the training samples, and extract image-question fusion features through an image encoder, a question encoder, and a multimodal fusion module; Step 2: Train the basic visual question answering model and the question bias branch, image bias branch, and question type bias branch. After the training of the three types of bias branches is completed, freeze their weights. Calculate the multi-source bias intensity score of the training samples based on the output of the three types of bias branches and generate bias representations. Divide the training samples into high-bias samples and low-bias samples according to the multi-source bias intensity score. Step 3: Cluster the deviation representations of the high deviation samples to obtain multiple deviation clusters, and use the center of each deviation cluster as the deviation prototype. At the same time, construct a low deviation benchmark prototype based on the mean deviation representation of the low deviation samples, and freeze the weights of the deviation model after pre-training the corresponding deviation model for each deviation prototype. Step 4: Construct a corresponding debiasing expert for each deviation prototype and a low-bias expert. Jointly train the debiasing expert, the low-bias expert, and the gating network on the full training samples. The frozen deviation model provides a deviation suppression signal for the corresponding debiasing expert. Step 5: Extract the deviation representation of the input sample during the testing phase, calculate the similarity between the deviation representation and each deviation prototype and the low deviation benchmark prototype, output the routing weight of each expert by the gating network, and perform weighted fusion on the probability distribution of the answers output by each expert according to the routing weight to obtain the final visual question answer.

2. The method according to claim 1, characterized in that, The image-question fusion features are obtained using the following formula: in, Indicates the training sample index; The input image, the natural language question, and the candidate answers are represented in that order. The components are represented in sequence: image encoder, question encoder, multimodal fusion module, and answer classifier. These are represented, in order, image features, question features, and image-question fusion features; This represents the probability distribution of the predicted candidate answers by the basic visual question answering model.

3. The method according to claim 1, characterized in that, The outputs and deviation scores of the three types of deviation branches are calculated using the following formulas: in, These represent the probabilities of candidate answers output by the question bias branch, image bias branch, and question type bias branch, respectively. These are represented, in order, the problem bias branch, the image bias branch, and the problem type bias branch; The problem type mapping function and problem type characteristics are represented in sequence. These represent the candidate answer set, candidate answers, and answer label scores, respectively. These represent the problem deviation score, image deviation score, and problem type deviation score, respectively.

4. The method according to claim 3, characterized in that, The multi-source deviation intensity fraction is calculated using the following formula: in, This represents the multi-source bias intensity score of the training samples; These represent, in order, the problem bias score, the image bias score, and the problem type bias score; The non-negative weights corresponding to the three types of bias scores are represented, and their sum is one; when the multi-source bias intensity score is not less than the bias intensity threshold, it is classified as a high-bias sample, otherwise it is classified as a low-bias sample.

5. The method according to claim 1, characterized in that, The deviation characterization is constructed using the following formula: in, The bias representation of the training samples; the feature concatenation operation in the formula represents vector concatenation; The outputs of the three types of bias branches and the multi-source bias intensity scores are represented in sequence. These represent the high-bias sample set, the low-bias sample set, and the bias intensity threshold, respectively; the bias characterization of the high-bias samples is used for bias prototype clustering and gated routing during the testing phase.

6. The method according to claim 5, characterized in that, The deviation prototype and the low-deviation baseline prototype are obtained through the following formula: In the formula, the clustering operation represents the clustering process for the bias representation of high-biased samples; The numbers represent the order of the numbers. One deviation cluster, the number of deviation clusters, and the deviation cluster index; Indicates the first A deviation prototype; This represents a low-deviation baseline prototype; The numbers represent the number of samples in the corresponding set, respectively; the cluster centers obtained by clustering are used as the corresponding deviation prototypes, and the mean deviation representation of the low-biased samples is used as the low-biased baseline prototype.

7. The method according to claim 6, characterized in that, For each deviation prototype, a corresponding deviation model and a debiasing expert are constructed. The deviation model adopts the same image and question input structure as the basic visual question answering model and is pre-trained using high-deviation samples within the corresponding deviation cluster. When training the corresponding debiasing expert, the weights of the bias model are kept frozen, so that its output serves as the bias suppression signal for the corresponding debiasing expert.

8. The method according to claim 7, characterized in that, The deviation suppression loss corresponding to the deviation suppression signal is: in, Indicates the first The bias suppression loss corresponding to each debiasing expert; The numbers represent the order of the numbers. The first deviation model and the second A specialist who goes astray; The probabilities of the bias model and the bias removal expert's predictions of the candidate answers are represented in sequence. Suppression weights are used to characterize non-true candidate answers; by minimizing the above-mentioned bias suppression loss, the dependence of debiasing experts on high-confidence non-true answers corresponding to biased models is reduced.

9. The method according to claim 8, characterized in that, The joint training employs the following total loss: in, This represents the expert differentiation loss; the similarity calculation function in the formula is used to measure how close the probability distributions of the two expert answers are. The numbers represent the order of the numbers. The and the first The probability distribution of the answers output by biased experts; The weights of total loss, answer supervision loss, bias suppression loss, and expert differentiation loss are represented in that order. The training of the gating network and the bias-reducing experts is carried out by jointly constraining the answer supervision loss, bias suppression loss, and expert differentiation loss. The low-bias experts are trained using the answer supervision loss.

10. The method according to claim 1, characterized in that, The gated network outputs the routing weights of each expert and performs answer fusion, including: in, The deviation characteristics of the test samples are represented sequentially, and the first... Each prototype and its corresponding cosine similarity; The gating network, expert routing weight vector, and the first... The routing weight of each expert; In order, they represent low-bias experts and the first... A specialist who goes astray; The fused answer probability distribution, the final output answer, and the candidate answer set are represented sequentially. Expert routing weights are generated based on the cosine similarity between the input sample and each prototype, and the candidate answer with the highest probability is used as the final output answer.

Images