Visual question-answering method and system based on sample difficulty perception and multi-path fusion

Abstract

The invention discloses a visual questioning and answering method and system based on sample difficulty perception and multipath fusion, and relates to the technical field of visual questioning and answering, and the method comprises the steps: inputting a visual questioning and answering sample into a visual questioning and answering network, coding an image and a question through a backbone network in the network, carrying out the multi-modal fusion, and then sending the image and the question into a backbone head, generating a basic prediction result; a sample difficulty coefficient is calculated based on the true confidence degree of single-mode prediction through a sample difficulty adaptive module, a class-level boundary and an instance-level boundary are fused, and a prediction result is output through an angle boundary head; and through a causal path consistency module, respectively calculating and predicting fusion features under a complete path, a knowledge suppression path and a problem-only indirect effect path, taking a prediction difference value of the complete fusion path and the knowledge suppression path as a prediction result output by a residual head, and finally integrating prediction results of a trunk head, an angle boundary head and the residual head in the network. And outputting a final answer to realize unbiased visual question answering.

Description

Technical Field

[0001] This invention relates to the field of visual question answering technology, and in particular to a visual question answering method and system based on sample difficulty perception and multi-path fusion. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Visual question answering (VQA), a core task of multimodal understanding, requires models to generate accurate answers based on input images and natural language questions. This task integrates the capabilities of computer vision and natural language processing, demanding not only the recognition of image content but also the understanding of question semantics and cross-modal reasoning. Therefore, it has gradually become an important direction in vision-language research and is widely applied in scenarios such as intelligent interaction and autonomous driving. However, in practical applications, existing models generally suffer from severe language prior dependence, tending to rely on high-frequency language patterns in the training data for answers. This makes the models perform poorly on questions that must rely on image evidence, resulting in limited robustness and generalization ability.

[0004] To alleviate the language prior problem, existing debiasing research mainly falls into two categories: one is data redistribution methods, which balance data distribution through resampling / reweighting to reduce the influence of high-frequency text patterns. For example, biased instances and label imbalance can be used to perform sample-level debiasing at the data level, reducing the dominance of high-frequency language patterns in training. The other is model constraint methods, which suppress language shortcuts through counterfactual or contrastive learning. For example, in a counterfactual framework, language bias can be split into shortcut bias and distribution bias and suppressed separately, or language-guided bias can be generated internally by the target model and constrained by a contrastive strategy, weakening the dependence on external statistical branches. However, both of these methods generally lack explicit modeling of sample difficulty, and training is still dominated by easy samples (i.e., samples that can be directly answered by language). They also tend to focus on the overall consistency of modalities, making it difficult to establish fine-grained causal dependencies between questions and image regions, resulting in poor accuracy and low precision in the final visual question answering. Summary of the Invention

[0005] To address the issues of poor accuracy in existing visual question answering methods due to easy sample dominance and insufficient region dependence, this invention provides a visual question answering method and system based on sample difficulty perception and multi-path fusion. This method can dynamically adapt to sample difficulty and strengthen region-level causal relationships, achieving unbiased visual question answering and improving the applicability and accuracy of visual question answering models in complex real-world scenarios.

[0006] In a first aspect, the present invention provides a visual question answering method based on sample difficulty perception and multi-path fusion.

[0007] A visual question answering method based on sample difficulty perception and multi-path fusion includes: Obtain visual question answering samples; The image and question from the visual question answering sample are input into the visual question answering network. The prediction results of the backbone head, angle boundary head and residual head in the network are integrated to output the final answer. The visual question answering network includes a backbone network, a sample difficulty adaptive module and a causal path consistency module. In the backbone network, an image encoder and a question encoder are used to encode the image and the question respectively. After multimodal fusion, the results are sent to the backbone head to generate basic prediction results. In the sample difficulty adaptive module, the sample difficulty coefficient is calculated based on the true confidence of single-modal prediction, class-level boundary and instance-level boundary are fused, and the prediction result is output through the angle boundary header; In the causal path consistency module, the fusion features under the complete path, the knowledge suppression path, and the indirect effect path of the problem only are calculated and predicted respectively. The prediction difference between the complete fusion path and the knowledge suppression path is the prediction result output by the residual head.

[0008] In a further technical solution, the sample difficulty adaptive module calculates the sample difficulty coefficient based on the true confidence level of single-modal prediction, as follows: Based on image encoder and question encoder, the images and questions in the samples are encoded respectively. A unimodal prediction score is generated based on the image encoding and question encoding, and the unimodal confidence of the true class is extracted. Based on the true confidence level of the single-modal prediction, the original difficulty coefficient of the sample is calculated, and the final sample difficulty coefficient is obtained after pruning and normalization.

[0009] A further technical solution integrates class-level and instance-level boundaries, and outputs the prediction results through an angle boundary header, as follows: The class boundaries are obtained by statistically analyzing the occurrence probabilities of each problem type and category in the training set, followed by rarity normalization, cropping, and normalization. The answer prediction value of each sample is masked using a deterministic masking operator to obtain the basic boundary after masking the invalid class. Then, the sample difficulty coefficient is combined and the instance-level boundary is generated by temperature normalization. It integrates class-level and instance-level boundaries, balances class priors and sample adaptation capabilities through piecewise decreasing coefficients, applies boundary constraints only at the true class index, and outputs predicted scores through angular boundary headers.

[0010] In a further technical solution, the loss function of the visual question answering network is the sum of the standard classification loss of fusion prediction, the angle boundary loss of difficulty perception, and the causal path consistency loss; Among them, the standard classification loss of the fusion prediction is the standard cross-entropy loss of the backbone network; The angle boundary loss for difficulty perception is the sum of the angle loss calculated based on the dynamic boundary of the sample difficulty adaptive module, combined with the frequency-weighted loss and the inter-sample comparison loss; The causal path consistency loss is the sum of the path-level cross-entropy loss and the KL distillation loss of the causal path consistency module.

[0011] A further technical solution is that the training process of the causal path consistency module is as follows: Calculate the fusion features under the complete path, the knowledge-suppressed path, and the indirect effect path of only the problem; Apply cross-entropy loss to each path to monitor the accuracy of the prediction; The complete fusion path is used as the teacher branch, and the indirect effect path is used as the student branch. Knowledge transfer is carried out through gradient blocking KL distillation loss. Training is performed using path-level cross-entropy loss and KL distillation loss to achieve consistency constraints in multi-path prediction.

[0012] A further technical solution integrates the prediction results from the backbone, angle boundary, and residual heads in the network to output the final answer, including: Based on the prediction results of the backbone, angle boundary, and residual heads in the network, the confidence level of each prediction result is calculated after temperature normalization. Then, the final answer is output through weighted fusion calculation.

[0013] Secondly, the present invention provides a visual question answering system based on sample difficulty perception and multi-path fusion.

[0014] A visual question answering system based on sample difficulty perception and multi-path fusion includes: The data acquisition module is used to acquire visual question-answering samples; The visual question answering output module is used to input the images and questions from the visual question answering samples into the visual question answering network, integrate the prediction results of the backbone head, angle boundary head and residual head in the network, and output the final answer; the visual question answering network includes a backbone network, a sample difficulty adaptive module and a causal path consistency module. In the backbone network, an image encoder and a question encoder are used to encode the image and the question respectively. After multimodal fusion, the results are sent to the backbone head to generate basic prediction results. In the sample difficulty adaptive module, the sample difficulty coefficient is calculated based on the true confidence of single-modal prediction, class-level boundary and instance-level boundary are fused, and the prediction result is output through the angle boundary header; In the causal path consistency module, the fusion features under the complete path, the knowledge suppression path, and the indirect effect path of the problem only are calculated and predicted respectively. The prediction difference between the complete fusion path and the knowledge suppression path is the prediction result output by the residual head.

[0015] Thirdly, the present invention also provides an electronic device, comprising: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the above-mentioned visual question answering method based on sample difficulty perception and multi-path fusion.

[0016] Fourthly, the present invention also provides a computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the above-described visual question answering method based on sample difficulty perception and multi-path fusion.

[0017] Fifthly, the present invention also provides a computer program product comprising executable instructions stored in a computer-readable storage medium; wherein, when the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the aforementioned visual question answering method based on sample difficulty perception and multi-path fusion is implemented.

[0018] The above one or more technical solutions have the following beneficial effects: This invention proposes a visual question answering method and system based on sample difficulty perception and multi-path fusion. To address the issues of excessive linguistic prior and insufficient regional causal dependence, a difficulty-sensitive debiasing framework is designed. Firstly, a learned angle space is introduced: the sample difficulty is characterized by the confidence levels of the question and view branches towards the true answer. The decision boundary is adaptively adjusted based on the sample difficulty, that is, this difficulty level is injected into the angle loss as an instance-level adaptive boundary to dynamically adjust the decision boundary. This applies stronger angle discrimination constraints to easy samples and reduces the angle interval to difficult samples, adaptively adjusting the strictness of the discrimination boundary, thereby suppressing the dominance of easy sample text and strengthening the contribution of visual evidence from difficult samples. Secondly, a causal graph-guided multi-path consistency fusion mechanism is introduced: the question path, visual path, and fusion path are combined to form a path-level effect estimate. Path-level fusion is performed in the probability space using an Explain-Away logarithmic-product approach, thereby suppressing spurious and irrelevant associations and highlighting visual evidence truly relevant to the answer. Through these strategies, this invention can effectively mitigate linguistic bias, thus achieving more accurate visual question answering.

[0019] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 The diagrams illustrate two typical failure scenarios of existing visual question answering models; where (a) represents a simple sample and (b) represents a difficult sample. Figure 2 This is an overall architecture diagram of the visual question answering network model based on sample difficulty and causal consistency in an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed descriptions are exemplary and are intended only to describe specific embodiments and to provide further explanation of the invention, and are not intended to limit the scope of exemplary embodiments of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0023] Terminology Explanation VQA (Visual Question Answering) is a task that combines computer vision and natural language processing. It requires the model to generate accurate answers based on the input image and natural language question. The core is to achieve the understanding and fusion of cross-modal information.

[0024] OOD (Out-Of-Distribution): This refers to a scenario where the distribution of the test data is inconsistent with the distribution of the training data.

[0025] IID (Independent and Identically Distributed): This means that the samples in the dataset are independent of each other and follow the same probability distribution.

[0026] Faster R-CNN (Faster Region-Based Convolutional Neural Network): A fast region-based convolutional neural network and one of the classic object detection algorithms. It efficiently generates candidate regions through a region proposal network, and then combines a convolutional neural network to classify regions and regress bounding boxes. It is widely used to extract region-level features from images.

[0027] GRU (Gated Recurrent Unit) encoding: GRU is a simplified variant of recurrent neural networks that controls the flow and memory of information by updating and resetting gates. It is often used to process sequential data such as natural language. GRU encoding refers to the process of using GRU to semantically represent sequential information such as text.

[0028] NDE (Non-Direct-Effect): In causal reasoning, it refers to the influence between variables that is not transmitted through a direct path.

[0029] EA (Explain-Away Fusion Function): An explanation-away fusion function, a multimodal fusion mechanism. Its core idea is to highlight the contribution of key evidence by suppressing redundant or spurious related information. It is often used to strengthen the real correlation between modalities and reduce noise interference.

[0030] The overall idea of ​​the method proposed in this invention is as follows: Existing bias-free visual question answering methods / models suffer from two major problems: first, the training process is dominated by easy samples, resulting in severe linguistic prior bias; second, they lack fine-grained causal dependencies between questions and image regions, leading to spurious correlations interfering with prediction results. For example... Figure 1 As shown, the above problems manifest in practice as two typical failures: (a) sample dominance, in Figure 1 In the text "How many people are there in the picture?" scenario shown in (a), the distribution of answers trained / tested by the Question Answering Model (VQA Model) is consistent. Even if the model falls into the wrong region, it can still answer correctly based on linguistic priors, i.e., answer "two". These types of samples are dominant in training, causing the model training to be guided by easy samples that can be directly answered by language; (b) there is insufficient region dependence. Figure 1 In the text "What color is the plane?" scenario shown in (b), the color judgment relies on local evidence such as the nose / tail, but the model is more likely to stay in irrelevant or overly large areas and give the wrong answer "White". Only by focusing on the key areas related to the question can the correct answer "Orange" be obtained. This is consistent with the conclusion of its robustness evaluation: under real perturbations such as noise, blur, and compression, failure to establish regional dependencies will significantly amplify the performance decline.

[0031] To address the aforementioned issues, this embodiment proposes a visual question answering method and system based on sample difficulty perception and multi-path fusion. A difficulty-sensitive de-biasing framework is designed, introducing a learned angle space: first, the confidence levels of the question and view branches towards the true answer characterize the sample difficulty; then, the decision boundary is adaptively adjusted based on the sample difficulty. This difficulty level is injected into the angle loss as an instance-level adaptive boundary to dynamically adjust the decision boundary, imposing stronger angle discrimination constraints on easy samples and reducing angle intervals on difficult samples. This adaptively adjusts the strictness of the discrimination boundary, thereby suppressing text dominance in easy samples and strengthening the contribution of visual evidence in difficult samples. Simultaneously, a causal graph-guided multi-path consistency fusion mechanism is introduced: the question path, visual path, and fusion path are combined to form a path-level effect estimate. Path-level fusion is performed in the probability space using an Explain-Away logarithmic-product approach, thereby suppressing spurious and irrelevant associations and highlighting visual evidence truly relevant to the answer. Through these strategies, language bias is effectively mitigated, resulting in more accurate visual question answering.

[0032] Example 1 This embodiment proposes a visual question answering method based on sample difficulty perception and multi-path fusion, which specifically includes the following steps: Step S1: Obtain visual question answering samples; Step S2: Input the image and question from the visual question answering sample into the visual question answering network, integrate the prediction results of the backbone head, angle boundary head and residual head in the network, and output the final answer.

[0033] like Figure 2 As shown, the architecture of the visual question answering network model built in this embodiment is based on the standard VQA architecture, mainly including a backbone network, a sample difficulty adaptive module, and a causal path consistency module. The following content will provide a more detailed introduction to the visual question answering network architecture.

[0034] Specifically, given a small batch As a sample set, "Always" represents the sample, and "B" represents the number of samples in the mini-batch, i.e., the batch size. and Representing images and questions, It is an answer set soft target vector on, Indicates sample The answer belongs to the first Confidence of similar answers Let be the number of answer categories. The input visual question-answering samples include images and questions. They are first processed through a backbone network, where image encoders and question encoders encode the images and questions respectively. After multimodal fusion, these are sent to the backbone head. and For image and problem encoders, set For a multimodal fusion module, the calculation is as follows: ; The fused feature representation Feed into the classifier head (i.e., the backbone), then the prediction result will be output: ; ; In the above formula, , For learnable parameters, This represents the weight parameters of the classifier head (i.e., the fully connected layer). It can be split into weight vectors according to categories. In the learning of angle discrimination, regarded as the first The directional prototype of a class in the feature space.

[0035] As one implementation method, a soft objective and a standard cross-entropy objective are used to train the backbone network, with the loss function being: ; In the above formula, Index for answer categories; Used to indicate the number in the current mini-batch One sample, .

[0036] In addition to the backbone network, this implementation also includes two complementary components: a SampleDifficulty Adaptation (SDA) module, which injects instance-level learnable angular intervals into the classifier to adjust the decision boundary; and a Causal Path Consistency (CPC) module, which enforces path-level consistency between predictions for question-only, image-only, and fusion problem-image pairs. Both components are trained end-to-end with the backbone network.

[0037] Specifically, such as Figure 2 As shown, since early training is mainly dominated by simple cases, their gradients suppress the gradients of difficult cases. The Sample Difficulty Adaptive Module (SDA) addresses this issue by deriving the difficulty of each instance from the unimodal prediction evidence and applying an instance-aware angular interval using this difficulty. This interval gives a smaller interval to difficult cases (i.e., loosens the boundaries for difficult samples, making them more lenient) and a larger interval to simple cases (making them more lenient), thereby decoupling their influence.

[0038] In this embodiment, an image encoder and a question encoder are used to encode the image and question in the sample, respectively, given a question. and image encoding Then, a single-peak prediction score, i.e., a single-mode prediction score, is generated using a lightweight header, which is expressed as: ; set up Indicates the first The true class index of each sample is defined, and the true confidence score of the unimodal is extracted as follows: ; To avoid fusion bias, the original difficulty coefficient of the sample is calculated based on the true confidence score of the single-modal prediction, as follows: ; In the above formula, V branch represents the monovision prediction branch, and V branch exists means that if the monovision prediction branch exists, the first calculation formula is executed, otherwise the second calculation formula is executed.

[0039] Using the above formula, when the visual unimodal branch is enabled (e.g., through a switch parameter), the sample difficulty coefficient is defined as follows: When the visual single-modal branch is not enabled, the difficulty level is: Therefore, more difficult samples will produce larger results. .

[0040] To ensure numerical stability and prevent excessive perturbations in the early stages of training, The samples are cropped and broadcast to each sample with its corresponding difficulty coefficient, resulting in the final sample difficulty coefficient: ; When calculating the difficulty metric, the single-modal prediction score and coefficient are considered. Stopping gradient propagation isolates the difficulty estimator from the backpropagation process, preventing boundary modules from manipulating the difficulty signal.

[0041] Furthermore, SDA will define class-level boundaries. Compared to instance-level boundaries generated by a learnable boundary generator Combined, and through Injecting difficulty information involves fusing the sample difficulty coefficients with class-level and instance-level boundaries, and outputting the prediction results through the angle boundary header.

[0042] In this embodiment, class-level boundaries The extraction process is as follows: Each question type With category Count the number of times it appears in the training set. Then, calculate its occurrence probability, and based on this, define the rarity normalization boundary. for: ; ; In the above formula, Indicates the problem type index. Indicates the category index of candidate answers; This indicates that the problem type is In the training sample subset, the first The number of occurrences of each answer category. For normalization, The category summation index is used to iterate through the candidate answer set. All categories in In subsequent instance-level calculations, This is the sample index, used to identify the first sample in the mini-batch. One sample, ; It is a smoothing constant used to avoid zero denominators and improve computational stability.

[0043] To reduce the sensitivity of class-level prior boundaries to frequency statistical noise in the training set and to improve the robustness and generalization ability of boundary estimation, this embodiment... Based on this, randomized smoothing by introducing a Gaussian perturbation can be expressed as: ; Next, clipping and normalization are performed to ensure valid values, represented as: ; ; Broadcasting this to the sample dimension yields the class-level boundary vector for each sample, as follows: .

[0044] Furthermore, the process of extracting instance-level boundaries is as follows: First, the visual question-answering backbone network (using the UpDn structure) outputs the predicted answer logits (i.e., the original value) for each sample, denoted as . ,in This represents the number of samples in the mini-batch. This represents the number of candidate answer categories. To retain only valid candidate categories, this embodiment introduces a deterministic masking operator before normalization. Masking of logits: For the first Each sample, based on its problem type Select the corresponding class-level prior boundary vector and according to the threshold To determine category validity, when At that time, it was believed that the first If a class is valid, it is considered invalid; otherwise, it is considered invalid. This yields the mask `logits` after masking invalid classes. Obtained by masking Defined as: ; ; ; In the above formula, This indicates a masking operation used to remove invalid candidate categories before normalization. For a pre-defined minimum constant (equivalent to) (numerical implementation). Therefore, in the case of After softmax normalization, the probability mass of the invalid class in the probability space is approximately zero, i.e. , This avoids invalid categories interfering with instance-level boundary estimation.

[0045] Then, after considering the invalid categories, the result was... This is merely the basic boundary learned by the model. Therefore, this embodiment also adds a sample difficulty coefficient and applies temperature-normalized softmax to obtain the instance-level residual distribution, expressed as: ; in, An intensity coefficient is injected into the difficulty level to adjust the degree of influence of sample difficulty on the instance-level boundary distribution; This is a temperature coefficient used to control the smoothness of the distribution during softmax normalization.

[0046] Finally, the class-level and instance-level boundaries are merged in the truth index. Above, while maintaining stability at the non-target location class level, it is as follows: ; ; in, The training process is divided into stages, initially relying more on stable class priors, and then gradually introducing sample adaptation capabilities.

[0047] By introducing a sample difficulty coefficient to adjust instance-level boundary assignment in the above manner, difficult samples receive higher ground-value class weights in instance-level boundary assignment, which is then combined with subsequent steps 1. The mapping effect of m further transforms the weight into a smaller angular interval, thereby relaxing the discrimination boundary on difficult samples and applying a smaller angular interval, while applying a larger angular interval on simple samples, thus achieving the purpose of decoupling.

[0048] Furthermore, classification is performed on the unit hypersphere: fusion features of the samples. With the Weight vectors corresponding to each answer category Perform separately Normalization, expressed as: , ; Therefore, its dot product is equivalent to the cosine of the included angle: ; Therefore, the basic angle prediction score is defined as: ; in, , where is the scale factor, is used to linearly scale the angle prediction scores to enhance numerical stability during softmax training. The above formula yields the base prediction scores without introducing angle boundary constraints, facilitating subsequent causal path consistency modeling.

[0049] By balancing class prior and sample adaptability through the aforementioned piecewise decreasing coefficients, and applying boundary constraints only at the true class index, the predicted score of the main branch can be obtained through the angular boundary head, which can be expressed as: ; .

[0050] Preferably, these predicted scores supervise the main classifier using cross-entropy loss.

[0051] As one implementation method, the design of the loss function of the aforementioned Sample Difficulty Adaptive Module (SDA) during training includes: To avoid ambiguity, the predicted scores from different heads are distinguished: the main head is... , angle boundary head is The aforementioned backbone header and its features are reused, including: baseline prediction score. Soft targets Fusion characteristics and question type tags Given the prediction scores of the injection boundary, the main loss, i.e., the angle loss calculated based on the dynamic boundary, is defined as follows: ; To emphasize long-tail answers while keeping the main head unchanged, a frequency-weighted loss is designed, and... The frequency weight for each sample (with rare answers receiving a larger weight) is: .

[0052] Additionally, regularization is performed using fusion features, assuming... The temperature coefficient is Sample similarity is defined as The positive sample mask is defined as The global negative sample partition within the batch range is Minimizing the following objective function, i.e., minimizing the comparison loss between samples, can be expressed as: ; .

[0053] like Figure 2 As shown, to achieve causal consistency constraints among multiple paths, the causal path consistency module designs three path configurations: full path (QKV, full-modal reasoning), disabled multimodal knowledge path, and... (i.e., the QV path, also known as the knowledge inhibition path, the vision-only path) and the problem-based indirect-effect path (Non-Direct-Effect, NDE-Q), let... Let EA represent the Explain-Away Fusion Function (EA). Then, for a sample... The fusion feature outputs for the three corresponding paths are calculated as follows: .

[0054] Correspondingly, the logits of the complete path (QKV), the debiased path (QV), and the indirect effect path (NDE-Q) are denoted as follows: This path partitioning helps to explicitly distinguish between problems – direct dependencies between images and indirect associations generated through multimodal knowledge channels.

[0055] Furthermore, for any path The prediction probability vector for this path is defined as follows: Their single-mode distributions are respectively and In this embodiment, the prediction probabilities are as follows: (The three representative fusion paths used are not explicitly stated in the original text.) .

[0056] As one implementation method, the loss function of the aforementioned causal path consistency module (CPC) during training includes path-level cross-entropy loss and KL distillation loss, and training in this way can achieve multi-path prediction consistency constraints.

[0057] The path-level cross-entropy is: using the true class index. , for any path The loss of a single cross-entropy can be defined as: ; Apply this loss to the complete path and the single-modal head: , And only if a visual monomodal head exists: The accuracy of the prediction is monitored by applying cross-entropy loss to each path.

[0058] The KL distillation loss is as follows: In the path distillation stage, to achieve path-level knowledge transfer, the prediction results of the complete path (QKV) are used as the teacher branch, and the output of the indirect effect path (NDE-Q) is used as the student branch. Let their corresponding prediction scores be respectively... and Then we have: ; in, This indicates a gradient blocking operation, used to prevent the teacher branch parameters from being updated in the distillation loss.

[0059] Based on knowledge transfer through gradient blocking KL distillation loss, the path-level KL divergence loss is defined as: ; The loss of the causal path consistency module is: .

[0060] By designing the aforementioned unidirectional KL distillation constraint, the NDE-Q branch maintains consistency with the full-modal path output during the learning process, while avoiding gradient backpropagation to the teacher branch, thereby stabilizing the multimodal fusion features and enhancing the robustness of weak modes.

[0061] Based on the above-mentioned modules, the training objective of visual question answering consists of three parts: the standard classification loss based on fusion prediction, the angle boundary loss for difficulty perception, and the causal path consistency loss, which can be expressed as: .

[0062] Finally, the prediction results of the backbone head, angle boundary head, and residual head in the integrated network are used to output the final answer. Specifically, three calibrated posterior distributions are integrated: (1) the backbone head based on fusion features, (2) the angle boundary head, and (3) the residual head (the single plug is defined as the difference between the complete path and the knowledge-suppressed path, i.e., ...). ),set up express Normalization The temperature coefficient is calculated using temperature normalization: ; Subsequently, the confidence levels of each prediction result are calculated and weighted non-negatively. To merge, and satisfy (Equal weights are used by default) (Settings), the calculation yields: ; The final predicted answer is .

[0063] Preferably, the above-mentioned solution proposed in this embodiment is verified to further verify its superiority.

[0064] Specifically, the dataset and evaluation metrics are first set up. Evaluation is performed on the same VQA v2 corpus, using two commonly used setups: VQA-CP v2 and VQA v2 corpus. VQA-CP v2 is a re-partitioned version of VQA v2, intentionally altering the prior distribution of answers by question type between the training and test sets, thus often used to evaluate the robustness of the model in language deviation (OOD) scenarios. VQA v2 maintains the consistent distribution (IID) between the training and validation sets and is used for general performance evaluation. Both setups use standard VQA accuracy as the evaluation metric, and, where necessary, classification results for different question types (Yes / No, Number, Other) are also provided.

[0065] To ensure fair comparison, a uniform preprocessing workflow was adopted across all datasets: image features were extracted at the region level by a pre-trained Faster R-CNN model, and the problem was represented sequentially through word segmentation, word embedding, and GRU encoding. This standardized configuration ensured the comparability of model training and evaluation across different experimental settings.

[0066] The final experimental results are shown in Table 1 below.

[0067] Table 1. Comparison of test results on VQA-CPv2 and validation results on VQAv2

[0068] In Table 1 above, "Methods" lists the existing visual question answering models / methods used and the abbreviation of the model (SACC) proposed in this embodiment. UpDn (Bottom-Up and Top-Down Attention) is a classic visual question answering model that combines bottom-up region feature extraction with top-down attention mechanisms; AdvReg is a visual question answering method that introduces adversarial constraints during training to reduce the model's dependence on statistical bias in the question text; LMH is a bias mitigation method that assists visual question answering model training by introducing prompts generated by a language model; CSS is a visual question answering method that enhances model robustness by constructing counterfactual samples to perturb visual or linguistic input; GGE-DQ is a debiased learning method that guides the model to focus on samples that are difficult to explain by bias by integrating multiple gradient update directions; RMLVQA is a loss design that introduces boundary constraints. Visual question answering methods that improve model stability under linguistic bias: GenB is a method that uses generative models to characterize the biased relationship between questions and answers, and uses this to assist in training visual question answering models; CVIV is a visual question answering method that models the relationship between visual features and answers based on the idea of ​​instrumental variables; PWVQA is a visual question answering method that models visual and linguistic information from the perspective of "possible worlds" to improve the model's generalization ability; LEGO is a visual question answering method that uses linguistic information to guide the construction of biased samples and combines contrastive learning for training; PDGH is a question answering method that improves the robustness of visual question answering models by introducing prompt information and imposing geometric consistency constraints on multimodal features.

[0069] Additionally, Reference lists the abbreviation of the conference or journal to which the article was published, plus the year; Overall indicates the overall accuracy, i.e., the overall VQA accuracy for all questions; Y / N indicates the accuracy for Yes / No questions; Num indicates the accuracy for Number questions; and Others indicates the accuracy for Other question types, i.e., all question types other than Yes / No and Number questions.

[0070] As shown in Table 1 above, on VQA-CP v2, the overall accuracy of our method reaches 62.14, exceeding the current state-of-the-art method's 61.68. The most significant improvements are seen in the Yes / No (90.54) and Other (50.66) question classes: by reducing the weight of language-dominated simple samples through difficulty-aware angle boundaries and combining cross-path consistency constraints, the model effectively suppresses the shortcut priors common in Yes / No questions, while enhancing the visual-semantic alignment capabilities required in the "Other" class. Under the IID setting (VQA v2 validation set), the model's overall accuracy is 60.56, and the results across all types are balanced, indicating that improving OOD robustness did not come at the expense of performance in standard scenarios.

[0071] In summary, the difficulty-aware angle boundary and causal-guided multi-path fusion mechanism proposed in this embodiment significantly improves the model's performance in OOD scenarios while maintaining stable performance in IID scenarios. It effectively alleviates the problem of easy sample dominance and strengthens fine-grained issues—region dependencies. These results fully validate the effectiveness of this method in addressing language bias and improving distribution transfer generalization ability.

[0072] Ablation experiments were also conducted, and the results are shown in Table 2 below. To evaluate the contribution of each component module, starting with the baseline model (60.41) without SDA and CPC enabled, enabling SDA improved the overall accuracy to 61.49 (+1.08), indicating that the difficulty-aware angle boundary effectively alleviated the easy-sample dominance problem by dynamically adjusting the decision boundary during training. Enabling CPC alone achieved 61.13 (+0.72), indicating that the causal path consistency constraint can reduce spurious correlations between the problem and the image during the inference stage. When SDA and CPC were enabled simultaneously, the model performance further improved to 62.14 (+1.73 from the baseline), almost equal to the sum of the gains of the two modules alone, indicating that their effects are complementary rather than redundant. In summary, the experimental results verify the necessity of the SDA and CPC modules, and their combined use can achieve the best robustness performance in distributed migration scenarios.

[0073] Table 2 Ablation experiments on the VQA-CP v2 dataset

[0074] In summary, this embodiment addresses two core issues in Visual Question Answering (VQA): training is dominated by easy samples driven by language priors, and there is a lack of mechanisms to establish stable causal dependencies at the question-candidate region level. It proposes a difficulty-aware debiasing framework, introducing a difficulty-adaptive angle boundary mechanism at the sample level to reduce the influence of textual bias in easy samples and strengthen the visual cue representation of difficult samples. A causal graph-guided multi-path consistency module is designed to estimate and fuse path-level effects among question-only paths, visual-only paths, and question-visual fusion paths, thereby effectively suppressing spurious correlations and highlighting visual evidence truly relevant to the answer.

[0075] Example 2 This embodiment provides a visual question answering system based on sample difficulty perception and multi-path fusion, specifically including: The data acquisition module is used to acquire visual question-answering samples; The visual question answering output module is used to input the images and questions from the visual question answering samples into the visual question answering network, integrate the prediction results of the backbone head, angle boundary head and residual head in the network, and output the final answer; the visual question answering network includes a backbone network, a sample difficulty adaptive module and a causal path consistency module. In the backbone network, an image encoder and a question encoder are used to encode the image and the question respectively. After multimodal fusion, the results are sent to the backbone head to generate basic prediction results. In the sample difficulty adaptive module, the sample difficulty coefficient is calculated based on the true confidence of single-modal prediction, class-level boundary and instance-level boundary are fused, and the prediction result is output through the angle boundary header; In the causal path consistency module, the fusion features under the complete path, the knowledge suppression path, and the indirect effect path of the problem only are calculated and predicted respectively. The prediction difference between the complete fusion path and the knowledge suppression path is the prediction result output by the residual head.

[0076] Example 3 This embodiment provides an electronic device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the method provided in this embodiment.

[0077] Example 4 This embodiment also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, will cause the processor to execute the method described above in this embodiment.

[0078] Example 5 This embodiment provides a computer program product including executable instructions, which are computer instructions; the executable instructions are stored in a computer-readable storage medium. When the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method described in this embodiment.

[0079] The steps and methods involved in Embodiments 2 to 5 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0080] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0081] The above description is only a preferred embodiment of the present invention. Although the specific implementation of the present invention has been described in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. A visual question answering method based on sample difficulty perception and multi-path fusion, characterized in that, include: Obtain visual question answering samples; The image and question from the visual question answering sample are input into the visual question answering network. The prediction results of the backbone head, angle boundary head and residual head in the network are integrated to output the final answer. The visual question answering network includes a backbone network, a sample difficulty adaptive module and a causal path consistency module. In the backbone network, an image encoder and a question encoder are used to encode the image and the question respectively. After multimodal fusion, the results are sent to the backbone head to generate basic prediction results. In the sample difficulty adaptive module, the sample difficulty coefficient is calculated based on the true confidence of single-modal prediction, class-level boundary and instance-level boundary are fused, and the prediction result is output through the angle boundary header; In the causal path consistency module, the fusion features under the complete path, the knowledge suppression path, and the indirect effect path of the problem only are calculated and predicted respectively. The prediction difference between the complete fusion path and the knowledge suppression path is the prediction result output by the residual head.

2. The visual question answering method based on sample difficulty perception and multi-path fusion as described in claim 1, characterized in that, In the sample difficulty adaptive module, the sample difficulty coefficient is calculated based on the true confidence level of the single-modal prediction, as follows: Based on image encoder and question encoder, the images and questions in the samples are encoded respectively. A unimodal prediction score is generated based on the image encoding and question encoding, and the unimodal confidence of the true class is extracted. Based on the true confidence level of the single-modal prediction, the original difficulty coefficient of the sample is calculated, and the final sample difficulty coefficient is obtained after pruning and normalization.

3. The visual question answering method based on sample difficulty perception and multi-path fusion as described in claim 1, characterized in that, By fusing class-level and instance-level boundaries, the prediction result is output through the angle boundary header as follows: The class boundaries are obtained by statistically analyzing the occurrence probabilities of each problem type and category in the training set, followed by rarity normalization, cropping, and normalization. The answer prediction value of each sample is masked using a deterministic masking operator to obtain the basic boundary after masking the invalid class. Then, the sample difficulty coefficient is combined and the instance-level boundary is generated by temperature normalization. It integrates class-level and instance-level boundaries, balances class priors and sample adaptation capabilities through piecewise decreasing coefficients, applies boundary constraints only at the true class index, and outputs predicted scores through angular boundary headers.

4. The visual question answering method based on sample difficulty perception and multi-path fusion as described in claim 1, characterized in that, The loss function of the visual question answering network is the sum of the standard classification loss of fusion prediction, the angle boundary loss of difficulty perception, and the causal path consistency loss. Among them, the standard classification loss of the fusion prediction is the standard cross-entropy loss of the backbone network; The angle boundary loss for difficulty perception is the sum of the angle loss calculated based on the dynamic boundary of the sample difficulty adaptive module, combined with the frequency-weighted loss and the inter-sample comparison loss; The causal path consistency loss is the sum of the path-level cross-entropy loss and the KL distillation loss of the causal path consistency module.

5. The visual question answering method based on sample difficulty perception and multi-path fusion as described in claim 1, characterized in that, During the training process of the causal path consistency module: Calculate the fusion features under the complete path, the knowledge-suppressed path, and the indirect effect path of only the problem; Apply cross-entropy loss to each path to monitor the accuracy of the prediction; The complete fusion path is used as the teacher branch, and the indirect effect path is used as the student branch. Knowledge transfer is carried out through gradient blocking KL distillation loss. Training is performed using path-level cross-entropy loss and KL distillation loss to achieve consistency constraints in multi-path prediction.

6. The visual question answering method based on sample difficulty perception and multi-path fusion as described in claim 1, characterized in that, The prediction results of the backbone, angular boundary head, and residual head in the ensemble network are used to output the final answer, including: Based on the prediction results of the backbone, angle boundary, and residual heads in the network, the confidence level of each prediction result is calculated after temperature normalization. Then, the final answer is output through weighted fusion calculation.

7. A visual question-answering system based on sample difficulty perception and multi-path fusion, characterized in that, include: The data acquisition module is used to acquire visual question-answering samples; The visual question answering output module is used to input the images and questions from the visual question answering samples into the visual question answering network, integrate the prediction results of the backbone head, angle boundary head and residual head in the network, and output the final answer; the visual question answering network includes a backbone network, a sample difficulty adaptive module and a causal path consistency module. In the backbone network, an image encoder and a question encoder are used to encode the image and the question respectively. After multimodal fusion, the results are sent to the backbone head to generate basic prediction results. In the sample difficulty adaptive module, the sample difficulty coefficient is calculated based on the true confidence of single-modal prediction, class-level boundary and instance-level boundary are fused, and the prediction result is output through the angle boundary header; In the causal path consistency module, the fusion features under the complete path, the knowledge suppression path, and the indirect effect path of the problem only are calculated and predicted respectively. The prediction difference between the complete fusion path and the knowledge suppression path is the prediction result output by the residual head.

8. An electronic device, characterized in that, include: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the visual question answering method based on sample difficulty perception and multi-path fusion as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the executable instructions to implement the visual question answering method based on sample difficulty perception and multi-path fusion as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the visual question answering method based on sample difficulty perception and multi-path fusion as described in any one of claims 1-6.

Images