A medical visual question answering method and system using visual evidence guided re-weighting

Abstract

The application discloses a medical visual question answering method and system based on visual evidence guidance and reweighting. A medical image, a clinical question and an answer label are acquired, and visual features and question features are extracted. In a first training stage, two types of features are fused in a cross-modal manner, a first clinical evidence region is determined according to global semantic features, local evidence features are aggregated, global and local answer prediction results are jointly constrained, and evidence positioning parameters are obtained. In a second training stage, the parameters are used to determine a second clinical evidence region, a spatial evidence mask is generated and visual features are modulated in an affine manner, the reweighted visual features and the question features are fused in a cross-modal manner, and question answering reasoning parameters are optimized. Based on the evidence and the reasoning parameters, visual evidence guidance and reweighting reasoning are performed on a medical image to be questioned and a clinical question, and a medical visual question answering result is output. The application enhances the attention to clinical evidence, suppresses language bias and shortcut learning, and improves the robustness and reliability of question answering under answer distribution changes.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a medical visual question-answering method and system guided by visual evidence and reweighted. Background Technology

[0002] Medical visual question answering (Med-VQA) is a core task in clinical computer-aided diagnosis, requiring models to provide accurate answers to clinical questions by integrating multimodal information. However, as... Figure 3 As shown, a key bottleneck in practical deployment lies in language prior bias: the model over-relies on the shallow statistical association between the question and the answer, rather than the actual visual content. In clinical settings, this bias stems from the inherent class imbalance in medical imaging data. For example, in routine screening, negative (normal) cases far outnumber positive (pathological) cases, leading to model overfitting to high-frequency label patterns. Therefore, for questions like "Is there an abnormality?", regardless of the image content, the model's answer is often dominated by the "no" label.

[0003] When faced with biased datasets, the performance of existing Med-VQA models such as SAN, BAN, M3AE, M2I2, MUMC, and LaPA typically degrades significantly. Figure 3 As shown in b, these models fall into a linguistic shortcut: they predict statistically dominant answers from question keywords, even when pathological evidence is absent. This kind of evidence-free reasoning is dangerous in clinical practice.

[0004] Existing general VQA debiasing methods primarily employ counterfactual reasoning or data distribution alignment. However, the significant domain gap between natural and medical images hinders direct transfer, as clinical diagnosis fundamentally relies on subtle pathological features. Medical debiasing methods such as CEDO, MEVE+SAN, MEVE+BAN, Med-BiasX, and DeCoCT largely depend on indirect interventions to improve global cross-modal alignment. While effective to some extent, these methods lack explicit spatial guidance on clinically relevant lesion regions. Without such spatial priors, the model remains unable to anchor key evidence and is susceptible to linguistic shortcuts. Summary of the Invention

[0005] The purpose of this invention is to address the problem that existing medical visual question answering models tend to rely on statistical correlations between clinical questions and answers and pay insufficient attention to relevant local clinical evidence, resulting in poor robustness of question answering reasoning when the distribution of answers changes. This invention provides a medical visual question answering method and system that uses visual evidence to guide reweighting.

[0006] To achieve the above objectives, the present invention provides a medical visual question-answering method guided by visual evidence reweighting, comprising: Acquire training samples including medical images, clinical questions, and answer labels, and extract corresponding visual features and question features; In the first training phase, visual features and question features are fused across modalities. The first clinical evidence region is determined based on the fused global semantic features. The visual features corresponding to the first clinical evidence region are aggregated to obtain local evidence features. Based on the answer label joint constraint, the global answer prediction results based on the global semantic features and the local answer prediction results based on the local evidence features are obtained to obtain evidence localization parameters. In the second training phase, the second clinical evidence region is determined using the evidence localization parameters, and a spatial evidence mask is generated based on the spatial position of the second clinical evidence region and the visual features. The visual features are then subjected to affine modulation based on the spatial evidence mask to obtain reweighted visual features. The reweighted visual features and question features are fused across modalities, and the question-answering inference parameters are optimized based on the answer labels. Using the aforementioned evidence localization parameters and question-answering inference parameters, visual evidence-guided reweighted question-answering inference is performed on the medical images to be answered and the corresponding clinical questions, and medical visual question-answering results are output.

[0007] As a further improvement to the above technical solution, the training samples include a first training sample set and a second training sample set; the first training sample set consists of training samples from the original medical visual question-answering dataset, used to train the evidence localization parameters; the second training sample set consists of training samples from a dataset obtained by reorganizing medical visual question-answering samples according to clinical question types, and the training samples and test samples in the reorganized dataset have different answer distributions under at least one of the same clinical question types, used to optimize the question-answering inference parameters; after the first training phase is completed, the evidence localization parameters are fixed, and only the question-answering inference parameters are updated in the second training phase, and the second clinical evidence region determined by the fixed evidence localization parameters is used as the spatial prior for generating the spatial evidence mask.

[0008] As a further improvement to the above technical solution, the first training sample set does not include clinical evidence region annotations; the extraction of visual features and problem features includes: dividing the medical image into multiple non-overlapping visual blocks and encoding them to obtain a visual block feature sequence, which is used as the visual features; encoding the clinical problem to obtain a problem feature sequence containing a global identifier, which is used as the problem features; the acquisition of global semantic features includes: performing multi-level cross-modal fusion on the visual block feature sequence and the problem feature sequence to obtain a joint feature representation, and determining the feature in the joint feature representation corresponding to the global identifier as the global semantic feature.

[0009] As a further improvement to the above technical solution, the determination of the first clinical evidence region includes: inputting the global semantic features into a multilayer perceptron for mapping, and normalizing the mapping result using the Sigmoid function to obtain continuous bounding box parameters; the continuous bounding box parameters include the center x-coordinate, center y-coordinate, width, and height of the first clinical evidence region; and determining the first clinical evidence region based on the continuous bounding box parameters.

[0010] As a further improvement to the above technical solution, the acquisition of the local evidence features includes: constructing a two-dimensional Gaussian kernel using Soft ROI pooling; using the center x-coordinate and center y-coordinate of the first clinical evidence region as the center of the two-dimensional Gaussian kernel, and using half of the width and half of the height as the horizontal and vertical scales of the two-dimensional Gaussian kernel, respectively; calculating the activation weight of each visual block feature using the two-dimensional Gaussian kernel based on the x-coordinate and y-coordinate of each visual block; and performing a weighted summation of the visual block features according to the activation weights to obtain the local evidence features.

[0011] As a further improvement to the above technical solution, the joint constraints of the first training phase include: classifying the answers for the global semantic features and local evidence features respectively to obtain the global answer prediction result and the local answer prediction result; determining the global binary cross-entropy loss and the local binary cross-entropy loss corresponding to the global answer prediction result and the local answer prediction result respectively based on the answer label; calculating the consistency loss used to constrain the semantic alignment of the two based on the global semantic features and the local evidence features, and determining the supervised contrast loss based on the answer label and the global semantic features and the local evidence features; adding the global binary cross-entropy loss, the local binary cross-entropy loss, the consistency loss and the supervised contrast loss to obtain the first-stage training objective; and performing backpropagation based on the first-stage training objective to optimize the parameters used to determine the first clinical evidence region to obtain the evidence localization parameters.

[0012] As a further improvement to the above technical solution, the visual features include visual block features that correspond one-to-one with multiple visual blocks; the second clinical evidence region is characterized by a central abscissa, a central ordinate, a width, and a height. The generation of the spatial evidence mask includes: obtaining the abscissa and ordinate of each visual block; for each visual block, when the absolute value of the difference between its abscissa and the central abscissa is not greater than half of the width, and the absolute value of the difference between its ordinate and the central ordinate is not greater than half of the height, the mask value corresponding to the visual block is set to 1, otherwise it is set to 0; a binary spatial evidence mask is generated based on each mask value, wherein the visual block with a mask value of 1 belongs to the second clinical evidence region, and the visual block with a mask value of 0 belongs to the background region.

[0013] As a further improvement to the above technical solution, the affine modulation includes: obtaining a spatial scaling factor and a shift bias, both of which are preset constants; for each visual block feature, multiplying the spatial scaling factor by the mask value corresponding to the visual block and adding it to the value 1 to obtain a feature scaling coefficient; multiplying the shift bias by the mask value to obtain a feature shift amount; multiplying the visual block feature by the feature scaling coefficient element by element and then superimposing the feature shift amount to obtain a reweighted visual block feature; and constructing the reweighted visual feature from the reweighted visual block features.

[0014] As a further improvement to the above technical solution, the affine modulation includes: using the center x-coordinate and center y-coordinate of the second clinical evidence region as the lateral mean and longitudinal mean of a two-dimensional Gaussian distribution, respectively, and determining the lateral and longitudinal scales based on the width and height to construct a two-dimensional Gaussian mask; using the Gaussian weight of each visual block in the two-dimensional Gaussian mask as the corresponding spatial scaling factor; for each visual block feature, adding the product of the value 1 and the spatial scaling factor and the mask value of the visual block in the binary spatial evidence mask to obtain a feature scaling coefficient; multiplying a preset shift bias by the mask value to obtain a feature translation amount; multiplying the visual block feature element-wise by the feature scaling coefficient and then superimposing the feature translation amount to obtain a reweighted visual block feature; and constructing the reweighted visual feature from each reweighted visual block feature.

[0015] The present invention also provides a visual evidence-guided reweighted medical visual question-answering system, comprising: The sample processing module is used to acquire training samples including medical images, clinical questions, and answer labels, and extract corresponding visual features and question features; The evidence localization training module is used in the first training phase to fuse the visual features and question features across modalities, determine the first clinical evidence region based on the fused global semantic features, aggregate the visual features corresponding to the first clinical evidence region to obtain local evidence features, and obtain evidence localization parameters based on the global answer prediction result based on the global semantic features and the local answer prediction result based on the local evidence features, according to the answer label joint constraint. The question-answering reasoning training module is used in the second training phase to determine the second clinical evidence region using the evidence localization parameters, generate a spatial evidence mask based on the spatial position of the second clinical evidence region and visual features, perform affine modulation on the visual features based on the spatial evidence mask to obtain reweighted visual features, fuse the reweighted visual features and question features across modalities, and optimize the question-answering reasoning parameters based on the answer labels. The question-and-answer output module is used to perform visual evidence-guided reweighted question-and-answer reasoning on the medical images to be answered and the corresponding clinical questions using the evidence localization parameters and question-and-answer reasoning parameters, and output medical visual question-and-answer results.

[0016] The beneficial effects of the medical visual question-answering method and system provided by this invention are as follows: 1. This invention, by identifying clinical evidence regions and aggregating corresponding visual features, enables the model to focus more on local clinical evidence related to clinical questions, reduces reliance on superficial statistical associations between question types and answers, and improves the reliability of medical visual reasoning.

[0017] 2. This invention employs a phased training approach for evidence localization parameters and question-answering inference parameters. It generates a spatial evidence mask based on the clinical evidence region and performs affine modulation on the visual features accordingly. This allows question-answering inference to be based on the visual evidence obtained from localization, thereby suppressing shortcut learning caused by language bias and improving the robustness of the model under conditions of changing answer distribution.

[0018] 3. Validated on VQA-RAD-CP and SLAKE-CP datasets, this invention can enhance the visual features corresponding to clinical evidence regions, improving question-answering performance in distributed offset scenarios while maintaining generalization ability in in-distribution scenarios. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the operation of the visual evidence-guided reweighted medical visual question-answering method in this embodiment of the invention. Figure 2 This is a structural diagram of the visual evidence-guided reweighted medical visual question-answering system module in an embodiment of the present invention; Figure 3 This is a schematic diagram comparing the reasoning mechanisms of existing medical visual question answering models and visual evidence-guided reweighting frameworks in embodiments of the present invention. Figure 4 This is a schematic diagram of the two-stage overall architecture of the visual evidence-guided reweighting framework in an embodiment of the present invention; Figure 5 This is an ablation result diagram showing the spatial scaling factor and shift bias in an embodiment of the present invention; Figure 6 This is a schematic diagram of the visual evidence localization workflow in an embodiment of the present invention; Figure 7 This is a schematic diagram comparing the attention heatmaps of the visual evidence-guided reweighting framework and the contrast model in an embodiment of the present invention. Figure 8 This is a schematic diagram of an off-distribution test sample debiasing case in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0021] Example 1 This embodiment provides a visual evidence-guided reweighting method for medical visual question answering, implemented using the Visual Evidence-Guided Reweighting for Debiased Medical Visual Question Answering framework (VEGRD). This framework follows an alignment-localization-visual cue enhancement approach, decoupling evidence localization and question-answering inference in medical visual question answering into two sequentially executed training phases, anchoring question-answering inference to local visual evidence relevant to the clinical question.

[0022] like Figure 1 As shown, the method in this embodiment includes steps S1 to S4. Figure 1 This is used to illustrate the sequential relationship between training sample processing, evidence location training, question-and-answer reasoning training, and actual question-and-answer output.

[0023] Step S1: Obtain training samples including medical images, clinical questions, and answer labels, and extract the corresponding visual features and question features.

[0024] Step S2: In the first training phase, visual features and question features are fused across modalities. The first clinical evidence region is determined based on the fused global semantic features. The visual features corresponding to the first clinical evidence region are aggregated to obtain local evidence features. Evidence localization parameters are obtained by jointly constraining the global answer prediction results based on the global semantic features and the local answer prediction results based on the local evidence features according to the answer labels.

[0025] Step S3: In the second training phase, the second clinical evidence region is determined using evidence localization parameters. A spatial evidence mask is generated based on the spatial position of the second clinical evidence region and the visual features. The visual features are affinely modulated based on the spatial evidence mask to obtain reweighted visual features. The reweighted visual features and question features are fused across modalities, and the question-answering inference parameters are optimized based on the answer labels.

[0026] Step S4: Using evidence localization parameters and question-answering inference parameters, visual evidence-guided reweighted question-answering inference is performed on the medical image to be answered and the corresponding clinical question, outputting the medical visual question-answering result. In the actual question-answering stage, the evidence localization parameters obtained in the first training stage and the question-answering inference parameters obtained in the second training stage are called to process the medical image to be answered and the corresponding clinical question. Specifically, the medical image to be answered is divided into multiple image blocks and input into a visual encoder to obtain visual block features; the clinical question is input into a text encoder to obtain text token features and global semantic features; the visual block features and global semantic features are input into the evidence localization branch to predict the clinical evidence region related to the clinical question. A spatial evidence mask is generated based on the predicted clinical evidence region, and the visual block features are affinely reweighted based on this spatial evidence mask to obtain reweighted visual features. Subsequently, the reweighted visual features and text token features are input into a cross-modal fusion network to obtain fused question-answering features, and the medical visual question-answering result is output through an answer classifier. Through the above processing, the model is explicitly guided to focus on lesions or anatomical regions related to clinical problems during the inference stage, reducing the risk of outputting answers based solely on statistical relevance of the question text, and improving the reliability and interpretability of medical visual question answering results.

[0027] like Figure 3 As shown, Figure 3 (a) shows the prior changes in answers between training and test samples; (b) shows the neglect of visual input when forming language shortcuts based solely on question keywords; and (c) shows the process of answer prediction in this embodiment by locating clinical evidence regions and enhancing corresponding visual features. This comparison illustrates the role of clinical evidence regions in suppressing the superficial statistical association between questions and answers.

[0028] like Figure 4 As shown, the upper part of the two-stage overall architecture is the first training stage, in which the instance-aware localization module IAL uses the features corresponding to the cross-modal global identifier to locate the clinical evidence region and aggregates local evidence features through Soft ROI pooling; the lower part of the overall architecture is the second training stage, in which the visual evidence-guided reweighting module VEGR uses the fixed localization prior to perform affine modulation on the visual block features. Figure 4 This is used to illustrate the parameter update relationship between the two training phases and the process by which clinical evidence regions are transformed from localization results into spatial intervention priors.

[0029] Through the above processing, the first training stage enables the evidence localization parameters to learn lesion-level spatial information related to clinical questions, and the second training stage enhances local visual evidence and optimizes answer reasoning under the guidance of this spatial information, thereby reducing the model's dependence on language shortcuts and maintaining the question-answering generalization ability in distributed scenarios.

[0030] 1. Applicable Scenarios and Basic Model In medical visual question answering, the base model can be any model capable of handling medical images and clinical questions, such as SAN, BAN, M3AE, M2I2, MUMC, or LaPA. The base model can obtain multimodal representations through attention mechanisms, mask reconstruction, or anatomical cues. This embodiment further enhances this by adding explicit localization and reweighting of local clinical evidence regions, allowing pathological features in medical images that are small in scale or have low contrast with surrounding tissues to be amplified during cross-modal inference.

[0031] This embodiment is particularly suitable for situations where training and test samples have different answer distributions for the same clinical question type, such as the out-of-distribution settings corresponding to VQA-RAD-CP and SLAKE-CP. Compared with indirect guidance methods that only perform counterfactual reasoning, data distribution alignment, feature alignment, or balanced sampling, this embodiment directly modulates visual features based on the clinical evidence region obtained from localization, so that answer reasoning is based on localized visual evidence.

[0032] 2. Training Sample Organization and Parameter Update In this embodiment, the training samples include a first training sample set and a second training sample set. The first training sample set consists of training samples from the original medical visual question-answering dataset and is used to train the evidence localization parameters. The second training sample set consists of training samples from a dataset obtained by reorganizing medical visual question-answering samples according to clinical question types. The training samples in the reorganized dataset and the test samples have different answer distributions under at least one of the same clinical question types, and are used to optimize the question-answering inference parameters. After the first training phase is completed, the evidence localization parameters are fixed. In the second training phase, only the question-answering inference parameters are updated, and the second clinical evidence region determined by the fixed evidence localization parameters is used as the spatial prior for generating the spatial evidence mask.

[0033] 3. Relationship between the two phases of training Traditional Med-VQA models learn end-to-end mappings Under a biased distribution, this direct mapping degenerates into a shortcut logic that only models the distribution. , This represents an end-to-end mapping function from medical images and clinical questions to answers. This represents the conditional probability of the corresponding answer given a clinical question. To address this issue, this embodiment decomposes the reasoning process into two decoupled stages and introduces intermediate variables. E (Clinical evidence) reshapes the causal chain as ,like Figure 4 As shown.

[0034] 1) The first stage is the clinical evidence anchoring stage. The goal of the first stage is to enable the model to locate pathological evidence under the guidance of multimodal semantics. This stage adopts a joint task learning approach. Given training samples... Model learns localization function Then we have: in, E This represents the predicted distribution of key clinical regions (such as bounding box parameters). This represents the parameter space for locating evidence at this stage. Representing medical images, Indicates a clinical problem. This indicates the corresponding answer label. Evidence location parameters. The parameters used to generate clinical evidence regions include visual encoder parameters, text encoder parameters, cross-modal fusion network parameters, feature interaction parameters that use global semantic features as semantic probes, multilayer perceptron parameters for predicting the location parameters of clinical evidence regions, and answer classifier parameters jointly optimized with evidence localization in the first training phase. The training objective is to establish a robust alignment between the visual feature space and clinical semantics, enabling the model to identify decisive pathological features in complex medical contexts.

[0035] 2) The second stage is the visual evidence-guided intervention stage. Building upon the stable evidence anchoring established in the first stage, the second stage optimizes the decision path under a biased training distribution. The core idea is to perform explicit intervention in the feature space. Given initial visual features... V Evidence priors are generated using the fixed evidence location parameters. E And define a reweighting function guided by visual evidence. To intervene in the reasoning process: in, Indicates the use of prior evidence Initial visual features The reweighted visual features obtained after reweighting.

[0036] Final Answer Prediction Generated under the visual feature distribution after intervention: in, This represents the question-answering reasoning parameters obtained from the second stage of optimization. This represents a question-answering inference function that generates predicted answer results based on reweighted visual features and clinical questions. In the second training phase, the evidence localization parameters used to generate the clinical evidence region in the first training phase are fixed, and these fixed parameters are used to generate evidence localization results for the input medical image and clinical question. The second training phase primarily updates the question-answering inference parameters. This includes trainable parameters in the visual evidence-guided reweighting module, the cross-modal fusion module, and the answer classification module.

[0037] By introducing The model no longer passively processes raw visual input, but is forced to reason under the bottleneck of explicit visual evidence. By decoupling the order of evidence perception localization and visual evidence intervention, VEGRD ensures that the second-stage optimization is anchored to real clinical features, structurally severing the model's dependence on non-causal linguistic priors.

[0038] Example 2 In the clinical evidence anchoring process of this embodiment The core task of the IAL module is to use a first training sample set that does not include annotations of the clinical evidence region, i.e., it does not require the provision of the true bounding box annotations of the first clinical evidence region, and to perform weakly supervised localization through a multi-instance learning (MIL) mechanism. Its processing includes unimodal encoding and cross-modal fusion, determination of the first clinical evidence region, aggregation of local evidence features, and joint constraint training. 1) Single-modal coding and cross-modal semantic detection. Given a medical image... I and clinical issues Q The medical image is divided into multiple non-overlapping visual blocks and encoded to obtain a visual block feature sequence as visual features. Specifically, a two-stream encoder can be used, where the image... I The Swin Transformer divides the data into non-overlapping blocks and maps them to... D Vision Block Token V : In the formula, Represents a visual encoder. Represents any visual block feature. The dimension representing the feature of each visual block. Represents the real number field.

[0039] in, N This refers to the number of visual blocks. (For clinical problems) Q Encoding is performed to obtain a sequence of question features containing a global [CLS] identifier, which serves as the question features; specifically, RoBERTa can be used to encode the clinical question Q into a token sequence. T : In the formula, Indicates a text encoder. This indicates the characteristic corresponding to the global [CLS] identifier. This represents any ordinary question token characteristic.

[0040] Visual block feature sequences V and problem feature sequence T Inputting the data into a multi-layer cross-modal fusion network yields joint feature representations. Z Joint feature representation Z The features corresponding to the global [CLS] identifier absorb multimodal context and are used as global semantic features. Used to identify visual blocks in visual space that are relevant to clinical problems: In the formula, This represents a multi-level cross-modal fusion function. This represents the first feature in the joint feature representation that corresponds to the global [CLS] identifier.

[0041] 2) Determining the first clinical evidence region. This involves defining global semantic features. The input is mapped using a multilayer perceptron (MLP), and the mapping result is normalized using a sigmoid function to obtain the parameters of the continuous bounding box. : in The function is Sigmoid. The continuous bounding box parameters include the center x-coordinate and center y-coordinate of the first clinical evidence region. And width and height The parameters are normalized to [0,1]; the first clinical evidence region is determined based on the continuous bounding box parameters.

[0042] 3) Differentiable feature aggregation based on Soft ROI. This involves backpropagating the gradient of the answer prediction to aggregate the parameters of continuous bounding boxes. E End-to-end optimization was performed, employing Soft ROI pooling to construct a two-dimensional Gaussian kernel. The center x-coordinate and center y-coordinate of the first clinical evidence region were used as the center of the two-dimensional Gaussian kernel, with half the width and half the height used as the horizontal and vertical scales, respectively. (Regarding location...) The first k A visual block Using a two-dimensional Gaussian kernel Calculate its activation weights: in, and These represent the corresponding visual blocks with the same parameters as the continuous bounding boxes. The x and y coordinates in a normalized coordinate system This represents the natural exponential function.

[0043] The features of each visual block are weighted and summed according to their activation weights to obtain the continuous clinical evidence features as local evidence features. : This Gaussian weight-based mapping makes the entire process from bounding box prediction to feature aggregation completely differentiable.

[0044] 4) Multi-instance joint optimization objective. Given continuous evidence features. This is then jointly optimized with multimodal global features for weakly supervised training. Global semantic probe and local evidence features Input the question-answering classifier separately .

[0045] Cosine similarity and consistency loss They can be represented as: in, The formula for cosine similarity is represented by... Indicates consistency loss. Represents a global semantic probe. These are local evidence features. The similarity between features of evidence regions can be expressed as: Based on this, the supervised comparison loss can be expressed as: in, For the current anchor point sample index, To be consistent with the sample The set of sample indices that constitute positive sample relations. , , These represent the evidence region features of the current sample, the positive sample, and the comparison sample, respectively. The index of the samples to be compared in the batch and This indicates that the sample itself is excluded. For a sample set in a training batch, the temperature coefficient Used to adjust the smoothness of the similarity distribution. It is a natural exponential function. Supervised contrastive loss brings the evidence region features corresponding to positive samples closer together in the feature space and pushes the evidence region features corresponding to negative samples further apart, thereby improving the class discrimination ability of the evidence localization results.

[0046] The goal of the first phase is : Two of them These represent the global binary cross-entropy loss determined based on the answer labels. and local binary cross-entropy loss This ensures accurate reasoning within a global context and local visual cues. The consistency loss is calculated based on global semantic features and local evidence features to constrain the semantic alignment between them; the supervised contrast loss is determined based on the answer label, global semantic features, and local evidence features. Each weight coefficient is set to 1.

[0047] The aforementioned losses work together to drive the activation weights. The system accurately covers the lesion area, achieving deep alignment between the visual feature space and clinical semantics. The global binary cross-entropy loss, local binary cross-entropy loss, consistency loss, and supervised contrast loss are added together to obtain the first-stage training objective. Backpropagation is then performed based on this objective to optimize the parameters used to determine the first clinical evidence region, thus obtaining the evidence localization parameters.

[0048] like Figure 6 As shown, Figure 6 This further illustrates the localization and feature aggregation process for the aortic aneurysm detection question, "Is there evidence of an aortic aneurysm?". During this process, the global semantic probe within IAL... Chest X-ray Figure 6 a) When interacting with the question, a weakly supervised search is performed in the visual feature space. The resulting ROI attention heatmap ( Figure 6 b) Precisely highlight the mediastinal and aortic arch regions, consistent with clinical anatomy. The MLP subsequently predicts a bounding box that tightly surrounds the lesion area. Figure 6 c). Finally, SoftROI pooling assigns continuous, differentiable Gaussian attention weights to the image grid ( Figure 6 d) Aggregating visual blocks into high-fidelity clinical evidence This process, supervised solely by answer labels, transforms clinical problems into fully differentiable spatial location priors.

[0049] Example 3 In the visual evidence-guided intervention process of this embodiment The VEGR module modulates the bounding box parameters of the second clinical evidence region determined by the fixed evidence localization parameters through spatial modulation. This is transformed into explicit visual cues. The core idea is to create a visual bottleneck to block ineffective shortcuts caused by statistical language bias.

[0050] 1) Spatial evidence mask generation. In the second training phase, the evidence localization parameters obtained in the first training phase are fixed. Using the aforementioned evidence localization parameters, a second clinical evidence region characterized by a central abscissa, a central ordinate, width, and height is obtained, with its bounding box parameters being: Obtain the x and y coordinates of each visual patch, calculate the spatial relationship between each visual patch and the second clinical evidence region, and construct a binary space evidence mask. Regarding location The first k Each block, its mask value for: Specifically, for each visual block, if the absolute value of the difference between the horizontal coordinate of the visual block and the central horizontal coordinate of the second clinical evidence region is not greater than half of the width, and the absolute value of the difference between the vertical coordinate of the visual block and the central vertical coordinate is not greater than half of the height, the corresponding mask value is set to 1; otherwise, it is set to 0. A binary space evidence mask is generated based on each mask value, where visual blocks with a mask value of 1 belong to the second clinical evidence region, and visual blocks with a mask value of 0 belong to the background region.

[0051] 2) Affine modulation based on preset constants. Unlike implicit attention mechanisms, VEGR uses direct affine modulation to apply visual features. V Perform explicit intervention. For each visual token Get the space scaling factor and shift bias The spatial scaling factor and shift bias are both preset constants: in, This represents the corresponding reweighted visual block feature. and During addition, broadcasting is performed according to each dimension of the visual block features. This represents element-wise multiplication. For each visual block feature, the spatial scaling factor is multiplied by the corresponding mask value and then added to the value 1 to obtain the feature scaling coefficient; the shift bias is multiplied by the mask value to obtain the feature shift; the visual block feature is then multiplied element-wise by the feature scaling coefficient and the feature shift is added to obtain the reweighted visual block feature, and the reweighted visual features are composed of the reweighted visual block features. This affine transformation amplifies the response within the mask region while preserving the original distribution of background features, constructing explicit visual evidence constraints before multimodal fusion.

[0052] 3) Affine Modulation Based on Two-Dimensional Gaussian Mask. As another implementation, the center x-coordinate and center y-coordinate of the second clinical evidence region are used as the horizontal and vertical mean values ​​of a two-dimensional Gaussian distribution, respectively. The horizontal and vertical scales are determined based on the width and height to construct a two-dimensional Gaussian mask. The Gaussian weight of each visual block in the two-dimensional Gaussian mask is used as the corresponding spatial scaling factor. For each visual block feature, the product of the value 1, the spatial scaling factor, and the mask value of the visual block in the binary evidence mask is added to obtain the feature scaling coefficient. A preset shift bias is multiplied by the mask value to obtain the feature translation amount. The visual block feature is multiplied element-wise by the feature scaling coefficient and then the feature translation amount is superimposed to obtain the reweighted visual block feature. The reweighted visual features are then composed of these reweighted visual block features.

[0053] Specifically, in the affine modulation of a two-dimensional Gaussian mask, for the first... i A visual block, with its center coordinates being the x and y coordinates. The visual block is calculated based on a two-dimensional Gaussian distribution. The corresponding original Gaussian weights. Two-dimensional Gaussian weights. The specific calculation formula is as follows: in, Indicates the first i The original Gaussian weights corresponding to each visual block are based on the center of the second clinical evidence region. As the peak, with For width and height, the closer the center of the visual block is to the center of the evidence region, the greater the Gaussian weight; the farther away, the smaller the Gaussian weight. To ensure that the weight values ​​are consistent for different evidence region sizes, this embodiment normalizes the original Gaussian weights so that the normalized Gaussian weights range from 0 to 1.

[0054] Gaussian weight normalization The formula is: Furthermore, the effective evidence weights are obtained by combining the binary space evidence mask. The binary space evidence mask represents the... iWhether a visual patch lies within the spatial evidence region defined by the predicted bounding box; multiplying the normalized Gaussian weights by the binary spatial evidence mask preserves both the spatial constraints of the predicted bounding box and the center-smoothing enhancement properties of the two-dimensional Gaussian distribution. Effective evidence weights. With binary space evidence mask Combination relationship for: In affine modulation, the scaling factor and feature shift of visual block features are determined based on the effective evidence weights. The scaling intensity factor controls the enhancement magnitude of the visual block features, while the shift bias controls the translation amount. Both the scaling intensity factor and the shift bias can be preset hyperparameters or learnable parameters. Feature scaling factor and feature translation It can be represented as: Furthermore, element-wise scaling and translation are performed on each visual block feature to obtain the reweighted visual block features. The affine-modulated reweighted visual block features can be represented as: in, Indicates the first i The weight of valid evidence corresponding to each visual block The scaling intensity coefficient can be a preset hyperparameter or a learnable parameter. The shift bias can be a preset bias or a learnable bias. For the first i The original visual block features of a visual block. Indicates the first i The reweighted visual block features are obtained by affine modulation of a visual block using a two-dimensional Gaussian mask.

[0055] 4) Evidence-driven debiasing. Feature sequences after intervention. Then with text tokens T They are fed together into the cross-modal fusion layer. This is due to the reweighting of key regions in the visual features. The model was augmented, forced to construct a cross-modal attention dependency centered on clinical evidence. (Final answer) A* Generate from the joint representation after intervention: in, Represents the answer classifier. The cross-modal fusion function is represented by equation (22); the fusion and classification processing shown in equation (22) constitutes the question-answering inference function in equation (3). One specific implementation of this mechanism is as follows: Answer reasoning must be based on enhanced block features, effectively severing shortcut connections between question keywords and high-frequency answers. Therefore, VEGR not only enhances out-of-distribution (OOD) generalization ability but also endows the reasoning process with explicit clinical interpretability.

[0056] Example 4 like Figure 2 As shown, this embodiment provides a medical visual question-answering system with visual evidence-guided reweighting. Figure 2 This describes the data transfer relationship between the sample processing module, the evidence localization training module, the question-answering reasoning training module, and the question-answering output module. Each module performs the following processing: (1) Sample processing module, used to perform sample acquisition and feature extraction processing as described in step S1, form visual features and question features, and provide them to evidence location training module and question answering reasoning training module.

[0057] (2) Evidence localization training module, used to perform the first training stage processing described in step S2, form a joint constraint between the global answer prediction result and the local answer prediction result based on visual features and question features, and output evidence localization parameters.

[0058] (3) Question answering reasoning training module, used to perform the second training stage processing described in step S3, receive evidence location parameters and form a spatial evidence mask, thereby performing affine modulation and cross-modal fusion on visual features and outputting question answering reasoning parameters.

[0059] (4) Question-answering output module, used to perform the actual question-answering processing described in step S4, jointly calling the evidence localization parameters and question-answering inference parameters to process the medical image to be answered and the corresponding clinical questions, and output the medical visual question-answering results. The feature extraction, cross-modal fusion, clinical evidence region determination, spatial evidence mask generation and affine modulation methods adopted by each module can adopt the corresponding methods described in Examples 1 to 3.

[0060] Example 5 1. Dataset and Evaluation Metrics To evaluate the effectiveness and generalization ability of VEGRD, this embodiment conducts experiments on four datasets: VQA-RAD, SLAKE, VQA-RAD-CP, and SLAKE-CP. VQA-RAD contains 315 radiographic images and 3,064 question-answer (QA) pairs, covering both closed-ended and open-ended questions. SLAKE contains 642 images and 14,028 bilingual QA pairs, divided into training, validation, and test sets in a 70:15:15 ratio. VQA-RAD-CP and SLAKE-CP are constructed following the Changing Priors (VQA-CP) principle: the data is reorganized by question type, resulting in a significant difference in the answer distribution between the training and test sets, thus creating an out-of-distribution (OOD) setting that exposes the model's dependence on statistical shortcuts. All datasets are evaluated using the M3AE evaluation protocol to ensure fair comparison.

[0061] 2. Implementation details This embodiment uses M3AE as the baseline. The visual encoder is Swin Transformer, the text encoder is RoBERTa, and multimodal feature interactions are handled by a multi-layer Transformer encoder. The input image is resized to 224×224, the maximum question token length is 32, and both visual and text feature dimensions are mapped to 768. Model parameters are initialized using M3AE weights pre-trained on ROCO and MedICaT. All experiments are run on a single NVIDIA RTX A4000 GPU (16GB) with FP16 mixed precision training. The AdamW optimizer is used with a learning rate of 1e-5. The first stage is trained on unbiased datasets (VQA-RAD and SLAKE); the second stage is trained on biased datasets (VQA-RAD-CP and SLAKE-CP). Evaluation covers closed and open QA accuracy. Notably, the first stage uses a weakly supervised paradigm without ground truth bounding box annotations, thus excluding traditional localization metrics (such as IoU). Accordingly, this embodiment utilizes biased OOD benchmarks VQA-RAD-CP and SLAKE-CP to implicitly validate the visual evidence localization capability. Qualitative evaluation using attention heatmaps further confirms the model's lesion localization capability.

[0062] 3. Comparative Experiment To comprehensively evaluate VEGRD, this embodiment compares it with 12 state-of-the-art methods, divided into two groups: classic Med-VQA models (SAN, BAN, M3AE, M2I2, MUMC, LaPA) and biased VQA models (MEVE+SAN, MEVE+BAN, RMLVQA, DeCoCT, Med-BiasX, CEDO). RMLVQA is a general method, while MEVE, DeCoCT, Med-BiasX, and CEDO are methods specific to the medical field.

[0063] Table 1. Performance comparison (%) of VEGRD with state-of-the-art methods on the OOD datasets VQA-RAD-CP and SLAKE-CP. Evaluation was conducted using open, closed, and global settings. Best results are highlighted in bold, and second-best results are indicated by underline.

[0064] 1) Performance on Biased Datasets. Table 1 reports the results on VQA-RAD-CP and SLAKE-CP. Classical Med-VQA models suffer a significant drop in accuracy on these OOD benchmarks due to their reliance on language shortcuts in the training distribution. For example, M3AE (the infrastructure used in this embodiment) achieves only 69.53% overall accuracy on VQA-RAD-CP. Introducing a debiasing mechanism significantly restores performance: the general debiased model RMLVQA achieves 76.42% on SLAKE-CP, while the medical-specific model CEDO achieves 92.07% on VQA-RAD-CP. VEGRD outperforms all comparable methods on both benchmarks, achieving 97.30% on VQA-RAD-CP and 83.48% on SLAKE-CP. Compared to its strongest competitor, CEDO, it improves by 5.23 percentage points on VQA-RAD-CP. These results confirm that VEGRD fundamentally reduces its reliance on language shortcuts by forcing attention to localized visual evidence.

[0065] Table 2. Performance comparison (%) of VEGRD with state-of-the-art methods on the in-distribution (ID) datasets VQA-RAD and SLAKE. Best results are highlighted in bold, and second-best results are indicated by underline. VEGRD performs well and enhances multimodal inference.

[0066] 2) Performance on the original unbiased dataset. Table 2 shows the results on the in-distribution VQA-RAD and SLAKE benchmarks. Bias-reduction methods often sacrifice in-distribution accuracy for OOD robustness. VEGRD avoids this trade-off: it achieves a highest overall accuracy of 79.82% on VQA-RAD, outperforming M3AE (76.50%) and all bias-reduction baselines, and achieves an overall accuracy of 85.02% and an open-ended accuracy of 82.79% on SLAKE. These results demonstrate that the visual evidence intervention mechanism effectively filters background noise, providing accurate and robust visual inference for routine clinical scenarios.

[0067] 4. Ablation test To verify the independent contributions and synergistic effects of each component of VEGRD, ablation experiments were conducted on VQA-RAD-CP and SLAKE-CP in this embodiment.

[0068] Table 3. Ablation study of the core components of the VEGRD framework. This example evaluates the independent and collaborative contributions of the two-stage training paradigm, the IAL module, and the VEGR module on the VQA-RAD-CP and SLAKE-CP datasets. Best results are highlighted in bold.

[0069] 1) Ablation of core components. Table 3 shows the contribution of each component. The baseline (global feature fusion only) achieved only 69.53% overall accuracy on VQA-RAD-CP, confirming that models without explicit visual guidance are highly susceptible to language shortcuts. Independent activation of IAL or VEGR each brought an improvement of approximately 20 percentage points, indicating that lesion localization and spatial feature weighting are effective in resisting bias. Compared to end-to-end joint training of IAL and VEGR (Method 5, 95.44%), the two-stage decoupling paradigm (VEGRD) improved accuracy to 97.30%. This confirms that decoupling evidence anchoring from visual evidence intervention avoids gradient conflicts, allowing the second-stage spatial intervention to more effectively reconstruct the causal inference chain.

[0070] 2) Spatial intervention factor ablation. Hyperparameters (scaling factor) and (Shift bias) adjusts the balance between lesion area magnification and background preservation. This embodiment uses a shift bias within the range {0.0, 0.1, 0.2, 0.5, 1.0}. and Perform a grid search, and in Figure 5 The overall accuracy of the report is as follows. On VQA-RAD-CP, peak performance (97.30%) is achieved. , The results obtained from [location] indicate that strong lesion magnification and minimal background preservation are optimal for this dataset. On SLAKE-CP, the optimal combination is transferred to [location]. , (83.48%), reflecting that SLAKE images typically cover the whole body or multiple organs, thus requiring richer anatomical context. These results indicate that affine space modulation can flexibly adapt to the clinical characteristics of different datasets.

[0071] Table 4. Performance comparison of different spatial intervention methods on scaling factors under different shift bias values ​​(%). The spatial scaling factor, shift bias, and Gaussian weighting method mentioned in the table are represented by the following symbols, and the Gaussian weighting method uses the bounding box parameters predicted in the first stage. E Constructing a two-dimensional Gaussian mask: The best results in the table are highlighted in bold.

[0072] 3) Intervention form and robustness. Table 4 sets the fixed constants. Gaussian weighted The bounding box parameters output in the first stage are compared. As the mean and scale of a two-dimensional Gaussian mask. Fixed constants. Reaching 97.30%, better than the best Variants (95.95%). This result demonstrates that uniform magnification within the bounding box avoids the edge decay effect of the Gaussian distribution, which can weaken the intervention on fine-grained pathological features located near lesion boundaries. This result further illustrates the importance of spatially uniform feature enhancement in the face of extreme linguistic biases.

[0073] 5. Visualization Results This section provides a qualitative analysis of VEGRD's interpretability, localization accuracy, and robust decision path. Since the first stage of weakly supervised training lacks real bounding box annotations, the standard IoU metric is not applicable. Instead, this example demonstrates a weakly supervised feature aggregation workflow, attention heatmap comparison, and a bias removal case study.

[0074] like Figure 7 As shown, Figure 7Attention heatmaps of VEGRD, M3AE, LaPA, M2I2, and MUMC were compared in the same aortic aneurysm cases. While all methods returned the correct answer "yes," their internal reasoning processes differed. Methods relying on global cross-attention (such as M3AE and M2I2) exhibited attention diffusion: high response weights diffused to the periphery of the lung and irrelevant soft tissues, failing to locate the actual lesion. LaPA showed more focused attention but still failed to cover the critical aortic arch region. VEGRD, used in this example, exhibited high spatial specificity: its attention was tightly focused on the mediastinum and great vessel regions with minimal background interference. This suggests that VEGRD is based on genuine pathological evidence rather than spurious relevance for clinical reasoning.

[0075] like Figure 8 As shown, Figure 8 Two OOD (Out-of-Depth) cases are presented. In the first case (question: "Is there airspace consolidation on the left side?"), the baseline model was misled by the high-frequency training label "No," ignoring the obvious consolidation shadow in the left lung. The VEGRD used in this embodiment locates the lesion area using a bounding box, and forces the model to examine this area through VEGR spatial magnification, correctly predicting "Yes." In the second case (question: "Where is the most prominent part of the colon in this view?"), the baseline model defaults to a biased answer "Right." The explicit spatial affine enhancement of the VEGRD aligns the multimodal features with the located abdominal colon area, giving a correct prediction of "Left." These two cases demonstrate that VEGRD successfully cuts off non-causal shortcuts and establishes reliable causal inference anchored to localized clinical evidence.

[0076] 6. Verification Conclusion The above experiments demonstrate that this embodiment, through weakly supervised clinical evidence localization in the first training phase and affine reweighting of visual features in the second training phase, can improve debiasing robustness under out-of-distribution conditions while maintaining in-distribution question-answering performance. Since the first training phase utilizes only answer labels without requiring ground-value bounding box annotations, the method can be applied to medical visual question-answering datasets lacking spatial annotations. Furthermore, the two-stage decoupling intervention can be extended to 3D medical visual question answering and alignment of medical multimodal large language models.

[0077] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A visual evidence-guided reweighted medical visual question-answering method, characterized in that, include: Acquire training samples including medical images, clinical questions, and answer labels, and extract corresponding visual features and question features; In the first training phase, visual features and question features are fused across modalities. The first clinical evidence region is determined based on the fused global semantic features. The visual features corresponding to the first clinical evidence region are aggregated to obtain local evidence features. Based on the answer label joint constraint, the global answer prediction results based on the global semantic features and the local answer prediction results based on the local evidence features are obtained to obtain evidence localization parameters. In the second training phase, the evidence localization parameters are used to determine the second clinical evidence region, and a spatial evidence mask is generated based on the spatial location of the second clinical evidence region and the visual features. Based on the spatial evidence mask, the visual features are affinely modulated to obtain reweighted visual features; The reweighted visual features and question features are fused across modalities, and the question-answering inference parameters are optimized based on the answer labels; Using the aforementioned evidence localization parameters and question-answering inference parameters, visual evidence-guided reweighted question-answering inference is performed on the medical images to be answered and the corresponding clinical questions, and medical visual question-answering results are output.

2. The medical visual question-answering method guided by visual evidence reweighting according to claim 1, characterized in that, The training samples include a first training sample set and a second training sample set; the first training sample set consists of training samples from the original medical visual question-answering dataset and is used to train the evidence localization parameters; the second training sample set consists of training samples from a dataset obtained by reorganizing medical visual question-answering samples according to clinical question types, and the training samples and test samples in the reorganized dataset have different answer distributions under at least one of the same clinical question types, and are used to optimize the question-answering inference parameters; after the first training phase is completed, the evidence localization parameters are fixed, and only the question-answering inference parameters are updated in the second training phase, and the second clinical evidence region determined by the fixed evidence localization parameters is used as the spatial prior for generating the spatial evidence mask.

3. The medical visual question-answering method guided by visual evidence and reweighted according to claim 2, characterized in that, The first training sample set does not include clinical evidence region annotations; the extraction of visual features and question features includes: dividing the medical image into multiple non-overlapping visual blocks and encoding them to obtain a visual block feature sequence, which is used as the visual features; encoding the clinical question to obtain a question feature sequence containing a global identifier, which is used as the question features; the acquisition of global semantic features includes: performing multi-level cross-modal fusion on the visual block feature sequence and the question feature sequence to obtain a joint feature representation, and determining the feature in the joint feature representation corresponding to the global identifier as the global semantic feature.

4. The medical visual question-answering method guided by visual evidence and reweighted according to claim 3, characterized in that, The determination of the first clinical evidence region includes: inputting the global semantic features into a multilayer perceptron for mapping, and normalizing the mapping result using a sigmoid function to obtain continuous bounding box parameters; the continuous bounding box parameters include the center x-coordinate, center y-coordinate, width, and height of the first clinical evidence region; and determining the first clinical evidence region based on the continuous bounding box parameters.

5. The medical visual question-answering method guided by visual evidence and reweighted according to claim 4, characterized in that, The acquisition of the local evidence features includes: constructing a two-dimensional Gaussian kernel using Soft ROI pooling; using the center x-coordinate and center y-coordinate of the first clinical evidence region as the center of the two-dimensional Gaussian kernel, and using half of the width and half of the height as the horizontal and vertical scales of the two-dimensional Gaussian kernel, respectively; calculating the activation weight of each visual block feature using the two-dimensional Gaussian kernel based on the x-coordinate and y-coordinate of each visual block; and performing a weighted summation of the visual block features according to the activation weights to obtain the local evidence features.

6. The medical visual question-answering method guided by visual evidence and reweighted according to claim 5, characterized in that, The joint constraints of the first training phase include: classifying the answers for the global semantic features and local evidence features respectively to obtain the global answer prediction result and the local answer prediction result; determining the global binary cross-entropy loss and the local binary cross-entropy loss corresponding to the global answer prediction result and the local answer prediction result respectively based on the answer label; calculating the consistency loss used to constrain the semantic alignment of the two based on the global semantic features and the local evidence features, and determining the supervised contrast loss based on the answer label and the global semantic features and the local evidence features; adding the global binary cross-entropy loss, the local binary cross-entropy loss, the consistency loss and the supervised contrast loss to obtain the first phase training objective; and performing backpropagation based on the first phase training objective to optimize the parameters used to determine the first clinical evidence region to obtain the evidence localization parameters.

7. The medical visual question-answering method guided by visual evidence and reweighted according to claim 2, characterized in that, The visual features include visual block features that correspond one-to-one with multiple visual blocks; the second clinical evidence region is characterized by a central horizontal coordinate, a central vertical coordinate, width, and height. The generation of the spatial evidence mask includes: obtaining the abscissa and ordinate of each visual block; for each visual block, when the absolute value of the difference between its abscissa and the central abscissa is not greater than half of the width, and the absolute value of the difference between its ordinate and the central ordinate is not greater than half of the height, the mask value corresponding to the visual block is set to 1, otherwise it is set to 0; a binary spatial evidence mask is generated based on each mask value, wherein the visual block with a mask value of 1 belongs to the second clinical evidence region, and the visual block with a mask value of 0 belongs to the background region.

8. The medical visual question-answering method guided by visual evidence and reweighted according to claim 7, characterized in that, The affine modulation includes: acquiring a spatial scaling factor and a shift bias, both of which are preset constants; for each visual block feature, multiplying the spatial scaling factor by the mask value corresponding to the visual block and adding it to the value 1 to obtain a feature scaling coefficient; multiplying the shift bias by the mask value to obtain a feature shift amount; multiplying the visual block feature by the feature scaling coefficient element by element and then adding the feature shift amount to obtain a reweighted visual block feature; and constructing the reweighted visual feature from the reweighted visual block features.

9. The medical visual question-answering method guided by visual evidence and reweighted according to claim 7, characterized in that, The affine modulation includes: using the center x-coordinate and center y-coordinate of the second clinical evidence region as the lateral mean and lateral mean of a two-dimensional Gaussian distribution, respectively, and determining the lateral and lateral scales based on the width and height to construct a two-dimensional Gaussian mask; using the Gaussian weights of each visual block in the two-dimensional Gaussian mask as corresponding spatial scaling factors; for each visual block feature, adding the product of the value 1 and the spatial scaling factor and the mask value of the visual block in the binary spatial evidence mask to obtain a feature scaling coefficient; multiplying a preset shift bias by the mask value to obtain a feature translation amount; multiplying the visual block feature element-wise by the feature scaling coefficient and then superimposing the feature translation amount to obtain a reweighted visual block feature; and constructing the reweighted visual feature from the reweighted visual block features.

10. A medical visual question-answering system guided by visual evidence and reweighted, characterized in that, include: The sample processing module is used to acquire training samples including medical images, clinical questions, and answer labels, and extract corresponding visual features and question features; The evidence localization training module is used in the first training phase to fuse the visual features and question features across modalities, determine the first clinical evidence region based on the fused global semantic features, aggregate the visual features corresponding to the first clinical evidence region to obtain local evidence features, and obtain evidence localization parameters based on the global answer prediction result based on the global semantic features and the local answer prediction result based on the local evidence features, according to the answer label joint constraint. The question-and-answer reasoning training module is used in the second training phase to determine the second clinical evidence region using the evidence location parameters and to generate a spatial evidence mask based on the spatial location of the second clinical evidence region and visual features. Based on the spatial evidence mask, the visual features are affinely modulated to obtain reweighted visual features; The reweighted visual features and question features are fused across modalities, and the question-answering inference parameters are optimized based on the answer labels; The question-and-answer output module is used to perform visual evidence-guided reweighted question-and-answer reasoning on the medical images to be answered and the corresponding clinical questions using the evidence localization parameters and question-and-answer reasoning parameters, and output medical visual question-and-answer results.

Images