Medical visual question answering method and system based on delay cross-modal alignment and reverse reasoning

Abstract

The application provides a medical visual question answering method and system based on delay cross-modal alignment and reverse reasoning. The method obtains medical images and question texts, extracts initial visual features, text features and corresponding global labels through a pre-training encoder; a dynamic alignment adaptation module is introduced at the top of the multi-modal fusion layer to perform delay cross-modal alignment, and a dynamic scalar gate is used to control the update strength of the residual alignment signal; an attribute perception reverse reasoning module is constructed, the clinical prior attribute pseudo-label is mapped to attribute embedding in the forward prediction stage, the aligned visual features are injected into the attribute embedding, and the text features are combined to predict the candidate answers; in the reverse reconstruction stage, the candidate answer embedding and the original visual global features without attribute embedding are spliced to reversely reconstruct the global semantic representation of the question. The model is optimized end-to-end by using a joint loss function. The application avoids the early assimilation of low-level visual features, and improves the bidirectional logical consistency and result accuracy of the question and answer reasoning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical image processing technology, and in particular to a medical visual question answering method and system based on delayed cross-modal alignment and reverse reasoning. Background Technology

[0002] Medical visual question answering aims to automatically output answers relevant to clinical diagnostic scenarios based on input medical image content and natural language question text. This task involves medical image feature representation, natural language text feature representation, and cross-modal feature fusion. Existing techniques typically introduce cross-modal interaction computation directly into the shallow network stage after feature extraction when constructing multimodal fusion networks, causing alignment between the visual and textual modalities in the low-level feature space. Medical images often contain subtle lesion textures, local abnormal regions, and structurally complex diagnostic clues. Early cross-modal interactions in the low-level feature space can easily lead to the smoothing or masking of these fine-grained lesion features during the fusion process.

[0003] Conventional one-way inference processes rarely include a reverse consistency verification step for the prediction results in the input-output mapping. When dealing with open-ended questions or complex diagnostic scenarios, pure forward computation is prone to semantic shifts, resulting in model answers lacking sufficient visual or pathological evidence. Considering the timing of feature fusion and the integrity of the inference chain, it is necessary to explore corresponding structural optimization strategies to improve the reliability of question-answering results. Summary of the Invention

[0004] To address the technical problems in existing technologies, such as the ease with which fine-grained lesion features are smoothed in early fusion and the lack of consistency verification in unidirectional reasoning, the present invention aims to provide a medical visual question answering method and system based on delayed cross-modal alignment and reverse reasoning.

[0005] In a first aspect, the present invention provides a medical visual question answering method based on delayed cross-modal alignment and backward reasoning, comprising the following steps: Step S1: Obtain the medical image and the corresponding question text, and extract the initial visual features and their global visual labels, and the initial text features and their global text labels through the pre-trained encoder. Step S2: A dynamic alignment adaptation module is introduced at the top layer of the multimodal fusion layer. The initial visual features and initial text features are input into the multimodal fusion layer to perform delayed cross-modal alignment. Specifically, the average value of the pooled text and visual features is calculated as a shared alignment anchor point, and the difference between the shared alignment anchor point and the center of each modality is calculated to obtain the residual alignment signal. At the same time, a dynamic scalar gating is generated using the text global marker, visual global marker and their difference in the current layer to control the intensity of the residual alignment signal updated in the features. Step S3: Construct an attribute-aware reverse reasoning module. In the forward prediction stage, obtain pseudo-labels of clinical prior attributes and map them as attribute embeddings. Through a gating mechanism based on global visual features, inject the attribute embeddings into the aligned visual features to generate an enhanced visual representation. Combine the aligned text features to predict candidate answers. Step S4: In the reverse reconstruction stage, the embedded representation of the predicted candidate answer is concatenated with the original visual global features that have been aligned but not injected with the attribute embeddings, and the global semantic representation of the original question is reconstructed through the reverse reconstruction network. Step S5: The model is optimized end-to-end using a joint loss function, which includes the main task loss, the cosine alignment loss between the text and the visual global labels, the clinical attribute-assisted classification loss, and the mean squared error loss used to supervise the consistency between the reconstructed global semantic representation and the global representation of the original question.

[0006] Optionally, in step S2, before performing the cross-modal hierarchical alignment, an intra-modal adaptation step is also included: In the selected fusion layer, parallel low-rank residual adapters are introduced on the text branch and the visual branch respectively, and learnable scalar gating initialized to zero is used to progressively activate independent residual updates for text features and visual features during training.

[0007] Optionally, the dynamic alignment and adaptation module is exclusively located in the last two layers of the multimodal fusion layer to avoid premature assimilation of superficial pathological details in cross-modal interactions, thereby preserving fine-grained lesion information in medical images.

[0008] Optionally, the clinical prior attributes in step S3 include organ category pseudo-labels and imaging modality pseudo-labels generated offline from training images using an external biomedical multimodal pre-trained model.

[0009] Optionally, the specific method for embedding the attribute into the aligned visual feature in step S3 is as follows: The visual global representation after delay cross-modal alignment is input into the gating network along with the organ category attribute embedding and the imaging modality attribute embedding, respectively. The corresponding relevance weights are calculated, and the attribute embeddings are superimposed on the visual global representation by soft injection.

[0010] Optionally, in step S4, the answer with the highest forward prediction score is selected as the semantic anchor point, and it is concatenated with the visual global features without injected clinical prior attributes and then input into the multilayer perceptron. The mean squared error loss is used to guide the model to backtrack and verify the questioning intent through visual evidence.

[0011] Optionally, in the joint loss function described in step S5, the weight of the cosine alignment loss is set to 0.2, and the weights of the clinical attribute-assisted classification loss and the mean square error loss of reverse intent reconstruction are both set to 0.1.

[0012] Optionally, in step S1, before the visual features and text features enter the multimodal fusion layer, modality type embeddings are added to both, so that the fusion network can distinguish different modality sources.

[0013] Optionally, the visual encoder uses a pre-trained ViT, and the text encoder uses a pre-trained RoBERTa.

[0014] Secondly, the present invention provides a medical visual question answering system based on delayed cross-modal alignment and reverse reasoning, comprising: The feature extraction module is used to obtain the initial visual and textual features of medical images and question text; The dynamic alignment and fusion module is used to perform delayed cross-modal alignment and intramodal adaptation as described in the first aspect; An attribute-aware reverse reasoning module is used to perform forward prior injection and reverse intent reconstruction as described in the first aspect. And an answer output module, used to provide medical visual question answering prediction results based on the output of the attribute-aware reverse reasoning module.

[0015] The present invention has achieved the following beneficial effects: This invention protects fine-grained lesion textures extracted from early network layers, such as nodule edge information and interstitial space textures, by delaying cross-modal alignment operations to the upper semantic layer of the multimodal fusion network. This prevents these low-frequency details, crucial for diagnosis, from being assimilated during shallow feature fusion. In the inference prediction stage, this scheme constructs a forward anchoring mechanism based on clinical prior attributes and a reverse question intent reconstruction mechanism based on predicted answers. By mapping the answer with the highest forward prediction score to a semantic anchor point and concatenating it with visual global features without injected clinical prior attributes, reverse intent reconstruction is performed, simulating the diagnostic process of reverse backtracking verification. This allows the mean squared error loss to guide the network in correcting feature mappings during model training. This bidirectional inference design enhances the bidirectional logical consistency of feature representations, improves the semantic shifts and insufficient evidence in answers that easily arise in complex diagnostic scenarios due to unidirectional inference, and facilitates the construction of a logical causal closed loop between pathological evidence and diagnostic answers.

[0016] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the overall network architecture of the medical visual question answering method and system provided in the embodiments of the present invention; Figure 2 A flowchart illustrating a medical visual question answering method based on delayed cross-modal alignment and backward reasoning, provided as an embodiment of the present invention; Figure 3 This is a schematic diagram showing the structural details of the Dynamic Alignment Adaptation (DAA) module in an embodiment of the present invention; Figure 4 This is a schematic diagram showing the structural details of the attribute-aware reverse reasoning (A2R2) module in an embodiment of the present invention; Figure 5 This is a comparison chart of the question-answering prediction results of the embodiments of the present invention and the baseline model in different medical diagnostic scenarios; Figure 6 This is a comparison chart of the attention heatmaps of the embodiment of the present invention and the baseline model during the feature extraction process; Figure 7 This is a sensitivity analysis curve of each hyperparameter (alignment weight, attribute weight, reconstruction weight) of the joint loss function in an embodiment of the present invention. Detailed Implementation

[0019] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] Medical visual question answering aims to automatically output answers relevant to clinical diagnostic scenarios based on input medical image content and natural language question text. This task involves medical image feature representation, natural language text feature representation, and cross-modal feature fusion. Existing technologies, when constructing multimodal fusion networks, typically introduce cross-modal interaction computation directly in the shallow network stage after feature extraction, aligning the visual and textual modalities in the low-level feature space. However, medical images often contain subtle lesion textures, local abnormal regions, and structurally complex diagnostic clues. Premature cross-modal interaction in the low-level feature space can lead to the smoothing or masking of these fine-grained lesion features during the fusion process. Furthermore, conventional one-way inference processes lack reverse consistency verification of prediction results, leading to semantic shifts and insufficient basis for answers in complex diagnostic scenarios. To address these issues, this embodiment provides a medical visual question answering method based on delayed cross-modal alignment and reverse inference. Figure 1 The diagram shows the overall network architecture of this method. This method delays cross-modal alignment operations to the upper semantic layer of the multimodal fusion network and constructs a forward anchoring mechanism based on clinical prior attributes and a reverse question intent reconstruction mechanism based on predicted answers during the inference and prediction phase, thereby enhancing the bidirectional logical consistency of feature representations.

[0021] like Figure 2 As shown, the method includes the following steps: Step S1: Obtain the medical image and the corresponding question text, and extract the initial visual features and their global visual labels, and the initial text features and their global text labels through the pre-trained encoder.

[0022] In the specific implementation process, the data acquisition phase obtains sample data from medical image archiving and communication systems or standard medical datasets. The acquired medical images include, but are not limited to, X-ray images, computed tomography (CT) images, magnetic resonance imaging (MRI) images, and ultrasound images. The corresponding natural language question text is either an open-ended question-and-answer text or a closed-ended classification question-and-answer text posed to the medical image. Before inputting the data into the model, the medical images undergo preprocessing. Specifically, the pixel size of the original medical images is adjusted to a preset fixed resolution size, single-channel grayscale images or multi-channel color images undergo channel unification processing, and the image pixel values ​​are normalized using preset mean and standard deviation parameters to eliminate differences in illumination and contrast caused by different imaging device acquisition conditions. Specifically, the preset mean and standard deviation parameters are adopted from the statistical benchmark parameters of a large-scale public image dataset (such as ImageNet) corresponding to the visual pre-trained model (e.g., RGB channel mean is [0.485, 0.456, 0.406], standard deviation is [0.229, 0.224, 0.225]), or are the exact mean and standard deviation constants calculated globally for all pixels in each color channel of the current medical image training set. For the corresponding question text, a word segmenter is used to segment the text string, converting the natural language text into a discrete word tag sequence. If the length of the word tag sequence is less than the preset maximum text length, a placeholder is added to the end of the sequence; if the length of the word tag sequence is greater than the preset maximum text length, the sequence is truncated.

[0023] After preprocessing the medical images and the question text, they are input into separate pre-trained encoders. Specifically, the visual encoder uses a pre-trained visual Transformer model. First, the resolution-adjusted medical images are divided into multiple non-overlapping image patch sequences. A linear projection layer is used to perform feature dimensionality reduction and mapping on each image patch, generating a corresponding patch embedding representation. Based on this, a learnable global visual classification label is concatenated at the beginning of the patch embedding sequence, and a one-dimensional learnable position encoding tensor is superimposed on all elements in the sequence to preserve the spatial location information of the image patches in the original two-dimensional image space. Subsequently, the concatenated sequence with added position encoding is input into a stacked Transformer module containing a multi-layer multi-head self-attention mechanism and a feedforward neural network for feature extraction. The initial visual features and global visual label output by the visual encoder are expressed mathematically as follows: ; In the above formula, This represents the initial visual feature sequence matrix output by the visual encoder; This indicates the length of the visual label sequence generated after the input medical image has been segmented, which includes all image patch features and the spliced ​​global visual labels; This indicates the channel dimension of the visual feature in the hidden layer; Represents the real number space. This initial visual feature sequence Includes corresponding global visual markers This global visual marker It aggregates the global structure and appearance information of the entire medical image.

[0024] Simultaneously, the text encoder employs a pre-trained RoBERTa model. The segmented and aligned sequence of question text word tags is input into the text encoder's embedding layer to extract word embedding features, which are then superimposed with positional and paragraph embedding information. Subsequently, the fused text embedding sequence is input into a network containing multiple bidirectional Transformer encoding layers, where a self-attention mechanism captures the semantic dependencies between words within the question text's context. The initial text features and global text tags output by the text encoder are mathematically represented as follows: ; In the above formula, This represents the initial text feature sequence matrix output by the text encoder; This indicates the length of the sequence of word tags in the question text, which includes all lexical tags as well as specific global text tags; This represents the hidden layer channel size of the text features. This is the initial text feature sequence. Includes the corresponding global text tag This global text tag It captures the overall global semantic information of the input question. It is understandable that the initial visual features... and initial text features Before being passed to the subsequent multimodal fusion layer, modality type embedding tensors are added to both. These modality type embedding tensors have the same characteristics as... and Same channel dimension This is used to explicitly distinguish whether features originate from medical image modalities or natural language text modalities in subsequent multimodal networks. The modality type embedding tensor specifically consists of two independent learnable parameter matrices (corresponding to visual and text modalities, respectively) initialized by random sampling from a truncated normal distribution with a mean of 0 and a standard deviation of 0.02. During the end-to-end training phase of the model, these two parameter matrices are updated synchronously and iteratively as part of the model parameters through the backpropagation gradient of the joint loss function.

[0025] Step S2: A dynamic alignment adaptation module is introduced at the top layer of the multimodal fusion layer. The initial visual features and initial text features are input into the multimodal fusion layer to perform delayed cross-modal alignment. Specifically, the average value of the pooled text and visual features is calculated as a shared alignment anchor point, and the difference between the shared alignment anchor point and the center of each modality is calculated to obtain the residual alignment signal. At the same time, a dynamic scalar gating is generated using the text global marker, visual global marker and their difference in the current layer to control the intensity of the residual alignment signal updated in the features.

[0026] like Figure 3 The DAA module details shown illustrate that in a multimodal fusion network, the network as a whole comprises multiple stacked fusion layers. Traditional cross-modal fusion introduces cross-attention mechanisms in all layers to achieve dense interaction of visual and textual features. However, in this embodiment, the backbone of the multimodal fusion layer retains a certain degree of independent computational logic when processing visual and textual features. To avoid premature assimilation of superficial pathological details in cross-modal interaction, this invention exclusively places the dynamic alignment and adaptation module in the last two layers of the multimodal fusion layer. This means that before reaching the penultimate layer, the initial visual and initial textual features are primarily updated with self-attention within their respective modal spaces, without forced cross-modal space constraint alignment.

[0027] When entering a specific upper semantic layer equipped with a dynamic alignment adaptation module (e.g., the last two layers of a multimodal fusion network), an intramodal adaptive adaptation step is first performed. Parallel low-rank residual adapters are introduced on both the text feature processing branch and the visual feature processing branch. These low-rank residual adapters consist of sequentially connected dimensionality-reducing linear layers, nonlinear activation functions, and dimensionality-increasing linear layers. By compressing high-dimensional features into a low-dimensional bottleneck space and then restoring them, the specific residual representation of the current modality in the deep network can be learned while reducing the number of parameters.

[0028] In specific implementation, the dimension reduction linear layer in the low-rank residual adapter will have an input dimension of... Feature vectors compressed to the bottleneck dimension ,in The preset dimensionality reduction ratio (preferably, The value is set to 4 or 8 to filter out redundant background noise and force the model to learn compact modality-specific representations. Subsequently, GELU (Gaussian Error LinearUnit) is used as the nonlinear activation function to introduce nonlinear mapping capabilities while preserving subtle pathological features. Finally, the feature dimension is restored to its original value through an upscaling linear layer. This low-rank bottleneck design not only significantly reduces the computational cost of parameters in deep multimodal interactions, but also effectively avoids overfitting of sparse, weak lesion clues in medical images in complex cross-modal projections.

[0029] When adding the adapter's output to the original features, control is achieved using a learnable scalar gating initialized to zero. The text branch update formula for intra-modal adaptive adaptation is as follows: ; In the above formula, This represents the updated text feature matrix after intramodal adaptive adaptation; This represents the text feature matrix input to the current layer; The learnable scalar gating parameter representing the text branch is initialized to 0. This represents the hyperbolic tangent activation function, used to map the values ​​of scalar gate parameters to the interval between -1 and 1; This represents a low-rank bottleneck residual adaptation network function deployed on the text branch, used to extract text modality-specific residual update information. Because... Initialized to 0, the output of this adaptation branch does not affect the main features in the early stages of training, thus ensuring the stability of pre-trained feature loading. As the backpropagation optimization process progresses, the model progressively activates independent residual updates for text features.

[0030] Simultaneously, the intramodal adaptive adaptation update formula for the visual branch is as follows: ; In the above formula, This represents the updated visual feature matrix after intramodal adaptive adaptation. This represents the visual feature matrix input to the current layer; This represents the learnable scalar gating parameter of the visual branch, which is initialized to 0. Represents the hyperbolic tangent activation function; This represents a low-rank bottleneck residual adaptation network function deployed on the vision branch, used to capture specific information for the deep semantics of medical images within the vision modality.

[0031] After completing the intramodal adaptive update respectively, based on the... and Perform cross-modal hierarchical alignment. First, perform average pooling on the sequence dimension for the text and visual branch features after intramodal adaptation to calculate the feature center representations for each modality. The center representation of the text branch is denoted as... The central representation of the visual branch is denoted as Subsequently, the two center representation vectors are added together and their average is taken to construct a shared semantic anchor point located in the middle of the two modal feature subspaces. The formula for constructing the shared alignment anchor point is as follows: ; In the above formula, This represents the calculated shared semantic anchor vector, whose dimension is the same as the dimension of the single-layer hidden features. ; Represents the text feature sequence matrix of the current layer The text center vector is obtained by averaging along the sequence dimension; Represents the visual feature sequence matrix of the current layer The visual center vector is obtained by averaging along the sequence dimension. This shared semantic anchor serves as an intermediate reference target in the cross-modal alignment process.

[0032] Next, the difference vectors between the shared semantic anchor point and the text center and visual center are calculated separately to obtain the residual signals that need to be aligned. The residual alignment signal on the text side is calculated as follows: The residual alignment signal on the visual side is calculated as follows: After acquiring the residual alignment signal, instead of directly superimposing it onto each marker in the sequence, a dynamic scalar gating mechanism is generated to control the strength of the alignment feature fusion. The generation of this gating mechanism depends on the global markers of the current layer's visual and textual data. Specifically, the global textual markers of the current layer are extracted. and visual global markers The difference vector between the two values ​​is calculated, and then these three vectors are concatenated as the input to the gating network. The generation formula for dynamic scalar gating is as follows: ; In the above formula, This represents the dynamic alignment strength coefficient predicted by the gating network, and its value ranges from 0 to 1. This represents the Sigmoid non-linear activation function; This represents a multilayer perceptron network, which includes fully connected layers and activation layers, used to capture the nonlinear mapping relationships between multiple input vectors; Indicates the current number The text global marker feature vector of the layer; Indicates the current number The visual global labeled feature vector of the layer; This represents an element-level difference vector between text global markers and visual global markers; This represents the learnable bias parameters in a multilayer perceptron network.

[0033] In this gating mechanism, a difference vector is introduced. It has specific medical protective significance. In medical visual question answering, when the input question involves extremely small or marginal lesions (such as a small nodule in the left lower lung), the initial visual global labeling... Often, the local details are not sufficiently focused on, causing them to clash with global text markers. A significant semantic gap exists. In this case, a large difference vector input will cause the multilayer perceptron network to output a smaller value. The value is adjusted to actively reduce the alignment and fusion strength of the current layer. This adaptive suppression mechanism based on semantic gap can effectively prevent the already weak fine-grained lesion features in medical images from being smoothed or swallowed up by the dominant semantics of the text during forced cross-modal alignment, ensuring that the model has sufficient hierarchical depth to gradually mine and retain these key diagnostic evidences.

[0034] After generating dynamic scalar gating coefficients, these coefficients are broadcast along the feature sequence dimension, then multiplied by the corresponding residual alignment signals, and finally superimposed onto the intramodal adapted feature sequence. The final hierarchical alignment update formula for text features is as follows: ; In the above formula, This represents the new text feature sequence matrix that is finally output after the complete delayed cross-modal hierarchical alignment operation; This represents the text feature matrix that has undergone intramodal adaptive updating in the early stage; The dynamic scalar gating coefficient representing the control of residual strength; This represents the text-side residual alignment signal tensor calculated from the shared alignment anchor and the text center representation.

[0035] The final hierarchical alignment update formula for visual features is as follows: ; In the above formula, This represents the new visual feature sequence matrix that is finally output after the complete delayed cross-modal hierarchical alignment operation; This represents the visual feature matrix that has undergone intramodal adaptive updating in the early stage; Represents the dynamic scalar gating coefficient; This represents the residual alignment signal tensor on the visual side. Through the above steps, a consistent adjustment of cross-modal semantic distribution is achieved at the top layer of multimodal fusion. Delaying this alignment operation to a deeper layer can effectively protect the fine-grained lesion textures of medical images extracted in early network layers (such as nodule edge information, interstitial space texture, etc.), preventing these low-frequency details that are crucial for diagnosis from being assimilated during shallow feature fusion.

[0036] Step S3: Construct an attribute-aware reverse reasoning module. In the forward prediction stage, obtain pseudo-labels of clinical prior attributes and map them as attribute embeddings. Through a gating mechanism based on global visual features, inject the attribute embeddings into the aligned visual features to generate an enhanced visual representation. Combine the aligned text features to predict candidate answers.

[0037] Combination Figure 4 The A2R2 module details shown demonstrate that forward inference is performed after processing by the multimodal fusion layer. Clinical medical diagnosis often relies on specific anatomical locations and imaging acquisition techniques; therefore, this method introduces a clinical prior attribute mechanism. Specifically, clinical prior attributes include organ category pseudo-labels and imaging modality pseudo-labels generated from training images offline using an external biomedical multimodal pre-trained model (such as the BiomedCLIP model). During offline data preprocessing, all medical image samples are input into the pre-trained model, which outputs the highest-confidence organ classification (e.g., liver, lung, brain) and imaging modality classification (e.g., CT, MRI, X-ray) for each image. These offline-generated pseudo-labels are stored in a label library as coarse-grained clinical prior knowledge. In this embodiment, the external model used is specifically a biomedical multimodal pre-trained network based on the PubMedCLIP architecture. The candidate space for the organ category pseudo-labels consists of a predefined set of medical anatomical terms, limited to 79 standard organ classifications; the classification dictionary for the imaging modality pseudo-labels is limited to 10 mainstream clinical imaging modalities such as CT, MRI, X-ray, ultrasound, and PET. The aforementioned fixed total number of categories precisely constitutes the discrete dimensional space of the label dictionary mapping.

[0038] To ensure the reliability of clinical prior attributes and avoid noise from external models interfering with the diagnostic logic of the question-answering task, a dynamic confidence threshold determination mechanism is introduced when generating the pseudo-labels. Specifically, a pseudo-label confidence threshold is set. (Preferably, set) When the highest confidence level of the output of the externally pre-trained model is greater than or equal to... When the corresponding category is specified, it is stored in the label library as a pseudo-label for the organ or imaging modality; when the highest confidence level is less than 10%, the corresponding category is stored in the label library. If the prior attributes of the image are not specified, they are uniformly labeled as the Unknown category. Correspondingly, the learnable vocabulary embedding matrix contains zero vector embeddings specifically initialized for the unknown category, so that when the confidence level is insufficient, the relevance fusion weights calculated by the above formula are close to zero, thereby adaptively closing the injection path of unreliable prior attributes and ensuring the stability of forward prediction.

[0039] In the online forward prediction phase, organ category pseudo-labels and imaging modality pseudo-labels corresponding to the current medical image are retrieved from the database. Then, using the learnable vocabulary embedding matrix contained in the network, the discrete classification label identifiers are mapped into continuous dense attribute embedding vectors to obtain the organ attribute embedding representation. Modal attribute embedding representation Subsequently, the aligned visual global representation located at the top layer of the network after processing in step S2 is... (i.e., from) The extracted global category labels, along with the aforementioned attribute embedding vectors, are input into the gating mechanism network to adaptively calculate the attribute relevance fusion weights. The formula for calculating the organ relevance weights is as follows: ; In the above formula, This represents the organ attribute correlation fusion weights calculated adaptively by the model, and it is a scalar value; This represents a gated unit module containing the Sigmoid activation function; This represents a multilayer perceptron network whose input is the visual global representation. Embedded representation of organ attributes The splicing tensor is used to calculate the correlation between visual image features and provided prior knowledge of organs.

[0040] Similarly, the formula for calculating the modal correlation weight is as follows: ; In the above formula, This represents the adaptively calculated imaging modal attribute correlation fusion weight scalar; Indicates a gating unit; Represents a multilayer perceptron network; This represents the aligned visual global representation vector; The embedding vector represents the imaging modal attributes. After calculating these two dynamic weights, the prior attribute information is superimposed onto the global visual representation using a soft-injection additive superposition method, generating an enhanced visual representation optimized for the current problem diagnosis. The calculation formula for the enhanced visual representation is as follows: ; In the above formula, This represents an enhanced visual representation vector that incorporates prior clinical attributes; This represents the initial visual global representation features after alignment without the injection of prior knowledge; Indicates the weight of organ attribute fusion; Represents the organ attribute embedding vector; Indicates the modal attribute fusion weights; This represents the imaging modality attribute embedding vector. Through this weighted soft injection mechanism, the model can adaptively and selectively absorb externally provided organ category and imaging modality information according to the needs of the current visual features.

[0041] Gain enhanced visual representation Then, it is aligned with the global text representation. (i.e., from) The extracted global category labels are concatenated or fused element-wise, and then fed into a Visual Question Answering (VQA) Head composed of fully connected layers. This head outputs a dimension equal to the size of the entire predefined candidate answer vocabulary. Multi-class logistic prediction vectors (Logits distribution): ; In the above formula, Indicates that the parameter is The forward prediction network computes the candidate answer prediction probability vector output by the network. The enhanced visual representation features input to the classification head; This represents the aligned text global representation features input to the classification head. Each element in this prediction distribution vector represents the predicted probability score for the corresponding candidate answer index. The category index with the highest score is selected as the candidate answer predicted by the model in the forward pass.

[0042] Step S4: In the reverse reconstruction stage, the embedded representation of the predicted candidate answer is concatenated with the original visual global features that have been aligned but not injected with the attribute embedding, and the global semantic representation of the original question is reconstructed through the reverse reconstruction network.

[0043] Medical visual question answering tasks suffer from a serious problem of unidirectional logic. To build bidirectional consistency, this method simulates the diagnostic process of a clinician's "backtracking verification." In the forward prediction output... Then, the highest-scoring answer category index is selected using the Argmax operation. Using a separately configured answer word embedding lookup table, this highest-scoring answer index is mapped to a continuous semantic anchor embedding vector, denoted as... In the initialization phase, the weight matrix of the answer word embedding lookup table is pre-loaded by extracting static context word embedding features from all natural language text strings in the predefined candidate answer dictionary using a pre-trained language model (such as pre-trained RoBERTa) that is from the same source as the text encoder; or it is based on this as the initial weight and fine-tuned and updated under the supervision of the mean squared error loss of the subsequent reverse intent reconstruction task, so as to ensure that the semantic anchor embedding vector has accurate and rich natural language prior semantic information.

[0044] Subsequently, the semantic anchor was embedded Aligned with uninjected visual global features Perform tensor splicing operations along the channel dimension. During this process, strict adherence to... Rather than containing prior information This approach strips away strong pseudo-label attributes, forcing the model to rely directly and purely on the original medical visual evidence and the currently derived answer conclusion to reconstruct the original question intent in the reverse path. The concatenated tensor is input into the reverse reconstruction network to calculate the global semantic representation of the original question. The reverse reconstruction formula is as follows: ; In the above formula, This represents the global semantic representation vector of the problem intent predicted by the reverse reconstruction network output, and its dimension is the same as the dimension of the global label of the original text. This represents an inverse multilayer perceptron network unit with multiple hidden layers and nonlinear activation functions; This represents the original visual global feature vector after delayed cross-modal alignment without any prior attribute soft injection. This represents the semantic anchor embedding vector obtained by mapping the highest-scoring candidate answer predicted from the forward pass.

[0045] To achieve strict semantic space alignment, the reverse reconstruction network A dimensional funnel structure was employed. This was due to the input of raw visual global features. With semantic anchor embedding The total dimension after concatenating the channel dimensions is: (in (for the feature dimension of a single-modal hidden layer), the First, the dimensions of the stitched features are reduced by the first fully connected layer. Dimensionality reduction mapping to intermediate transition dimensions Subsequently, a joint representation is extracted through a nonlinear activation layer, and finally, a second fully connected layer is applied to the dimension. Perform linear rotation and projection of the semantic space within the inner dimension, with the output dimension precisely defined as... Reconstructed vector This dimension mapping mechanism ensures that the reconstructed semantic representation can be matched with the true initial global text markup output by the text encoder. Being in the same dimensional space, the alignment of subjective and objective intentions based on mean square error has a solid mathematical tensor foundation.

[0046] Step S5: Perform end-to-end optimization of the model using a joint loss function. The joint loss function includes the main task loss, the cosine alignment loss between text and visual global labels, the clinical attribute-assisted classification loss, and the mean squared error loss used to supervise the consistency between the reconstructed global semantic representation and the original problem global representation.

[0047] This embodiment employs an end-to-end joint training mode, where the backpropagation gradient during the training phase is composed of multiple constraints. First, the main task loss for medical visual question answering is calculated using a binary cross-entropy loss function to determine the consistency between the predicted probability distribution and the soft-label target. The main task loss formula is as follows: ; In the above formula, This represents the scalar value of the loss in the main visual question answering task. This indicates a calculation operation that incorporates the binary cross-entropy loss function computed using the Sigmoid algorithm. This represents the prediction distribution tensor of the network's forward output without Softmax activation; The soft label vector represents the corresponding true labeled answer. In the medical question-and-answer dataset, since some questions have multiple valid reference answers, the soft label is obtained by normalizing the voting scores of multiple doctors. This represents the total number of categories in the answer space. Multiplying this value by the loss scalar scales the loss by an order of magnitude to prevent the cross-entropy gradient from being too small in a large answer space.

[0048] Secondly, a delayed alignment loss is used to supervise the dynamic alignment adaptation module, achieved by constraining the cosine similarity between the text and the visual global marker. The formula is as follows: ; In the above formula, This represents the scalar value of the delayed cross-mode alignment loss; This is represented as the set multimodal fusion layer. The normalized weight coefficients assigned to the layers are non-zero in the last two layers due to the dynamic alignment configuration. This represents the operation of calculating the cosine similarity of two input tensors along the channel dimension; and They represent the first time. The text global label feature vector output by the layer is compared with the visual global label feature vector. By applying this loss term, the cosine distance between the visual and text representations in the high-level semantic space is explicitly reduced.

[0049] Simultaneously, for the attribute injection process in the forward prediction stage, separate organ classification heads and modality classification heads are configured for the aligned visual global features, and cross-entropy loss, i.e., organ loss, is calculated based on the pseudo-labels generated offline. and modal loss For the reverse reconstruction stage, the mean squared error loss is calculated to constrain the reverse reconstruction result to approximate the original problem intent. The mean squared error loss is denoted as... ,calculate Compared with the actual initial global text markup output by the encoder The mean of the element-wise squared differences between them. Integrating all the above loss units, a final joint loss function model is constructed for end-to-end optimization. The formula for the total loss function is as follows: ; In the above formula, This represents the total joint loss scalar value used for backpropagation parameter updates in the entire deep learning model; This represents the binary cross-entropy loss of the main task; This represents the delayed alignment cosine similarity loss; and These represent the cross-entropy loss of organ prior classification and the cross-entropy loss of modality prior classification, respectively. This represents the mean squared error loss of the reverse intention reconstruction; This represents the hyperparameter weight values ​​set for the delay alignment loss; This represents the weight values ​​of the hyperparameters uniformly set for the clinical attribute-assisted classification loss; This represents the hyperparameter weight values ​​set for the mean squared error loss in reverse reconstruction. For example... Figure 7 As shown, in a specific embodiment, after hyperparameter optimization, the weight of the cosine alignment loss is set. Weights of clinical attribute-assisted classification loss Mean squared error loss weights for reverse intention reconstruction This ratio of hyperparameter settings ensures that reverse intent reconstruction and attribute-assisted classification provide a gentle regularization guidance for the main network feature updates, avoiding gradient overwhelming the main task loss.

[0050] Specifically, the weight ratios are set based on the differences in the gradient magnitudes generated by the various losses during backpropagation. (Main task loss) The gradient of this component, responsible for fitting the final medical answer classification, directly determines the model's diagnostic accuracy and is therefore used as a baseline weight (implicitly 1.0); delayed alignment loss. Used to align view features at deeper levels, setting This is because excessively high alignment constraints cause visual branches to lose modality specificity (i.e., pathological texture) in medical images, while excessively low constraints fail to produce effective cross-modal collaboration; clinical attribute-assisted classification loss ( and ) and reverse intention reconstruction loss All belong to auxiliary regularization terms, set and The aim is to control the magnitude of the gradient generated by it to be within the magnitude of the gradient of the main task. arrive The ratio is between [specific values]. This ratio provides sufficient structural priors and logical backtracking constraints for the network, while strictly preventing auxiliary tasks from interfering with the convergence trajectory of the main diagnostic task in the later stages of end-to-end optimization.

[0051] During the end-to-end optimization of the network based on the aforementioned joint loss function, the gradient of the total loss with respect to all learnable parameters of the network is calculated, and the parameters are updated using the AdamW optimizer. Simultaneously, a cosine annealing learning rate scheduling mechanism with a warm-up strategy is employed to control the decay rate of the learning rate. During the inference and testing phase, gradient backpropagation is stopped, and the network directly executes the forward path from steps S1 to S3. The final result of the medical visual question answering is based on the category with the highest probability output from the classification head and is output to the interactive page.

[0052] Combination Figure 5 and Figure 6 As shown, the effective technical effect of this solution can be observed through specific experiments. When receiving the question "Does the image contain more than one organ system?" from a brain CT image, the baseline model and the model provided in this embodiment are compared and analyzed. Due to global feature confusion and lack of logical backtracking verification, the baseline model generates the "yes" result with the highest probability of answering. However, by exclusively setting the dynamic alignment and adaptation module in the last two layers of the multimodal fusion layer and verifying the rationality of the answer in the reverse reconstruction, this method outputs "no", avoiding excessive smoothing and assimilation of microscopic anatomical features. When judging the distribution of lesions, when asked "Which lobe of the brain is the lesion located in?", the attention weight distribution matrix of the baseline model is discrete and the extreme value is focused on the right parietal lobe region. However, the attribute prior used in this embodiment not only pre-anchors the current "brain" macrostructure distribution, but its forward output combines the visual image matrix and organ correlation fusion weights, so that the attention weight distribution matrix strictly converges on high-frequency lesion features, and finally outputs the highest-scoring category index, which accurately corresponds to "right frontal lobe", effectively avoiding attention shift. When faced with highly open-ended and complex questions, such as "What is below the right diaphragm?", the baseline model mismatches visual evidence with the "gastric bubble," which is typically located in a similar area. This method, by imposing a reverse mean squared error penalty during inference, requires the answer to effectively reverse-generate the original question's encoding matrix, prompting the network to correct the feature mapping and ultimately output "free gas." This technically demonstrates the decisive role of reverse constraints in constructing a logical causal loop between complex pathological evidence and diagnostic answers.

[0053] Based on the same inventive concept, this embodiment also provides a medical visual question-answering system based on delayed cross-modal alignment and reverse reasoning. This system specifically includes a data interface unit, a single-modal feature extraction module, a dynamic alignment fusion module, an attribute-aware reverse reasoning module, and an answer output module. The data interface unit connects to an image storage system to extract the target medical image and accompanying text. The single-modal feature extraction module deploys a visual Transformer and RoBERTa network to map the image and text to a high-dimensional feature space. The dynamic alignment fusion module loads a multi-layer attention network and deploys a low-rank tensor decomposition network structure and alignment anchor point calculation unit on the top two specified network nodes. The attribute-aware reverse reasoning module calls a parallel gated neural network to load the modality category pseudo-label probability vector calculated offline, and completes the reverse reconstruction calculation of the predicted intent through feature tensor concatenation and fully connected mapping of the multi-layer perceptron. The answer output module performs logistic regression calculation and looks up the data in a dictionary, converting the floating-point array index with the highest probability back into a natural language phrase, and outputs the final medical question-answering conclusion.

[0054] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A medical visual question answering method based on delayed cross-modal alignment and backward reasoning, characterized in that, Includes the following steps: Step S1: Obtain the medical image and the corresponding question text, and extract the initial visual features and their global visual labels, and the initial text features and their global text labels through the pre-trained encoder. Step S2: A dynamic alignment adaptation module is introduced at the top layer of the multimodal fusion layer. The initial visual features and initial text features are input into the multimodal fusion layer to perform delayed cross-modal alignment. Specifically, the average value of the pooled text and visual features is calculated as a shared alignment anchor point, and the difference between the shared alignment anchor point and the center of each modality is calculated to obtain the residual alignment signal. At the same time, a dynamic scalar gating is generated using the text global marker, visual global marker and their difference in the current layer to control the intensity of the residual alignment signal updated in the features. Step S3: Construct an attribute-aware reverse reasoning module. In the forward prediction stage, obtain pseudo-labels of clinical prior attributes and map them as attribute embeddings. Through a gating mechanism based on global visual features, inject the attribute embeddings into the aligned visual features to generate an enhanced visual representation. Combine the aligned text features to predict candidate answers. Step S4: In the reverse reconstruction stage, the embedded representation of the predicted candidate answer is concatenated with the original visual global features that have been aligned but not injected with the attribute embeddings, and the global semantic representation of the original question is reconstructed through the reverse reconstruction network. Step S5: The model is optimized end-to-end using a joint loss function, which includes the main task loss, the cosine alignment loss between the text and the visual global labels, the clinical attribute-assisted classification loss, and the mean squared error loss used to supervise the consistency between the reconstructed global semantic representation and the global representation of the original question.

2. The method according to claim 1, characterized in that, In step S2, before performing the cross-modal hierarchical alignment, an intra-modal adaptation step is also included: In the selected fusion layer, parallel low-rank residual adapters are introduced on the text branch and the visual branch respectively, and learnable scalar gating initialized to zero is used to progressively activate independent residual updates for text features and visual features during training.

3. The method according to claim 1, characterized in that, The dynamic alignment and adaptation module is exclusively located in the last two layers of the multimodal fusion layer to prevent superficial pathological details from being assimilated prematurely in cross-modal interactions, thereby preserving fine-grained lesion information in medical images.

4. The method according to claim 1, characterized in that, The clinical prior attributes in step S3 include organ category pseudo-labels and imaging modality pseudo-labels generated offline from training images using an external biomedical multimodal pre-trained model.

5. The method according to claim 1, characterized in that, The specific method for embedding the attribute into the aligned visual feature in step S3 is as follows: The visual global representation after delay cross-modal alignment is input into the gating network along with the organ category attribute embedding and the imaging modality attribute embedding, respectively. The corresponding relevance weights are calculated, and the attribute embeddings are superimposed on the visual global representation by soft injection.

6. The method according to claim 1, characterized in that, In step S4, the answer with the highest forward prediction score is selected as the semantic anchor point. This anchor point is then concatenated with the visual global features that have not been injected with clinical prior attributes and input into a multilayer perceptron. The mean squared error loss is used to guide the model to backtrack and verify the questioning intent through visual evidence.

7. The method according to claim 1, characterized in that, In the joint loss function described in step S5, the weight of the cosine alignment loss is set to 0.2, and the weights of the clinical attribute-assisted classification loss and the mean square error loss of reverse intent reconstruction are both set to 0.

1.

8. The method according to claim 1, characterized in that, In step S1, before the visual features and text features enter the multimodal fusion layer, modality type embeddings are added to both of them to allow the fusion network to distinguish different modality sources.

9. The method according to claim 1, characterized in that, The visual encoder uses a pre-trained ViT, and the text encoder uses a pre-trained RoBERTa.

10. A medical visual question-answering system based on delayed cross-modal alignment and backward reasoning, characterized in that, include: The feature extraction module is used to obtain the initial visual and textual features of medical images and question text; A dynamic alignment and fusion module is used to perform delayed cross-modal alignment and intramodal adaptation as described in any one of claims 1-3; An attribute-aware reverse reasoning module is used to perform forward prior injection and reverse intent reconstruction as described in any one of claims 1 and 4-6; And an answer output module, used to provide medical visual question answering prediction results based on the output of the attribute-aware reverse reasoning module.

Images