Multi-modal fusion medical intelligent reasoning method and device
By employing a multimodal fusion-based intelligent medical reasoning method, features from medical images and text are extracted and collaborative representation is achieved using a self-attention mechanism. This solves the problem of single-modal information input, realizes deep semantic alignment and joint reasoning between medical images and text, and improves the accuracy and interpretability of medical diagnosis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-05
- Publication Date
- 2026-04-07
AI Technical Summary
Existing medical AI systems mostly rely on single-modal information input, resulting in a disconnect between medical images and clinical text information. They lack cross-modal information fusion mechanisms, making it difficult for visual models to understand medical record semantics. General multimodal models have insufficient feature extraction capabilities in medical imaging tasks and lack structured features and decision interpretability.
By extracting the structured deep features of medical images and the semantic encoding of patient information, we can achieve the collaborative representation of medical visual features and text features. We can use a multimodal self-attention mechanism for disease prediction and assisted diagnosis, and combine images, text and physiological indicators for deep semantic alignment.
It significantly improves the model's adaptability, accuracy, and interpretability in medical scenarios, enhances the feature extraction capabilities in the medical field, and achieves unified cognition and decision-making consistency between images and text.
Smart Images
Figure CN121809652A_ABST
Abstract
Description
Technical Field
[0001] This document relates to the interdisciplinary field of artificial intelligence and medical informatics, and in particular to a multimodal fusion medical intelligent reasoning method and device. Background Technology
[0002] Existing medical AI systems largely rely on single-modal information input, such as using only text medical records or only image models for analysis. These "single-modal" systems have significant limitations when handling complex clinical scenarios: Information fragmentation: Text models (such as LLM) cannot capture the spatial and pathological features in medical images, while visual models have difficulty understanding the semantics of medical records.
[0003] Insufficient domain adaptability: Although general multimodal models (such as Qwen3-VL, GPT-4V, etc.) have cross-modal capabilities, their feature extraction capabilities in medical imaging tasks are limited because the visual encoder training data is biased towards natural images, resulting in insufficient reliability of diagnostic inference results.
[0004] Specifically, (1) the problem of the separation between medical images and clinical text information.
[0005] Most mainstream medical intelligent reasoning systems currently rely on a single modality (such as being based solely on text medical records or solely on images), lacking cross-modal information fusion mechanisms. While large text models (LLMs) possess powerful language understanding capabilities, they cannot effectively analyze lesions, morphology, and pathological clues in medical images; and while visual models can extract image features, they struggle to understand clinical semantics and contextual logic, resulting in a lack of global consistency and clinical interpretability in model output.
[0006] (2) The problem that general multimodal models lack adaptability to the medical field.
[0007] Existing general-purpose multimodal models (such as Qwen3-VL, GPT-4V, etc.) primarily train their visual encoders based on natural images. They lack the ability to extract professional features from medical images (especially fundus, CT, MRI, etc.), resulting in insufficient sensitivity to features such as subtle lesions, tissue layers, and optical density, which limits the effectiveness of the models in medical scenarios.
[0008] (3) The medical reasoning process lacks structure and decision interpretability.
[0009] Traditional "pure text-driven" medical models cannot effectively incorporate the joint representation of structured physiological indicators (such as blood pressure, blood sugar, eGFR, etc.) and imaging features during the reasoning process. This leads to a disconnect between the reasoning logic and the doctor's diagnostic process, making the results difficult to interpret and verify, and failing to reflect the hierarchical relationship of medical knowledge.
[0010] Therefore, there is an urgent need for an intelligent reasoning framework that can deeply integrate professional medical visual features with clinical semantic information in order to achieve unified cognition and decision-making based on "images, indicators, and text". Summary of the Invention
[0011] The purpose of this invention is to provide a multimodal fusion medical intelligent reasoning method and device, which aims to solve the above-mentioned problems in the prior art.
[0012] This invention provides a multimodal fusion-based medical intelligent reasoning method, comprising: Extract structured deep features from medical images to obtain medical visual features; semantically encode patient information to obtain medical text features; The medical text features and the medical visual features are aligned and weighted within the fusion layer to achieve collaborative representation of medical visual features and medical text features. Based on the aforementioned collaborative representation, disease prediction, risk assessment, or assisted diagnosis can be achieved through a multimodal self-attention mechanism.
[0013] This invention provides a multimodal fusion medical intelligent reasoning device, comprising: The extraction and encoding module is used to extract the structured deep features of medical images and obtain medical visual features; and to semantically encode patient information to obtain medical text features. The fusion module is used to perform feature alignment and weight optimization of the medical text features and the medical visual features within the fusion layer, so as to achieve collaborative representation of medical visual features and medical text features. The processing module is used to complete the output of disease prediction, risk assessment or assisted diagnosis based on the collaborative representation through a multimodal self-attention mechanism.
[0014] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-described multimodal fusion medical intelligent reasoning method.
[0015] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described multimodal fusion medical intelligent reasoning method.
[0016] The embodiments of the present invention achieve professional extraction of visual features; and through a multimodal fusion mechanism, medical image features, physiological indicators and medical record text are deeply semantically aligned and jointly inferred, thereby significantly improving the model's adaptability, accuracy and interpretability in medical scenarios. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of the multimodal fusion medical intelligent reasoning method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a multimodal fusion medical intelligent reasoning device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0019] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.
[0020] Method Implementation Examples According to embodiments of the present invention, a multimodal fusion medical intelligent reasoning method is provided. Figure 1 This is a flowchart of the multimodal fusion medical intelligent reasoning method according to an embodiment of the present invention, such as... Figure 1 As shown, the multimodal fusion medical intelligent reasoning method according to an embodiment of the present invention specifically includes: Step S101: Extract structured deep features from medical images to obtain medical visual features; semantically encode patient information to obtain medical text features; specifically including: The RetFound model is used to extract structured deep features from medical images to obtain medical visual features; Patient information is semantically encoded using the Qwen3-VL language module to obtain medical text features.
[0021] The patient information specifically includes: patient medical records, patient vital signs data, and patient physiological parameters.
[0022] Step S102 involves aligning and optimizing the medical text features and medical visual features within the fusion layer to achieve collaborative representation of medical visual features and medical text features; specifically including: The medical text features and the medical visual features are concatenated together, and modal identifiers and positional encodings are introduced. Feature alignment and weight optimization are performed through multi-layer fully connected layers to achieve collaborative representation of medical visual features and medical text features.
[0023] In one example, RetFound is used to extract the visual token (corresponding to the aforementioned medical visual features), and a linear projection is performed through a linear layer to obtain the visual token (corresponding to the aforementioned medical visual features, denoted as V). The word segmenter of the Qianwen Big Language Model itself is used. The input text is converted into a series of text token IDs and mapped to word embedding vectors to obtain the text token (corresponding to the aforementioned medical text features, denoted as T). The visual token (V) and text token (T) are concatenated and positional encoding is introduced, followed by feature alignment through multiple fully connected layers.
[0024] Step S103: Based on the collaborative representation, the output of disease prediction, risk assessment or assisted diagnosis is completed through a multimodal self-attention mechanism.
[0025] The network structure of the multimodal self-attention mechanism specifically includes: N Transformer blocks, each containing multi-head self-attention + feedforward network FFN + residual and LayerNorm, and inserting bidirectional Cross-Attention layers to achieve finer-grained interaction between text and vision. Different resolution visual tokens are used in different Transformer blocks to take into account both local details and global context. Some attention heads are used to constrain only within the same modality, and some attention heads are used to constrain cross-modality. Attention capacity is adaptively allocated during training. Local window attention is used for medical visual features to reduce computation, while global attention is used for cross-modality.
[0026] Step S103 specifically includes: based on the collaborative representation, after processing through all interaction layers of the multimodal self-attention mechanism, the last collaborative representation in the sequence is sent to the language model output head, wherein the structure of the language model output head is a linear layer + Softmax; The hidden state is mapped to a probability distribution across the entire vocabulary through the output head of the language model, and the answer is generated in an autoregressive manner: the probability distribution of the first output token is predicted based on the entire input sequence, and the most likely token is sampled / selected from it. This generated token is then appended to the end of the input sequence as new input to predict the next token. This process is repeated until an end token is generated, forming a complete answer. In the entire generation process, each step of the prediction depends on the original image token and all the generated text tokens.
[0027] In one instance, the specific processing includes the following: (1) Input indicates preparation.
[0028] The visual token V (N_v × D) and the text token T (N_t × D) are concatenated into a joint sequence X = [T; V], and a modality token type embedding and position encoding are added to each token.
[0029] (2) Design of multimodal self-attention layer.
[0030] Basic structure: N Transformer blocks, each layer containing multi-head self-attention + feedforward network (FFN) + residuals and LayerNorm. Self-attention computation allows cross-modal information interaction (text tokens can directly attend to visual tokens): Attn(Q,K,V) = softmax(QK^T / sqrt(d_k)) V, where Q / K / V all come from X (joint sequence). Where [1], Attn(Q,K,V) means Q, K, V, T, and sqrt(d_k); A bidirectional Cross-Attention layer (text query → visual key / value) is inserted to enable finer-grained interaction between text and vision. Some attention heads are specifically constrained to operate only within the same modality (intra-modal), while others are dedicated to cross-modal (inter-modal) interactions, with attention capacity adaptively allocated during training. To reduce computational burden, sparse attention is introduced, using local window attention for visual tokens to reduce computation, while global attention is used for cross-modal interactions. Multi-scale fusion: visual tokens of different resolutions are used in different Transformer layers to balance local details with global context.
[0031] (3) Text generation output.
[0032] After processing through all interaction layers, the last token in the sequence is fed into a language model output head, which is a linear layer plus softmax. The output head maps the hidden states to a probability distribution across the entire vocabulary, and the model generates the answer in an autoregressive manner. The model predicts the probability distribution of the first output token based on the entire input sequence (image + text) and samples / selects the most likely token from it. This generated token is then appended to the end of the input sequence as new input to predict the next token. This process is repeated until an end marker is generated, forming the complete answer. Throughout the generation process, each prediction step relies on the original image token and all generated text tokens, ensuring the coherence and consistency of the reasoning.
[0033] The medical intelligent reasoning method based on the fusion of specialized medical visual models and general multimodal large models in this invention is applicable to scenarios such as medical image analysis, assisted diagnosis, and intelligent interpretation of multimodal health data.
[0034] The technical solution of this invention, while maintaining the powerful language understanding and multimodal interaction capabilities of Qwen3-VL, introduces specialized medical visual features from the RetFound model, giving the system the following advantages: Enhanced medical adaptability: The visual module is transformed from a general encoder to a medically specialized model, significantly improving the feature extraction and representation capabilities of medical images. Breaking through unimodal limitations: Through deep multimodal fusion of images, text, and physiological indicators, a leap from "unimodal text reasoning" to "multimodal joint decision-making" is achieved. Improved reasoning reliability and interpretability: Joint features make the reasoning results closer to clinical logic, enabling the interpretation of the image and indicator basis for the model's judgments.
[0035] The technical solutions of this invention are widely used in intelligent diagnostic systems, health management platforms, hospital information systems, etc., and are especially suitable for scenarios that require joint decision-making by combining imaging and clinical indicators, such as cardiovascular risk prediction, diabetic retinopathy screening, and intelligent management of chronic diseases.
[0036] Device Example 1 According to embodiments of the present invention, a multimodal fusion medical intelligent reasoning device is provided. Figure 2 This is a schematic diagram of a multimodal fusion medical intelligent reasoning device according to an embodiment of the present invention, as shown below. Figure 2 As shown, the multimodal fusion medical intelligent reasoning device according to an embodiment of the present invention specifically includes: The extraction and encoding module 20 is used to extract the structured deep features of medical images to obtain medical visual features; and to semantically encode patient information to obtain medical text features. Specifically, it is used to: extract the structured deep features of medical images using the RetFound model to obtain medical visual features; and to semantically encode patient information using the Qwen3-VL language module to obtain medical text features. The patient information specifically includes: patient medical records, patient vital signs data, and patient physiological parameters.
[0037] The fusion module 22 is used to perform feature alignment and weight optimization of the medical text features and the medical visual features within the fusion layer to achieve collaborative representation of medical visual features and medical text features; specifically, it is used to: concatenate the medical text features and the medical visual features together and introduce modal identifiers and positional encodings, and perform feature alignment and weight optimization through multi-layer fully connected layers to achieve collaborative representation of medical visual features and medical text features. Processing module 24 is used to complete the output of disease prediction, risk assessment or assisted diagnosis based on the collaborative representation through a multimodal self-attention mechanism. Specifically, it is used to: after processing through all interaction layers of the multimodal self-attention mechanism based on the collaborative representation, the last collaborative representation in the sequence is sent to the language model output head, wherein the structure of the language model output head is a linear layer + Softmax; The hidden state is mapped to a probability distribution across the entire vocabulary through the output head of the language model, and the answer is generated in an autoregressive manner: the probability distribution of the first output token is predicted based on the entire input sequence, and the most likely token is sampled / selected from it. This generated token is then appended to the end of the input sequence as new input to predict the next token. This process is repeated until an end token is generated, forming a complete answer. In the entire generation process, each step of the prediction depends on the original image token and all the generated text tokens.
[0038] The network structure of the multimodal self-attention mechanism specifically includes: N Transformer blocks, each containing multi-head self-attention + feedforward network FFN + residual and LayerNorm, and inserting bidirectional Cross-Attention layers to achieve finer-grained interaction between text and vision. Different resolution visual tokens are used in different Transformer blocks to take into account both local details and global context. Some attention heads are used to constrain only within the same modality, and some attention heads are used to constrain cross-modality. Attention capacity is adaptively allocated during training. Local window attention is used for medical visual features to reduce computation, while global attention is used for cross-modality.
[0039] The embodiments of the present invention are device embodiments corresponding to the above method embodiments. The specific operation of each module can be understood with reference to the description of the method embodiments, and will not be repeated here.
[0040] Device Example 2 This invention provides an electronic device, such as... Figure 3 As shown, it includes: a memory 30, a processor 32, and a computer program stored in the memory 30 and executable on the processor 32, wherein the computer program, when executed by the processor 32, performs the steps as described in the method embodiment.
[0041] Device Example 3 This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor 32, performs the steps described in the method embodiment.
[0042] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.
[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multimodal fusion medical intelligent reasoning method, characterized in that, include: Extracting structured deep features from medical images to obtain medical visual features; The patient information is semantically encoded to obtain medical text features; The medical text features and the medical visual features are aligned and weighted within the fusion layer to achieve collaborative representation of medical visual features and medical text features. Based on the aforementioned collaborative representation, disease prediction, risk assessment, or assisted diagnosis can be achieved through a multimodal self-attention mechanism.
2. The method according to claim 1, characterized in that, Extracting structured deep features from medical images to obtain medical visual features; semantically encoding patient information to obtain medical text features, specifically including: The RetFound model is used to extract structured deep features from medical images to obtain medical visual features; Patient information is semantically encoded using the Qwen3-VL language module to obtain medical text features.
3. The method according to claim 1, characterized in that, The medical text features and medical visual features are aligned and weighted within the fusion layer to achieve collaborative representation of medical visual features and medical text features. Specifically, this includes: The medical text features and the medical visual features are concatenated together, and modal identifiers and positional encodings are introduced. Feature alignment and weight optimization are performed through multi-layer fully connected layers to achieve collaborative representation of medical visual features and medical text features.
4. The method according to claim 1, characterized in that, The network structure of the multimodal self-attention mechanism specifically includes: N Transformer blocks, each containing multi-head self-attention + feedforward network FFN + residual and LayerNorm, and inserting bidirectional Cross-Attention layers to achieve finer-grained interaction between text and vision. Different resolution visual tokens are used in different Transformer blocks to take into account both local details and global context. Some attention heads are used to constrain only within the same modality, and some attention heads are used to constrain cross-modality. Attention capacity is adaptively allocated during training. Local window attention is used for medical visual features to reduce computation, while global attention is used for cross-modality.
5. The method according to claim 1, characterized in that, Based on the aforementioned collaborative representation, the outputs for disease prediction, risk assessment, or assisted diagnosis achieved through a multimodal self-attention mechanism specifically include: Based on the aforementioned collaborative representation, after processing through all interaction layers of the multimodal self-attention mechanism, the last collaborative representation in the sequence is sent to the language model output head, wherein the structure of the language model output head is a linear layer + Softmax; The hidden state is mapped to a probability distribution across the entire vocabulary through the output head of the language model, and the answer is generated in an autoregressive manner: the probability distribution of the first output token is predicted based on the entire input sequence, and the most likely token is sampled / selected from it. This generated token is then appended to the end of the input sequence as new input to predict the next token. This process is repeated until an end token is generated, forming a complete answer. In the entire generation process, each step of the prediction depends on the original image token and all the generated text tokens.
6. A multimodal fusion medical intelligent reasoning device, characterized in that, include: The extraction encoding module is used to extract the structured deep features of medical images and obtain medical visual features; The patient information is semantically encoded to obtain medical text features; The fusion module is used to perform feature alignment and weight optimization of the medical text features and the medical visual features within the fusion layer, so as to achieve collaborative representation of medical visual features and medical text features. The processing module is used to complete the output of disease prediction, risk assessment or assisted diagnosis based on the collaborative representation through a multimodal self-attention mechanism.
7. The apparatus according to claim 6, characterized in that, The extraction and encoding module is specifically used to: extract structured deep features of medical images using the RetFound model to obtain medical visual features; and semantically encode patient information using the Qwen3-VL language module to obtain medical text features. The fusion module is specifically used to: concatenate the medical text features and the medical visual features together and introduce modal identifiers and positional codes, and perform feature alignment and weight optimization through multi-layer fully connected layers to achieve collaborative representation of medical visual features and medical text features; The processing module is specifically used for: based on the collaborative representation, after processing through all interaction layers of the multimodal self-attention mechanism, the last collaborative representation in the sequence is sent to the language model output head, wherein the structure of the language model output head is a linear layer + Softmax; The hidden state is mapped to a probability distribution across the entire vocabulary through the output head of the language model, and the answer is generated in an autoregressive manner: the probability distribution of the first output token is predicted based on the entire input sequence, and the most likely token is sampled / selected from it. This generated token is then appended to the end of the input sequence as new input to predict the next token. This process is repeated until an end token is generated, forming a complete answer. In the entire generation process, each step of the prediction depends on the original image token and all the generated text tokens.
8. The apparatus according to claim 6, characterized in that, The network structure of the multimodal self-attention mechanism specifically includes: N Transformer blocks, each containing multi-head self-attention + feedforward network FFN + residual and LayerNorm, and inserting bidirectional Cross-Attention layers to achieve finer-grained interaction between text and vision. Different resolution visual tokens are used in different Transformer blocks to take into account both local details and global context. Some attention heads are used to constrain only within the same modality, while some attention heads are used to constrain cross-modality. Attention capacity is adaptively allocated during training. Local window attention is used for medical visual features to reduce computation, while global attention is used for cross-modality.
9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the multimodal fusion medical intelligent reasoning method as described in any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the multimodal fusion medical intelligent reasoning method as described in any one of claims 1 to 5.