A disease classification method based on multi-modal medical images
By combining the image representation module, the cross-modal semantic alignment module, and the fusion classification module, the problems of information interference and quality fluctuation in multimodal medical images are solved, achieving robust disease classification and improving accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- EAST CHINA NORMAL UNIV
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-21
AI Technical Summary
Multimodal medical imaging suffers from problems such as intermodal information interference, dilution of key information, influence of spurious correlation features, and fluctuations in modal quality in disease classification, making it difficult to achieve robust disease classification.
By employing an image representation module, a cross-modal semantic alignment module, an uncertainty prediction module, and a fusion classification module, robust classification of multimodal images is achieved through shared feature extraction, modality-private feature extraction, instance-level attention aggregation, and adaptive weighted fusion.
It improves the accuracy and robustness of disease classification, maintains good classification performance under conditions of modal quality fluctuations or omissions, and fully leverages the complementary advantages of multimodal images.
Smart Images
Figure CN122435331A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image analysis and artificial intelligence-assisted diagnosis technology, and in particular to a disease classification method based on multimodal medical images. Background Technology
[0002] Disease classification, subtyping, and risk stratification are crucial in clinical diagnosis and treatment, providing a basis for diagnostic decisions, treatment selection, efficacy evaluation, and follow-up management. Traditionally, such assessments rely on clinical symptoms, laboratory indicators, and pathological examination results. However, obtaining some of this information can be invasive, delayed, or limited by sampling constraints, and it's difficult to fully reflect lesion heterogeneity, tissue microenvironment differences, and disease progression before treatment. Therefore, utilizing medical imaging for earlier and more comprehensive disease identification and risk assessment under non-invasive or minimally invasive conditions has significant application value.
[0003] Medical imaging technologies, including but not limited to magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), and ultrasound (US), can reflect the structure and functional state of human tissues from different dimensions. Different imaging modalities, or different sequences, phases, and parameter settings within the same imaging modality, can provide complementary information, such as differences in anatomical structure, tissue density or signal intensity, enhancement characteristics, diffusion properties, perfusion, and metabolism. Building disease classification models based on multimodal medical imaging holds promise for integrating multi-source information to improve discriminative performance and achieve more reliable assisted diagnosis and classification support.
[0004] However, multimodal medical imaging disease classification still faces several challenges in practical applications: First, the imaging mechanisms and signal intensity distributions of different modalities differ significantly, and direct splicing or simple fusion can easily lead to mutual interference between modal information, making it difficult to fully leverage complementary advantages; Second, clinical images are often presented in the form of multiple slices or multiple phases, and the effective discriminative information for the same subject is often concentrated in a few key slices or key phases, and average convergence is easily diluted by a large amount of irrelevant information; Third, images usually contain a large area of background tissue, and the structures around organs are complex and prone to introducing spurious correlation features, affecting the model's generalization ability and stability; Fourth, affected by factors such as scanning protocols, equipment differences, artifacts, field of view coverage, and examination deficiencies, the quality and usability of different modalities fluctuate, and fixed-weight fusion is difficult to adapt to individual differences and the lack of actual clinical data.
[0005] To address the aforementioned issues, there is an urgent need for a multimodal medical imaging disease classification method that can simultaneously model cross-modal commonalities and modality-specific information, automatically focus on key information at the slice (or instance) level, and be robust to modality loss or quality fluctuations. Summary of the Invention
[0006] To address the problems existing in the background technology, the present invention aims to provide a disease classification method based on multimodal medical images. This method can adaptively weight and fuse information based on the contribution and uncertainty of each modality, give full play to the complementary advantages of multimodality, and maintain robust disease classification even under conditions of modality quality fluctuations or missing information.
[0007] The specific technical solution for achieving the purpose of this invention is as follows:
[0008] A disease classification method based on multimodal medical images, the method comprising: an image representation module, a cross-modal semantic alignment module, an uncertainty prediction module, and a fusion classification module; wherein the number of the image representation module and the uncertainty prediction module corresponds to the total number of modalities;
[0009] Each image representation module includes a shared feature extraction submodule, a private feature extraction submodule, a gated fusion submodule, and an instance-level attention aggregation submodule. Each image representation module takes the preprocessed medical image data corresponding to a single modality as input, uses the shared feature extraction submodule and the private feature extraction submodule to extract the common representations and specific information of the single modality, respectively, and obtains instance-level fusion features through the gated fusion submodule. Furthermore, the instance-level attention aggregation submodule is introduced to perform weighted aggregation of the instance-level fusion features to form an object-level modality representation of the single modality.
[0010] The preprocessing includes image size unification, intensity normalization, key image screening, and image enhancement operations;
[0011] The shared feature extraction submodule is a shared encoder across modalities, enabling a unified mapping of different modalities to extract common representations. The shared encoder can be a convolutional neural network, a Transformer encoder, or a hybrid encoder combining convolution and attention; this invention is not limited to any particular type. If the input is represented as... The common characteristics are then represented as ,satisfy:
[0012]
[0013] Where m represents the m-th mode, and k represents the slice index of that mode. K is the number of input instances. For shared encoders.
[0014] The private feature extraction submodule is a parameter-independent network that corresponds one-to-one with each modality. It is used to learn specific information of each modality. ,satisfy:
[0015]
[0016] The gated fusion submodule is used to integrate the common representations. With the aforementioned special information The concatenation is performed along the channel dimension, followed by 1×1 convolution to generate a gated mapping, and then normalized using the Sigmoid function to obtain the gate coefficients. The specific information is weighted element by element, and then added element by element to the common representation to obtain the instance-level fusion feature. The specific process is as follows:
[0017]
[0018]
[0019] in This is the Sigmoid function.
[0020] The instance-level attention aggregation submodule maps the instance-level fused features into slice embedding vectors. Attention scores are then generated through a learnable attention network. The attention network consists of two nonlinear mapping branches: one using a tanh nonlinear transformation and the other using a sigmoid nonlinear transformation. The attention scores are normalized to obtain the slice weights. The object-level modal representation is obtained by weighted convergence of the slice embedding vectors. The process is as follows:
[0021]
[0022]
[0023]
[0024] Among them, V, U, These are learnable parameters.
[0025] The cross-modal semantic alignment module performs shared semantic space alignment on the object-level modal representations of each modality to obtain the aligned object-level semantic representations of each modality.
[0026] The cross-modal semantic alignment module performs the shared semantic space alignment by: setting up a set of N learnable prototype vectors containing a preset number. The prototype vector is used to represent the semantic reference representation in the shared semantic space; the object-level modal representation of modality m is calculated. Distance metric between each prototype vector in the prototype vector set A temperature coefficient is used for the distance measurement. After scaling and negation, Softmax normalization is performed to obtain the soft-assigned weights of the object-level modal representation on each prototype vector. The object-level semantic representation is obtained by weighting and summing the prototype vector set according to the soft-assignment weights. .
[0027]
[0028]
[0029]
[0030] Each uncertainty prediction module outputs the task score and reliability parameters for each modality based on the object-level semantic representation of the corresponding modality;
[0031] The uncertainty prediction module It includes at least one layer of learnable mapping network, and sets task scoring output branch and reliability parameter output branch; the uncertainty prediction module performs object-level semantic representation for each modality. Perform mapping and output the task score for each modality on the target task. and the reliability parameter corresponding to the task score. .
[0032]
[0033] The fusion classification module generates modality fusion weights based on the task scores and reliability parameters of each modality, and performs adaptive weighted fusion of the object-level semantic representations of each modality to obtain a fusion representation, thereby outputting the classification result of the inspected object; wherein, in the case of missing modalities, the missing modalities are masked and the modality fusion weights are calculated based on the existing modalities, so that the fusion classification module can still output the classification result.
[0034] The fusion classification module specifically performs task scoring for each modality. With the aforementioned reliability parameters Uncertainty correction is performed using a preset function to obtain a correction score. The preset function is a function that monotonically decreases with respect to the reliability parameter, so that low reliability modalities correspond to low correction scores. The correction scores of each modality are combined into a correction score vector and normalized to obtain the modality fusion weights. The normalization is Softmax normalization, and masking is supported for missing modalities in the case of missing modalities. The modality fusion weights are weighted and summed with the object-level semantic representations of each modality to obtain the fusion representation. The fusion representation is then input into a classifier or predictor to output the classification result.
[0035] The beneficial effects of the technical solution provided by this invention are as follows: The disease classification method based on multimodal medical images proposed in this invention combines shared feature extraction and modality-specific feature extraction at the feature level, taking into account both cross-modal consistent representation and modality-specific discriminative information; at the object level, it introduces instance-level attention aggregation to automatically select key instances to form object-level modal representations; simultaneously, it performs semantic alignment of the object-level modal representations of each modality through a cross-modal semantic alignment module, and combines an uncertainty prediction module and a fusion classification module to output stable object-level classification results, thereby improving the accuracy and robustness of disease classification and risk stratification, and maintaining good classification performance even under conditions of modality quality fluctuations or missing data. Using the method of this invention, the complementary advantages of multimodal images can be fully utilized, thereby improving the accuracy and robustness of disease classification. Attached Figure Description
[0036] Figure 1 This is a schematic diagram of the structure of the present invention;
[0037] Figure 2 This is a schematic diagram of input image data preprocessing according to an embodiment of the present invention;
[0038] Figure 3 This is a schematic diagram of the classification network structure according to an embodiment of the present invention;
[0039] Figure 4 This is a schematic diagram of the modality private feature extraction network structure according to an embodiment of the present invention. Detailed Implementation
[0040] To more clearly illustrate the technical means, technical improvements, and beneficial effects of this invention, the invention will be described in detail below with reference to the accompanying drawings.
[0041] Example
[0042] This embodiment uses multimodal magnetic resonance imaging data of patients with endometrial cancer as an example. The multimodal imaging data includes three modalities of data: T2-weighted imaging (T2WI), contrast-enhanced T1-weighted imaging (CE-T1WI), and diffusion-weighted imaging (DWI) for the same subject. Patient-level prognostic risk classification labels and uterine masks are also provided.
[0043] See Figure 1 - 4. The prognostic classification method based on multimodal magnetic resonance imaging provided by this invention includes an image representation module, a cross-modal semantic alignment module, an uncertainty prediction module, and a fusion classification module, such as... Figure 1 As shown. The specific steps include:
[0044] The preprocessed multimodal images of the same subject are input into the image representation module, which includes a shared feature extraction submodule, a private feature extraction submodule, a gated fusion submodule, and an instance-level attention aggregation submodule.
[0045] The preprocessing includes operations such as image size unification, intensity normalization, key image screening, and image enhancement.
[0046] In this embodiment, the image size is standardized to 256×256 for all modal images.
[0047] In this embodiment, the intensity normalization scales the pixel values of the image uniformly to the range of [0, 1000].
[0048] The key image screening involves sorting the images by the area of the uterine mask and selecting the top 5 images with the largest area in each modality as the effective data for that modality. Thus, the T2WI, CE-T1WI, and DWI modalities each contain only 5 images, avoiding irrelevant images from diluting the effective information.
[0049] The image enhancement involves weighted suppression of the image using a uterine mask before it enters the network, reducing the weight of regions outside the mask to weaken irrelevant background interference. Figure 2 As shown, specifically:
[0050]
[0051] Among them, I input M represents the image, and M represents the uterine mask. For element-wise multiplication, These are the weighting coefficients outside the mask; in this embodiment... =0.5.
[0052] After completing the preprocessing of each modality of image, input it into the classification network (see...). Figure 3 ).
[0053] The shared feature extraction submodule performs unified feature encoding on the preprocessed images of each modality. In this embodiment, the shared feature extraction submodule is built with a pre-trained ResNet-18 backbone to achieve unified mapping of different modalities, thereby extracting common representations. Let the k-th preprocessed slice of the m-th modality be denoted as... K The common characteristic extraction results are as follows:
[0054]
[0055] in For pre-trained ResNet-18, This is a common characteristic.
[0056] For each of the three modalities, a modality-specific feature extraction submodule is set up and outputs modality-specific information. The structural diagram is as follows: Figure 4 As shown, where Figure 4 (a) shows the structure of the private feature extraction module for the T2WI modality. Figure 4 (b) shows the structure of the private feature extraction module of the CE-T1WI mode. Figure 4 (c) shows the private feature extraction module structure for the DWI modality. The T2WI branch focuses on multi-scale structure and low-frequency information enhancement to suppress artifacts; the CE-T1WI branch focuses on second-order geometry and center-periphery difference to highlight and enhance boundaries; and the DWI branch focuses on denoising, monotonic mapping, and directional filtering to extract diffusion-related anisotropic features. Let the independent network for each modality parameter be denoised as... ,but:
[0057]
[0058] in This is special information.
[0059] The gated fusion submodule will use the common characteristics and To perform fusion, the common characteristics are... With the aforementioned special information The concatenation is performed along the channel dimension, followed by 1×1 convolution to generate a gated mapping, and then normalized using the Sigmoid function to obtain the gate coefficients. The specific information is weighted element by element, and then added element by element to the common representation to obtain the instance-level fusion feature. The specific process is as follows:
[0060]
[0061]
[0062] in Indicates channel-dimensional splicing. This is a 1×1 convolution operation. This is the Sigmoid function.
[0063] The instance-level attention aggregation submodule integrates the instance-level fused features. The slice embedding vector is obtained by linear projection after pooling. Then, an attention score corresponding to the slice embedding vector is generated through a learnable attention network. The attention network consists of two nonlinear mapping branches: one using a tanh nonlinear transformation and the other using a sigmoid nonlinear transformation. The attention scores are normalized to obtain the slice weights. The object-level modal representation is obtained by weighted convergence of the slice embedding vectors. The process is as follows:
[0064]
[0065]
[0066]
[0067] Among them, V, U, These are learnable parameters.
[0068] The cross-modal semantic alignment module aligns the object-level modal representations of each modality using a shared semantic space to obtain aligned object-level semantic representations for each modality. Specifically, it sets a set of learnable prototype vectors containing a preset number N=16. The prototype vector is used to represent the semantic reference representation in the shared semantic space; the object-level modal representation of modality m is calculated. Distance metric between each prototype vector in the prototype vector set A temperature coefficient is used for the distance measurement. Scaling, where the temperature coefficient is fixed at 1. After negation, Softmax normalization is performed to obtain the soft-assigned weights of the object-level modal representation on each prototype vector. :
[0069]
[0070]
[0071] The object-level semantic representation is obtained by weighting and summing the prototype vector set according to the soft-assignment weights. :
[0072]
[0073] The uncertainty prediction module is based on the object-level semantic representation of each modality. Output task scores for each modality With reliability parameters The uncertainty prediction module This includes a multi-layer fully connected network, where the task score is output by a linear layer, and the reliability parameter is output by another linear layer and constrained to a positive value by a softplus function.
[0074]
[0075] The fusion classification module generates modality fusion weights based on the task scores and reliability parameters of each modality, and performs adaptive weighted fusion of the object-level semantic representations of each modality to obtain a fused representation, thereby outputting the classification result of the examined object. Specifically, uncertainty correction is applied to the task scores and reliability parameters of each modality to obtain a corrected score:
[0076]
[0077] Furthermore, missing modalities are suppressed before normalization by combining missing modal masks, and masked softmax normalization is performed on the correction scores to obtain modality fusion weights. The fused representation is obtained by weighting and fusing the object-level semantic representations of each modality according to the modality fusion weights:
[0078]
[0079]
[0080] The fusion representation F is input to the classifier to obtain the patient-level prognostic risk prediction classification result. .
[0081] In summary, the disease classification method based on multimodal medical images proposed in this invention can effectively integrate complementary information from different modalities. It achieves collaborative modeling of common representations and specific information through shared feature extraction and modality-specific feature extraction, reducing intermodal interference. It combines instance-level attention aggregation to form object-level modal representations and achieves unified semantic space mapping through a cross-modal semantic alignment module. Furthermore, it introduces an uncertainty prediction module and a fusion classification module to perform reliability modeling and weighted fusion of information from each modality, enabling the model to output stable results even under conditions of modality quality fluctuations or missing information, thereby improving the accuracy and reliability of prognostic classification.
[0082] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A disease classification method based on multimodal medical imaging, characterized in that, The classification method includes the following modules: an image representation module, a cross-modal semantic alignment module, an uncertainty prediction module, and a fusion classification module; wherein the number of the image representation module and the uncertainty prediction module corresponds to the total number of modalities. Each image representation module includes a shared feature extraction submodule, a private feature extraction submodule, a gated fusion submodule, and an instance-level attention aggregation submodule. Each image representation module takes the preprocessed data of the corresponding single-modality medical image as input, uses the shared feature extraction submodule and the private feature extraction submodule to extract the common representation and specific information of the single modality, respectively, and obtains instance-level fusion features through the gated fusion submodule. The instance-level attention aggregation submodule is further introduced to perform weighted aggregation of the instance-level fusion features to form an object-level modality representation of the corresponding single modality. The cross-modal semantic alignment module performs shared semantic space alignment on the object-level modal representations to obtain the object-level semantic representations after each modality alignment. Each uncertainty prediction module outputs the task score and reliability parameters for each modality based on the object-level semantic representation of the corresponding modality; The fusion classification module generates modality fusion weights based on the task scores and reliability parameters of each modality, and performs adaptive weighted fusion of the object-level semantic representations of each modality to obtain a fusion representation, thereby outputting the classification result of the inspected object; wherein, in the case of missing modalities, the missing modalities are masked and the modality fusion weights are calculated based on the existing modalities, so that the fusion classification module can still output the classification result.
2. The disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The preprocessing includes image size unification, intensity normalization, key image screening, and image enhancement operations.
3. The disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The shared feature extraction submodule is a shared encoder across modalities, achieving a unified mapping of different modalities to extract common representations; wherein, the shared encoder is a convolutional neural network, a Transformer encoder, or a hybrid encoder combining convolution and attention. If the input is represented as... The common characteristics are then represented as ,satisfy: Where m represents the m-th mode, and k represents the instance index of that mode. K is the number of input instances. For shared encoders.
4. The disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The private feature extraction submodule is a parameter-independent network that corresponds one-to-one with each modality. It is used to learn specific information of each modality. ,satisfy: 。 5. A disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The gated fusion submodule is used to integrate the common representations. With the aforementioned special information The concatenation is performed along the channel dimension, followed by 1×1 convolution to generate a gated mapping, and then normalized using the Sigmoid function to obtain the gate coefficients. The specific information is weighted element by element, and then added element by element to the common representation to obtain the instance-level fusion feature. The specific process is as follows: in Indicates channel splicing. This is a 1×1 convolution operation. This is the Sigmoid function.
6. The disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The instance attention aggregation module maps the instance-level fused features into slice embedding vectors. Attention scores are then generated through a learnable attention network. The attention network consists of two nonlinear mapping branches: one using a tanh nonlinear transformation and the other using a sigmoid nonlinear transformation. The attention scores are then normalized to obtain slice weights. The object-level modal representation is obtained by weighted convergence of the slice embedding vectors. The process is as follows: Among them, V, U, These are learnable parameters.
7. The disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The cross-modal semantic alignment module performs the shared semantic space alignment by: setting up a set of N learnable prototype vectors containing a preset number. The prototype vector is used to represent the semantic reference representation in the shared semantic space; The object-level modal representation of the computational modality m Distance metric between each prototype vector in the prototype vector set A temperature coefficient is used for the distance measurement. After scaling and negation, Softmax normalization is performed to obtain the soft-assigned weights of the object-level modal representation on each prototype vector. ; The object-level semantic representation is obtained by weighting and summing the prototype vector set according to the soft-assignment weights. ; 。 8. A disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The uncertainty prediction module It includes at least one layer of learnable mapping network, and sets task scoring output branch and reliability parameter output branch; the uncertainty prediction module performs object-level semantic representation for each modality. Perform mapping and output the task score for each modality on the target task. and the reliability parameter corresponding to the task score. ; 。 9. A disease classification method based on multimodal medical imaging according to claim 1, characterized in that, The fusion classification module specifically includes: scoring the task for each modality. With the aforementioned reliability parameters Uncertainty correction is performed using a preset function to obtain a correction score. The preset function is a function that monotonically decreases with respect to the reliability parameter, so that low reliability modalities correspond to low correction scores. The correction scores of each modality are combined into a correction score vector and normalized to obtain the modality fusion weights. The normalization is Softmax normalization, and masking is supported for missing modalities in the case of missing modalities. The modality fusion weights are weighted and summed with the object-level semantic representations of each modality to obtain the fusion representation, and the fusion representation is input into a classifier or predictor to output the classification result.