A hybrid supervised fusion method based on multi-modal medical data

Abstract

The application discloses a kind of mixed supervision fusion methods based on multi-modal medical data, comprising the following steps: paired multi-modal medical data are collected, and pretreatment is carried out;Multi-modal feature fusion model is constructed based on convolutional neural network, including feature extraction module, feature decomposition module, uncertainty minimization module, multi-modal queue contrast learning module and fusion optimization module;Feature extraction module carries out feature extraction to the multi-modal medical data after pretreatment, obtains multi-modal feature;Feature decomposition module carries out feature decomposition to multi-modal feature, obtains the multi-modal feature after decomposition;Uncertainty minimization module and multi-modal queue contrast learning module, carry out alignment processing to the multi-modal feature after decomposition;Fusion optimization module fuses the multi-modal feature after alignment, obtains the multi-modal medical data after fusion.The application can reduce the heterogeneity between different modal features on medical public multi-modal data set, and better carry out feature fusion.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a hybrid supervised fusion method based on multimodal medical data. Background Technology

[0002] In medical image processing, multimodal data often provides information from multiple dimensions compared to unimodal data. With the development of medical imaging technology and artificial intelligence, multimodal fusion methods have made some progress, but some common limitations still exist.

[0003] First, existing research typically fuses features from different modalities directly into the model. However, multimodal information can be decomposed into two distinct parts: modality-shared information and modality-specific information. In the process of multimodal alignment and fusion, shared information often dominates, potentially suppressing modality-specific information and thus neglecting the unique perspectives offered by different modalities.

[0004] Secondly, different data modalities often exhibit significant differences. For example, digital pathology images are typically gigapixel-level images, usually represented as a collection of tens of thousands of image patches; while genomic data is typically represented as a 1×1 table. This difference in representation leads to significant heterogeneity in the feature spaces between different modalities, thus increasing the complexity of their fusion. Existing research has shown that directly fusing heterogeneous features often faces considerable challenges in the absence of effective alignment strategies.

[0005] Therefore, in order to address the above problems, there is an urgent need to propose a hybrid supervision and fusion method based on multimodal medical data. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a hybrid supervised fusion method based on multimodal medical data. This method employs a multimodal feature decomposition module to decompose input features into general features and modality-specific features, thereby providing a comprehensive multimodal perspective. To alleviate the heterogeneity of the feature space during the fusion process, an uncertainty minimization module and a multimodal queue contrastive learning module are designed to ensure the consistency of prediction results across different modal features, thereby improving the heterogeneity of multimodal features. Finally, through joint training, effective multimodal feature fusion is achieved.

[0007] This invention proposes a hybrid supervised fusion method based on multimodal medical data, comprising the following steps:

[0008] Collect paired multimodal medical data and perform preprocessing;

[0009] A multimodal feature fusion model is constructed based on a convolutional neural network. The multimodal feature fusion model includes a multimodal feature extraction module, a multimodal feature decomposition module, an uncertainty minimization module, a multimodal queue contrastive learning module, and a fusion optimization module.

[0010] Based on the multimodal feature extraction module, feature extraction is performed on the preprocessed multimodal medical data to obtain multimodal features;

[0011] The multimodal features are decomposed based on the multimodal feature decomposition module to obtain the decomposed multimodal features;

[0012] Based on the uncertainty minimization module and the multimodal queue contrast learning module, the decomposed multimodal features are aligned.

[0013] Based on the fused and aligned multimodal features obtained by the fusion optimization module, fused multimodal medical data is obtained.

[0014] Optionally, the process of extracting features from the preprocessed multimodal medical data based on the multimodal feature extraction module to obtain multimodal features includes:

[0015] The multimodal medical data includes pathological images and genetic data;

[0016] The thumbnail of the preprocessed pathological image is converted into a grayscale image. The threshold of the grayscale image is determined based on the Otsu thresholding method to obtain an initial binary mask. Morphological opening is performed on the initial binary mask, and connected components with an area smaller than a preset pixel are removed to obtain a foreground mask. The foreground mask is mapped back from the thumbnail to the original image to obtain a set of effective image patch coordinates. Based on the image patches, ResNet-50 is used as the feature extraction backbone to extract features and obtain pathological image features.

[0017] A neural network is used to extract features from the preprocessed gene data to obtain gene features.

[0018] Optionally, the process of performing feature decomposition on the multimodal features based on the multimodal feature decomposition module to obtain the decomposed multimodal features includes:

[0019] Using gene features and pathological image features as input, cosine similarity is used to measure the distance between features of different modalities to obtain similarity; based on the similarity, a similarity threshold is adaptively calculated to generate a feature mask; when the similarity is higher than the threshold, it is identified as a strongly correlated feature, and the generated feature mask is multiplied element-wise with the original features to retain the identified strongly correlated features; the strongly correlated features are subtracted from the original features of their respective modalities to obtain the corresponding modality-specific difference features.

[0020] Optionally, the process of aligning the decomposed multimodal features based on the uncertainty minimization module includes:

[0021] The decomposed multimodal features are input into the fully connected neural network structure corresponding to each modality for processing to obtain the prediction probability distribution of each modality. Based on the prediction probability distribution of each modality, the prediction group of each modality is determined. The prediction probability of each modality and its corresponding prediction group are input into the uncertainty minimization loss function. When the prediction groups of two modalities are consistent, the consistency of multimodal prediction is achieved.

[0022] Optionally, the process of aligning the decomposed multimodal features based on the multimodal queue contrastive learning module includes:

[0023] Two dual-queue structures are constructed to store feature representations of different modalities, resulting in a pathological modality queue and a gene modality queue. Based on prediction grouping, the pathological modality queue and the gene modality queue are grouped and initialized using a normal distribution, and dynamically updated using a momentum coefficient. During the dynamic update process, the distance between pathological features and gene features with the same group label is minimized, while the distance between pathological features and other group features in the pathological modality queue is maximized, thereby achieving multimodal feature alignment.

[0024] Optionally, the process of obtaining fused multimodal medical data based on the fused and aligned multimodal features obtained by the fusion optimization module includes:

[0025] The strongly correlated and differential features corresponding to each modality are concatenated, and the concatenated features are dimensionality reduced by feature projection. Then, a fully connected layer with a Sigmoid activation function is used to fuse multimodal features.

[0026] Optionally, it also includes parameter optimization of the multimodal feature fusion model:

[0027] We assign weights to the uncertainty minimization loss and the multimodal queue comparison loss, construct an overall loss function for the multimodal feature fusion model, and optimize and train the parameters of the multimodal feature fusion model based on the overall loss function.

[0028] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0029] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0030] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0031] Compared with the prior art, the present invention has the following advantages and technical effects:

[0032] This invention achieves effective fusion of multimodal medical data based on techniques such as multimodal feature decomposition, uncertainty minimization, and multimodal queue contrastive learning. The multimodal feature decomposition module divides input features into general and modality-specific features, providing a comprehensive modality perspective and preventing modality suppression issues caused by direct fusion. The uncertainty minimization module constrains feature learning at the individual level, enhancing the consistency of prediction results; the multimodal queue contrastive learning module guides and constrains feature alignment through cross-modal contrastive learning, reducing heterogeneity between features from different modalities. Ultimately, on public medical multimodal datasets, this method can better perform feature fusion, improving the utilization efficiency and predictive performance of multimodal data. Attached Figure Description

[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0034] Figure 1 This is a schematic diagram of the structure of the multimodal feature decomposition module according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the multimodal feature fusion model according to an embodiment of the present invention;

[0036] Figure 3 This is a diagram showing the cosine similarity results of an embodiment of the present invention;

[0037] Figure 4 This is an integral gradient heatmap of an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0041] Example 1

[0042] like Figure 5 As shown, this embodiment provides a hybrid supervised fusion method based on multimodal medical data, including the following steps:

[0043] Collect paired multimodal medical data and perform preprocessing;

[0044] A multimodal feature fusion model is constructed based on a convolutional neural network. The multimodal feature fusion model includes a multimodal feature extraction module, a multimodal feature decomposition module, an uncertainty minimization module, a multimodal queue contrastive learning module, and a fusion optimization module.

[0045] Based on the multimodal feature extraction module, feature extraction is performed on the preprocessed multimodal medical data to obtain multimodal features;

[0046] The multimodal features are decomposed based on the multimodal feature decomposition module to obtain the decomposed multimodal features;

[0047] Based on the uncertainty minimization module and the multimodal queue contrast learning module, the decomposed multimodal features are aligned.

[0048] Based on the fused and aligned multimodal features obtained by the fusion optimization module, fused multimodal medical data is obtained.

[0049] An feasible process for acquiring and preprocessing paired multimodal medical data includes:

[0050] In this embodiment, a public dataset on kidney cancer was collected and used. This dataset contains pathological images and genetic data.

[0051] Data preprocessing: First, check whether pathological images and gene data are paired and exclude data with missing modalities. The specific operations are as follows: (1) Pathological modality validity check: check whether the diagnostic pathological images exist and whether they are clearly readable; (2) Genomics modality validity check: check whether mRNA and methylation data exist and whether there are missing values; (3) Generate the final pairing list: only when both of the above modalities (pathology and genes) pass the judgment are they included; otherwise, record and exclude data with missing modalities.

[0052] The feasible process of extracting features from preprocessed multimodal medical data based on the multimodal feature extraction module to obtain multimodal features includes:

[0053] The screened pathological images P imgLoading into memory. This embodiment uses the open-source OpenSlide package to read each whole slide image (WSI) and selects 40× magnification for foreground detection. The specific operation is as follows: Convert the WSI thumbnail to a grayscale image I. gray The initial binary mask is obtained using the Otsu thresholding method, where the threshold is automatically determined by Otsu. To remove background noise and small fragments, morphological opening operations (structural element radius 3) are used to remove connected components with an area less than 2000 pixels. Finally, the foreground mask is obtained and mapped back from the thumbnail to 40× to obtain a set of effective image patch coordinates. Then, the foreground region is cropped according to a certain stride. In this embodiment, a stride size of 512 is selected to obtain image patches of the same size. Then, a pre-trained convolutional neural network (CNN) is used to extract the pattern feature vector of each image patch. That is, ResNet-50 is used as the backbone for image patch feature extraction, and the weights are initialized using the ImageNet pre-trained model. The obtained features are numerical representations of the input image patches after multiple layers of convolution, pooling, and nonlinear transformation. They are usually in vector form and are used to describe key information related to the task in the image (such as shape, texture, structure, etc.) to support subsequent analysis and prediction.

[0054] For gene data G seq Feature extraction is performed using a neural network, and the calculation method for the processing is as follows:

[0055] P fea =CNN(Crop(Otsu(P) img )));

[0056] G fea =NN(G seq );

[0057] Where P fea and G fea These represent the feature representations of pathological images and gene data after feature extraction, respectively. Otsu indicates thresholding using the OpenCV library, Crop indicates cropping of digital pathological images, and NN indicates a neural network.

[0058] The feasible process of performing feature decomposition on multimodal features based on the multimodal feature decomposition module to obtain the decomposed multimodal features includes:

[0059] The multimodal feature decomposition module can obtain a more comprehensive multimodal perspective and prevent the modality suppression problem caused by direct fusion. First, the multimodal feature decomposition module is constructed using features from extracted pathological images and gene data. For example... Figure 1As shown, the module uses genomic features G fea and pathological image features P fea As input, cosine similarity is used to measure the distance between features of different modalities. Then, the similarity values ​​are input into a sigmoid function to adaptively calculate a similarity threshold. A feature mask is generated based on this threshold: when the similarity is higher than the threshold, the corresponding feature is identified as a strongly correlated feature G. com and P com The strongly correlated features are preserved by element-wise multiplication of the generated mask with the original features. Finally, the modality-specific differential features G are obtained by subtracting the strongly correlated features from the original features of their respective modalities. spe and P spe .

[0060] The feasible process of aligning the decomposed multimodal features based on the uncertainty minimization module includes:

[0061] The uncertainty minimization module enhances the consistency of prediction results by constraining feature learning at the individual level. First, the features derived from the pathological and genetic data are input into the classifier head corresponding to their respective modalities to obtain the probabilities of predicting N groups. The classifier head is a fully connected neural network structure with an input dimension equal to the dimension of the corresponding modality's feature vector and an output dimension of N. Specifically, the input features first undergo a linear transformation (fully connected layer) to map the features to the hidden layer dimension. Then, a ReLU activation function introduces non-linearity. Next, a second fully connected layer maps the features to the class space (dimension = N). Finally, a Softmax function transforms the output into a prediction probability distribution for each modality. The maximum prediction probability p for the pathological modality is... m The maximum predictive probability g of gene modality n These are used to determine the prediction groups m and n, respectively. Then, the predicted probabilities and their corresponding groups are input into the uncertainty minimization loss function. When the prediction groups of the two modalities are consistent, it indicates that individuals are consistent in multimodal predictions. In this case, the confidence of multimodal predictions is improved by minimizing the average uncertainty of the prediction groups. Conversely, when the prediction groups of the two modalities are inconsistent, it indicates that there are differences in individual predictions, and a greater penalty needs to be applied. Finally, the uncertainty minimization loss function can be expressed as:

[0062]

[0063] The feasible process of aligning the decomposed multimodal features based on the multimodal queue contrastive learning module includes:

[0064] This embodiment designs a multimodal contrastive learning module to guide and constrain feature alignment. Unlike image classification methods that rely on data augmentation, the method in this embodiment utilizes the highly correlated features G generated in the above steps. com and P com To achieve cross-modal contrastive learning, this embodiment designs a dual-queue structure Q. g and Q p The multimodal feature queue Q is used to store feature representations from different modalities and inputs features into the corresponding groups for each modality based on the group label i∈N. Specifically, the pathology modality queue and the gene modality queue are each divided into N groups. The multimodal feature queue Q is initialized using a normal distribution and dynamically updated using a momentum coefficient M. Its update process can be represented as follows:

[0065]

[0066] in, Q represents the queue before the update. new This represents the queue after momentum update. Let x ∈ {P, G} represent the features input at the current stage, where x ∈ {P, G} represents the pathological modality and the genetic modality, respectively, and i represents the feature group label. The update of the momentum parameter is determined by the exponential decay of the training epoch E, and its expression is as follows:

[0067] M = Min[(1-0.5)] E ),0.9];

[0068] Following the principles of contrastive learning, the pathological modality queue and the gene modality queue are updated independently. During training, this embodiment minimizes the distance between pathological features and gene features with the same group label, while maximizing the distance between pathological features and other group features in the pathological modality queue. This process is achieved through the following contrastive learning loss function:

[0069]

[0070] Where sim represents the cosine similarity score, and p i and g i These represent the input pathological features and gene features assigned to the i-th group based on clinical information, respectively. and These represent the characteristics of the i-th group in the gene modality cohort and the pathology modality cohort, respectively. and These represent the characteristics of the j-th group in the pathological modality cohort and the gene modality cohort, respectively.

[0071] The feasible process of obtaining fused multimodal medical data based on the fused and aligned multimodal features of the fusion optimization module includes:

[0072] After mitigating feature heterogeneity through multimodal feature alignment, a fusion optimization process is required. Specifically, firstly, G... com P com G spe and P spe The concatenation is performed at the 0th dimension, followed by dimensionality reduction via feature projection Φ, using a single fully connected layer as the feature projection layer. Subsequently, a fully connected layer S with a sigmoid activation function is applied to aggregate the multimodal features. This process can be expressed mathematically as follows:

[0073] F h =S(Φ(G) com ,P com G spe ,P spe )).

[0074] like Figure 2 As shown, the multimodal feature fusion module constructed in this embodiment fully embodies a multimodal perspective in its design. That is, in each stage of feature extraction, decomposition, alignment, and fusion, different modalities (pathology and genes) information are explicitly utilized and interacted to leverage their complementary advantages and improve prediction performance. First, in the multimodal feature extraction stage, pathology modality feature encoders and gene modality feature encoders are constructed separately to independently extract high-dimensional representations from the two modalities, fully preserving key information from each domain. Second, in the multimodal feature decomposition stage, the features of the two modalities are simultaneously decomposed into general features and modality-specific features. General features reflect shared biological patterns across modalities, while modality-specific features capture information unique to each modality, thus ensuring complementary cross-modal information. In the multimodal alignment stage, to map features from the two modalities into a shared latent space, two complementary mechanisms are introduced: one is an individual-level uncertainty minimization loss, used to reduce the difference in prediction results between different modalities for the same patient; the other is a feature-level multimodal cohort contrastive loss, used to enhance the similarity between cross-modal features and improve the consistency of representations between modalities.

[0075] To address the heterogeneity between features from different modalities, this embodiment uses cosine similarity for measurement. The idea is to measure the similarity between two feature vectors by calculating the angle between them in a high-dimensional space. For two feature vectors x and y from different modalities, the closer their directions, the more similar the features are, and the lower the heterogeneity; conversely, the larger the angle, the more significant the feature differences, and the higher the heterogeneity. The cosine distance is defined as follows:

[0076]

[0077] like Figure 3The cosine similarity results are shown in the diagram. Compared to direct modality fusion in the base model, the uncertainty minimization module and queue contrastive learning module of the proposed method in this embodiment both increase the cosine similarity between multimodalities, indicating that multimodal heterogeneity is reduced to some extent. Furthermore, this embodiment uses integral gradients to visually demonstrate the proposed method's focus on modal information. Integral gradients are gradient-based feature attribution methods used to interpret the prediction results of deep models. The idea is to measure the importance of each input feature to the result by calculating the gradient of the model output with respect to the input features and integrating it along the path from a baseline input (usually a zero vector) to the actual input. For example... Figure 4 The integral gradient heatmap shown indicates that using only single-modal data results in darker colors representing lower values, suggesting that single-modal data contributes less to the model. However, using multimodal data increases the values ​​in the heatmap, indicating that the model pays more attention to modal information and can utilize multimodal information more comprehensively.

[0078] Feasible implementation also includes parameter optimization of the multimodal feature fusion model:

[0079] The multimodal feature fusion model uses the loss function (uncertainty minimization loss L) from the above stages. e Multimodal queue comparison loss L p +L g The model parameters are optimized and trained using the following formula:

[0080] L=αL e +β(0.5×(L p +L g ));

[0081] Where L represents the overall loss function, and α and β are hyperparameters, which are set to 1 and 0.5 respectively in this embodiment.

[0082] Example 2

[0083] This embodiment also discloses a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0084] Example 3

[0085] This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0086] Example 4

[0087] This embodiment also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0088] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hybrid supervised fusion method based on multimodal medical data, characterized in that, Includes the following steps: Collect paired multimodal medical data and perform preprocessing; A multimodal feature fusion model is constructed based on a convolutional neural network. The multimodal feature fusion model includes a multimodal feature extraction module, a multimodal feature decomposition module, an uncertainty minimization module, a multimodal queue contrastive learning module, and a fusion optimization module. Based on the multimodal feature extraction module, feature extraction is performed on the preprocessed multimodal medical data to obtain multimodal features; The multimodal features are decomposed based on the multimodal feature decomposition module to obtain the decomposed multimodal features; Based on the uncertainty minimization module and the multimodal queue contrast learning module, the decomposed multimodal features are aligned. Based on the fused and aligned multimodal features obtained by the fusion optimization module, fused multimodal medical data is obtained; The process of performing feature decomposition on multimodal features based on the multimodal feature decomposition module to obtain the decomposed multimodal features includes: Using gene features and pathological image features as input, cosine similarity is used to measure the distance between features of different modalities to obtain similarity; based on the similarity, a similarity threshold is adaptively calculated to generate a feature mask; when the similarity is higher than the threshold, it is identified as a strongly correlated feature, and the generated feature mask is multiplied element-wise with the original features to retain the identified strongly correlated features; the strongly correlated features are subtracted from the original features of their respective modalities to obtain the corresponding modality-specific difference features; The process of aligning the decomposed multimodal features based on the uncertainty minimization module includes: The decomposed multimodal features are input into the fully connected neural network structure corresponding to each modality for processing to obtain the prediction probability distribution of each modality; based on the prediction probability distribution of each modality, the prediction group of each modality is determined; the prediction probability of each modality and its corresponding prediction group are input into the uncertainty minimization loss function, and the consistency of multimodal prediction is achieved when the prediction groups of two modalities are consistent. The process of aligning the decomposed multimodal features based on the multimodal queue contrast learning module includes: Two dual-queue structures are constructed to store feature representations of different modalities, resulting in a pathological modality queue and a gene modality queue. Based on prediction grouping, the pathological modality queue and the gene modality queue are grouped and initialized using a normal distribution, and dynamically updated using a momentum coefficient. During the dynamic update process, the distance between pathological features and gene features with the same group label is minimized, while the distance between pathological features and other group features in the pathological modality queue is maximized, thereby achieving multimodal feature alignment.

2. The method according to claim 1, characterized in that, The process of extracting features from preprocessed multimodal medical data based on the multimodal feature extraction module to obtain multimodal features includes: The multimodal medical data includes pathological images and genetic data; The thumbnail of the preprocessed pathological image is converted into a grayscale image. The threshold of the grayscale image is determined based on the Otsu thresholding method to obtain an initial binary mask. Morphological opening is performed on the initial binary mask, and connected components with an area smaller than a preset pixel are removed to obtain a foreground mask. The foreground mask is mapped back from the thumbnail to the original image to obtain a set of effective image patch coordinates. Based on the image patches, ResNet-50 is used as the feature extraction backbone to extract features and obtain pathological image features. A neural network is used to extract features from the preprocessed gene data to obtain gene features.

3. The method according to claim 1, characterized in that, The process of obtaining fused multimodal medical data based on the fused and aligned multimodal features obtained by the fusion optimization module includes: The strongly correlated and differential features corresponding to each modality are concatenated, and the concatenated features are dimensionality reduced by feature projection. Then, a fully connected layer with a Sigmoid activation function is used to fuse multimodal features.

4. The method according to claim 1, characterized in that, This also includes parameter optimization for the multimodal feature fusion model: We assign weights to the uncertainty minimization loss and the multimodal queue comparison loss, construct an overall loss function for the multimodal feature fusion model, and optimize and train the parameters of the multimodal feature fusion model based on the overall loss function.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-4.

7. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-4.

Images