Tumor survival analysis method based on multi-modal medical data fusion

Abstract

The invention provides a tumor survival analysis method based on multi-modal medical data fusion, which comprises the following steps: collecting and preprocessing multi-modal medical data of a glioma patient through a data collection and preprocessing module, the multi-modal medical data comprising an MRI image, a WSI image and RNAseq gene data; a survival analysis model is constructed, the survival analysis model comprises a feature extraction module, a feature fusion module and a survival analysis module, feature extraction is conducted on the preprocessed MRI image, the preprocessed WSI image and the preprocessed RNAseq gene data through the feature extraction module, and MRI image features, WSI image features and RNAseq gene data features are obtained; performing dimension reduction and tensor fusion on the MRI image features, the WSI image features and the RNAseq gene data features through a cross-modal Autoencoder network in a feature fusion module to generate fusion features; and inputting the fusion features into an FCN model and a Cox model through a survival analysis module, and carrying out survival probability analysis. According to the method, the survival prediction accuracy of the glioma patient is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a tumor survival analysis method based on multi-modal medical data fusion. Background Art

[0002] Glioblastoma is the most common primary intracranial tumor in adults, with characteristics such as high incidence, high recurrence rate, and high mortality rate. It is considered one of the tumors with relatively high treatment difficulty in the field of neurosurgery. Accurate preoperative quantitative assessment, grading diagnosis, gene typing, and survival prediction of glioblastoma are crucial for the treatment plan and prognosis prediction of patients.

[0003] In the survival analysis of glioblastoma patients, most traditional methods rely on single-modal data (such as pathological WSI images, MRI images, or gene RNAseq data), and it is difficult to comprehensively reflect the complexity of glioblastoma and its diverse biological characteristics, resulting in limited prediction accuracy. For example, MRI images can provide morphological information of tumors, but cannot reflect the molecular heterogeneity of tumors; pathological WSI images can reveal the cytomorphological characteristics of tumors, but lack information at the functional or molecular level; gene RNAseq data provides gene expression information of tumors, but lacks spatial information and cannot directly reflect the morphology or tissue structure of tumors. Therefore, single-modal data often cannot meet the needs of comprehensively analyzing glioblastoma.

[0004] In view of the complexity of glioblastoma, more and more studies in existing technical solutions have begun to focus on survival analysis methods based on multi-modal data fusion. Multi-modal data fusion aims to improve the accuracy and robustness of survival analysis by integrating information from different data sources. Existing multi-modal data fusion technologies mainly focus on how to effectively extract meaningful features from data of different modalities and fuse these features. Currently, there are two most common existing technical solutions: one is feature-level fusion. Feature-level fusion methods usually involve independently extracting features from each modality first, and then combining these features into a unified feature vector. Feature extraction methods mostly rely on manually designed feature extraction algorithms or use deep learning models (such as convolutional neural networks) for automatic feature learning. The extracted features are directly concatenated and then used for survival analysis modeling. The other is model-based fusion. Model-based fusion methods design a joint model to simultaneously process data from different modalities. These methods can directly perform feature extraction, fusion, and prediction within a unified network framework. For example, some deep learning models receive data of different modalities through a multi-input structure and perform information fusion through shared layers or branch networks within the model.

[0005] However, the existing technologies have the following disadvantages:

[0006] (1) The differences between data modalities are significant, resulting in high difficulty in feature extraction and inability to ensure the alignment of features from different modalities.

[0007] (2) The dimensionality of features from multiple modalities is large. The current tensor concatenation method is simple but fails to consider the heterogeneity between modalities, making it difficult to fully utilize the information between different modalities. While using the tensor fusion method can take into account the different information of data from different modalities, due to the use of the vector outer product fusion method, the data scale increases rapidly with the fused modalities, increasing the computational complexity. Existing data fusion methods fail to fully utilize the advantages of each modality, resulting in low accuracy of survival prediction.

[0008] (3) Existing methods lack strong evidence in terms of model interpretability, making it difficult to judge the key role played by multiple data in result inference and lacking interpretability. Summary of the Invention

[0009] To solve the above problems, the object of the present invention is to provide a tumor survival analysis method based on multi-modal medical data fusion. Through a task-specific modal task network, features of each modality are extracted, and then a cross-modal Autoencoder network is used for dimensionality reduction and alignment of modal features. The tensor fusion method is used to comprehensively utilize multi-modal data, and the fused features are used for downstream tasks such as survival analysis of glioma patients, improving the prediction ability of the survival analysis model and enhancing the CI value, thereby achieving accurate prognosis prediction and survival analysis of glioma patients.

[0010] According to the first aspect of the present invention, there is provided a tumor survival analysis method based on multi-modal medical data fusion, including the following steps: Step S1, collecting and preprocessing multi-modal medical data of glioma patients through a data collection and preprocessing module, where the multi-modal medical data includes MRI images, WSI images, and RNAseq gene data; Step S2, constructing a survival analysis model, where the survival analysis model includes a feature extraction module, a feature fusion module, and a survival analysis module. Respectively extract features from the preprocessed MRI images, preprocessed WSI images, and preprocessed RNAseq gene data through the feature extraction module to obtain MRI image features, WSI image features, and RNAseq gene data features; Step S3, perform dimensionality reduction and tensor fusion on the MRI image features, the WSI image features, and the RNAseq gene data features through the cross-modal Autoencoder network in the feature fusion module to generate fused features; Step S4, input the fused features into the FCN and Cox models through the survival analysis module for survival probability analysis.

[0011] Optionally, in step S2, the feature extraction module extracts features from the preprocessed MRI image, the preprocessed WSI image, and the preprocessed RNAseq gene data respectively to obtain MRI image features, WSI image features, and RNAseq gene data features, including: using the ResNet50 network to extract features from the preprocessed MRI image to obtain MRI image features; using the HIPT network to extract features from the preprocessed WSI image to obtain WSI image features; using the gene-typing pre-trained FCN network to extract features from the preprocessed RNAseq gene data to obtain RNAseq gene data features.

[0012] Optionally, in step S3, the cross-modal Autoencoder network reduces the dimensions of the MRI image features, the WSI image features, and the RNAseq gene data features into three low-dimensional feature tensors and aligns them, and performs tensor fusion on the aligned three low-dimensional feature tensors to obtain fused features.

[0013] Optionally, the method further includes training the survival analysis model using a deep learning framework and avoiding overfitting through cross-validation and early stopping strategies.

[0014] Optionally, the training parameters of the survival analysis model are a learning rate of 0.001, the Adam optimizer is used as the optimizer, the number of iterations is 150 times, and 10-fold cross-validation is used for cross-validation.

[0015] Optionally, the method further includes performing model interpretability analysis on the survival analysis model through Attention Map and SHAP.

[0016] According to the second aspect of the present invention, there is provided a tumor survival analysis system based on multi-modal medical data fusion, including: a data collection and preprocessing module for collecting and preprocessing multi-modal medical data of glioma patients, the multi-modal medical data including MRI images, WSI images, and RNAseq gene data; a feature extraction module for extracting features from the preprocessed MRI image, the preprocessed WSI image, and the preprocessed RNAseq gene data respectively to obtain MRI image features, WSI image features, and RNAseq gene data features; a feature fusion module for reducing the dimensions and performing tensor fusion on the MRI image features, the WSI image features, and the RNAseq gene data features using a cross-modal Autoencoder network to generate fused features; a survival analysis module for inputting the fused features into an FCN and a Cox model to perform survival probability analysis.

[0017] According to a third aspect of the present invention, there is provided an electronic device, including a processor and a memory for storing a program. Wherein, the program includes instructions which, when executed by the processor, cause the processor to perform the steps performed by the method of the first aspect as described above.

[0018] According to a fourth aspect of the present invention, there is provided a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method of the first aspect as described above.

[0019] Adopting the above technical solutions, the present invention has the following beneficial effects:

[0020] The present invention effectively integrates and improves the accuracy of MRI, WSI, and RNAseq modal data fusion. For the feature extractor of a single modality, it is replaceable and can be replaced at any time with a feature extractor that has a better effect for that modality. At the same time, a cross-modal AutoEncoder is designed, which can reduce the high-dimensional modal features to low-dimensional features and align the features of each modality, so that the fusion vector generated after passing through the feature fusion network can be as small as possible, solving the problem of information mismatch between modalities and effectively improving the prediction accuracy of the survival analysis model. The present invention shows stronger generalization ability on different data sets, can effectively avoid overfitting, and improves the accuracy of survival prediction for glioma patients. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 It is a flowchart of the steps of the tumor survival analysis method based on multi-modal medical data fusion of the present invention;

[0023] Figure 2 For Figure 1 The overall flowchart of the corresponding tumor survival analysis method based on multi-modal medical data fusion. Detailed Embodiments

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0025] The objective of the present invention is to provide a tumor survival analysis method based on multi-modal medical data fusion, mainly to solve the problem of difficult prediction of the prognosis survival period of glioma patients. Glioma, as the most common primary intracranial tumor, accounts for about 80% of adult malignant tumors. Due to its characteristics such as high mortality, incidence, and recurrence rate, it is considered one of the most challenging and refractory tumors in neurosurgical treatment. Statistics show that the overall prognosis of glioblastoma is poor, the median survival period of patients is less than 15 months, and there is a large difference in the survival period of patients. The length of their survival period is affected by multiple factors such as tumor molecular factors, patient age, and tumor location and size. Therefore, accurate prognosis prediction and survival analysis are of great significance for the individualized treatment of patients. To make the above objectives, features, and advantages of the present invention more obvious and understandable, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] See Figure 1 , Figure 2 , a tumor survival analysis method based on multi-modal medical data fusion of the present invention includes the following steps:

[0027] Step S1: Collect and preprocess multi-modal medical data of glioma patients through a data collection and preprocessing module. The multi-modal medical data includes MRI images, WSI images, and RNAseq gene data;

[0028] Step S2: Construct a survival analysis model. The survival analysis model includes a feature extraction module, a feature fusion module, and a survival analysis module. Respectively extract features from the preprocessed MRI images, preprocessed WSI images, and preprocessed RNAseq gene data through the feature extraction module to obtain MRI image features, WSI image features, and RNAseq gene data features;

[0029] Step S3: Perform dimensionality reduction and tensor fusion on the MRI image features, the WSI image features, and the RNAseq gene data features through the cross-modal Autoencoder network in the feature fusion module to generate fusion features;

[0030] Step S4: Input the fusion features into the FCN and Cox models through the survival analysis module for survival probability analysis.

[0031] The method of the present invention first extracts features from each modality data, then uses a cross-modal Autoencoder to perform dimensionality reduction and alignment of the features of each modality, generates a unified feature vector through tensor fusion, and finally uses a Cox regression model for survival analysis. This method can not only integrate the information of different modality data, but also effectively improve the prediction performance of survival analysis.

[0032] Optionally, in step S2, the feature extraction module respectively extracts features from the preprocessed MRI images, preprocessed WSI images, and preprocessed RNAseq gene data to obtain MRI image features, WSI image features, and RNAseq gene data features, including: using a ResNet50 network to extract features from the preprocessed MRI images to obtain MRI image features; using a HIPT network to extract features from the preprocessed WSI images to obtain WSI image features; using a gene-typing pre-trained FCN network to extract features from the preprocessed RNAseq gene data to obtain RNAseq gene data features.

[0033] Optionally, in step S3, the cross-modal Autoencoder network reduces the MRI image features, the WSI image features, and the RNAseq gene data features into three low-dimensional feature tensors and aligns them, and performs tensor fusion on the three aligned low-dimensional feature tensors to obtain fused features.

[0034] Optionally, the method further includes training the survival analysis model using a deep learning framework and avoiding overfitting through cross-validation and early stopping strategies.

[0035] Optionally, the training parameters of the survival analysis model are a learning rate of 0.001, the optimizer uses an Adam optimizer, the number of iteration rounds is 150 times, and the cross-validation uses 10-fold cross-validation.

[0036] Optionally, the method further includes performing model interpretability analysis on the survival analysis model through Attention Map and SHAP.

[0037] Specifically, the solution of the present invention is further described according to the following example:

[0038] 1. Data collection and preprocessing

[0039] Collect multi-modal medical data of glioma patients, including: MRI images, WSI images, and RNAseq gene data.

[0040] Specifically: Collect the magnetic resonance MR images of the brain of glioma patients, that is, MRI images, and uniformly save the data format as a.nii format file; collect the digital pathological images of H&E stained sections of glioma patients (Whole Slide Image, WSI), that is, WSI images, and uniformly save the data format as a.svs format file; collect the RNAseq gene data of glioma patients, and uniformly save the data format as an xlsx format.

[0041] For MRI images, for the T2 modality and FlAIR modality volume data, extract 5 slices at equal intervals respectively, and compress each single slice into an image of 224*224 pixels. Therefore, each patient corresponds to an MRI data of 10*1*224*224 (1 is the number of channels, because MRI is a grayscale image and the image channel is 1);

[0042] For WSI images, at a magnification of 20 times, the images are cropped into three sizes, namely 4096*4096 tissue-level patches, 256*256 cell-tissue-level patches, and 16*16 cell-feature-level patches;

[0043] For RNAseq gene data, obtain its standardized FPKM (Fragments Per Kilobase of exon per Million reads mapped) data. WGCNA (Weighted Gene Co-expression Network Analysis) is a tool for exploring the interactions between genes and constructing a gene co-expression network. By associating the glioma classification phenotype data with genes, different gene modules are clustered hierarchically, and key genes within the modules are selected as potential biomarkers. In the survival analysis model of the present invention, WGCNA is used to screen for core genes strongly related to the glioma type of patients.

[0044] 2. Construction of the survival analysis model

[0045] a. Feature extraction module:

[0046] MRI feature extraction: The ResNet50 network can effectively avoid the problem of gradient disappearance, improve the depth and accuracy of feature extraction, has better accuracy compared with ResNet34, and at the same time has a relatively small computational complexity. In the survival analysis model of the present invention, it is used for the feature extraction of magnetic resonance MR images. The ResNet50 network pre-trained by MRI classification is used for the feature extraction of patients' MRIs, and a 1*2048-dimensional feature vector is obtained.

[0047] WSI Feature Extraction: HIPT (Hierarchical Image Pretraining Transformer) provides an efficient solution for processing WSI images, especially suitable for ultra-high-resolution images. Through hierarchical self-supervised learning, HIPT can reduce computational overhead and memory consumption when processing high-resolution images, while extracting fine-grained local features and global structure information. Its architecture based on Vision Transformer (ViT) can better model long-range dependencies, capture the complex relationships between cells and tissue structures in the image, and has a powerful self-supervised learning ability to perform pre-training on unlabeled data.

[0048] In addition, HIPT shows excellent accuracy in multiple medical image tasks, has strong transfer learning ability, and can adapt to different types of medical image data. It is an efficient and accurate feature extraction method for processing WSI images. In the survival analysis model of the present invention, the WSI is subjected to HIPT feature extraction to obtain a 1*384-dimensional feature vector.

[0049] RNAseq Feature Extraction: For the key genes that have been screened, a gene-typing pre-trained FCN (Fully Connected Network) is used for feature extraction to obtain a 1*512-dimensional feature vector.

[0050] b. Feature Fusion Module:

[0051] After feature extraction, the present invention realizes dimensionality reduction of the three-modal features and alignment and fusion between modalities through a cross-modal AutoEncoder. The feature vectors of each modality are connected to an AutoEncoder, and the output at the Encoder end is a 1*16-dimensional tensor used to represent the dimensionality-reduced features of each modality. Therefore, the data of the three modalities of MRI, WSI, and RNAseq each correspond to a dimensionality-reduced 1*16-dimensional tensor. The three dimensionality-reduced feature tensors are fused to obtain a final 1*4096-dimensional fusion feature for downstream tasks.

[0052] c. Survival Analysis Module:

[0053] The fusion feature is input into an FCN and a Cox model for survival probability analysis, and finally the survival probability of glioma patients is obtained.

[0054] It should be understood that the survival analysis model adopted by the present invention is a Cox regression model.

[0055] 3. Model Training

[0056] Train the established deep learning model, i.e., the survival analysis model, using the Pytorch (1.4.0) deep learning framework. Its training parameters are as follows: learning rate 0.001, Adam optimizer is adopted, the number of iterative rounds is 150, 10-fold cross-validation is used, and a learning strategy of early stopping is applied to avoid overfitting.

[0057] 4. Model interpretability

[0058] Interpretability analysis is carried out through Attention Map and SHAP (SHapley Additive explanation), which can show the data attention focus of the model's prediction of the patient's survival probability, as well as the contribution degree of different modality features to the result inference, having clinical significance and model interpretability.

[0059] In summary, the present invention effectively integrates and improves the accuracy of the fusion of MRI, WSI, and RNAseq modality data. For the feature extractors of individual modalities, they are replaceable and can be replaced at any time with feature extractors that have better effects for that modality. At the same time, a cross-modal AutoEncoder is designed, which can reduce the high-dimensional modality features to low-dimensional features and align the features of each modality, so that the fusion vector generated after passing through the feature fusion network can be as small as possible, solving the problem of information mismatch between modalities and effectively improving the prediction accuracy of the survival analysis model. The present invention shows stronger generalization ability on different data sets, can effectively avoid overfitting, and improves the accuracy of survival prediction for glioma patients.

[0060] As another example, the present invention also provides a tumor survival analysis system based on multi-modal medical data fusion, including:

[0061] A data collection and preprocessing module for collecting and preprocessing multi-modal medical data of glioma patients, where the multi-modal medical data includes MRI images, WSI images, and RNAseq gene data;

[0062] A feature extraction module for respectively extracting features from the preprocessed MRI images, preprocessed WSI images, and preprocessed RNAseq gene data to obtain MRI image features, WSI image features, and RNAseq gene data features;

[0063] A feature fusion module for using a cross-modal Autoencoder network to perform dimensionality reduction and tensor fusion on the MRI image features, the WSI image features, and the RNAseq gene data features to generate fusion features;

[0064] A survival analysis module, which is used to input the fused features into an FCN and a Cox model for survival probability analysis.

[0065] It should be understood that the tumor survival analysis system based on multimodal medical data fusion in the embodiments of the present invention is used to implement the corresponding methods in the foregoing multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated herein.

[0066] It should be noted that the present invention has been verified through experiments. On the glioma dataset, the method of the present invention has a significant improvement in the CI value index compared with the traditional method, indicating the feasibility and superiority of this method in survival analysis.

[0067] As another example, the present invention also provides an electronic device. Now, the electronic device that can be used as the server or client of the present invention will be described. It is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described herein and / or claimed.

[0068] The electronic device may include: a processor, a communications interface, a memory, and a communication bus.

[0069] The processor, the communications interface, and the memory communicate with each other through the communication bus. The communications interface is used to communicate with other electronic devices or servers.

[0070] The processor is used to execute a program, and specifically can execute the relevant steps in the foregoing method embodiments.

[0071] Specifically, the program may include program code, and the program code includes computer operation instructions.

[0072] The processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. One or more processors included in the intelligent device may be of the same type, such as one or more CPUs; or may be of different types, such as one or more CPUs and one or more ASICs.

[0073] The memory is used to store programs. The memory may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

[0074] When the program is executed by the processor, it is used to cause the electronic device to execute the tumor survival analysis method based on multi-modal medical data fusion of the present invention.

[0075] In addition, for the specific implementation of each step in the program, reference may be made to the corresponding steps and the corresponding descriptions in the units in the foregoing method embodiments, which will not be elaborated herein. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the devices and modules described above may refer to the corresponding process descriptions in the foregoing method embodiments, which will not be elaborated herein.

[0076] An exemplary embodiment of the present invention further provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the methods of the embodiments of the present invention are implemented. Reference may be made to the corresponding process descriptions in the foregoing method embodiments, which will not be elaborated herein.

[0077] The method according to the embodiments of the present invention described above may be implemented in hardware, firmware, or may be implemented as software or computer code stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or may be implemented as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium and downloaded through a network and stored in a local recording medium, so that the method described herein may be processed by such software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or an FPGA). It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component (such as RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods described herein are implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0078] Thus far, specific embodiments of the present invention have been described. Other embodiments are within the scope of the appended claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous.

[0079] It should be understood that although this specification is described in terms of various embodiments, not every embodiment contains only a single independent technical solution. This narrative style of the specification is merely for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the embodiments of the present invention, and not to limit the embodiments of the present invention. Those of ordinary skill in the relevant technical field can also make various changes and modifications without departing from the spirit and scope of the embodiments of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present invention, and the patent protection scope of the embodiments of the present invention should be defined by the claims.

Claims

1. A tumor survival analysis method based on multi-modal medical data fusion, characterized in that, It includes the following steps: Step S1: Collect and preprocess multi-modal medical data of glioma patients through a data collection and preprocessing module. The multi-modal medical data includes MRI images, WSI images, and RNAseq gene data; Step S2: Construct a survival analysis model. The survival analysis model includes a feature extraction module, a feature fusion module, and a survival analysis module. Respectively extract features from the preprocessed MRI images, preprocessed WSI images, and preprocessed RNAseq gene data through the feature extraction module to obtain MRI image features, WSI image features, and RNAseq gene data features; Step S3: Reduce the dimensions and perform tensor fusion on the MRI image features, the WSI image features, and the RNAseq gene data features through a cross-modal Autoencoder network in the feature fusion module to generate fused features; Step S4: Input the fused features into an FCN and a Cox model through the survival analysis module to perform survival probability analysis.

2. The method according to claim 1, wherein In step S2, respectively extract features from the preprocessed MRI images, preprocessed WSI images, and preprocessed RNAseq gene data through the feature extraction module to obtain MRI image features, WSI image features, and RNAseq gene data features, including: Use a ResNet50 network to extract features from the preprocessed MRI images to obtain MRI image features; Use a HIPT network to extract features from the preprocessed WSI images to obtain WSI image features; Use a pre-trained FCN network for gene typing to extract features from the preprocessed RNAseq gene data to obtain RNAseq gene data features.

3. The method according to claim 2, characterized in that, In step S3, the cross-modal Autoencoder network reduces the MRI image features, the WSI image features, and the RNAseq gene data features into three low-dimensional feature tensors and aligns them, and performs tensor fusion on the aligned three low-dimensional feature tensors to obtain fused features.

4. The method according to claim 1, wherein The method further includes training the survival analysis model using a deep learning framework and avoiding overfitting through cross-validation and early stopping strategies.

5. The method according to claim 4, wherein The training parameters of the survival analysis model are a learning rate of 0.001, an Adam optimizer is used as the optimizer, the number of iteration rounds is 150 times, and 10-fold cross-validation is used for cross-validation.

6. The method according to claim 1, characterized in that The method further includes performing model interpretability analysis on the survival analysis model through Attention Map and SHAP.

7. A tumor survival analysis system based on multi-modal medical data fusion, characterized in that, It includes: A data collection and preprocessing module for collecting and preprocessing multi-modal medical data of glioma patients. The multi-modal medical data includes MRI images, WSI images, and RNAseq gene data; A feature extraction module for respectively extracting features from the preprocessed MRI images, preprocessed WSI images, and preprocessed RNAseq gene data to obtain MRI image features, WSI image features, and RNAseq gene data features; A feature fusion module, which is used to reduce the dimensions and perform tensor fusion on the MRI image features, the WSI image features, and the RNAseq gene data features by using a cross-modal Autoencoder network to generate fused features; A survival analysis module, which is used to input the fused features into an FCN and a Cox model for survival probability analysis.

8. An electronic device, characterized in that, Comprising: A processor; A memory for storing programs; Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor is caused to execute the steps performed by the method according to any one of claims 1-6.

9. A computer storage medium, characterized in that, A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1-6 is implemented.

Images