Prognosis prediction method based on multi-modal intermediate fusion

Abstract

The invention relates to a prognosis prediction method based on multi-modal intermediate fusion, and the method comprises the steps: extracting pathological features from a pathological image through a deep convolutional neural network and predefined omics, and carrying out the aggregation of the features through multi-instance learning, so as to form a pathological representation, the method comprises the following steps: extracting radioactive features by using a deep convolutional neural network and predefined omics to form a radiology representation, and then integrating a pathological representation, the radiology representation and clinical variables by using a deep survival network to generate a multi-modal prognosis prediction score. Compared with the prior art, information of three modals of pathology, radiology and clinic is integrated through the deep survival network, and traditional prognosis prediction indexes and prediction performance of a single-modal model are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of tumor image analysis, and in particular to a prognosis prediction method based on multimodal intermediate fusion. Background Art

[0002] Renal cell carcinoma is considered to be one of the most common malignant tumors in the urinary system. Among them, clear cell renal cell carcinoma (ccRCC), as the main subtype of RCC, accounts for about 70% of renal cancer cases. Although surgery is the main initial treatment method, it is worth noting that about 20%-30% of ccRCC patients will experience recurrence and metastasis after surgery, which has a significant impact on the overall survival rate of patients. Therefore, accurately identifying people at high risk of recurrence is of great significance for clinicians to make better decisions and intervene in a timely manner.

[0003] Currently, researchers have developed a variety of tools to assess the risk of tumor recurrence after surgery for renal clear cell carcinoma based on clinical characteristics (including TNM stage and histopathological grade). Among them, the Leibovich score and UISS score are the most commonly used evaluation criteria. However, in the clinical decision-making process, doctors have observed significant differences in the survival outcomes of patients in the same risk subgroup, indicating that traditional clinical characteristics may not be sufficient to fully capture the intrinsic heterogeneity of tumors. In addition, a prospective study of eight prognostic prediction models widely used in a clinical trial cohort showed that the predictive accuracy of all developed models was significantly lower than the results initially reported.

[0004] In recent years, some studies have begun to apply artificial intelligence for prognosis prediction. Most models use single-modality data (for example, only pathology, imaging or genetic information). In contrast, multimodal models can integrate comprehensive and complementary medical data to improve the accuracy of predicting postoperative recurrence in patients. One study developed a multimodal recurrence model for ccRCC by integrating clinical features, single nucleotide polymorphisms, and whole-section images of tumor tissue. However, the single nucleotide polymorphism sequencing process is complex and costly. At present, there is still a lack of a prognosis prediction model that can effectively integrate multimodal information. Summary of the invention

[0005] The purpose of the present invention is to provide a prognosis prediction method based on multimodal intermediate fusion, which extracts pathological features from pathological images through residual deep convolutional neural networks and predefined omics, and aggregates these features using multi-instance learning to form a pathological representation. At the same time, residual deep convolutional neural networks and predefined omics are used to extract radiological features to form an image representation, and then a deep survival network is used to integrate the pathological representation, image representation and clinical information to generate a multimodal prognosis prediction score for accurate prognosis prediction.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A prognosis prediction method based on multimodal intermediate fusion, comprising the following steps:

[0008] Step S1, data acquisition: obtaining whole slide images (WSI) stained with hematoxylin-eosin (HE) as pathological images, obtaining arterial phase three-dimensional computed tomography (3D CT) sequences as radiological images, and obtaining clinical variables of patients;

[0009] Step S2, data preprocessing: performing block cutting, staining normalization processing and filtering preprocessing on the pathological image, performing data format conversion and tumor area delineation preprocessing on the radiological image, and performing coding preprocessing on the clinical variables;

[0010] Step S3, pathological image feature representation: extract manually defined features and semantic features based on deep convolutional neural network for each pathological patch after data preprocessing, and aggregate features of different pathological patches through multi-instance learning method to obtain feature representation of pathological images;

[0011] Step S4, radiological image feature representation: based on the radiological image and the outlined tumor contour, manually define features and extract semantic features based on a deep convolutional neural network, and perform dimensionality reduction processing through principal component analysis to obtain a feature representation of the radiological image;

[0012] Step S5, constructing a multimodal prognostic prediction model: using a deep survival network to integrate the features of pathological images, radiological images, and clinical variables, and output a multimodal prognostic prediction score.

[0013] The preprocessing of the pathological image in step S2 specifically includes the following steps:

[0014] Cutting: Cut the pathological image into small blocks of preset size;

[0015] Color normalization: Perform color normalization on each pathological patch in RGB three-channel format;

[0016] Filtering: The OTSU method is used to calculate the proportion of tissue area in each pathological block, and the pathological blocks with a tissue area proportion less than a preset value are filtered.

[0017] The preprocessing of the radiological image in step S2 specifically includes the following steps:

[0018] Format conversion: convert radiology images in DICOM format to NII format;

[0019] Resampling: Resample radiological images to set slice intervals;

[0020] Grayscale normalization: normalize the grayscale of radiological images to a preset range;

[0021] Tumor area delineation: Delineate the tumor contour for each 2D scan section in the 3D sequence of radiological images;

[0022] Region of interest extraction: By outlining the tumor area, the 3D region of interest containing the tumor is cropped from the original radiological image;

[0023] Selection of the largest tumor section: Based on the cropped 3D region of interest containing the tumor and the tumor annotation, the 2D image section containing the largest tumor is selected.

[0024] The tumor region delineation uses the deep segmentation learning model nnU-Net pre-trained in public data to automatically delineate the renal tumor, followed by manual film reading to modify the tumor boundaries that are incorrectly segmented by the model.

[0025] The clinical variables include basic vital signs indicators and tumor-related indicators, wherein the basic vital signs indicators include gender, age and ECOGPS score, and the tumor-related indicators include tumor size, pT stage, pN stage, TNM stage, necrosis condition, and whether it is sarcomatoid renal cell carcinoma.

[0026] In step S2, the basic vital signs indicators are coded and preprocessed as follows:

[0027] For the gender indicator, male was coded as 1 and female was coded as 0;

[0028] For the age indicator, the age exceeding the preset age value is coded as 1, and the age not exceeding the preset age value is coded as 0;

[0029] Regarding the ECOG PS score, the ECOG PS score is used to assess the patient's physical condition, with ECOG PS ≥ 1 being coded as 1 and ECOG PS < 1 being coded as 0.

[0030] In step S2, encoding preprocessing of tumor-related indicators is specifically performed as follows:

[0031] For tumor size, the maximum diameter of the tumor is greater than or equal to the preset value and is coded as 1, and the maximum diameter of the tumor is less than the preset value and is coded as 0;

[0032] For pT staging, the pT staging is used to evaluate the size and infiltration of the tumor, with T1 coded as 1, T2 coded as 2, and T3 coded as 3;

[0033] For pN staging, the pN staging is used to evaluate lymph node status, with N0 coded as 0 and N1 coded as 1;

[0034] For TNM staging, stage I is coded as 1, stage II is coded as 2, and stage III is coded as 3;

[0035] For necrosis, the presence of tumor necrosis was coded as 1, and the absence of necrosis was coded as 0;

[0036] As for whether it is sarcomatoid renal cell carcinoma, it is coded as 1 if it is sarcomatoid renal cell carcinoma, and it is coded as 0 if it is not sarcomatoid renal cell carcinoma.

[0037] In step S3, the semantic features based on the deep convolutional neural network are obtained by extracting the representation of the image through the ResNet network; the specific steps of extracting the manually defined features include:

[0038] Use the pre-trained HoverNet model to segment the cell nucleus and predict the cell type;

[0039] Based on the cell nucleus segmentation results and pathological images, the HistomicsTK package in Python was used to extract features;

[0040] For each pathological patch, the average value of the HistomicsTK feature of the most frequently occurring cell type is calculated and concatenated with the one-hot encoding of the cell type to form the manually defined features extracted for each pathological patch.

[0041] The multi-instance learning for aggregating features of different pathological patches in step S3 adopts a Transformer structure.

[0042] In the step S4, the semantic features based on the deep convolutional neural network are extracted through the ResNet network; the manually defined features include shape features, first-order statistical features, texture features, filtering features and wavelet features defined in the Pyradiomics package; when performing principal component analysis on the shape, first-order statistical, texture, filtering, and wavelet features, only the principal components whose explanation ratio exceeds the preset value are retained.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) Compared with the existing prognosis prediction models that use pathological images or genetic information, the present invention uses information from three modalities, namely pathology, imaging and clinical information, which are easier to obtain and more comprehensive in clinical practice. Information acquisition is easier and the data that can be obtained is richer, which can improve the training accuracy of the prognosis prediction model.

[0045] (2) The present invention utilizes an intermediate fusion approach to take into account both the situation of missing data and the performance of the model.

[0046] (3) The present invention uses a deep survival network to integrate multimodal prognostic information, significantly improving the performance of traditional prognostic prediction indicators and single-modality models in predicting recurrence risk. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2 A flow chart of sample screening in an embodiment of the present invention;

[0049] Figure 3 A consistency calibration curve between the nomogram prediction and the actual result established by the multimodal prognostic prediction model in the embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the effect of using the multimodal prognostic prediction model for patient risk stratification in an embodiment of the present invention, where Part A is the stratification results of all cohorts, Part B is the results of the training set, Part C is the results of the validation set, and Part D is the results of the test set. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0052] This embodiment takes the prognosis prediction of patients with renal clear cell carcinoma as an example to provide a prognosis prediction method based on multimodal intermediate fusion. Figure 1 As shown, the method comprises the following steps:

[0053] Step S1, data acquisition: obtain the 40-fold magnified whole slide image (WSI) stained with hematoxylin-eosin (HE) of the renal clear cell carcinoma patient after surgery as the pathological image, obtain the patient's preoperative arterial phase three-dimensional computed tomography (3D CT) sequence from the database as the radiological image, and obtain the patient's clinical variables.

[0054] The pathological images in this embodiment are derived from HE staining images of renal clear cell carcinoma patients after surgery in multiple hospitals and renal tumor diagnostic slice images in the TCGA database. The radiological images in this embodiment are derived from preoperative arterial phase 3D CT sequences of renal clear cell carcinoma patients collected in multiple hospitals.

[0055] Step S2, data preprocessing: performing block cutting, staining normalization processing and filtering preprocessing on the pathological image, performing data format conversion and tumor area delineation preprocessing on the radiological image, and performing coding preprocessing on the clinical variables.

[0056] Step S21, preprocessing the pathological image, specifically includes the following steps:

[0057] Cutting: Cut the 40x magnified pathological image into small pathological blocks, each with a size of 768×768 pixels;

[0058] Color normalization: Color normalization is performed on each pathological patch of 768×768 pixels in RGB three-channel format;

[0059] Filtering: The OTSU method was used to calculate the ratio of tissue area in each pathological block, and the pathological blocks with a tissue area ratio less than 0.3 were filtered.

[0060] Step S22, preprocessing the radiological image, specifically includes the following steps:

[0061] Format conversion: convert radiology images in DICOM format to NII format;

[0062] Resampling: Radiological images were resampled with a slice interval of 1 mm;

[0063] Grayscale normalization: Normalize the grayscale of radiological images to [-140, 260];

[0064] Tumor area delineation: Delineate the tumor contour for each 2D scan section in the 3D sequence of radiological images;

[0065] Region of interest extraction: By outlining the tumor area, the 3D region of interest containing the tumor is cropped from the original radiological image;

[0066] Selection of the largest tumor section: Based on the cropped 3D region of interest containing the tumor and the tumor annotation, the 2D image section containing the largest tumor is selected.

[0067] In this embodiment, the tumor region is delineated using the deep segmentation learning model nnU-Net pre-trained in public data to automatically delineate the renal tumor, followed by manual film reading to confirm or modify the tumor boundaries that are incorrectly segmented by the model.

[0068] Step S23: preprocessing of clinical variables, specifically:

[0069] In this embodiment, clinical variables include basic vital signs and tumor-related indicators, wherein the basic vital signs include gender, age and ECOGPS score, and the tumor-related indicators include tumor size, pT stage, pN stage, TNM stage, necrosis, and whether it is sarcomatoid renal cell carcinoma.

[0070] For the gender indicator, male was coded as 1 and female was coded as 0;

[0071] For the age indicator, those aged over 60 years were coded as 1, and those under 60 years were coded as 0;

[0072] For the ECOGPS score (used to assess the patient's physical condition), ECOGPS ≥ 1 was coded as 1, and ECOGPS < 1 was coded as 0.

[0073] For tumor size, a tumor with a maximum diameter ≥10 cm was coded as 1, and a tumor with a maximum diameter <10 cm was coded as 0;

[0074] For pT staging (used to assess tumor size and invasion), T1 is coded as 1, T2 as 2, and T3 as 3;

[0075] For pN staging (used to assess lymph node status), N0 is coded as 0 and N1 is coded as 1;

[0076] For TNM staging, stage I is coded as 1, stage II is coded as 2, and stage III is coded as 3;

[0077] For necrosis, the presence of tumor necrosis was coded as 1, and the absence of necrosis was coded as 0;

[0078] As for whether it is sarcomatoid renal cell carcinoma, it is coded as 1 if it is sarcomatoid renal cell carcinoma, and it is coded as 0 if it is not sarcomatoid renal cell carcinoma.

[0079] Step S3, pathological image feature representation: extract manually defined features and semantic features based on deep convolutional neural networks for each pathological patch after data preprocessing, and aggregate the features of different pathological patches through the multi-instance learning method to obtain the feature representation of the pathological image.

[0080] Specifically, the semantic features based on the deep convolutional neural network are obtained by extracting the image representation through the ResNet network. Specifically, the ResNet network pre-trained on the ImageNet dataset is used to obtain the corresponding image representation by removing its last classification layer.

[0081] The specific steps to extract manually defined features include:

[0082] Use the HoverNet model pre-trained on the PanNuke dataset to segment the cell nucleus and predict the cell type (can be any of tumor cells, inflammatory cells, junction cells, dead cells and normal epithelial cells);

[0083] Based on the cell nucleus segmentation results and pathological images, the HistomicsTK package in Python was used to extract features;

[0084] For each pathological patch, the average value of the HistomicsTK feature of the most frequently occurring cell type is calculated and concatenated with the one-hot encoding of the cell type to form the manually defined features extracted for each pathological patch.

[0085] After completing the feature representation of each pathological block, the Transformer-based multi-instance learning method is used to aggregate the features of different blocks to generate the feature representation of the entire pathological slice.

[0086] In this embodiment, a pathology-based prognosis prediction result is obtained by connecting linear layers. The prediction result is compared with the patient's actual survival label, and the parameters of the multi-instance learning part are optimized by calculating the negative logarithmic loss.

[0087] Step S4, radiological image feature representation: Based on the radiological image and the outlined tumor contour, manually define features and extract semantic features based on a deep convolutional neural network, and perform dimensionality reduction processing through principal component analysis to obtain a feature representation of the radiological image.

[0088] Among them, the semantic features based on deep convolutional neural networks are extracted through the ResNet network; specifically, the extracted semantic features take the maximum tumor section obtained in the radiological image preprocessing stage as input, and use the ResNet model pre-trained on the ImageNet dataset, removing the last classification layer to obtain the semantic feature representation of the image.

[0089] Manually defined features used the three-dimensional region of interest and tumor annotations containing the tumor obtained in the radiological image preprocessing stage to extract shape features, first-order statistical features, texture features, filtering features, and wavelet features defined in the Pyradiomics package; when performing principal component analysis on shape, first-order statistical, texture, filtering, and wavelet features, only principal components with an explanation ratio exceeding 0.8 were retained.

[0090] The principal components extracted from each feature dimension are spliced ​​to generate summary radiological features. The obtained summary features are input into the Cox proportional hazard model to obtain the prognostic prediction results, which are compared with the actual survival labels of the patients, and the model parameters of the radiological image feature extraction part are optimized.

[0091] Step S5, constructing a multimodal prognostic prediction model: using a deep survival network to integrate the features of pathological images, radiological images, and clinical variables, and output a multimodal prognostic prediction score.

[0092] In this embodiment, a deep survival network is used to expand the linear covariate modeling of the traditional proportional hazard model Cox to the nonlinear form of a neural network.

[0093] The present embodiment uses the time-dependent AUC under the receiver operating characteristic curve based on different time points to evaluate the model effect. In addition, for low-risk group and high-risk group, survival estimation is performed using the Kaplan-Meier method, risk stratification is performed based on the median of the predicted score of the training set, and its significance is evaluated using the log-rank test. Statistical analysis and hypothesis testing are implemented based on software packages such as survival, survcomp, survminer, timeROC and rms in R4.2.2.

[0094] In order to verify the accuracy of the present invention, the example verification content of this embodiment is as follows:

[0095] 1) Dataset

[0096] The data comes from patients with renal clear cell carcinoma in six hospitals between 2014 and 2021. The discovery cohort included 788 patients from three centers, among which some patients had low-quality CT images or tumor tissue pathology sections, resulting in the lack of complete modality information for these patients. 134 patients were randomly selected from the cohort with complete multimodal information as the validation cohort, and the remaining 654 patients were assigned to the training cohort to ensure that patients lacking multimodal information were only assigned to the training cohort. In addition, in the construction of the pathological model, the present invention also used diagnostic slice data of 503 patients with renal clear cell carcinoma in the Cancer Genome Atlas (TCGA) database. The independent external test cohort included 357 patients from the remaining three hospitals, such as Figure 2 shown.

[0097] Table 1 shows the basic information of all internal center patients. Baseline characteristics were evenly distributed among the training, validation, and testing cohorts, and there was no statistically significant difference between the groups (p>0.05). Overall, most patients were male (67.4%), and more than half of the patients were over 60 years old (50.5%). In terms of pathological characteristics, most patients presented stage I (79.0%) and grade 2 pathology (63.5%). A few patients showed poor prognostic features such as tumor necrosis (14.0%) and sarcomatoid differentiation (1.7%). In different risk stratification models, most patients were classified as low or intermediate risk. The median follow-up time was 49 months, during which 197 patients (17.2%) experienced recurrence of renal cell carcinoma. Among them, the P values ​​in the table were obtained by Pearson's Chi-squared test, Fisher's exact test, or Kruskal-Wallis rank sum test.

[0098] Table 1 Patient demographics and baseline characteristics

[0099]

[0100]

[0101] 2) Comparison of different prediction models

[0102] In order to verify the advantages of the multimodal input (MPRS) proposed in the present invention in predicting the recurrence of renal clear cell carcinoma, this embodiment compares a variety of single-modal models, including a pathology-based recurrence prediction model (HPRS), an image-based recurrence prediction model (RPRS), and a clinical data-based recurrence prediction model (CPRS). At the same time, the present invention also compares the Leibovich score and UISS score commonly used in assessing the severity of renal cancer.

[0103] In each cohort, the MPRS model was superior to the traditional single-modality models (HPRS, RPRS, CPRS) and Leibovich and UISS scores in predicting recurrence of renal clear cell carcinoma (Table 2). Specifically, in the training cohort, the MPRS model had an AUC of 0.907 (95% CI, 0.844-0.969) in 3-year recurrence prediction and 0.835 (95% CI, 0.780-0.890) in the validation cohort. In the 5-year survival prediction, MPRS still showed the highest discrimination ability, with an AUC of 0.897 (95% CI, 0.820-0.974) in the validation cohort and 0.829 (95% CI, 0.768-0.891) in the test cohort.

[0104] Table 2 Comparison of different prediction models

[0105]

[0106] In addition, the MPRS showed close alignment between the 3-year and 5-year calibration curves of each cohort and the 45° reference line, further indicating that there was significant consistency between the MPRS predictions and the actual observations, such as Figure 3 By stratifying the high-risk and low-risk groups, the recurrence-free survival between the high-risk group and the low-risk group defined by MPRS showed significant differences in the training, validation, and test cohorts (p < 0.001, as shown in Figure 4 This fully demonstrates the effectiveness of deep learning-based multimodal input in renal clear cell carcinoma recurrence prediction modeling.

[0107] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A prognosis prediction method based on multimodal intermediate fusion, characterized in that: The following steps are involved: Step S1, data acquisition: obtaining HE-stained whole-slice images as pathological images, obtaining arterial phase three-dimensional computed tomography sequences as radiological images, and obtaining clinical variables of patients; Step S2, data preprocessing: performing block cutting, staining normalization processing and filtering preprocessing on the pathological image, performing data format conversion and tumor area delineation preprocessing on the radiological image, and performing coding preprocessing on the clinical variables; Step S3, pathological image feature representation: extract manually defined features and semantic features based on deep convolutional neural networks from each pathological block after data preprocessing, and aggregate features of different pathological blocks through a multi-instance learning method to obtain feature representation of the pathological image; Step S4, radiological image feature representation: based on the radiological image and the outlined tumor contour, manually define features and extract semantic features based on a deep convolutional neural network, and perform dimensionality reduction processing through principal component analysis to obtain a feature representation of the radiological image; Step S5, constructing a multimodal prognostic prediction model: using a deep survival network to integrate the features of pathological images, radiological images, and clinical variables, and output a multimodal prognostic prediction score.

2. The prognosis prediction method based on multimodal intermediate fusion according to claim 1, characterized in that: The preprocessing of the pathological image in step S2 specifically includes the following steps: Cutting: Cut the pathological image into small blocks of preset size; Color normalization: Perform color normalization on each pathological patch in RGB three-channel format; Filtering: The OTSU method is used to calculate the proportion of tissue area in each pathological block, and the pathological blocks with a tissue area proportion less than a preset value are filtered.

3. The prognosis prediction method based on multimodal intermediate fusion according to claim 1, characterized in that: The preprocessing of the radiological image in step S2 specifically includes the following steps: Format conversion: convert radiology images in DICOM format to NII format; Resampling: Resample radiological images to set slice intervals; Grayscale normalization: normalize the grayscale of radiological images to a preset range; Tumor area delineation: Delineate the tumor contour for each 2D scan section in the 3D sequence of radiological images; Region of interest extraction: By outlining the tumor area, the 3D region of interest containing the tumor is cropped from the original radiological image; Selection of the largest tumor section: Based on the cropped 3D region of interest containing the tumor and the tumor annotation, the 2D image section containing the largest tumor is selected.

4. The prognosis prediction method based on multimodal intermediate fusion according to claim 3, characterized in that: The tumor region delineation uses the deep segmentation learning model nnU-Net pre-trained in public data to automatically delineate the renal tumor, followed by manual film reading to modify the tumor boundaries that are incorrectly segmented by the model.

5. The prognosis prediction method based on multimodal intermediate fusion according to claim 1, characterized in that: The clinical variables include basic vital signs indicators and tumor-related indicators, wherein the basic vital signs indicators include gender, age and ECOGPS score, and the tumor-related indicators include tumor size, pT stage, pN stage, TNM stage, necrosis condition, and whether it is sarcomatoid renal cell carcinoma.

6. The prognosis prediction method based on multimodal intermediate fusion according to claim 5, characterized in that: In step S2, the basic vital signs indicators are coded and preprocessed as follows: For the gender indicator, male was coded as 1 and female was coded as 0; For the age indicator, the age exceeding the preset age value is coded as 1, and the age not exceeding the preset age value is coded as 0; Regarding the ECOG PS score, the ECOG PS score is used to assess the patient's physical condition, with ECOG PS ≥ 1 being coded as 1 and ECOG PS < 1 being coded as 0.

7. The prognosis prediction method based on multimodal intermediate fusion according to claim 5, characterized in that: In step S2, encoding preprocessing of tumor-related indicators is specifically performed as follows: For tumor size, the maximum diameter of the tumor is greater than or equal to the preset value and is coded as 1, and the maximum diameter of the tumor is less than the preset value and is coded as 0; For pT staging, the pT staging is used to evaluate the size and infiltration of the tumor, with T1 coded as 1, T2 coded as 2, and T3 coded as 3; For pN staging, the pN staging is used to evaluate lymph node status, with N0 coded as 0 and N1 coded as 1; For TNM staging, stage I is coded as 1, stage II is coded as 2, and stage III is coded as 3; For necrosis, the presence of tumor necrosis was coded as 1, and the absence of necrosis was coded as 0; As for whether it is sarcomatoid renal cell carcinoma, it is coded as 1 if it is sarcomatoid renal cell carcinoma, and it is coded as 0 if it is not sarcomatoid renal cell carcinoma.

8. The prognosis prediction method based on multimodal intermediate fusion according to claim 1, characterized in that: In step S3, the semantic features based on the deep convolutional neural network are obtained by extracting the representation of the image through the ResNet network; the specific steps of extracting the manually defined features include: Use the pre-trained HoverNet model to segment the cell nucleus and predict the cell type; Based on the cell nucleus segmentation results and pathological images, the HistomicsTK package in Python was used to extract features; For each pathological patch, the average value of the HistomicsTK feature of the most frequently occurring cell type is calculated and concatenated with the one-hot encoding of the cell type to form the manually defined features extracted for each pathological patch.

9. The prognosis prediction method based on multimodal intermediate fusion according to claim 1, characterized in that: The multi-instance learning for aggregating features of different pathological patches in step S3 adopts a Transformer structure.

10. The prognosis prediction method based on multimodal intermediate fusion according to claim 1, characterized in that: In the step S4, the semantic features based on the deep convolutional neural network are extracted through the ResNet network; the manually defined features include shape features, first-order statistical features, texture features, filtering features and wavelet features defined in the Pyradiomics package; when performing principal component analysis on the shape, first-order statistical, texture, filtering, and wavelet features, only the principal components whose explanation ratio exceeds the preset value are retained.

Images