Renal clear cell carcinoma prognosis prediction method based on multi-mode MRI image and digital pathomics fusion

Abstract

The invention discloses a renal clear cell carcinoma prognosis prediction method based on multi-mode MRI (Magnetic Resonance Imaging) image and digital pathological omics fusion. The method comprises the following steps: S1, collecting a training data set based on an MR image and a pathological image; s2, feature extraction of MR radiomics; s3, deep learning feature extraction of the pathological image; s4, an MR-pathological feature fusion module; and S5, deploying the network. According to the method, depth features with prognosis information are obtained from two scales of pre-treatment images and post-operation pathology, effective features are extracted by adopting image omics and a convolutional neural network mode according to data characteristics of MR images and pathology images, and depth fusion of the two types of features is completed in a hidden space through a multi-task guiding mode, so that the accuracy of the MR image and the pathology image is improved. A precise prognosis model with multi-scale information is provided, and the method has a relatively strong clinical application prospect and is of great significance for realizing precise immunotherapy and improving prognosis of a patient.

Description

Technical Field

[0001] The present invention relates to biomedical technology, and in particular to a prognosis prediction method for renal clear cell carcinoma based on the fusion of multimodal MRI imaging and digital pathology omics. Background Art

[0002] Clear cell renal cell carcinoma is the most common subtype of renal cancer, accounting for approximately 75%. The natural history of clear cell renal cell carcinoma is relatively complex. Patients with localized disease can be cured by surgical resection, but there is still a 20-30% recurrence or metastasis rate after surgery. Metastatic renal cancer is highly insensitive to chemotherapy and radiotherapy, and the benefits of targeted therapy have reached a plateau due to the existence of drug resistance. In recent years, the discovery of immune checkpoint inhibitors such as PD-L1 can significantly prolong the survival of some patients, but the objective response rate of single-agent use in advanced patients who have failed targeted therapy ranges from 15-29%, and the prognosis is poor. Therefore, how to identify these high-risk patients early is the key to improving patient survival and prognosis.

[0003] In the clinical setting, there is still a lack of assessment criteria for patient prognosis. On the one hand, imaging can evaluate the morphology and surrounding invasion of renal cancer in vitro, mainly reflecting the macroscopic characteristics of the tumor. Postoperative pathology can observe the morphological structure of microscopic cell tissue by taking local samples from key areas. On the other hand, with the application of artificial intelligence in medical images, machine vision can obtain a large number of image features that are beyond the range of human eye evaluation, providing technical support for quantifying sub-visual tissue heterogeneity. Therefore, the integration of imaging omics and pathology omics provides an opportunity to quantify tumor heterogeneity at both macro and micro scales, and can complement each other's advantages. The integrated imaging pathology omics features can obtain a mapping relationship with genetic information (tumor immune microenvironment), thereby providing a possibility for prognostic assessment of renal clear cell carcinoma. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a prognosis prediction method for renal clear cell carcinoma based on the fusion of multimodal MRI images and digital pathology omics, a new MR image-pathology image joint latent space representation method based on multi-task guidance, to construct the most predictive joint features for the prediction task of renal clear cell carcinoma prognosis, and to establish an accurate prognosis model in combination with clinical indicators, so as to achieve early identification of high-risk renal clear cell carcinoma patients and make prognostic judgments.

[0005] Technical solution: The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI imaging and digital pathology omics described in the present invention comprises the following steps:

[0006] Step S1: Collection of training datasets based on MR images and pathological images;

[0007] Step S2: Feature extraction of MR radiomics;

[0008] Step S3: deep learning feature extraction of pathological images;

[0009] Step S4: MR-pathological feature fusion module;

[0010] Step S5: Network deployment.

[0011] Furthermore, step S1: Collection of training datasets. For MR images, radiologists manually delineated the 3D shape of the segmented tumor layer by layer using 3D Slicer software. Inter-observer variability was assessed by performing more than two delineations, which were then used as the ground truth mask for the segmentation model. For pathology images, pathologists first delineated regions of interest (ROIs) on digital pathology images (Whole Slide Images (WSIs)) for feature extraction. Finally, we statistically analyzed the immunophenotyping of renal cancer patients to serve as classification supervision for the immunophenotyping prediction model.

[0012] Furthermore, step s2: MR radiomics feature extraction. For multimodal MR imaging data of clear cell renal cell carcinoma, the present invention utilizes a cascaded segmentation network based on the nn-UNet framework to segment the kidney and renal cancer. Subsequently, images of different phases are registered with the parenchymal phase of T1WI, and the parenchymal phase kidney segmentation results are referenced to further segment the renal cancer. Quantitative texture features are extracted from the segmented 3D tumor. Texture features describe the high-order spatial distribution of density within the 3D tumor.

[0013] Furthermore, step S3: Deep Learning Feature Extraction for Pathology Images. This module employs transfer learning for feature extraction, using the ResNet-50 image encoder, a PLIP model pre-trained on a large dataset of pathology images and text, as the feature extractor. The output of the ResNet-50 average pooling layer is used as the feature vector for each image patch to extract the most representative high-dimensional semantic features. To reduce the size of the extracted deep learning features and avoid overfitting, this method also constructs an autoencoder for feature dimensionality reduction.

[0014] Furthermore, step S4: MR-pathology feature fusion module. This section integrates the omics features of preoperative MRI and the deep learning features of postoperative pathology, and on this basis, completes the prediction of immunophenotyping. To address the risk of overfitting caused by high-throughput features, the present invention uses a multi-task guided autoencoder to achieve fusion and dimensionality reduction of multimodal imaging features. This module specifically includes an encoder, a decoder, and an immunophenotyping prediction network.

[0015] (1) Encoder

[0016] The encoder inputs MRI omics features and pathological image depth features into two independent encoder modules E through a fully connected neural network structure. MR and E WSI The latent space representation is obtained and further input into the fusion module to obtain the joint latent space representation of MR image-pathology. The set of omics features screened by MR images is recorded as in is the kth MR image feature, and the deep learning feature set screened out from the pathological image is in is the kth pathological deep learning feature, then the latent space vector representations of the MR imaging omics features and the deep learning features of the pathological image after being encoded by the encoder are respectively and Joint latent space representation of MR imaging and pathology Where FC is a fully connected neural network used for latent space fusion. The encoder module consists of a 5-layer fully connected network and a ReLu activation layer, and the fusion module consists of a feature merging layer and a 2-layer fully connected neural network layer.

[0017] (2) Decoder

[0018] The decoder inputs the fused MR-pathological feature z into two different decoders D MR and D WSI , used for downstream immunophenotyping prediction tasks. In order to prevent the over-fitting problem caused by too many network parameters, we adopt the idea of ​​variational autoencoder and provide regularization constraints on the joint latent space by constructing a decoder. That is, we hope to be able to recover the MR image features P from the joint latent space representation z. MR and pathological image features P WSI . This corresponds to and Among them D MR and D WSI They are image feature and pathological feature decoders, respectively, consisting of a 5-layer fully connected network and a ReLU activation layer.

[0019] (3) Immunophenotyping prediction network

[0020] On the basis of the joint latent space representation z, we add a prediction branch network F for the prediction of immunophenotyping, that is, in The predicted results for immunophenotyping are shown in Figure 5. The prediction network F consists of a three-layer fully connected network and a ReLU activation layer. Step S5: Network deployment. The trained network is deployed to prepare for the patient's immunophenotyping prediction.

[0021] The system network architecture of the present invention includes the following modules:

[0022] MR radiomics feature extraction module. To assess renal cancer morphology and peripheral invasion, this method uses a cascaded segmentation network based on nn-Unet to segment the kidney and renal cancer. Based on this, MR radiomics feature extraction and screening are performed. This module explores renal cancer morphology and peripheral invasion from a macro perspective, providing an opportunity to quantify tumor heterogeneity.

[0023] Pathology Image Deep Learning Feature Extraction Module. This module aims to leverage the powerful feature modeling capabilities of deep convolutional networks to create high-dimensional feature representations for pathology images, effectively extracting key microscopic tumor information from pathology images. Specifically, this module employs transfer learning for feature extraction, using the image encoder ResNet-50, a PLIP model pre-trained on a large dataset of pathology images and text, as the feature extractor. The output of the ResNet-50 average pooling layer is used as the feature vector for each image patch to extract the most representative high-dimensional semantic features.

[0024] MR-Pathology Feature Fusion Module. This module integrates the omics features of preoperative MRI with deep learning features of postoperative pathology to predict immunophenotyping. To address the risk of overfitting caused by high-throughput features, this module uses a multi-task-guided autoencoder to achieve fusion and dimensionality reduction of multimodal imaging features.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] This project proposes a novel joint latent space representation method of MR images and pathological images based on multi-task guided construction, constructs the most predictive joint features for the prognosis prediction task of renal clear cell carcinoma, and establishes an accurate prognostic model in combination with clinical indicators to achieve early identification of high-risk renal clear cell carcinoma patients and make prognostic judgments.

[0027] Currently, imaging and pathology are important bases for assessing the prognosis of renal clear cell carcinoma, but they have not yet been standardized and quantified. This paper proposes to obtain deep features with prognostic information from two scales: pre-treatment imaging and post-treatment pathology. Based on the data characteristics of MR imaging and pathology images, radiomics and convolutional neural networks are used to extract effective features, respectively. The two types of features are deeply fused in the latent space through a multi-task guided approach. This paper proposes a precise prognostic model with multi-scale information, which has strong clinical application prospects and is of great significance for achieving precision immunotherapy and improving patient prognosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Flow chart of the method of the present invention;

[0029] Figure 2 Schematic diagram of the network structure for fusion of MR image and pathological image features and downstream immunophenotyping prediction;

[0030] Figure 3 This is a schematic diagram of the encoder. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further described below.

[0032] The network framework diagram of the present invention is as follows Figure 2 As shown, it includes 5 implementation steps:

[0033] Step S1: Collection of training dataset. For MR modality images, we first standardized the shape and density features of the MR images and extracted gross features to ensure model reproducibility. Each voxel was isotropically divided into 1 × 1 × 1 mm³ voxel blocks using linear interpolation. Using the parenchymal phase images as templates, each patient's images were screened and analyzed using information similarity. Imaging physicians manually outlined the segmented tumor's three-dimensional shape layer by layer using 3D Slicer software. Interobserver variability was assessed by performing more than two outlines, which were then used as the ground truth mask for the segmentation model. Voxels with segmented tumor density outside the μ ± 3σ range (μ and σ represent the mean and standard deviation of the tumor region density) were removed to conform to a normal distribution and minimize variability caused by different scanning instruments.

[0034] For pathological modality images, a pathologist first delineates a region of interest (ROI) on a digital whole slide image (WSI) for feature extraction. Due to the high resolution and large size of the WSI, to reduce the time and space complexity of subsequent analysis and improve accuracy, the ROI region of the entire WSI at 40× magnification was sampled as small tiles (512*512 fixed pixels). Individual-level analysis of each patient was obtained by integrating the results of multiple small tiles. Due to the influence of scanning equipment, lighting, and operator, image color can vary significantly between patients. Therefore, color correction was performed on the sampled small tiles, and RGB normalization was performed to minimize the impact of color aberration on sampling. Finally, statistical immunophenotyping of renal cancer patients was used as classification supervision for the immunophenotyping prediction model.

[0035] Step S2: MR radiomics feature extraction. For multimodal MR image data of clear cell renal cell carcinoma, this paper employs a cascaded segmentation network based on the nn-UNet framework to segment the kidney and renal cancer. A 3×3×3 convolution kernel (deconvolution is performed in the upsampling segment) with a stride of 2 is used. The image resolution is halved layer by layer, while the number of feature maps is doubled. Each convolution module employs a residual interlayer connection design, effectively avoiding the vanishing gradient problem and facilitating convergence and optimization. The kidney contrast is greatest during the parenchymal phase of T1WI. The kidney segmentation network is first trained and tested on this phase. Images at different phases are then registered with the parenchymal phase of T1WI, and the renal cancer is further segmented using the parenchymal phase kidney segmentation results. Quantitative texture features are extracted from the segmented 3D tumor. Texture features describe the high-order spatial distribution of the 3D tumor's internal density. These texture features include interpretable features such as tumor shape and size: 3D maximum diameter, surface area, volume, surface area / volume ratio, spherical asymmetry, sphericity, and compactness. First-order statistical histogram features: grayscale mean, maximum, minimum, variance and percentile, entropy, energy, kurtosis, skewness. Texture features: absolute gradient, grayscale co-occurrence matrix, grayscale run matrix, grayscale size area matrix, and neighbor grayscale difference matrix.

[0036] We then eliminated features with poor repeatability and evaluated the intra- and inter-observer repeatability of each feature. Features with an intraclass correlation coefficient less than 0.85 were eliminated. Based on repeatability, features were further screened for relevance. Because the random forest algorithm is suitable for feature evaluation and requires less parameter tuning, it is widely used in feature screening tasks. This project will use the Boruta algorithm based on random forests to screen for relevant features. This involves introducing randomness into the system and collecting results from a random sample set. The goal is to screen all feature sets that are correlated with the dependent variable, reducing the misleading effects of random fluctuations and correlations.

[0037] Step S3: Deep learning feature extraction of pathological images. This module aims to use the powerful feature modeling capabilities of deep convolutional networks to perform high-dimensional feature representation of pathological images, effectively mining key tumor information at the microscopic scale from pathological images. Specifically, this module uses transfer learning for feature extraction, using the image encoder ResNet 50 of the base model PLIP, which has been pre-trained on massive pathological images and texts, as a feature extractor. We use the average pooling layer output of ResNet-50 as the feature vector of each image block to extract the most representative high-dimensional semantic features. In order to reduce the size of the extracted deep learning features and avoid overfitting, this method additionally constructs an autoencoder for feature dimensionality reduction.

[0038] Resnet-50 and autoencoders obtain deep features at the image block level, and each patient-level feature is obtained by averaging each dimension of a large number of block features. Subsequently, a two-step method is proposed to screen features closely related to the immune cell typing of renal clear cell carcinoma. First, 50% of the patients are randomly selected for training as the feature selection subset. Through a variety of feature selection methods (elastic net elasticnet, LASSO, L2-norm), the key features are screened in this subset in combination with the COX regression algorithm model. The same method is used to repeatedly select different subsets to extract features. According to the frequency of the key features being selected, the features with the highest frequency (top 15%) are used as the deep learning features with the most significant mapping. The two-step screening method based on patient subsets is more robust than the conventional method based on full feature filtering of all patients, and can effectively prevent overfitting.

[0039] Step S4: MR-Pathology Feature Fusion Module. This module fuses the omics features of preoperative MRI with the deep learning features of postoperative pathology, and based on this, completes the prediction of immunophenotyping. To address the risk of overfitting caused by high-throughput features, the present invention uses a multi-task-guided self-implementation to fuse and reduce the dimensionality of multimodal imaging features. This module specifically includes an encoder, a decoder, and an immunophenotyping prediction network.

[0040] (1) Encoder

[0041] like Figure 3 As shown in the figure, the encoder inputs the MRI omics features and pathological image depth features into two independent encoder modules E through a fully connected neural network structure. MR and E WSI The latent space representation is obtained and further input into the fusion module to obtain the joint latent space representation of MR image-pathology. The set of omics features screened by MR images is recorded as in is the kth MR image feature, and the deep learning feature set screened out from the pathological image is in is the kth pathological deep learning feature, then the latent space vector representations of the MR imaging omics features and the deep learning features of the pathological image after being encoded by the encoder are respectively and Joint latent space representation of MR imaging and pathology Where FC is a fully connected neural network used for latent space fusion. The encoder module consists of a 5-layer fully connected linear network and a ReLu activation layer, and the fusion module consists of a feature merging layer and a 2-layer fully connected neural network layer.

[0042] (2) Decoder

[0043] like Figure 3 As shown in the figure, the decoder inputs the fused MR-pathological features z into two different decoders D MR and D WSI , used for downstream immunophenotyping prediction tasks. In order to prevent the over-fitting problem caused by too many network parameters, we adopt the idea of ​​variational autoencoder and provide regularization constraints on the joint latent space by constructing a decoder. That is, we hope to be able to recover the MR image features P from the joint latent space representation z. MR and pathological image features P WSI . This corresponds to and Among them D MR and D WSI They are image feature and pathological feature decoders, respectively, consisting of a 5-layer fully connected Linear network and a Relu activation layer.

[0044] (3) Immunophenotyping prediction network

[0045] On the basis of the joint latent space representation z, we add a prediction branch network F for the prediction of immunophenotyping, that is, in is the prediction result of immunophenotyping. The prediction network F consists of a 3-layer fully connected network and a ReLU activation layer.

[0046] To supervise the proposed fusion of MR image and pathological image features and the downstream immunophenotyping prediction network, three training losses are designed:

[0047] (a) Feature reconstruction loss function: This part of the loss function represents the similarity between the features restored by the joint latent space representation z and the original features:

[0048]

[0049] Where ||·|| is the feature distance, and the L1 norm will be used in the present invention.

[0050] (b) Spatial regularization function: This part of the loss function represents the distance between the joint latent space and the prior Gaussian distribution, measured by the Kullback-Leibler divergence:

[0051]

[0052] Among them D KL is the Kullback-Leibler divergence, and N(0, I) is the Gaussian prior distribution of the assumed joint latent space.

[0053] (c) Immunoassay prediction loss function: This part of the loss function represents the performance of the z classifier based on the joint latent space representation, and is measured by cross entropy:

[0054]

[0055] Combining these three loss functions, we obtain the total loss function of the entire fusion and prediction module:

[0056] L=L BCE +α·L BCE +β·L KL

[0057] Among them, α and β are the weight coefficients of the feature reconstruction branch loss function and the DL divergence loss function, respectively. The hyperparameters can be selected through the validation set.

[0058] Step S5: Network deployment: Deploy the trained network and connect it to the local terminal to prepare for the patient's immunophenotyping prediction.

[0059] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.

Claims

1. A method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI imaging and digital pathology, characterized in that: The steps include: Step S1: Collection of training datasets based on MR images and pathological images; Step S2: Feature extraction of MR radiomics; Step S3: deep learning feature extraction of pathological images; Step S4: MR-pathological feature fusion module; Step S5: Network deployment.

2. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI imaging and digital pathology according to claim 1, characterized in that: The method of step S1 is as follows: standardizing the shape and density features of the MR image, extracting features, isotropically dividing each voxel into voxel blocks using linear interpolation, using the parenchymal phase image as a template, screening and analyzing the images of each patient using information similarity, outlining the three-dimensional shape of the segmented tumor, and using it as a true mask for the segmentation model. If the segmented tumor density exceeds the range of μ±3σ, the voxel is eliminated, where μ and σ represent the mean and standard deviation of the tumor area density, respectively.

3. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI imaging and digital pathology according to claim 1, characterized in that: The method of step S1 is as follows: for the pathological modality image, a region of interest (ROI) is first outlined on the digital pathological image WSI for feature extraction of the pathological image, and the ROI area of ​​the entire WSI is sampled.

4. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI imaging and digital pathology according to claim 3, characterized in that: The method of step S2 is as follows: for renal clear cell carcinoma MR multimodal imaging data, a cascaded segmentation network based on the nn-UNet framework is used to complete the segmentation of the kidney and renal cancer. Then, the images of different phases are aligned with the T1WI parenchymal phase, and the renal cancer is further segmented with reference to the kidney segmentation results of the parenchymal phase. For the segmented three-dimensional tumor, quantitative texture features are extracted. The texture features describe the high-order spatial distribution of the internal density of the three-dimensional tumor.

5. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI images and digital pathology according to claim 4, characterized in that: The Boruta algorithm based on random forest is used to screen relevant features, that is, randomness is added to the system, results are collected from a set of random samples, and all feature sets that are correlated with the dependent variable are screened out.

6. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI imaging and digital pathology according to claim 1, characterized in that: The method of step S3 is: feature extraction is performed using transfer learning, using the image encoder ResNet 50 of the base model PLIP pre-trained with pathological images and text as the feature extractor, and using the average pooling layer output of ResNet-50 as the feature vector of each image block to extract high-dimensional semantic features. Resnet-50 and autoencoders were used to obtain deep features at the image block level. Each dimension of the block features was averaged to obtain patient-level features. Subsequently, a two-step method was used to screen features closely related to the immune cell typing of renal clear cell carcinoma. First, 50% of the patients were randomly selected for training as a feature selection subset. Key features were screened in this subset using the elasticnet, LASSO, or L2-norm feature selection methods combined with the COX regression algorithm model.

7. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI images and digital pathology according to claim 1, characterized in that: In step S4, a multi-task guided autoencoder and decoder are used to achieve fusion and dimensionality reduction of multimodal image features, including an encoder, a decoder and an immunophenotyping prediction network.

8. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI images and digital pathology according to claim 7, characterized in that: The encoder inputs MRI omics features and pathological image depth features into two independent encoder modules E through a fully connected neural network structure. MR and E WSI The latent space representation is obtained and further input into the fusion module to obtain the joint latent space representation of MR image-pathology. The set of omics features screened by MR image is recorded as in is the kth MR image feature, and the deep learning feature set screened out from the pathological image is in is the kth pathological deep learning feature, then the latent space vector representations of the MR imaging omics features and the deep learning features of the pathological image after being encoded by the encoder are respectively and Joint latent space representation of MR imaging and pathology Among them, FC is a fully connected neural network used for latent space fusion, the encoder module consists of a 5-layer fully connected network and a ReLu activation layer, and the fusion module consists of a feature merging layer and a 2-layer fully connected neural network layer.

9. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI images and digital pathology according to claim 7, characterized in that: The decoder inputs the fused MR-pathological feature z into two different decoders D MR and D WSI , used for downstream immunophenotyping prediction tasks, adopts the idea of ​​variational autoencoder, and provides regularization constraints on the joint latent space by constructing a decoder, that is, it is expected to be able to restore the MR image features P from the joint latent space representation z. MR and pathological image features P WSI , which correspond to and Among them D MR and D WSI They are image feature and pathological feature decoders, respectively, consisting of a 5-layer fully connected network and a ReLU activation layer.

10. The method for predicting the prognosis of renal clear cell carcinoma based on the fusion of multimodal MRI images and digital pathology according to claim 7, characterized in that: The immunophenotyping prediction network: Based on the joint latent space representation z, a prediction branch network F is added for the prediction of immunophenotyping, that is, in The prediction result of immunophenotyping is composed of a 3-layer fully connected network and a Relu activation layer. To supervise the proposed fusion of MR image and pathological image features and the downstream immunophenotyping prediction network, three training losses are designed: (a) Feature reconstruction loss function: This part of the loss function represents the similarity between the features restored by the joint latent space representation z and the original features: Where ||·|| is the feature distance, and the L1 norm will be used in the present invention. (b) Spatial regularization function: This part of the loss function represents the distance between the joint latent space and the prior Gaussian distribution, measured by the Kullback-Leibler divergence: Among them D KL is the Kullback-Leibler divergence, N(0, I) is the Gaussian prior distribution of the assumed joint latent space, (c) Immunoassay prediction loss function: This part of the loss function represents the performance of the z classifier based on the joint latent space representation, and is measured by cross entropy: Combining these three loss functions, we can obtain the total loss function of the entire fusion and prediction module: L=L BCE +α·L BCE +β·L KL Among them, α and β are the weight coefficients of the feature reconstruction branch loss function and the DL divergence loss function, respectively.

Images