A multi-modal deep learning-based method for predicting the prognosis risk of renal cell carcinoma patients

Abstract

The application discloses a kind of kidney cell cancer patient prognosis risk prediction methods based on multi-modal deep learning.The present application processes whole field digital pathology section (WSI) image and pathological diagnosis report by deep learning algorithm, extracts text features using natural language processing model, extracts image features using computer vision model and ABMIL network, and carries out multi-modal feature fusion through Cross-attention Transformer network, and finally outputs the survival risk curve of patient changes with time in combination with Weibull survival function, for assisting doctor to carry out more accurate prognosis evaluation and survival analysis.

Description

Technical Field

[0001] This invention relates to an intelligent prediction technology that uses multimodal deep learning technology (combining full-view digital pathological slides and pathological diagnostic reports) to predict the prognostic survival risk of patients with renal cell carcinoma. Background Technology

[0002] Prognostic prediction for malignant tumors is crucial for developing treatment plans. Currently, clinical prognostic assessments primarily rely on pathological TNM staging or physician experience. Existing intelligent prognostic studies have the following limitations:

[0003] Monomodal approach: Most studies rely solely on whole-scan images (WSI) or solely on clinical text data. Using images alone may overlook key semantic information contained in the report (such as specific immunohistochemical markers), while using text alone loses the spatial morphological information of the tumor microenvironment.

[0004] Unstructured text processing is difficult: Pathology reports are usually unstructured natural language text with complex formats, making it difficult to extract effective features for direct computation.

[0005] The fusion mechanism is simple: existing multimodal methods mostly use simple feature splicing, which fails to fully explore the deep interaction relationship between image features and text features.

[0006] Therefore, there is an urgent need for a method that can efficiently integrate unstructured pathology reports with high-resolution pathology images and accurately predict patient survival risk trends. Summary of the Invention

[0007] The present invention aims to solve the problems of insufficient utilization of single-modal information and inadequate fusion of multimodal features in existing intelligent assisted prognostic prediction methods.

[0008] To address the aforementioned technical problems, the present invention discloses a method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning, characterized by comprising the following steps: Step 1: Collect clinical case data of patients with renal cell carcinoma, including pathological slides and pathology reports; Step 2: Digitize the data collected in Step 1; Step 3: Multimodal data preprocessing and initial feature extraction, including text processing and image processing branches, wherein: The text processing branch uses the large language model Qwen3 to perform natural language understanding and information extraction on the original unstructured pathology report obtained in step 2, and organizes the messy text into structured text containing fixed fields (such as pathological subtype, nuclear grade, tumor size, vascular tumor thrombus status, etc.). Then, the BERT-base-Chinese model is used to encode each structured attribute and extract the corresponding text feature vector. The image processing branch extracts the foreground of the pathological slide image obtained in step 2, removes the blank background, and cuts the foreground region of the slide into fixed-size (e.g., 224 pixels × 224 pixels) patches in a non-overlapping sliding window manner at a preset scaling resolution (e.g., 0.5 mpp (microns per pixel) scaling resolution). Using a pre-trained pathological basic model as a feature extractor, it encodes the features of each patch to obtain a patch-level feature vector sequence. Step 4: Input the tile-level feature vector sequence obtained from the image processing branch into the ABMIL (Attention-based Multiple Instance Learning) network. Through the attention mechanism, aggregate to generate a slice-level image feature vector that can represent the entire slice. Perform dimensionality reduction on the slice-level image feature vector using a fully connected layer to adapt it to the dimension of the text feature vector output by the text processing branch. Then, concatenate it with the text feature vector to obtain multimodal features. Step 5: Input the spliced ​​multimodal features into the Cross-attention Transformer network. Use the cross-attention mechanism to capture the deep association between image morphological features and text description features. The output of the Cross-attention Transformer network is processed by a multilayer perceptron (MLP). A prediction model is built using the parametric survival analysis method Weibull. The prediction model outputs the scale and shape parameters of the Weibull distribution. Step 6: Substitute the scale parameter and shape parameter into the Weibull survival function formula to obtain the survival probability curve of the current patient over time.

[0009] Preferably, in step 2, during digitization: for the pathological slides, the glass slides are converted into full-view digital pathological slides using a digital pathology scanner; for the pathology report, electronic text data corresponding to the pathological slides is collected.

[0010] Preferably, when training and evaluating the prediction model and constructing the training cohort, validation cohort, and independent test cohort, patients who have undergone partial or radical nephrectomy and whose pathological diagnosis is clearly renal cell carcinoma are included, while cases with missing clinical pathological data or poor image quality are excluded. Complete follow-up data of patients are collected, and the data are divided into the training cohort, validation cohort, and independent test cohort.

[0011] Preferably, when training and evaluating the prediction model, after obtaining digital pathological slides and pathological reports of multiple patients using steps 1 and 2, privacy desensitization processing is performed to construct a local multimodal dataset containing unstructured text and high-resolution images.

[0012] Preferably, the prediction model is evaluated for accuracy by comparing the risk score output by the model with the patient’s actual survival follow-up data. The concordance index (C-index) is used as the main evaluation indicator, and the area under the receiver operating characteristic curve (Time-dependent AUC) at different time points is calculated.

[0013] Preferably, the effectiveness and superiority of the prognostic risk prediction method for renal cell carcinoma patients as an independent prognostic factor are verified by multivariate Cox regression analysis.

[0014] This invention processes whole-view digital pathology slide (WSI) images and pathology reports using deep learning algorithms, extracts text features using a natural language processing model, extracts image features using a computer vision model and the ABMIL network, and performs multimodal feature fusion using a Cross-attention Transformer network. Combined with the Weibull survival function, it finally outputs the patient's survival risk curve over time, which is used to assist doctors in making more accurate prognostic assessments and survival analyses.

[0015] Compared with existing technical solutions, the present invention has the following beneficial effects: 1. Multimodal deep fusion: It combines the latest large language model (Qwen3) with computer vision technology, and can provide more comprehensive prognostic assessment information compared with single-modal methods; 2. Intelligent text processing: The introduction of Qwen3 transforms messy pathology reports into structured data, greatly improving the effectiveness of feature extraction by the downstream BERT model; 3. Enhanced Interactivity Features: Instead of simple concatenation, the Cross-attention Transformer is used, enabling the model to learn the implicit association between images and text, such as the correspondence between the description of "necrosis" in the report and the feature of "necrotic area" in the image.

[0016] 4. Intuitive output results: The output shows a risk curve that changes over time, which provides a more dynamic reference for clinical practice compared to a single survival time prediction. Attached Figure Description

[0017] Figure 1 This is a flowchart of the overall framework of the present invention, which includes multiple processes such as data collection, pathological slide preprocessing, pathological slide feature extraction, pathological report feature extraction, multimodal combination, and prediction of patient prognostic risk. Figure 2 This is a schematic diagram of the multimodal combined network of the present invention. Detailed Implementation

[0018] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0019] This invention discloses a method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning. It collects pathological images and text data of patients diagnosed with renal cell carcinoma using the standard data collection process described in this application, and obtains feature data usable for multimodal networks through standard data processing procedures (including text structure cleaning and WSI slice tile processing). Finally, it constructs a deep learning model based on the cross-attention mechanism to predict the survival risk curve of renal cell carcinoma patients over time. Figure 1 Specifically, it includes the following steps: Step 1: Standard Data Collection and Inclusion. In one embodiment of the present invention, the data inclusion criteria are patients who have undergone partial or radical nephrectomy in a hospital, with a confirmed postoperative pathological diagnosis of renal cell carcinoma, and who have complete follow-up information (including overall survival and progression-free survival). Two modalities of patient data need to be collected simultaneously: the first modal data is the whole-field digital pathology slide (WSI) generated from the digital scanning of the surgical specimen; the second modal data is the corresponding electronic pathology report text.

[0020] Step 2: Structure the pathology report using a large language model. In one embodiment of the invention, to address the issues of unstructured and complex descriptions in pathology reports, the Qwen3 large language model is used to clean and extract information from the original text. Specifically, the original pathology report obtained in Step 1 is input into the Qwen3 model, and based on preset prompts, the text is transformed into structured text containing key diagnostic and treatment elements. In one embodiment of the invention, the extracted key fields include, but are not limited to: pathological subtype, nuclear grade (WHO / ISUP), tumor size, tumor number, presence of necrosis, presence of sarcomatoid degeneration, and immunohistochemical status. Subsequently, a pre-trained BERT-base-Chinese model is used as a text encoder. The text description of each of the above structured attributes is input into the text encoder to extract the corresponding text feature vectors, forming a high-dimensional text feature sequence.

[0021] Step 3: Patient prognostic risk labeling. In one embodiment of the present invention, survival analysis labeling is performed based on the patient's postoperative follow-up information. Specifically, it includes two labels: the first label is a time label, which is the time interval from the surgery date to the last follow-up or the date of death / recurrence; the second label is a status label, which is whether the patient has experienced an endpoint event (such as death or recurrence) at the last follow-up. Patients who have not experienced an event are marked as censored.

[0022] Step 4: WSI Image Preprocessing and Patch Cropping. In one embodiment of the invention, the WSI images collected in Step 1 undergo quality control and preprocessing. First, the Otsu thresholding algorithm is used to automatically identify tissue regions (foreground) and blank regions (background), removing background regions lacking information. In one embodiment of the invention, to adapt to the input requirements of the deep learning model and retain sufficient cell morphology details, the foreground tissue regions are meshed at a scaling resolution of 0.5 mpp (microns per pixel). Finally, a large number of non-overlapping image patches are obtained, with each patch's pixel size normalized to 224 pixels × 224 pixels.

[0023] Step 5: Image Feature Extraction and Aggregation Based on Pathology Foundation Model and ABMIL. In one embodiment of the invention, to extract effective features from the massive image patches obtained in Step 4, a pathology foundation model (such as CTransPath or UNI) pre-trained on massive histopathological images is used as a feature extractor. The 224×224 image patches obtained in Step 4 are input into this feature extractor, and the feature vector (e.g., 1024-dimensional) corresponding to each image patch is output. In one embodiment of the invention, since a WSI image contains thousands of image patches, they need to be aggregated into slice-level features. The feature vector sequence of all image patches is input into the ABMIL (Attention-based Multiple Instance Learning) network. The ABMIL network automatically learns the importance weight of each image patch for prognostic prediction through an attention mechanism (e.g., assigning higher weights to necrotic areas or high-grade areas), and performs weighted summation according to the weights, finally outputting a slice-level image feature vector that can represent the entire pathological slice.

[0024] Step 6: Multimodal Fusion and Weibull Risk Prediction Based on Cross-attention Transformer. First, the slide-level image feature vector obtained in Step 5 is dimensionality-reduced using a fully connected layer to match the dimension of the text feature vector obtained in Step 2. Then, the dimensionality-reduced slide-level image feature vector and the text feature vector sequence are concatenated (Concat) to construct a multimodal feature sequence. The multimodal feature sequence is input into the Cross-attention Transformer network. To obtain a continuous risk prediction curve, this embodiment employs a parametric survival analysis strategy. The last layer (output layer) of the Cross-attention Transformer network is designed to output two positive real numbers, corresponding to the scale parameter and shape parameter of the Weibull distribution, respectively. During training, a negative log-likelihood loss function based on the Weibull distribution is used. This loss function can simultaneously utilize uncensored data (patients who experienced death / relapse events) and censored data (censored data, i.e., patients still alive at the end of follow-up), maximizing the likelihood probability of the observed data, thereby optimizing the network parameters.

[0025] Risk curve generation: During the inference phase, the scale and shape parameters output by the model are substituted into the Weibull survival function formula to plot the patient's survival probability curve (i.e., the risk curve) over time. Doctors can use this curve to visually view the patient's survival probability at any postoperative time point (such as 12 months, 24 months, 60 months, etc.), thereby enabling more accurate risk stratification.

[0026] Pathology reports contain a summary of the pathologist's expert knowledge, while pathology images contain rich sub-visual morphological information. The technical solution disclosed in this invention leverages Qwen3's powerful understanding capabilities to solve the problem of unstructured text, utilizes a pathology-based model and the ABMIL architecture to address the feature aggregation problem of ultra-high resolution images, simulates the diagnostic thinking process of a doctor combining report information when reviewing images through a cross-attention mechanism, and finally outputs the patient's risk score curve over time based on the Weibull survival function.

[0027] As an example, the prognostic risk prediction method for renal cell carcinoma patients disclosed in this embodiment of the invention can be used through the following steps: S1. Clinical Data Recruitment. The method disclosed in this invention first requires the recruitment of surgical specimens and pathological data from patients who have undergone partial or radical nephrectomy. The input data must include two modalities: full-field digital pathological sections (WSI) generated from the patient's surgical slides after scanning, and the corresponding electronic pathology report text.

[0028] S2. Structured Processing of Pathology Reports Based on a Large Language Model. When using the method disclosed in this invention, preprocessing of unstructured pathology reports collected clinically is required. Specifically, the deployed Qwen3 large language model is invoked on a local server, and pre-defined prompts are used to transform the chaotic pathological descriptions into standardized structured text (containing key fields such as pathological subtype, nuclear grade, necrosis, and tumor size). The structured output results should be quickly reviewed by senior pathologists to ensure the accuracy of extracted key information and to manually correct any erroneous fields extracted by the model.

[0029] S3. Automatic Segmentation and Patch Cropping of WSI Images. The method disclosed in this invention requires standardization of the WSI image on a local GPU server or workstation. Specifically, the Otsu thresholding algorithm is used to automatically identify and segment the tissue foreground region in the WSI image, removing blank backgrounds. Subsequently, the foreground region is resampling at spatial resolution. At a scaling resolution of 0.5 mpp (microns per pixel), the tissue region is cut into 224 pixel × 224 pixel image patches using a non-overlapping sliding window, which serve as input units for subsequent deep learning networks.

[0030] S4. Extraction and Aggregation of Multimodal Features. When using the method disclosed in this invention, a pre-trained text and pathological slide feature extraction network needs to be deployed on a local GPU server. For the structured text obtained in step S2, the text feature vector is extracted by inputting it into a locally deployed BERT-base-Chinese model; for the image patches obtained in step S3, patch-level features are extracted by inputting them into a pathological base model (such as CTransPath), and then aggregated through the ABMIL network to generate slide-level image feature vectors. In this step, it is necessary to ensure that the feature vectors of all dimensions are aligned and normalized to facilitate subsequent fusion.

[0031] S5. Predict the Weibull survival risk curve using a multimodal cross-attention transformer network. When using the above method disclosed in this invention, the multimodal prognostic prediction network described in this invention needs to be deployed on a local GPU server, and the pre-trained weights need to be read. The slide-level image features extracted in step S4 are concatenated with the text feature vector and input into the network. The network deeply fuses the image and text features through a cross-attention mechanism. The network output no longer outputs a single classification label, but instead outputs two key parameters of the Weibull distribution: scale parameter and shape parameter. The system uses these two parameters to automatically plot the patient's survival risk curve over time. Doctors can directly view this curve to intuitively assess the patient's survival probability at 1 year, 3 years, 5 years, or any other time point after surgery, thereby developing personalized follow-up and treatment plans.

Claims

1. A method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning, characterized in that, Includes the following steps: Step 1: Collect clinical case data of patients with renal cell carcinoma, including pathological slides and pathology reports; Step 2: Digitize the data collected in Step 1; Step 3: Multimodal data preprocessing and initial feature extraction, including text processing and image processing branches, wherein: The text processing branch uses the large language model Qwen3 to perform natural language understanding and information extraction on the original unstructured pathology report obtained in step 2. After obtaining the structured text, the BERT-base-Chinese model is used to encode each structured attribute and extract the corresponding text feature vector. The image processing branch extracts the foreground of the pathological slide image obtained in step 2, removes the blank background, and cuts the foreground area of ​​the slide into fixed-size patches in a non-overlapping sliding window manner at a preset scaling resolution. Using a pre-trained pathological basic model as a feature extractor, it encodes the features of each patch to obtain a patch-level feature vector sequence. Step 4: Input the tile-level feature vector sequence obtained from the image processing branch into the ABMIL network. Through the attention mechanism, aggregate to generate a slice-level image feature vector that can represent the whole slice. Perform dimensionality reduction on the slice-level image feature vector through a fully connected layer to adapt it to the dimension of the text feature vector output by the text processing branch. Then, concatenate it with the text feature vector to obtain multimodal features. Step 5: Input the spliced ​​multimodal features into the Cross-attention Transformer network. Use the cross-attention mechanism to capture the deep association between image morphological features and text description features. The output of the Cross-attention Transformer network is processed by a multilayer perceptron. A prediction model is constructed using the parametric survival analysis method Weibull. The prediction model outputs the scale and shape parameters of the Weibull distribution. Step 6: Substitute the scale parameter and shape parameter into the Weibull survival function formula to obtain the survival probability curve of the current patient over time.

2. The method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning as described in claim 1, characterized in that, In step 2, during the digitization process: for the pathological slides, the glass slides are converted into full-view digital pathological slides using a digital pathology scanner; for the pathology report, electronic text data corresponding to the pathological slides is collected.

3. The method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning as described in claim 1, characterized in that, When training and evaluating the prediction model, and constructing the training cohort, validation cohort, and independent test cohort, patients who have undergone partial or radical nephrectomy and whose pathological diagnosis is confirmed as renal cell carcinoma are included, while cases with missing clinicopathological data or poor image quality are excluded. Complete follow-up data of patients are collected, and the data are divided into the training cohort, validation cohort, and independent test cohort.

4. The method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning as described in claim 1, characterized in that, When training and evaluating the prediction model, after obtaining digital pathological slides and pathological reports of multiple patients using steps 1 and 2, privacy desensitization processing is performed to construct a local multimodal dataset containing unstructured text and high-resolution images.

5. The method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning as described in claim 1, characterized in that, The accuracy of the prediction model was evaluated by comparing the risk score output by the model with the actual survival follow-up data of the patients. The consistency index was used as the main evaluation indicator, and the area under the receiver operating characteristic curve at different time points was calculated.

6. The method for predicting the prognostic risk of renal cell carcinoma patients based on multimodal deep learning as described in claim 1, characterized in that, The effectiveness and superiority of the proposed method for predicting the prognostic risk of renal cell carcinoma patients as an independent prognostic factor were verified by multivariate Cox regression analysis.

Images