Artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system

Abstract

The present invention provides a multimodal prostate cancer biochemical recurrence risk stratification prediction system based on artificial intelligence. Based on the Xgboost framework, the panoramic pathology section scan images of postoperative patients are analyzed in an end-to-end, multi-scale, multi-center, and large-sample manner to maximize the use of pathological information. At the same time, combined with clinical indicators such as the CAPRA-S score, the patient's prognostic risk can be more comprehensively evaluated, which has obvious advantages over traditional models. This method is designed to be more in line with the hospital's usage scenarios. By integrating the pathological section characteristics and clinical characteristics after radical surgery, it can more efficiently and accurately predict the risk of prostate cancer recurrence in patients, and help to accurately predict the risk of recurrence in patients 3 years after surgery and longer. Further combined with an interpretable module, it assists doctors in interpreting the results, thereby achieving accurate stratification and personalized follow-up of BCR risk in prostate cancer patients, reducing the risk of overtreatment and missed diagnosis, and has good application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of prostate cancer biochemical recurrence risk prediction and analysis, and specifically relates to an artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system. Background Art

[0002] After radical prostatectomy, 30-40% of prostate cancer patients experience elevated prostate-specific antigen (PSA) levels, indicating disease recurrence. Patients whose PSA levels exceed 0.2 ng / mL twice or more after surgery are considered to have experienced biochemical recurrence (BCR). BCR is an early sign of metastasis and cancer recurrence, often requiring interventions such as targeted radiotherapy and hormone blockade. Currently, there are no reliable indicators for predicting BCR to guide the frequency of postoperative follow-up. The traditional prostate cancer prognosis assessment paradigm relies primarily on the pathological Gleason grading system, which is considered to be a reference standard for tumor BCR risk assessment. However, a large amount of clinical practice data has fully confirmed that even patients with relatively low Gleason scores may still experience biochemical recurrence within a very limited time period. Predictive models that rely solely on Gleason scores are no longer able to meet the needs of accurate clinical risk stratification.

[0003] To address these issues, based on extensive clinical data, prognostic assessments have been developed by combining preoperative PSA levels, Gleason score, and clinical stage. This has led to the development of predictive models such as the CAPRA-S-Score. However, due to significant heterogeneity between individuals, the widespread presence of missing data across multiple dimensions in clinical data, and systematic differences in data collection standards across different medical institutions, these factors further increase the cognitive and technical challenges of accurate prediction. Using clinical data alone to predict biochemical recurrence remains suboptimal.

[0004] Artificial intelligence (AI) is an emerging discipline that excels at simulating, extending, and expanding human intelligence through theories, methods, and techniques. With the rapid development of AI, its powerful capabilities are playing a crucial role in processing and integrating complex, multi-parameter, and multi-dimensional clinical information resources. Deep learning, a class of machine learning algorithms that includes neural network models, has shown initial application in the early diagnosis and prognostic assessment of cancer. Recent reports on deep learning models analyzing panoramic pathology slide scans (WSIs) for other cancer types have shown that they can accurately predict disease type and patient prognosis with excellent accuracy and stability. This AI-based digital pathology image analysis paradigm provides a new technical approach for the precise prediction of biochemical recurrence risk in prostate cancer.

[0005] Although artificial intelligence has made some progress in digital pathology image analysis, true clinical translation still faces numerous key technical challenges. Pathology image processing requires dealing with massive amounts of multidimensional data, and the algorithmic complexity of feature extraction and selection is extremely high. Image quality and staining variations between pathology sections pose significant challenges to model stability and generalization. Crucially, predictive models for clinical applications must be highly interpretable. Clinicians need a clear and transparent understanding of the model's decision-making mechanisms and a quantitative assessment of the marginal contribution of each feature to the prediction results. This requires that artificial intelligence should not be a closed "black box" prediction tool, but rather an intelligent decision-making support system that provides intuitive and understandable explanations. Therefore, how to effectively integrate clinical indicators and pathology image features to construct a multimodal prediction model that is both accurate and highly interpretable has become a core paradigm in current precision medicine research for prostate cancer.

[0006] At present, there is no artificial intelligence prediction system that can integrate clinical and pathological data and accurately stratify the risk of biochemical recurrence of prostate cancer. Existing studies are either limited to a single data modality or lack sufficient clinical interpretability. In view of the individual differences in the prognosis of prostate cancer patients and the decisive influence of biochemical recurrence on patient survival, there is an urgent need for a more intelligent, accurate and clinically translational value risk prediction methodological paradigm. The present invention aims to break through the cognitive limitations of traditional prediction models and construct a multimodal intelligent prediction system that can provide personalized prognostic assessment by innovatively integrating clinical big data, artificial intelligence technology and digital pathology image analysis. Summary of the Invention

[0007] This invention aims to fill this gap by developing an artificial intelligence-based, multimodal prostate cancer biochemical recurrence risk stratification prediction system. An AI-based prediction model was designed that integrates and analyzes clinical and pathological information from prostate cancer patients to achieve stratified prediction of biochemical recurrence risk. This provides a reliable basis for predicting the risk of biochemical recurrence of prostate cancer and offers powerful support and guidance for subsequently reducing the negative impact of postoperative disease and improving survival rates.

[0008] In order to achieve this purpose, the specific technical solutions of the present invention are as follows:

[0009] In its first aspect, the present invention provides an artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system. By integrating multimodal data such as panoramic pathology slide scans and clinical indicators, it can better predict biochemical recurrence in prostate cancer patients after surgery, helping clinicians develop more personalized postoperative follow-up and adjuvant treatment plans. The system has the following technical features, including:

[0010] An input display module is used to input the patient's original medical record report and the surgical pathology report containing the panoramic pathology slice scan image of the pathology slice, and to display the analysis results after the analysis is completed;

[0011] The clinical feature extraction and processing module automatically extracts the patient's clinical feature information from the original medical records and surgical pathology reports, automatically determines and fills in missing values, and then scores the filled information with a risk stratification algorithm based on the CAPRA-S scoring system to form a standardized clinical feature vector;

[0012] The pathology feature extraction and processing module uses a multi-instance algorithm based on a pseudo-packet strategy and recurrent cross-attention after image preprocessing to first extract deep features of tumor images at different objective magnifications. Next, a feature representation at the level of panoramic pathology slice scans is generated in a weakly supervised network. Microscopic cellular abnormalities and macroscopic tissue change features are extracted to obtain pathology feature representations, which are then integrated to generate a biochemical recurrence prediction score at the pathology level.

[0013] The fusion prediction module performs multimodal feature fusion on the pathological feature representation and standardized clinical feature vector information to build an end-to-end prostate cancer biochemical recurrence risk prediction model. The module uses an attention mechanism to highlight the weights of key features, analyzes the pathological feature representation and standardized clinical feature vector information, classifies patients into three risk levels: high-risk, intermediate-risk, and low-risk, and visualizes key predictive factors.

[0014] The explainability module provides in-depth interpretation from three levels: pathological images, fusion models, and similar cases. It quantitatively displays the contribution of each clinical indicator and pathological feature to the prediction results, and provides a basis for understanding the model from multiple dimensions: visual, quantitative analysis, and analogical reasoning.

[0015] The storage module is used to store the information processed by each module.

[0016] The control module integrates the manual judgment results and risk prediction results and transmits the final results to the input display module.

[0017] The preferred technical solutions for each module in the system are as follows:

[0018] (1) Input display module

[0019] The system is used to input the patient's original medical history report and surgical pathology report containing panoramic pathology slide scan images, and then displays the analysis results after completion. In this invention, the input and display module is the data entry point for the system, and an intelligent and adaptive interface is designed. Its intelligent design aims to achieve efficient and accurate collection of patient clinical data. For example, the module allows clinicians to directly upload or paste original medical records and surgical pathology reports.

[0020] (2) Clinical feature extraction and processing module

[0021] It includes a clinical feature extraction submodule and a clinical data processing submodule.

[0022] The clinical feature extraction submodule is a system-integrated large language model API (such as GPT-4o, Claude, etc.), which can intelligently parse text content for efficient collection and processing of patients' multi-dimensional clinical data, including patient age, preoperative PSA level, pathological Gleason score, clinical pathological stage, surgical margin status, seminal vesicle invasion status, and other key indicators that constitute the CAPRA-SScore.

[0023] During the extraction process, a built-in data validation mechanism ensures the accuracy of the extraction results. For ambiguous or uncertain fields, the system fluorescently marks and bolds the clinical features in the table, prompting clinicians to review them. This allows for a more intelligent and standardized integration of clinical and pathological data, laying a solid data foundation for subsequent AI-powered risk prediction.

[0024] The clinical data processing submodule first integrates the patient characteristics and the training cohort, and accurately fills the missing values ​​of the patient's key clinical variables through the multiple interpolation chain equation (MICE) algorithm, and sends a warning when the number of missing values ​​is not less than the preset number (such as >4 missing values); then the filled data is scored by the risk stratification algorithm based on the CAPRA-S scoring system, and the PSA level, Gleason score, clinical pathological stage, surgical margin status, and lymph node status are assigned as follows: the PSA level score range is 0-4 points (corresponding to <6, 6-10, 10-20, respectively). ,20-30,≥30), Gleason score range is 0-3 points (corresponding to ISUP1-5, among which ISUP4 and 5 are 3 points), clinical pathological stage is 0-1 point (0, T1 / T2 stage, 1, T3 / T4 stage), surgical margin status is 0-2 points (0, R0 status, 1, R1 status, 2, R2 status), lymph node involvement is 0-1 point (0, N0 status, 1, N1 status), and finally a continuous risk assessment of 0-7 points is achieved, and patients are accurately stratified into three levels: low risk (0-2 points), medium risk (3-5 points) and high risk (6 points and above).

[0025] (3) Pathological feature extraction and processing module

[0026] It includes feature vector processing submodule, cyclic cross attention submodule and multi-scale feature fusion submodule.

[0027] The eigenvector processing submodule first preprocesses the panoramic pathology slide scan image. In the present invention, the panoramic pathology slide scan image is preferably a panoramic pathology slide scan image of HE-stained tissue. The eigenvector processing submodule first annotates the tumor region of the HE-stained tissue pathology full slide image using QuPath software and exports the annotated information in GeoJSON format. It then performs preprocessing to exclude background areas. The specific method is as follows: by adjusting the mean and standard deviation of the image block data, the data is converted to a distribution with zero mean and unit variance, achieving data standardization and image intensity normalization. The tumor area was then cropped into non-overlapping image blocks of 256×256 pixels at 10×, 20×, and 40× objective lens magnifications, and features were automatically extracted from them using the self-supervised visual encoder UNI. After the last convolutional layer of UNI, the 1024-dimensional feature vector of each image block was extracted to generate a feature representation at the level of the panoramic pathology slice scan image. Each panoramic pathology slice scan image will generate a set of low-dimensional feature matrices of size {Ns×1024} at three magnification scales, where N represents the number of image blocks at that scale, and s∈{10×, 20×, 40×}.

[0028] The pseudo-package strategy is used to solve the overfitting problem while predicting the features of BCRs that may be sparse but critical, such as cell populations with high metastatic potential. The feature vector processing submodule refines the features based on the pseudo-package strategy, capturing the information of local key areas while retaining the overall structural features, thereby improving the model's ability to identify and discriminate heterogeneous lesions in the image. In the subsequent four-dimensional feature reconstruction and convolution processing of different sizes, the input panoramic pathology slice scan image-level feature representation or the pseudo-package and parent package feature package representation is first reconstructed into a four-dimensional tensor X reshaped : , then uses convolution kernels of different sizes (7×7, 5×5, and 3×3) to process the feature data to further extract features with spatial awareness and contextual information, providing a more comprehensive and richer feature representation. By preserving the spatial correlation in pathological images, it achieves modeling of tissue morphological features and extraction of spatial features.

[0029] The recurrent cross-attention submodule complements convolution kernels of different scales with each other, and based on the cross-attention iteration, gradually generates new feature maps with dense and rich contextual information, enabling the model to simultaneously identify microscopic cellular abnormalities and macroscopic tissue changes; such as microscopic cellular abnormalities include unequal cell size, nuclear atypia, increased chromatin, multiple nuclear division figures, and abnormal cell metabolism; the macroscopic tissue changes include irregular gland size, disordered arrangement, basement membrane destruction, perineural infiltration accompanied by necrosis and hemorrhage, interstitial fibrosis, etc.

[0030] The multi-scale feature fusion submodule fuses the feature information at three objective lens magnification scales, performs a weighted average of the biochemical recurrence probabilities obtained at different objective lens magnifications, and ultimately generates a predicted risk score for the patient, thereby obtaining a recurrence or non-recurrence prediction result.

[0031] Multiscale fusion is crucial in pathology image analysis because features at different scales complement each other and provide more comprehensive information. For example, a low-magnification objective lens (e.g., 10×) helps identify the overall distribution and extent of cancer lesions and their relationship to surrounding tissues, such as their size, proportion, and invasion. A high-magnification objective lens (e.g., 40×) reveals morphological details at the cellular level, such as cancer cell morphology, nuclear atypia, and nuclear-cytoplasmic ratio. Through multiscale fusion, the model can better understand hierarchical structures and improve its ability to recognize complex features. Furthermore, this approach enhances the model's robustness, reduces the impact of noise and information loss, and can simulate the pathologist's observation method at different magnifications, thereby improving diagnostic accuracy.

[0032] (4) Fusion prediction module

[0033] The Fusion Prediction Module aims to achieve precise stratification and personalized prediction of prostate cancer biochemical recurrence (BCR) risk. Through a multimodal feature deep fusion strategy, the module integrates deep learning features from panoramic pathology slide scans, clinical CAPRA-S scores, and key clinical information to build an end-to-end AI risk prediction system.

[0034] Specifically, the fusion prediction module first integrates CAPRA-S score, age, clinical pathological stage, seminal vesicle invasion, surgical margin status, preoperative PSA level and biochemical recurrence prediction score at the pathological level through the Xgboost model, and highlights the weight of key features through the attention mechanism; then calculates the attention score of each feature (W i × score), and after weighted fusion of the features, three risk levels of high risk, medium risk, and low risk are obtained based on the calculation and analysis results; then, the survival:aft accelerated failure time objective function model is used to predict the patient's survival time based on the relationship between the clinical features learned by the model and the survival time distribution.

[0035] In this invention, the Softmax algorithm is used to achieve dynamic weight allocation of the above features. That is, the attention mechanism automatically learns the weight of each feature in the current task by calculating the correlation or importance between features. The calculation formula is as follows:

[0036]

[0037] Among them, Q and K are the representations of features, K Trefers to the transpose of the matrix K, and d is the dimension.

[0038] The assignment of weights depends on the overall distribution of the input data, such as the combination of the above characteristics of a patient.

[0039] In the code implementation, the Xgboost extreme gradient boosting tree model is first used to integrate seven features, including CAPRA-S score, age, clinical pathological stage, seminal vesicle invasion, resection margin status, PSA level, and biochemical recurrence prediction score at the pathological level. The survival:aft accelerated failure time objective function is used for modeling, and parameter optimization is performed through grid search. Parameter selection is based on cross-validation and ROC-AUC evaluation indicators, and its calculation formula is: .

[0040] Based on the relationship between features and survival time, a prediction model based on the Xgboost framework was developed. This model takes clinical features as input and outputs a recurrence risk score. The objective function of the Xgboost model is survival:aft, which is an Accelerated Failure Time (AFT) model specifically designed for survival analysis. The AFT model directly models survival time, assuming that covariates (i.e., clinical features) influence survival time by accelerating or decelerating it. In this case, if BCR = 1, BCR_TimeMonths represents the actual observed time to biochemical recurrence. If BCR = 0, BCR_TimeMonths indicates that no biochemical recurrence was observed during follow-up and is therefore considered censored data, with BCR_TimeMonths corresponding to the censored time. The model predicts each patient's survival time by learning the relationship between clinical features and the survival time distribution. Because it uses the AFT model, the model actually predicts an acceleration factor, which describes how clinical features influence survival time. Therefore, the final predicted risk score actually reflects the patient's survival time, that is, the model's predicted time to biochemical recurrence. The lower the score, the shorter the patient's recurrence time and the higher the recurrence risk; conversely, the higher the risk score, the lower the patient's recurrence risk.

[0041] (5) Explainability module

[0042] The explainability module is an important component of the AI-based multimodal prostate cancer biochemical recurrence risk stratification prediction system, which includes a pathology image interpretable submodule, a fusion model interpretable submodule, and a similar case comparative analysis submodule. Its core design is to improve the transparency and clinical credibility of the prediction results. The module consists of three subsystems that work together, aiming to provide in-depth interpretation from three levels: pathology images, fusion models, and similar cases. It enables clinicians to understand the model's judgment basis from multiple dimensions simultaneously, including visual, quantitative analysis, and analogical reasoning. This significantly improves the transparency and credibility of AI-assisted decision-making, and provides stronger support for individualized follow-up and treatment plan formulation.

[0043] 1) The pathology image interpretability submodule intelligently analyzes pathology images using Grad-CAM technology. Using multi-scale heatmaps, it precisely locates pathological regions that contribute significantly to the predicted score. Key morphological features are visualized at different magnifications (10×, 20×, and 40×). Key pathological features extracted include glandular structure, nuclear atypia, and cellular infiltration. A built-in visual calibration mechanism ensures that the extracted results are consistent with the pathologist's judgment. For ambiguous or uncertain pathological regions, the system bolds important areas in the heatmap with fluorescent markers to prompt the pathologist for review. This allows for a better integration of clinical experience and AI analysis, providing more accurate pathological evidence for subsequent risk prediction.

[0044] 2) The interpretable submodule of the fusion model uses the SHAP value calculation method to quantitatively analyze the contribution of all input variables, generating waterfall charts and trying to intuitively display the positive and negative impact of each clinical indicator (PSA, Gleason score, resection margin status, etc.) and pathological characteristics on the final risk prediction results. The key clinical indicator contributions that are focused on include the core variables of the CAPRA-SScore, such as patient age, preoperative PSA level, Gleason score, clinical pathological stage, resection margin status, etc., and a built-in contribution verification mechanism ensures the accuracy of the SHAP value calculation results. For indicators with abnormal contributions or inconsistent with clinical experience, the system will fluorescently mark and bold the key variables in the waterfall chart, prompting clinicians to review, so as to better integrate clinical cognition and artificial intelligence analysis and enhance the credibility of the model's prediction results.

[0045] 3) The similar case comparative analysis submodule automatically retrieves 30 patient cases with the most similar feature expressions and prediction scores from the internal training set based on the KNN algorithm, and presents their key clinical baseline indicators, treatment plans and actual biochemical recurrence results in tabular form to form a risk reference spectrum based on real cases. The key clinical baseline indicators for reference include: age, preoperative PSA level, Gleason score, clinical TN stage, treatment plan, follow-up time and biochemical recurrence time, etc., and a built-in similarity verification mechanism is used to ensure the accuracy of the retrieval results. For features that differ significantly from similar cases and target patients, the system will fluorescently mark the key variables in the bold table to prompt clinicians to review, so as to better integrate individual characteristics and group experience, and provide a more comprehensive reference for formulating personalized treatment plans.

[0046] In a second aspect, the present invention provides a method for constructing the above-mentioned prediction system and its application in clinical practice, comprising the following steps:

[0047] (1) Dataset division: Clinical data and full pathological slide data of patients after radical prostatectomy were collected retrospectively from multiple centers. The panoramic pathological slide scan images of tissue HE staining that were judged by doctors were summarized as the total dataset, and the total dataset was divided into a test set and a validation set;

[0048] (2) Clinical feature extraction and processing module, by accessing the big prediction model API, automatically extracts the patient's clinical feature information (age, preoperative PSA level, Gleason score, clinical pathological stage, surgical margin status, lymph node status, seminal vesicle invasion) from the original medical record report and surgical pathology report, and automatically determines and fills in missing values. The filled information is then scored by the risk stratification algorithm based on the CAPRA-S scoring system to form a standardized clinical feature vector;

[0049] (3) Pathological feature extraction and processing module: For the pre-processed tissue HE-stained panoramic pathological section scan images, a multi-instance algorithm based on pseudo-packet strategy and cyclic cross-attention is used to first extract the deep features of tumor images with different objective magnifications. Then, a feature representation at the panoramic pathological section scan image level is generated in a weakly supervised network. The microscopic cell abnormalities and macroscopic tissue change features are extracted to obtain the pathological feature representation, which is then fused to generate a biochemical recurrence prediction score at the pathological level.

[0050] (4) Fusion prediction module, which performs multimodal feature fusion on the pathological feature representation and the standardized clinical feature vector information to build an end-to-end prostate cancer biochemical recurrence risk prediction model; highlights the weight of key features through the attention mechanism, and then calculates the attention score of each feature (W i× score), and after weighted fusion of the features, the patients were classified into three risk levels: high risk, medium risk, and low risk based on the calculation and analysis results, and the key predictive factors were visualized;

[0051] (5) The above evaluation system was trained and validated in multiple independent validation cohorts and compared with traditional clinical prognostic evaluation models.

[0052] In a third aspect, the present invention provides a non-transitory computer-readable storage medium that can be installed in a communication tool such as a computer, and stores a computer program that implements the above steps when executed by a processor.

[0053] Functions and effects of the invention

[0054] The present invention establishes a multimodal prostate cancer biochemical recurrence risk stratification prediction system based on artificial intelligence. It is based on the Xgboost framework and analyzes the panoramic pathological section scan images of postoperative patients through end-to-end, multi-scale, multi-center, and large-sample analysis. It can maximize the use of pathological information and provide reliable clinical facts with a complete and long-term follow-up process. At the same time, combined with clinical indicators such as the CAPRA-S score, it can more comprehensively evaluate the patient's prognostic risk, which has obvious advantages over traditional models. This method aims to be more in line with the hospital's usage scenarios in many aspects. By integrating the pathological section characteristics and clinical characteristics after radical resection, it can more efficiently and accurately predict the risk of prostate cancer recurrence in patients, and help to accurately predict the risk of recurrence in patients 3 years after surgery and longer. Combined with an interpretable module, it assists doctors in interpreting the results, and then accurately stratifies and personalizes the BCR risk of prostate cancer patients, reduces the risk of overtreatment and missed diagnosis, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Shows the composition framework diagram of the system of the present invention;

[0056] Figure 2 The pattern process of the training and verification phases of the system of the present invention is shown.

[0057] Figure 3 Internal cohort training-validation ROC curves are shown.

[0058] Figure 4 The ROC curves of the external validation cohorts are shown, from left to right: the Chinese multicenter cohort (A), TCGA (B), and PLCO (C) external validation ROC curves.

[0059] Figure 5 KM survival prediction curves of the four cohorts are shown.

[0060] Figure 6The analysis results of the explainability module are shown. DETAILED DESCRIPTION

[0061] The present invention will be described in detail below with reference to the examples and accompanying drawings. However, the following examples should not be considered as limiting the scope of the present invention.

[0062] 1. System Structure

[0063] The structure of the multimodal prostate cancer biochemical recurrence risk stratification prediction system 100 based on artificial intelligence in the present invention can be found in Figure 1 , including input display module 1, clinical feature extraction and processing module 2, pathological feature extraction and processing module 3, fusion prediction module 4, interpretability module 5, storage module 6, and control module 7.

[0064] Input and Display Module 1 is the system's data entry point, used to input the patient's original medical records and surgical pathology reports containing panoramic pathology slide scans. It then displays the analysis results after completion. This module is designed as an intelligent, adaptive interface to enable efficient and accurate collection of patient clinical data, allowing clinicians to directly upload or paste original medical records and surgical pathology reports.

[0065] The clinical feature extraction and processing module 2 includes a clinical feature extraction submodule 21 and a clinical data processing submodule 22 . The clinical feature extraction submodule 21 is a system-integrated large language model API (such as GPT-4o and Claude), which is used to efficiently collect and process multi-dimensional clinical data of patients, including patient age, preoperative PSA level, pathological Gleason score, clinical pathological stage, surgical margin status, seminal vesicle invasion status, and other key indicators that constitute the CAPRA-SScore. The clinical data processing submodule 22 first integrates patient characteristics with the training cohort and accurately fills missing values ​​for the patient's key clinical variables using the Multiple Interpolation Chained Equations (MICE) algorithm. It also issues a warning if the number of missing values ​​is no less than a preset number (e.g., >4 missing values). The filled data is then scored using a risk stratification algorithm based on the CAPRA-S scoring system, assigning scores to PSA level, Gleason score, clinical pathological stage, surgical margin status, and lymph node status. Ultimately, a continuous risk assessment of 0-7 points is achieved, and patients are accurately stratified into three levels: low risk (0-2 points), intermediate risk (3-5 points), and high risk (6 points or above).

[0066] The pathological feature extraction and processing module 3 includes a feature vector processing submodule 31, a cyclic cross-attention submodule 32, and a multi-scale feature fusion submodule 33. The feature vector processing submodule 31 first pre-processes the panoramic pathology slice scan image, and then crops the tumor area into non-overlapping image blocks of 256×256 pixels at 10×, 20×, and 40× objective lens magnifications. The self-supervised visual encoder UNI is then used to automatically extract features from it and refine the features based on the pseudo-package strategy, capturing local key area information while retaining overall structural features, thereby improving the model's ability to identify and discriminate heterogeneous lesions in the image. The cyclic cross-attention submodule 32 then performs four-dimensional feature reconstruction and convolution processing of different sizes on the input panoramic pathology slice scan image-level feature representation or the pseudo-package and parent package feature package representation, first reconstructing it into a four-dimensional tensor X reshaped : , and then use convolution kernels of different sizes (7×7, 5×5, 3×3) to process the feature data to further extract features with spatial perception and contextual information; the multi-scale feature fusion submodule 33 fuses the feature information at the three objective lens magnification scales, and performs a weighted average of the biochemical recurrence probabilities obtained under different objective lens magnifications to finally generate the patient's predicted risk score and obtain a recurrence or non-recurrence prediction result.

[0067] Fusion Prediction Module 4 aims to achieve precise stratification and personalized prediction of prostate cancer biochemical recurrence (BCR) risk. This module uses a multimodal feature deep fusion strategy to integrate deep learning features from panoramic pathology slide scans, clinical CAPRA-S scores, and key clinical information to build an end-to-end AI risk prediction system.

[0068] Specifically, the CAPRA-S score, age, clinical pathological stage, seminal vesicle invasion, surgical margin status, preoperative PSA level and biochemical recurrence prediction score at the pathological level were first integrated through the Xgboost model, and the weight of key features was highlighted through the attention mechanism; then the attention score of each feature (W i × score), and after weighted fusion of the features, three risk levels of high risk, medium risk, and low risk are obtained based on the calculation and analysis results; then, the survival:aft accelerated failure time objective function model is used to predict the patient's survival time based on the relationship between the clinical features learned by the model and the survival time distribution.

[0069] Preferably, the Softmax algorithm is used to implement dynamic weight allocation of the above features, that is, the attention mechanism automatically learns the weight of each feature in the current task by calculating the correlation or importance between features. The calculation formula is as follows:

[0070]

[0071] Among them, Q and K are the representations of features, K T refers to the transpose of the matrix K, and d is the dimension.

[0072] The assignment of weights depends on the overall distribution of the input data, such as the combination of the above characteristics of a patient.

[0073] In the code implementation, the Xgboost extreme gradient boosting tree model is first used to integrate seven features, including CAPRA-S score, age, clinical pathological stage, seminal vesicle invasion, resection margin status, PSA level, and biochemical recurrence prediction score at the pathological level. The survival:aft accelerated failure time objective function is used for modeling, and parameter optimization is performed through grid search. Parameter selection is based on cross-validation and ROC-AUC evaluation indicators, and its calculation formula is: .

[0074] Based on the relationship between features and survival time, a prediction model based on the Xgboost framework was developed. This model takes clinical features as input and outputs a recurrence risk score. The objective function of the Xgboost model is survival:aft, which is an Accelerated Failure Time (AFT) model specifically designed for survival analysis. The AFT model directly models survival time, assuming that covariates (i.e., clinical features) influence survival time by accelerating or decelerating it. In this case, if BCR = 1, BCR_TimeMonths represents the actual observed time to biochemical recurrence. If BCR = 0, BCR_TimeMonths indicates that no biochemical recurrence was observed during follow-up and is therefore considered censored data, with BCR_TimeMonths corresponding to the censored time. The model predicts each patient's survival time by learning the relationship between clinical features and the survival time distribution. Because it uses the AFT model, the model actually predicts an acceleration factor, which describes how clinical features influence survival time. Therefore, the final predicted risk score actually reflects the patient's survival time, that is, the model's predicted time to biochemical recurrence. The lower the score, the shorter the patient's recurrence time and the higher the recurrence risk; conversely, the higher the risk score, the lower the patient's recurrence risk.

[0075] The Interpretability Module 5 is a key component of the AI-based multimodal prostate cancer biochemical recurrence risk stratification prediction system. It includes a pathology image interpretability submodule 51, a fusion model interpretability submodule 52, and a similar case comparative analysis submodule 53. Its core design aims to enhance the transparency and clinical credibility of prediction results. Composed of three collaborative subsystems, this module aims to provide in-depth interpretation from three levels: pathology images, fusion models, and similar cases. This allows clinicians to understand the model's judgments from multiple perspectives, including visual, quantitative analysis, and analogical reasoning. This significantly improves the transparency and credibility of AI-assisted decision-making, providing stronger support for individualized follow-up and treatment planning.

[0076] 2. Model Training and Construction of an Artificial Intelligence-Based Multimodal Prostate Cancer Biochemical Recurrence Risk Stratification Prediction System

[0077] Modeling flow chart see Figure 2 The specific process is as follows:

[0078] 1. Data collection and processing: A multicenter retrospective study was conducted to collect original medical records of patients after radical prostatectomy and surgical pathology reports containing panoramic pathology slide scans of HE-stained sections. Data cleaning and preprocessing were then performed.

[0079] A total of 2,522 patients who underwent radical prostatectomy from four independent centers were included in the present invention, of which 849 cases were from Renji Hospital affiliated to Shanghai Jiao Tong University School of Medicine in the training set, and the validation set included 564 cases from a Chinese multicenter cohort, 341 cases from the TCGA cohort, and 768 cases from the PLCO cohort.

[0080] Inclusion criteria: (1) patients with pathologically confirmed prostate cancer; (2) patients who underwent radical prostatectomy; (3) patients who did not receive neoadjuvant therapy before surgery and did not receive adjuvant therapy within 6 months after surgery; (4) patients who had at least one full-section HE-stained pathological slide image; and (5) patients with complete follow-up data and biochemical recurrence outcome information.

[0081] Exclusion criteria: (1) poor specimen preservation, affecting feature extraction; (2) the tumor area is too small to extract at least one image block under 10x field of view; (3) the section is contaminated, seriously affecting the outcome interpretation.

[0082] Tumor regions were annotated in QuPath software for the 4,368 diagnostic HE-stained slides included in the model, and the annotations were exported as GeoJSON. After excluding background regions, the original panoramic pathology slide scans were cut into non-overlapping 256×256 pixel blocks at 10x, 20x, and 40x magnification. Tumor region blocks were extracted based on the annotations and used as digital prostate cancer pathology slides for subsequent analysis.

[0083] 2. Clinical feature extraction and processing

[0084] The system uses an integrated large language model API to collect and process multi-dimensional clinical data from patients, including patient age, preoperative PSA level, pathological Gleason score, clinical pathological stage, surgical margin status, and seminal vesicle invasion, which constitute the key indicators of CAPRA-SScore. During the extraction process, the accuracy of the extraction results is ensured based on a built-in data verification mechanism. For fields with ambiguity or uncertainty, the system will fluorescently mark the clinical features in the bold table to prompt clinicians to review, so as to achieve better intelligent and standardized integration of clinical and pathological data, laying a solid data foundation for subsequent artificial intelligence risk prediction.

[0085] The patient characteristics and the training cohort were then integrated, and the missing values ​​of the patient's key clinical variables were accurately filled using the Multiple Imputation Chained Equations (MICE) algorithm. A warning was issued when the number of missing values ​​was not less than the preset number (e.g., >4 missing values). The filled data were then scored using the risk stratification algorithm based on the CAPRA-S scoring system, and the PSA level, Gleason score, clinical pathological stage, surgical margin status, and lymph node status were assigned as follows: the PSA level score range was 0-4 points (corresponding to <6, 6-10, 10-20, 20-3 The system uses the Gleason score range of 0-3 points (corresponding to ISUP grades 1-5, of which ISUP grades 4 and 5 are 3 points), clinical pathological stage 0-1 point (0, T1 / T2 stage, 1, T3 / T4 stage), surgical margin status 0-2 points (0, R0 status, 1, R1 status, 2, R2 status), and lymph node involvement 0-1 point (0, N0 status, 1, N1 status). Ultimately, a continuous risk assessment of 0-7 points is achieved, and patients are accurately stratified into three levels: low risk (0-2 points), intermediate risk (3-5 points), and high risk (6 points and above).

[0086] 3. Pathological feature extraction and processing module

[0087] A self-supervised visual encoder UNI is used to automatically extract features from the above 256×256 pixel image blocks to generate a feature representation at the level of panoramic pathology slice scans, with a size of {Ns×1024}, where N represents the number of image blocks at this scale and s∈{10×, 20×, 40×}. A pseudo bag strategy is then used to extract key features. The feature bag representation is then reconstructed into a four-dimensional tensor X. reshaped : , and then use convolution kernels of different sizes (7×7, 5×5, 3×3) to process the feature data to further extract features with spatial perception and contextual information.

[0088] Convolution kernels of different scales complement each other, and new feature maps with dense and rich contextual information are gradually generated based on cross-attention iteration, enabling the model to simultaneously identify microscopic cellular abnormalities and macroscopic tissue changes; finally, the feature information at the three objective lens magnification scales is fused, and the weights of key features are highlighted through the attention mechanism. The biochemical recurrence probabilities obtained at different objective lens magnifications are weighted averaged to ultimately generate a biochemical recurrence prediction score at the pathological level for the patient, and a recurrence or non-recurrence prediction result is obtained.

[0089] 4. Integration of clinical and pathological features to construct a multimodal biochemical recurrence risk prediction model

[0090] First, the CAPRA-S score, age, clinical pathological stage, seminal vesicle invasion, surgical margin status, preoperative PSA level, and biochemical recurrence prediction score at the pathological level were integrated through the Xgboost model, and the weight of key features was highlighted through the attention mechanism; then the attention score of each feature (W i × score), and after weighted fusion of the features, three risk levels of high risk, medium risk, and low risk are obtained based on the calculation and analysis results; then, the survival:aft accelerated failure time objective function model is used to predict the patient's survival time based on the relationship between the clinical features learned by the model and the survival time distribution.

[0091] In this invention, both fusion operations use the Softmax algorithm to achieve dynamic weight allocation of the above features. That is, the attention mechanism automatically learns the weight of each feature in the current task by calculating the correlation or importance between features. The calculation formula is as follows:

[0092]

[0093] Among them, Q and K are the representations of features, K T refers to the transpose of the matrix K, and d is the dimension.

[0094] The assignment of weights depends on the overall distribution of the input data, such as the combination of the above characteristics of a patient.

[0095] 5. Verification

[0096] The predictive performance of the model was evaluated on multiple independent validation cohorts and compared with traditional clinical prognostic assessment models.

[0097] The C-index, area under the receiver operating characteristic curve (AUC), and KM survival prediction curve were used to evaluate the predictive performance of the model on the Renji Hospital internal validation set, the Chinese multicenter validation cohort, the TCGA validation cohort, and the PLCO validation cohort.

[0098] The PBRC-S model constructed by the present invention was also compared with the traditional clinical prognostic assessment model, the CAPRA-S score. The results, as shown in Table 1 below, show that in the training cohort, the C-index value of the CAPRA-S score alone was 0.783, while the C-index of the fusion model of the present invention reached 0.834, exceeding that of CAPRA-S. Furthermore, in different validation cohorts, the C-index of the fusion model was consistently higher than that of the CAPRA-S score, indicating that integrating pathological image information and clinical indicators can significantly improve the accuracy of prognostic prediction.

[0099] ROC results showed that in the four cohorts, the AUC area of ​​the PBRC-S model constructed by the present invention was higher than that of the traditional CAPRA-S score and the single-scale AI model, and the prediction performance was better ( Figure 3 、 Figure 4 ).

[0100] KM survival prediction curve analysis showed that the PBRC-S model can more accurately classify the total population of each cohort and different subgroups into three equal parts according to the risk of recurrence, namely high-risk, intermediate-risk, and low-risk patients ( Figure 5 ).

[0101] Table 1 Validation results of the model in different cohorts

[0102]

[0103] 6. Model interpretability analysis

[0104] In order to improve the transparency and clinical credibility of the model prediction results, the present invention constructs a multi-level interpretability analysis module. First, for pathological images, Grad-CAM technology is used to intelligently analyze pathological images, and multi-scale heat maps are generated to accurately locate pathological areas that contribute significantly to the prediction score. Secondly, for the fusion model, the SHAP value calculation method is used to quantitatively analyze the contribution of all input variables, and waterfall charts and force graphs are generated to intuitively display the positive and negative impact of various clinical indicators (such as PSA, Gleason score, surgical margin status, clinical pathological stage, etc.) and pathological characteristics on the final risk prediction results. At the same time, a built-in contribution verification mechanism ensures the accuracy of the SHAP value calculation results. In addition, based on KNN, patient cases with the most similar feature expressions and prediction scores are automatically retrieved from the internal training set, and their key clinical baseline indicators, treatment plans and actual biochemical recurrence results are presented in tabular form to form a risk reference spectrum based on real cases.

[0105] For example, for patient 561 in the test set, the model predicted that he had a higher risk of biochemical recurrence. The pathology image heat map found that the model may have focused on high-grade features and inflammatory cell areas, which are closely related to tumor aggressiveness ( Figure 6A). Combined with clinical information, we further used SHAP values ​​to analyze the impact of each feature on the prediction results. The results showed that the pathology prediction module score was the most critical risk factor, followed by CAPRA_S_score, psa_value, age_at_diagnosis, and pathologic_T_stage. Higher values ​​of these features increased the risk of biochemical recurrence ( Figure 6 B). Overall, factors such as pathologic_T_stage = 2, age_at_diagnosis = 70, pathology prediction module score = 0.309, and psa_value = 32.9 work together to increase the patient's predicted risk score from a baseline value of 1.9 to 3.8. CAPRA_S_score = 10 significantly reduces the patient's time to recurrence and increases the risk of recurrence ( Figure 6 C). Ultimately, the model predicts that this patient has a high risk of biochemical recurrence.

[0106] The interpretable model used the KNN algorithm to find 30 cases with characteristics most similar to that of the patient. The average age of these cases was 69.43 years, the median PSA value was 20.75 ng / mL, the average Gleason score was 9.03, the median CAPRA-S score was 8.00, the median pathology prediction module score was 0.27, and the median BCR_TimeMonths was 24.50. Compared with this patient, these similar cases had lower PSA values, lower CAPRA-S scores, lower pathology prediction module scores, and their average recurrence time was also shorter than that of this patient. Considering that the patient's PSA value and pathology prediction module score were higher than those of similar cases, and the CAPRA_S_score was higher, it suggests that his risk of recurrence may still be relatively high. For specific information, see Table 2 below:

[0107] Table 2 Comparison of characteristics between the test patients and 30 similar patients selected by the KNN algorithm in the training set

[0108]

[0109] In summary, compared with other methods, this method establishes a multimodal prostate cancer biochemical recurrence risk stratification prediction model. Based on the Xgboost framework, this model maximizes the use of pathological information through end-to-end, multi-scale, multi-center, and large-sample analysis of full pathology slides. Combined with the CAPRA-S score, it provides a more comprehensive assessment of a patient's prognostic risk and provides reliable clinical evidence through a complete and long-term follow-up process, offering significant advantages over traditional models. This method can assist physicians in the precise stratification and personalized follow-up of prostate cancer, potentially reducing the risks of overtreatment and missed diagnosis, and holds great promise for future application.

[0110] The basic principles, main features and significant advantages of the present invention have been fully described above. The purpose of the description in the embodiments and the specification is only to further illustrate the operating principles of the present invention. Without violating the core concept of the present invention and the scope of legal protection, the present invention allows and covers all kinds of reasonable changes and improvements. These changes and improvements will be deemed to fall within the scope of protection requested by the present invention. Specifically, the scope of protection of the present invention will be clearly and strictly defined by the attached claims and their equivalents with equal legal effect.

Claims

1. An artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system, characterized by: include: An input and display module is used to input the patient's original medical record report and the surgical pathology report containing the panoramic scan image of the pathological section, and to display the analysis results after the analysis is completed; The clinical feature extraction and processing module automatically extracts patients' clinical feature information from original medical records and surgical pathology reports, and automatically determines and fills in missing values. During the extraction process, a built-in data validation mechanism marks ambiguous or uncertain fields, prompting clinicians to review them. The filled-in information is then scored using a risk stratification algorithm based on the CAPRA-S scoring system to form a standardized clinical feature vector. The pathology feature extraction and processing module uses a multi-instance algorithm based on a pseudo-packet strategy and recurrent cross-attention after image preprocessing to first extract deep features of tumor images at different objective magnifications. Next, a feature representation at the level of panoramic pathology slice scans is generated in a weakly supervised network. Microscopic cellular abnormalities and macroscopic tissue change features are extracted to obtain pathology feature representations, which are then integrated to generate a biochemical recurrence prediction score at the pathology level. The fusion prediction module first uses the Xgboost model to fuse the CAPRA-S score, age, clinical pathological stage, seminal vesicle invasion, surgical margin status, preoperative PSA level, and biochemical recurrence prediction score at the pathological level. The attention mechanism then highlights the weights of key features. The attention score of each feature is then calculated, and after weighted fusion of the features, three risk levels (high, intermediate, and low) are determined based on the calculation and analysis results. The survival:aft accelerated failure time objective function is then used to model the patient's survival time based on the relationship between the clinical features learned by the model and the survival time distribution. The explainability module provides in-depth interpretation from three levels: pathological images, fusion models, and similar cases. It quantitatively displays the contribution of each clinical indicator and pathological feature to the prediction results, and provides a basis for understanding the model from multiple dimensions: visual, quantitative analysis, and analogical reasoning. The explainability module includes a pathological image explainable submodule, a fusion model explainable submodule, and a similar case comparative analysis submodule. The pathology image interpretability submodule intelligently analyzes pathology images based on Grad-CAM technology, accurately locates pathological areas that contribute significantly to the prediction score through multi-scale heat maps, and presents important morphological features at different magnifications; During the analysis process, a built-in visual calibration mechanism is used to ensure that the extracted results are basically consistent with the judgment of the pathologist. For ambiguous or uncertain pathological areas, the system marks the important areas in the heat map and prompts the pathologist to review. The interpretable submodule of the fusion model uses the SHAP value calculation method to quantitatively analyze the contribution of all input variables, intuitively displaying the positive and negative impact of each clinical indicator and pathological feature on the final risk prediction result; during the analysis process, the system marks indicators with abnormal contribution or inconsistent with clinical experience and prompts clinicians to review them. The similar case comparative analysis submodule automatically retrieves 30 patient cases with the most similar feature expressions and prediction scores from the internal training set, and presents their key clinical baseline indicators, treatment plans, and actual biochemical recurrence results in a tabular form to form a risk reference spectrum based on real cases; During the analysis process, the system marks any features that differ significantly from those of similar cases and target patients and prompts clinicians to review them.

2. The artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system according to claim 1, characterized in that: It also includes a storage module for storing information processed by each module.

3. The artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system according to claim 1, characterized in that: The clinical feature extraction and processing module includes a clinical feature extraction submodule and a clinical data processing submodule. The clinical feature extraction submodule is a system-integrated large language model API that extracts at least the following clinical feature information: age, preoperative PSA level, Gleason score, clinical pathological stage, surgical margin status, lymph node status, and seminal vesicle invasion. During the extraction process, ambiguous or uncertain fields are marked based on the built-in data verification mechanism, prompting clinicians to review. The clinical data processing submodule first integrates patient characteristics and training cohorts, and uses the MICE algorithm to accurately fill in the missing values ​​of the patient's key clinical variables, and sends a warning when the number of missing values ​​is not less than the preset number; the filled data is then scored using the risk stratification algorithm based on the CAPRA-S scoring system, and values ​​are assigned to PSA levels, Gleason scores, clinical pathological stages, surgical margin status, and lymph node status. Based on the comprehensive scoring results, patients are accurately stratified into three levels: low risk, medium risk, and high risk.

4. The artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system according to claim 1, characterized in that: The pathological feature extraction and processing module includes a feature vector processing submodule, a cyclic cross attention submodule and a multi-scale feature fusion submodule. The feature vector processing submodule first preprocesses the panoramic pathology slice scan image, then crops the tumor area into non-overlapping image blocks of 256×256 pixels at 10×, 20×, and 40× objective lens magnifications, and uses the self-supervised visual encoder UNI to automatically extract features from it, generating a feature representation at the panoramic pathology slice scan image level containing low-dimensional feature sets at three different magnifications; then, based on the pseudo-bag strategy, the features are refined, and four-dimensional reconstruction and convolution processing of different sizes are performed on the features. The recurrent cross-attention submodule complements convolutional kernels of different scales and iteratively generates new feature maps with dense and rich contextual information based on cross-attention, enabling the model to simultaneously identify microscopic cellular abnormalities and macroscopic tissue changes. The multi-scale feature fusion submodule fuses feature information at three objective lens magnification scales, performs weighted averaging on the biochemical recurrence probabilities obtained at different objective lens magnifications, and obtains a recurrence or non-recurrence prediction result.

5. The artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system according to claim 4, characterized in that: The preprocessing method of the panoramic pathology slice scan image is as follows: by adjusting the mean and standard deviation of the image block data, the data is converted into a distribution with zero mean and unit variance, thereby achieving data standardization and image intensity normalization; After the last convolutional layer of UNI, the 1024-dimensional feature vector of each image block is extracted to generate a feature representation at the level of the panoramic pathology slide scan image. Each panoramic pathology slide scan image generates a set of low-dimensional feature matrices at three multiple scales, with a size of {Ns×1024}, where N represents the number of image blocks at that scale, and s∈{10×, 20×, 40×}; The size of each convolution kernel corresponds to a different spatial range, which are 7×7, 5×5, and 3×3 respectively.

6. The artificial intelligence-based multimodal prostate cancer biochemical recurrence risk stratification prediction system according to claim 1, characterized in that: The key pathological features extracted by the pathological image interpretable submodule include glandular structure, nuclear atypia, and cell infiltration; The contribution of the key clinical indicators that the explained submodule of the fusion model focuses on covers the core variables that constitute the CAPRA-S Score; The key clinical baseline indicators that the similar case comparative analysis submodule focuses on include age, preoperative PSA level, Gleason score, clinical pathological stage, treatment plan, follow-up time and biochemical recurrence time.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the function of the system according to any one of claims 1 to 6 is realized.

Images