An interpretable analysis and decision sharing verification system for rectal cancer prognosis model

Abstract

The application discloses an interpretable analysis and decision sharing verification method and system for a colorectal cancer prognosis model, and relates to the field of medical artificial intelligence interpretability. The method comprises the following steps: performing gradient weighted class activation mapping analysis on the prognosis model to generate an image heat map; calculating the contribution degree of multi-modal features by using a SHAP interpreter; constructing an integrated visualization interface to present the patient data, model prediction and the above-mentioned explanation results to doctors; the doctors perform independent risk assessment based on the interface information; and finally, the decisions of the doctors and the model are compared to evaluate the auxiliary performance of the model. Through multi-level explanation and innovative doctor-model decision sharing verification mechanism, the transparency and clinical credibility of the complex AI prognosis model are significantly improved, the value of time series data in dynamic risk assessment can be verified, and the clinical landing application of the AI model is effectively promoted.

Description

Technical Field

[0001] This invention relates to the field of interpretability and clinical validation of medical artificial intelligence, specifically to a method and system for interpretability analysis and decision-sharing validation of a rectal cancer prognostic model. Background Technology

[0002] While deep learning-based medical prognostic models have demonstrated superior performance, their internal decision-making processes are often considered a "black box," lacking interpretability. This constitutes a significant barrier to application in the rigorous field of clinical medicine, as doctors find it difficult to understand and trust the model's predictions, thus hindering their practical translation.

[0003] In existing technologies, some general-purpose model interpretability tools (such as LIME and SHAP) can be used for machine learning models. However, for complex deep neural networks that integrate multimodal temporal data, especially those involving mixed inputs of image and structured data, there is a lack of targeted, hierarchical interpretation solutions. Furthermore, how to effectively present the results of interpretability analysis to clinicians and design scientific processes to verify the complementarity between model decisions and physician decisions, thereby improving the clinical acceptance and practical value of the model, are urgent problems to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for interpretability analysis and decision-sharing verification of a rectal cancer prognostic model, aiming to solve the problems of poor interpretability and insufficient clinical validation of complex AI prognostic models. Through multi-level interpretation and physician-model decision sharing, the transparency and clinical credibility of the model are improved.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an interpretability analysis and decision-sharing verification method for a rectal cancer prognostic model, characterized in that it is applied to a prognostic prediction model, which predicts the postoperative recurrence risk of rectal cancer patients based on multimodal time-series data, and the method includes: Interpretability analysis steps: a) Image visualization step: For the visual transformer network used to process magnetic resonance images in the prognostic prediction model, gradient-weighted class activation mapping technology is used to generate a regional heatmap related to the model's prediction decision, and it is overlaid on the original image; b) Feature contribution analysis step: Using a SHAP interpreter, the contribution values ​​of various features input to the prognostic prediction model for the prediction results of a single patient and for the overall model prediction results are calculated; Decision-sharing verification steps: c) Constructing a visual interactive interface, which simultaneously displays the patient's multi-time-series clinical and imaging information, the heatmap, and the feature contribution analysis results; d) Clinicians independently assess and score the patient's recurrence risk based on all the information provided by the visual interactive interface; e) The doctor's assessment score is compared with the automatic prediction results of the prognosis prediction model, and the effectiveness of the model in assisting decision-making is evaluated through statistical indicators.

[0006] Further: In the image visualization step, the specific steps include: extracting the output feature map and gradient of the last attention block of the visual transformer; calculating the weight of each feature channel; performing ReLU activation and upsampling on the weighted feature map to generate a heatmap that matches the size of the input image.

[0007] Further: In the feature contribution analysis step, for the tumor cell nuclear morphology features extracted from digital pathology images, a support vector machine linear model is additionally used to analyze the weights of two features, nuclear size and nuclear shape non-convexity score, in order to explain the contribution of nuclear atypia to the prediction.

[0008] Furthermore, the decision-sharing verification step also includes a multi-time-point comparison and evaluation sub-step: at different time points during the patient's postoperative follow-up, the prediction model constructed using time-series data up to that point is used respectively, and the changes in model performance at different time points are compared to evaluate the value of time-series data for long-term risk prediction.

[0009] Further: The multi-time-point comparative evaluation sub-step specifically involves: constructing a t0 time-point model, where t0 is before the start of follow-up, and t1 to t6 time-point models, where t1 to t6 are the follow-up nodes from the 6th month to the 36th month after surgery, and comparing the area under the curve, C-index, and Kaplan-Meier survival curves of each model.

[0010] This invention also provides an interpretability analysis and decision-sharing verification system for a rectal cancer prognostic model, comprising: an interpretability analysis engine for: performing gradient-weighted class activation mapping analysis on the image analysis network in the prognostic prediction model to generate a heatmap; calculating the contribution value of the model input features using a SHAP interpreter; a visualization interface generation module for integrating patient data, model prediction results, the heatmap, and the contribution values ​​to generate an interactive interface for physician review; and a decision comparison and evaluation module for receiving independent evaluation results input by physicians through the interactive interface, comparing them with the model prediction results, and calculating evaluation indicators.

[0011] Furthermore, it also includes a time-series performance evaluation module, which is used to select multiple nodes on the patient follow-up timeline, evaluate the predictive performance of the model at the corresponding time points, and analyze the trend of performance changes with time points.

[0012] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the described method.

[0014] The beneficial effects of this invention are as follows:

[0015] Multi-level interpretability: It provides multi-level interpretation from the "image visual level" (heatmap) to the "feature model level" (contribution value), enabling doctors to not only intuitively see the image areas that the model focuses on, but also understand the specific impact of different clinical, imaging, and pathological features on prediction.

[0016] Highly targeted: Specifically designed for complex prognostic models that integrate multimodal time-series data, with a particular customized interpretation scheme for ViT networks and pathological features.

[0017] Decision-sharing verification mechanism: It innovatively proposes a verification process for "doctor-model" shared decision-making. By constructing a visual interface, the model's interpretation results are presented together with the original data, allowing doctors to make independent judgments with sufficient information, thereby scientifically evaluating whether the model replaces, assists, or enhances the doctor's decision-making ability.

[0018] Dynamic evaluation: By comparing and evaluating at multiple time points, the model's performance was not only verified, but the incremental value of time series data in dynamic risk assessment was also demonstrated more intuitively, enhancing the model's persuasiveness in long-term follow-up management. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the module of the system for predicting the postoperative recurrence risk of rectal cancer patients based on multimodal time-series data involved in the present invention.

[0021] Figure 3 This is a schematic diagram of the U-Net automatic segmentation model network structure of the system for predicting postoperative recurrence risk in rectal cancer patients based on multimodal temporal data, which is involved in this invention.

[0022] Figure 4 The model involved in this invention is a Transformer network structure for a system that predicts the postoperative recurrence risk of rectal cancer patients based on multimodal temporal data.

[0023] Figure 5 This is a flowchart of tumor cell nucleus segmentation based on self-supervised pre-training for a system that predicts postoperative recurrence risk in rectal cancer patients based on multimodal temporal data, as described in this invention.

[0024] Figure 6 This is a flowchart of the pathological image spatial feature extraction network based on Graph Transformer for the system of predicting postoperative recurrence risk of rectal cancer patients based on multimodal temporal data involved in this invention.

[0025] Figure 7 This is a visualization diagram of Grad-CAM according to an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram of the SHAP interpretable mechanism in an embodiment of the present invention.

[0027] Figure 9 This is a schematic diagram of the doctor-model shared decision visualization interface according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below with reference to the accompanying drawings.

[0029] This also includes a detailed introduction to the model involved in this study, which uses multimodal time-series data to predict the postoperative recurrence risk of rectal cancer patients:

[0030] like Figure 2 As shown, the system for predicting postoperative recurrence risk of rectal cancer patients based on multimodal time-series data involved in this invention mainly includes three modules: data acquisition and preprocessing module 101, feature extraction module 102, and multimodal feature fusion and modeling module 103.

[0031] The data acquisition and preprocessing module 101 is responsible for retrospectively collecting cross-modal time-series datasets of stage III rectal cancer patients from the hospital information system, image archiving system, and pathology system. Data must meet the following inclusion criteria:

[0032] 1. Neoadjuvant therapy has been performed, and postoperative histopathology confirms stage III rectal cancer;

[0033] 2. Biopsy and pathological examination should be performed before neoadjuvant therapy;

[0034] 3. The last MR multiparameter images before neoadjuvant therapy and the last MR before radical surgery;

[0035] 4. The patient's clinical data is complete;

[0036] 5. Postoperative follow-up should be conducted for at least 3 years, and the follow-up plan and content should be complete.

[0037] Exclusion criteria:

[0038] 1. Spatiotemporal involvement accompanied by other primary tumors;

[0039] 2. Patients without pre-treatment MR examination or pre-operative MR evaluation images;

[0040] 3. Poor MR image quality;

[0041] 4. Patients with poor H&E staining quality in biopsy images, making cell identification impossible.

[0042] 5. Patients who do not have clinical, imaging, and pathological data simultaneously.

[0043] The dataset includes:

[0044] 1. Time-series clinical data: including the last clinical data before neoadjuvant therapy and the most recent clinical data before surgery. Follow-up strategy for stage III rectal cancer recommended by the CSCO (Chinese Society of Clinical Oncology) guidelines, including physical examinations every 3 months for a total of 3 years, serological markers (CEA, CA19-9, CA-125); treatment regimens such as medications; and major clinical events such as emergency hospitalization and bowel obstruction.

[0045] 2. Temporal magnetic resonance imaging data: including T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI), and dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) before neoadjuvant therapy, before surgery, and every 6 months after surgery.

[0046] 3. Pathological digital images: WSI images of biopsy tissue before neoadjuvant therapy, including routine pathological indicators such as TNM staging, tumor histological type, and tumor deposition.

[0047] The specific data arrangement plan is as follows:

[0048] Inclusion criteria:

[0049] 1. The patient has undergone neoadjuvant therapy and postoperative histopathology confirmed stage III rectal cancer;

[0050] 2. Perform biopsy and pathological examination before neoadjuvant therapy;

[0051] 3. The last MR multiparameter images before neoadjuvant therapy and the last MR before radical surgery;

[0052] 4. The patient's clinical data is complete;

[0053] 5. Postoperative follow-up should be conducted for at least 3 years, and the follow-up plan and content should be complete.

[0054] Exclusion criteria:

[0055] 1. Spatiotemporal involvement accompanied by other primary tumors;

[0056] 2. Patients without pre-treatment MR examination and pre-operative MR evaluation images;

[0057] 3. Poor MR image quality;

[0058] 4. Patients with poor H&E staining quality in biopsy images, making cell identification impossible.

[0059] 5. Patients who do not have clinical, imaging, and pathological data simultaneously.

[0060] Follow-up plan (Phase III):

[0061] The CSCO guidelines recommend that patients with stage III rectal cancer undergo in-hospital follow-up every three months, including physical examination and serological testing; and pelvic contrast-enhanced MRI every six months.

[0062] Feature extraction module 102 includes: (1) Filtering the time-series clinical data to obtain clinical features. Specifically:

[0063] Data Preprocessing: To fully utilize clinical information for patient prognostic assessment, the MinMaxScaler function was used to normalize all clinical features and scale them to a range of 0-1. Secondly, missing values ​​are common in clinical data, so interpolation methods were used to impute them. Finally, outliers may exist in clinical data, which could negatively impact the analysis results. Therefore, outlier checks were first performed, followed by correction using the median. Clinical Information Screening: To reduce overfitting in the prediction model, this embodiment employs a series of statistical and machine learning methods to screen the clinical data, specifically including:

[0064] 1. Spearman's correlation analysis: Spearman's rank correlation coefficient is used to measure the non-linear correlation between two variables. It is based on the rank of the variable rather than the magnitude of its actual value.

[0065]

[0066] Where ρ represents the Spearman correlation coefficient, d represents the difference in rank between the two variables, and n represents the number of samples.

[0067] 2. Variance Thresholding: Variance thresholding is a feature selection method used to remove features with low variance. By calculating the variance of a feature, its degree of variation across the entire sample can be determined. If the variance of a feature is below a preset threshold, it can be considered a low-variance feature, which may lack a significant impact on the target variable.

[0068]

[0069] Where n represents the number of samples, and xi represents the value of the i-th sample. This represents the mean of the sample.

[0070] 3. Univariate analysis based on logistic regression: Univariate analysis based on logistic regression is used to evaluate the impact of a single factor on binary classification problems. It measures the contribution of a factor to the classification result by fitting a logistic regression model and calculating the coefficients (or weights) of the factors.

[0071]

[0072] Where logit(p) represents the log odds, p represents the probability of the classification result, β0 represents the intercept, and β1 to β... The coefficients representing factors, from x1 to x This indicates the value of the factor. In logistic regression, the sign and magnitude of the coefficient β represent the direction and degree of influence of the factor on the classification result.

[0073] (2) Image data analysis: Tumor region segmentation is performed on the time-series magnetic resonance imaging data, and radiomics handcrafted features and deep learning features based on visual transformers are extracted respectively; specifically including:

[0074] Image standardization: Due to differences in MR equipment parameters between hospitals and centers, acquired images may exhibit variations in resolution, sampling rate, and voxel count, as well as spatial misalignment or distortion, all of which can impact subsequent medical image analysis and processing. Therefore, image standardization is often necessary during medical image preprocessing. First, B-spline interpolation is used to address differences in sampling rate and pixel size, smoothing the image data while adjusting resolution. Then, 1x1x1 resampling unifies the MRI image sampling rate and pixel spacing to 1mm between pixels, homogenizing the image in three directions. Finally, the resampled images are processed... Z -Score standardization ensures consistency between data; the conversion formula is as follows:

[0075]

[0076] in The mean of the original data. The standard deviation of the original data. These are individual observations.

[0077] Tumor segmentation was performed using a pre-trained U-Net network to automatically segment tumor regions in MRI images. Currently, many rectal cancer studies using 2D slices have demonstrated effectiveness in prognostic assessment of rectal cancer. To reduce the time and resources wasted on manual annotation, a pre-trained U-Net network was used for 2D pre-segmentation of the largest tumor area, followed by secondary correction by two radiologists (junior and intermediate). Due to significant differences in individual images, the image was segmented into 512×512 patches as input, with a batch size of 42 and a maximum layer of 30 feature maps. The U-Net network architecture is as follows. Figure 3 As shown.

[0078] In one downsampling operation, the following calculations were performed:

[0079]

[0080] In this operation, conv represents convolution, batch_norm represents batch normalization, relu represents the ReLU activation function, and max_pool represents pooling. This represents the input for this downsampling.

[0081] One upsampling operation performed the following calculations:

[0082]

[0083] Here, up_conv performs deconvolution, batch_norm performs batch normalization, relu represents the ReLU activation function, cat represents channel concatenation, and max_pool represents pooling. This represents the input for this upsampling. This represents the intermediate result of the corresponding downsampling convolution.

[0084] Image feature extraction is used to extract shape, histogram, and texture features from the segmented tumor region using the Pyradiomics package as the radiomics handcrafted features, and to extract high-dimensional features using a visual transformer as the deep learning features. Specifically, this includes...

[0085] Manual Feature Extraction in Radiomics: After tumor segmentation, three sets of handcrafted features—shape, histogram, and texture—were extracted from the three parametric MR images using the Pyradiomics 3.0.1 Python package. Subsequently, Spearman correlation analysis was employed to remove features with autocorrelation coefficients higher than 0.9, eliminating highly correlated features and reducing the impact of multicollinearity. We propose using the maximum correlation minimum redundancy (mRMR) algorithm and the Lasso algorithm for feature selection and dimensionality reduction to retain the features most influential on the prediction of the target variable, thereby achieving feature dimensionality reduction and model optimization.

[0086] Spearman correlation analysis can be expressed by the following formula:

[0087]

[0088] Where rho represents the Spearman correlation coefficient, d represents the rank difference between the two variables, and n represents the sample size of the variable.

[0089] The core formula of the mRMR algorithm is as follows:

[0090]

[0091] Where mRMR represents the maximum relevance minimum redundancy value of a feature, and G(i, S) represents the information gain between feature i and dataset S. Representation of features Redundancy is measured by the information gain, calculated as the difference between the entropy of the dataset and the conditional entropy of a given feature. A higher information gain indicates a greater contribution of the feature to the classification or prediction of the dataset. Redundancy is also measured by the average correlation coefficient between features. Features with high redundancy can be predicted using other features and can be removed to reduce redundant information.

[0092] Deep Learning Feature Extraction in Imaging: Segmented 2D regions of interest (ROIs) of the tumor are directly input into the ViT network. ViT is a deep learning model based on the Transformer architecture, widely used in computer vision tasks such as image classification, object detection, and image generation. Compared to traditional CNN networks, ViT offers a novel approach, representing images as sequential data and utilizing the Transformer's self-attention mechanism to learn global relationships within the images. ViT segments the input image, treating each segment as an element in the sequence, and uses the Transformer architecture to learn the relationships between these segments. The penultimate layer is used as a deep learning feature for subsequent analysis. The Transformer network structure diagram is shown below. Figure 4 As shown.

[0093] The coding layer represents the operation of encoding each small block of the input image. The specific algorithm is as follows:

[0094] (14)

[0095] Then, L Transformer encoder modules are input for computation. The first code representing the entire image is then input into the multilayer perceptron classification layer and the Softmax activation function layer to finally obtain the depth risk score y.

[0096]

[0097] (3) Pathological data analysis: tumor cell nuclei were segmented from the pathological data, and the morphological features of tumor cell nuclei and the spatial distribution features of tumor cells based on graph convolutional networks and transformers were extracted.

[0098] A wealth of information features for analysis and diagnosis can be extracted from pathological images. To accurately extract the nuclear features and spatial distribution features of tumor cells, this embodiment designs a tumor cell segmentation network. First, different tissue cells in the pathological image are segmented to locate tumor cells. Then, a tumor cell spatial feature extractor based on GCN and Transformer is constructed. Compared with deep learning pathological analysis models that only use Multi-Instance Learning (MIL), it can provide more sufficient and clinically interpretable predictive evidence.

[0099] Before analyzing the WSI of rectal cancer, it is necessary to segment and identify the tumor cells to locate the nuclei. Due to the high resolution and large size of pathological images, the workload of manual annotation by doctors is too large. Therefore, a cell nucleus segmentation process with strong generalization is constructed using a small number of labeled samples. Figure 5 ).

[0100] Pathological data analysis includes:

[0101] The pathology image enhancement unit is used to enhance digital pathology images using generative adversarial networks.

[0102] A deep learning network with good segmentation performance requires a large amount of training data. However, when the number of samples is small, data augmentation methods are needed to expand the training dataset. In this embodiment, a method of training a Generative Adversarial Network (GAN) to generate images similar to the samples is adopted to achieve data augmentation.

[0103] The cell nucleus segmentation unit is used to locate and segment tumor cell nuclei in enhanced pathological images using a network based on a masked autoencoder pre-trained architecture. To further explore the depth information between different tissue cells in the pathological images, a MAE (Masked Auto Encoders) architecture is adopted for cell nucleus segmentation. The MAE architecture is a self-supervised learning method in computer vision. The augmented dataset D from Research Content 1 is used as the MAE training input dataset to train an MAE encoder based on the pathological image's own information. The encoder with pre-trained parameters is used as the encoder for tumor cell nucleus segmentation. The decoder part of the segmentation network is designed based on Transformer and CNN. The loss function used by MAE is Mean Squared Error (MSE), as shown below:

[0104]

[0105] in Reconstruct the image for MAE. For the original image, the loss function used by the tumor cell nucleus segmentation network consists of the Dice loss function and the binary cross-entropy loss (CE), as shown below:

[0106]

[0107] in To predict the segmentation mask, This is the true mask for tumor cell nuclei.

[0108] The pathological feature extraction unit is used to calculate the nuclear area, entropy value and number ratio of the segmented tumor cell nuclei as the morphological features of the tumor cell nuclei, and to construct a cell feature map based on the location of the tumor cell nuclei. The map is then input into a network composed of a multi-layer graph convolutional network and a transformer module to extract the spatial distribution features of the tumor cells.

[0109] The aforementioned manual features of pathological cell nuclei do not include the spatial distribution features between cells. Therefore, this embodiment is based on the Graph Transformer pathological image spatial feature extraction network (…). Figure 6 Spatial features of tumor cells were extracted.

[0110] GCN networks are commonly used for image spatial feature analysis, but they are more sensitive to the spatial feature distribution of small images, while they tend to oversmooth large images with too many nodes, affecting prediction performance. Transformer, as a deep learning model based on attention mechanisms, can capture global information of input image sequences, but its ability to learn local information features is weak. By combining GCN and Transformer, we can fully learn the spatial features between tumor cells in pathological images and the local information correlation features within each tumor cell. In WSI, cell features are extracted from all cells based on whether the nuclear mask exists in the same connected region and the image patch size, forming a cell feature map. Then The input is a four-layer GraphTransformer module. The output is subjected to mean pooling, then passes through a multilayer perceptron (MLP) layer and another GraphTransformer module. The output feature map is then subjected to dimensionality reduction using PCA principal component analysis to obtain the spatial features of tumor cells. Simultaneously, it passes through a linear layer to output a binary classification prediction value for high and low recurrence risk (WSI). The loss function used is the binary cross-entropy loss commonly used in binary classification tasks, as shown in the following formula:

[0111]

[0112] in The predicted value is 1 for high recurrence risk and 0 for low recurrence risk. This is the actual label value.

[0113] Through the aforementioned intelligent quantification process of digital pathological images, the pathological features, comprised of tumor cell nuclear features and tumor cell spatial features, are obtained and used for subsequent model construction. An additional predicted value is also obtained, used in subsequent interpretative analysis to describe the importance of pathological features in predicting the risk of recurrence.

[0114] The multimodal feature fusion and modeling module 103 receives all filtered feature vectors from the feature extraction module 102. These features are then concatenated and input into a fusion network based on a Transformer encoder architecture. This network learns the intrinsic relationships between features of different modalities and between features at different time points through its self-attention mechanism. Finally, through a fully connected layer and a Softmax activation function, it outputs a binary probability indicating whether the patient belongs to "high recurrence risk" or "low recurrence risk".

[0115] The multimodal feature fusion and modeling module includes:

[0116] Multi-time-series dataset establishment: This embodiment aims to establish a multi-time-series dataset, encompassing multimodal time-series data generated before neoadjuvant therapy, before radical surgery, and during follow-up. This primarily includes multi-parameter MRI imaging data, including T2WI, DWI, and DCE-MRI, as well as clinical demographic information such as age distribution, serological markers CEA, CA19-9, and CA-125 levels, treatment regimens, and medication use. Furthermore, it includes significant clinical events, such as bowel obstruction and emergency admissions, for subsequent analysis.

[0117] Longitudinal model construction: The above-mentioned multimodal data analysis and quantification methods are used to initially screen time-series clinical features and extract features from time-series image data. Then, the radiomics manual and deep learning features are input into the Transformer network to extract time-series image features. The time-series clinical features, time-series image features, and biopsy pathology features are combined and input into the Transformer network for fusion to construct the final prediction model.

[0118] Based on the above multimodal time-series data system for predicting postoperative recurrence risk in rectal cancer patients, this invention provides an interpretability analysis and decision-sharing validation method for rectal cancer prognostic models. This method mainly includes two stages: an interpretability analysis step and a decision-sharing validation step (the overall method flow is as follows...). Figure 1 ).

[0119] The interpretability analysis step includes two parallel analyses:

[0120] a) Image visualization steps: This involves the Vision Transformer subnetwork processing MR images in the prognostic prediction model. Specifically, after forward propagation to obtain the predicted score for the target category (e.g., "high risk"), backpropagation calculates the gradient of this score relative to the output feature map of the last attention block in the ViT. Global average pooling is then applied to these gradients to obtain the importance weight for each feature channel. These weights are then used to weighted summation of the feature maps, and negative contributions are filtered out using the ReLU function. Finally, bilinear interpolation is used to enlarge the generated heatmap to the same size as the original T2WI, DWI, or DCE-MRI images and overlay them for display. The red area represents the location the model focuses on when making decisions. The specific steps are as follows: First, starting from the last convolutional layer of the ViT model, the gradient relative to the prediction result is calculated. Assuming the feature map of the last convolutional layer is A, the prediction result is y, and the loss function is L, the gradient can be obtained. Next, the gradient is spatially averaged to obtain the importance weight for each feature map.

[0121]

[0122] Where A represents the size of the output of the last convolutional layer of the network, Z represents the number of pixels in the feature map, and Aij represents the pixel value at position (i, j) of the k-th feature map. The features of the selected convolutional layer are weighted using the neuron importance weights obtained above, as shown in the following formula:

[0123]

[0124] Here, the ReLU function is the ReLU activation function. ∑ represents a weighted summation of all feature maps. Finally, the heatmap undergoes non-linear processing and interpolation to obtain a Grad-CAM map that matches the size of the input image, as shown below. Figure 7 As shown:

[0125] b) Feature contribution analysis steps: using the SHAP interpreter. A diagram is shown below. Figure 8 As shown. For the entire trained prognostic prediction model (considered a black-box function), the SHAP value is calculated for each input sample (i.e., all feature vectors of a patient). The calculation formula is as follows:

[0126]

[0127] Assume the first The sample is , No. The first sample The features are The model's predicted value for this sample is The baseline of the entire model (usually the mean of the target variable for all samples) is... .

[0128] This generates two views: 1) a "SHAP force map" for a single patient, showing whether each feature (such as "CEA value," "T2WI texture entropy," and "mean nuclear area") pushes the predicted value from baseline towards high or low risk, and the magnitude of that push; 2) a "summary map" for the entire dataset, sorted by the mean absolute SHAP value of all samples, showing which features have the greatest impact on the model output. Specifically, for the two key handcrafted features extracted from pathological images—"nuclear size" and "nuclear morphology non-convexity score"—an additional linear support vector machine model is trained, with its coefficients serving as a supplementary explanation of the feature weights. The nuclear size and shape of a single tumor cell are calculated using the following formula:

[0129]

[0130] The decision-sharing validation step aims to use the above interpretation results for clinical validation:

[0131] c) Construct an integrated, visual, and interactive interface. The left side of the interface displays the patient's multi-timepoint MR images (with switchable sequences) overlaid with Grad-CAM heatmaps. The center of the interface displays time-series clinical indicator curves. The left side of the bottom interface displays the SHAP analysis results, including the patient's feature contribution SHAP plot and a summary of overall feature importance. The right side of the interface displays the model's final risk score and classification results. Figure 9 As shown.

[0132] d) Invite experienced clinicians (such as colorectal surgeons or oncologists) to use the interface in an independent environment. The physician can view all raw data, time-series variations, and interpretive information provided by the model, but will not know the model's final classification result. The physician needs to independently assess the patient's recurrence risk based on all information (e.g., assigning a risk rating of 1-5).

[0133] e) Collect evaluation results from multiple physicians on a batch of test patients. Compare the physicians' evaluations (as a reference standard or comparison object) with the model's predictions. Evaluation metrics include: consistency between model predictions and physician judgments (e.g., Kappa coefficient), performance improvement based on physician judgments (e.g., increased AUC after incorporating physician ratings as a feature into the model), and physician satisfaction survey scores regarding the model's interpretation of the results. This comprehensively validates the model's effectiveness in assisting clinical decision-making.

[0134] In a preferred embodiment, the method further includes a multi-timepoint comparative evaluation sub-step. At multiple fixed points in the patient's postoperative follow-up (e.g., immediately postoperative t0, 6 months postoperative t1, ..., 36 months postoperative t6), the model is rerun using all time-series data up to that point (or using a specially trained sequence model) to obtain the risk prediction for that time point. The predictive performance (AUC, C-index) of the model at each time point from t0 to t6 is analyzed and compared. The expected result is that with the addition of more follow-up time-series data, the model performance (AUC) should steadily improve, which intuitively demonstrates the key value of time-series information for dynamic risk prediction and provides data support for determining the optimal timing for clinical assessment.

[0135] The present invention also provides a system for implementing the above method, including an interpretability analysis engine, a visual interactive interface generation module, a decision comparison and evaluation module, and a time-series performance evaluation module.

[0136] The present invention also provides an electronic device and a computer-readable storage medium for performing the above-described method.

[0137] This invention effectively opens the "black box" of complex AI prognostic models, establishes a bridge of trust between doctors and models, and verifies the clinical applicability of models through a scientific decision-making sharing process, greatly promoting the application of artificial intelligence in the field of precision medicine.

Claims

1. A method for interpretability analysis and decision-sharing validation of a prognostic model for rectal cancer, characterized in that, This is applied to a prognostic prediction model, which predicts the risk of postoperative recurrence in rectal cancer patients based on multimodal time-series data. The model includes three modules: a data acquisition and preprocessing module, a feature extraction module, and a multimodal feature fusion and modeling module. The data acquisition and preprocessing module is responsible for retrospectively collecting cross-modal time-series datasets of stage III rectal cancer patients from the hospital information system, image archiving system, and pathology system. The dataset includes time-series clinical data, time-series magnetic resonance imaging data, and digital pathology images; The feature extraction module includes: (1) filtering the time-series clinical data to obtain clinical features; (2) Image data analysis: The time-series magnetic resonance imaging data is segmented into tumor regions, and radiomics handcrafted features and deep learning features based on visual transformers are extracted respectively. This includes extracting shape, histogram and texture features from the segmented tumor regions using the Pyradiomics package as the radiomics handcrafted features, and extracting high-dimensional features as the deep learning features using the visual transformer; (3) Pathological data analysis: The digital pathological images are segmented into tumor cell nuclei, and the morphological features of tumor cell nuclei and the spatial distribution features of tumor cells based on graph convolutional networks and transformers are extracted; Specifically, it includes: a pathological image enhancement unit, used to enhance the digital pathological images using generative adversarial networks; a cell nucleus segmentation unit, used to locate and segment tumor cell nuclei in the enhanced pathological images using a network based on a mask autoencoder pre-trained architecture; and a pathological feature extraction unit, used to calculate the nuclear area, entropy value and number ratio from the segmented tumor cell nuclei as the morphological features of the tumor cell nuclei, and construct a cell feature map based on the location of the tumor cell nuclei, which is then input into a network composed of multi-layer graph convolutional networks and transformer modules to extract the spatial distribution features of the tumor cells; The method includes: an interpretability analysis step: a) an image visualization step: for the visual transformer network used to process magnetic resonance images in the prognostic prediction model, gradient-weighted class activation mapping technology is used to generate a regional heatmap related to the model's prediction decision, and the heatmap is overlaid on the original image; b) a feature contribution analysis step: using a SHAP interpreter, the contribution values ​​of various features input to the prognostic prediction model to the prediction results of a single patient and to the overall model prediction results are calculated; a decision sharing verification step: c) constructing a visual interactive interface, which simultaneously displays the patient's multi-temporal clinical and imaging information, the heatmap, and the feature contribution analysis results; d) clinicians independently assess and score the patient's recurrence risk based on all the information provided by the visual interactive interface; e) comparing the doctor's assessment score with the automatic prediction results of the prognostic prediction model, and evaluating the effectiveness of the model in assisting decision-making through statistical indicators; In the feature contribution analysis step, for the tumor cell nuclear morphology features extracted from digital pathology images, a support vector machine linear model is additionally used to analyze the weights of two features, nuclear size and nuclear shape non-convexity score, in order to explain the contribution of nuclear atypia to the prediction. The decision-sharing verification step also includes a multi-time-point comparison evaluation sub-step: at different time points during the patient's postoperative follow-up, a prediction model constructed using time-series data up to that point is used, and the changes in model performance at different time points are compared to evaluate the value of time-series data for long-term risk prediction.

2. The method according to claim 1, characterized in that, The image visualization step specifically includes: extracting the output feature map and gradient of the last attention block of the visual transformer; calculating the weight of each feature channel; performing ReLU activation and upsampling on the weighted feature map to generate a heatmap that matches the size of the input image.

3. The method according to claim 2, characterized in that, The specific steps of the multi-time-point comparative evaluation are as follows: construct a model at time t0, where t0 is before the start of follow-up, and models at times t1 to t6, where t1 to t6 are the follow-up nodes from the 6th month to the 36th month after surgery, and compare the area under the curve, C-index and Kaplan-Meier survival curve of each model.

4. A system for interpretability analysis and decision-sharing validation of a rectal cancer prognostic model, implementing the method as described in any one of claims 1-3, characterized in that, include: An interpretable analytics engine is used to: perform gradient-weighted class activation mapping analysis on the image analysis network in the prognostic prediction model to generate heatmaps; The system utilizes the SHAP interpreter to calculate the contribution value of the model input features; a visualization interface generation module integrates patient data, model prediction results, the heatmap, and the contribution values ​​to generate an interactive interface for physician review; a decision comparison and evaluation module receives independent evaluation results input by physicians through the interactive interface, compares them with the model prediction results, and calculates evaluation indicators; it also includes a time-series performance evaluation module, which selects multiple nodes on the patient follow-up timeline, evaluates the predictive performance of the model at the corresponding time points, and analyzes the trend of performance changes with time nodes.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

Images