A rectal cancer postoperative recurrence risk prediction system and method based on multi-modal time series data

Abstract

The application discloses a rectal cancer postoperative recurrence risk prediction system and method based on multi-modal time series data, and relates to the technical field of medical artificial intelligence. The system comprises a data acquisition and preprocessing module, a feature extraction module and a multi-modal feature fusion and modeling module. The method comprises the following steps: constructing a cross-modal data set comprising time series clinical data, time series MR images and biopsy digital pathology images; extracting clinical features, imageomics and deep learning features of MR images, and cell nucleus morphology and spatial distribution features of pathology images respectively; and fusing all the features by using a Transform network to construct a prediction model. The application integrates macroscopic images, microscopic pathology and dynamic time series information, comprehensively quantifies tumor heterogeneity, solves the problem of inaccurate prediction of a single modal static model, and can more accurately evaluate the postoperative recurrence risk of a stage III rectal cancer patient, thereby assisting clinical treatment decision-making.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, specifically to a system and method for predicting the risk of postoperative recurrence of rectal cancer based on multimodal time-series data. Background Technology

[0002] Colorectal cancer is a prevalent malignant tumor worldwide, with a high proportion of patients in stage III. The standard treatment is neoadjuvant therapy followed by radical surgery. Whether adjuvant chemotherapy is administered post-surgery depends heavily on the accurate assessment of the patient's recurrence risk. Currently, clinical practice mainly relies on static indicators such as AJCC pathological staging. However, the high heterogeneity of stage III tumors makes existing assessment methods insufficiently precise. Some high-risk patients may not receive timely treatment, while some low-risk patients suffer unnecessary chemotherapy toxicity.

[0003] In recent years, radiomics and deep learning technologies have provided new insights into quantifying tumor heterogeneity. However, existing studies are mostly based on single-time-point and single-modality data (such as MRI or pathology alone) to build predictive models, which are difficult to fully capture the heterogeneity of tumors in both macroscopic (imaging) and microscopic (pathological) dimensions, and even more so to reflect the dynamic evolution of the disease during treatment and follow-up. As a result, their predictive efficacy is limited and clinical application is difficult.

[0004] Therefore, there is an urgent need for a method that can integrate multi-time point and multi-dimensional information of patients before and after treatment to more comprehensively and dynamically characterize tumors, thereby achieving more accurate prediction of postoperative recurrence risk. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a system and method for predicting the risk of recurrence after rectal cancer surgery based on multimodal time-series data, so as to solve the problems of incomplete tumor heterogeneity characterization and inaccurate prediction of long-term recurrence risk caused by the use of single-modal and single-time-point data in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a system for predicting the risk of postoperative recurrence of rectal cancer based on multimodal time-series data, comprising: a data acquisition and preprocessing module, used to acquire a cross-modal time-series dataset of stage III rectal cancer patients, the dataset including: time-series clinical data, time-series magnetic resonance imaging data, and digital pathological images; a feature extraction module, connected to the data acquisition and preprocessing module, used to: a) filter the time-series clinical data to obtain clinical features; b) segment the tumor regions of the time-series magnetic resonance imaging data and extract radiomics handcrafted features and deep learning features based on visual transformers, respectively; c) segment the tumor cell nuclei of the digital pathological images and extract tumor cell nucleus morphological features and tumor cell spatial distribution features based on graph convolutional networks and transformers; a multimodal feature fusion and modeling module, connected to the feature extraction module, used to fuse the clinical features, the radiomics handcrafted features, the deep learning features, the tumor cell nucleus morphological features, and the tumor cell spatial distribution features, and input them into a prediction model based on a transformer architecture to output the prediction result of the patient's postoperative recurrence risk.

[0007] Furthermore, the time-series magnetic resonance imaging data includes at least T2-weighted imaging, diffusion-weighted imaging, and dynamic contrast-enhanced magnetic resonance imaging at multiple time points before neoadjuvant therapy, before radical surgery, and during postoperative follow-up.

[0008] Furthermore, the feature extraction module includes: a tumor segmentation unit, used to automatically segment tumor regions in magnetic resonance images using a pre-trained U-Net network; and an image feature extraction unit, used to extract shape, histogram, and texture features from the segmented tumor regions as the image omics handcrafted features using the Pyradiomics package, and to extract high-dimensional features as the deep learning features using a visual transformer.

[0009] Furthermore, the feature extraction module includes: a pathological image enhancement unit, used to perform data enhancement on digital pathological images using a generative adversarial network; a cell nucleus segmentation unit, used to locate and segment tumor cell nuclei in the enhanced pathological image using a network based on a mask autoencoder pre-trained architecture; and a pathological feature extraction unit, used to calculate the nuclear area, entropy value, and number ratio from the segmented tumor cell nuclei as the morphological features of the tumor cell nuclei, and to construct a cell feature map based on the location of the tumor cell nuclei, which is then input into a network composed of a multi-layer graph convolutional network and a transformer module to extract the spatial distribution features of the tumor cells.

[0010] Furthermore, the prediction model in the multimodal feature fusion and modeling module is a binary classification model based on a transformer encoder, which generates the final relapse risk score by associating and fusing features from different modalities and different time points.

[0011] This invention also provides a method for predicting the risk of postoperative recurrence of rectal cancer based on multimodal time-series data, comprising the following steps: S1: Constructing a cross-modal time-series dataset, the dataset including time-series clinical data, time-series magnetic resonance imaging data, and digital pathological images of stage III rectal cancer patients; S2: Preprocessing and feature filtering the time-series clinical data to obtain clinical features; S3: Segmenting the tumor regions in the time-series magnetic resonance imaging data, and extracting radiomics handcrafted features and deep learning features based on visual transformers, respectively; S4: Segmenting the tumor cell nuclei in the digital pathological images, and extracting morphological features of tumor cell nuclei and spatial distribution features of tumor cells based on graph convolutional networks and transformers; S5: Fusing all features extracted in steps S2 to S4, inputting them into a prediction model based on a transformer architecture for training and prediction, and outputting the patient's postoperative recurrence risk, etc.

[0012] The tumor region segmentation in step S3 specifically involves: using a pre-trained U-Net network for automatic initial segmentation, followed by manual correction by a radiologist.

[0013] Furthermore, the extraction of spatial distribution features of tumor cells in step S4 specifically includes: constructing a cell feature map based on the location of the segmented tumor cell nuclei, inputting the cell feature map sequentially into multiple graph convolution-transformer modules for spatial relationship modeling, and then obtaining the spatial distribution features of tumor cells after pooling and dimensionality reduction.

[0014] Furthermore, in step S5, the prediction model is trained using three-year disease-free survival as the clinical endpoint.

[0015] The present invention also provides an electronic device whose processor can execute the above-described method.

[0016] The beneficial effects of this invention are as follows:

[0017] Multimodal fusion: By jointly analyzing time-series MR images (macro), digital pathological images (micro), and clinical data, tumor heterogeneity is comprehensively quantified from multiple dimensions, overcoming the limitations of a single data source.

[0018] Temporal dynamic modeling: By utilizing multiple data points from before neoadjuvant therapy, before surgery, and during postoperative follow-up, a time-series dataset is constructed, enabling the model to capture the dynamic process of disease development and treatment response, thereby more accurately assessing the risk of long-term relapse.

[0019] Advanced Feature Extraction: Employing advanced deep learning techniques such as U-Net, visual transformers, masked autoencoders, and graph convolutional networks, we perform automated and intelligent deep feature mining tailored to the characteristics of both image and pathological data, resulting in more discriminative information.

[0020] Powerful fusion architecture: Multimodal feature fusion is performed based on a transformer network, and its self-attention mechanism can effectively model the complex relationships between different features, thereby improving the overall performance of the model. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the system modules in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the U-Net automatic segmentation model network structure according to an embodiment of the present invention.

[0024] Figure 4 This is the Transformer network structure according to an embodiment of the present invention.

[0025] Figure 5 This is a flowchart of tumor cell nucleus segmentation based on self-supervised pre-training, according to an embodiment of the present invention.

[0026] Figure 6 This is a flowchart of the pathological image spatial feature extraction network based on Graph Transformer according to an embodiment of the present invention.

[0027] Figure 7 This is a visualization diagram of Grad-CAM according to an embodiment of the present invention.

[0028] Figure 8 This is a schematic diagram of the SHAP interpretable mechanism in an embodiment of the present invention.

[0029] Figure 9 This is a schematic diagram of the doctor-model shared decision visualization interface according to an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in further detail below with reference to the accompanying drawings.

[0031] like Figure 2 As shown, the system of this embodiment mainly includes three modules: data acquisition and preprocessing module 101, feature extraction module 102, and multimodal feature fusion and modeling module 103.

[0032] The data acquisition and preprocessing module 101 is responsible for retrospectively collecting cross-modal time-series datasets of stage III rectal cancer patients from the hospital information system, image archiving system, and pathology system. Data must meet the following inclusion criteria:

[0033] 1. The patient has undergone neoadjuvant therapy and postoperative histopathology confirmed stage III rectal cancer;

[0034] 2. Biopsy and pathological examination should be performed before neoadjuvant therapy;

[0035] 3. The last MR multiparameter images before neoadjuvant therapy and the last MR before radical surgery;

[0036] 4. The patient's clinical data is complete;

[0037] 5. Postoperative follow-up should be conducted for at least 3 years, and the follow-up plan and content should be complete.

[0038] Exclusion criteria:

[0039] 1. Spatiotemporal involvement accompanied by other primary tumors;

[0040] 2. Patients without pre-treatment MR examination or pre-operative MR evaluation images;

[0041] 3. Poor MR image quality;

[0042] 4. Patients with poor H&E staining image quality from biopsies, making cell identification impossible;

[0043] 5. Patients who do not have clinical, imaging, and pathological data simultaneously.

[0044] The dataset includes:

[0045] 1. Time-series clinical data: including the last clinical data before neoadjuvant therapy and the most recent clinical data before surgery. Follow-up strategy for stage III rectal cancer recommended by CSCO (Chinese Society of Clinical Oncology) guidelines, including physical examinations every 3 months for a total of 3 years, serological markers (CEA, CA19-9, CA-125); treatment regimens such as medications; major clinical events such as emergency hospitalization, bowel obstruction, etc.

[0046] 2. Temporal magnetic resonance imaging data: including T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI), and dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) before neoadjuvant therapy, before surgery, and every 6 months after surgery.

[0047] 3. Pathological digital images: WSI images of biopsy tissue before neoadjuvant therapy, including routine pathological indicators such as TNM staging, tumor histological type, and tumor deposition.

[0048] The specific data arrangement plan is as follows:

[0049] Inclusion criteria:

[0050] 1. The patient has undergone neoadjuvant therapy and postoperative histopathology confirmed stage III rectal cancer;

[0051] 2. Biopsy and pathological examination should be performed before neoadjuvant therapy;

[0052] 3. The last MR multiparameter images before neoadjuvant therapy and the last MR before radical surgery;

[0053] 4. The patient's clinical data is complete;

[0054] 5. Postoperative follow-up should be conducted for at least 3 years, and the follow-up plan and content should be complete.

[0055] Exclusion criteria:

[0056] 1. Spatiotemporal involvement accompanied by other primary tumors;

[0057] 2. Patients without pre-treatment MR examination or pre-operative MR evaluation images;

[0058] 3. Poor MR image quality;

[0059] 4. Patients with poor H&E staining image quality in biopsies, making cell identification impossible.

[0060] 5. Patients who do not have clinical, imaging, and pathological data simultaneously.

[0061] Follow-up plan (Phase III):

[0062] The CSCO guidelines recommend that patients with stage III rectal cancer undergo in-hospital follow-up every three months, including physical examination and serological testing; and pelvic contrast-enhanced MRI every six months.

[0063] Feature extraction module 102 includes: (1) filtering the time-series clinical data to obtain clinical features. Specifically:

[0064] Data Preprocessing: To fully utilize clinical information for assessing patient prognosis, we used the MinMaxScaler function to normalize all clinical features and scale them to a range of 0-1. Secondly, clinical data often contains missing values, so missing value handling was performed using interpolation methods to impute them. Finally, clinical data may contain outliers, which can negatively impact the analysis results. Therefore, we first checked the data for outliers and then corrected them using the median. Clinical Information Screening: To reduce overfitting in the prediction model, this embodiment employs a series of statistical and machine learning methods to screen the clinical data, specifically including:

[0065] 1. Spearman's correlation analysis: Spearman's rank correlation coefficient is used to measure the non-linear correlation between two variables. It is based on the rank of the variable rather than the magnitude of its actual value.

[0066]

[0067] Where ρ represents the Spearman correlation coefficient, d represents the difference in rank between the two variables, and n represents the number of samples.

[0068] 2. Variance Thresholding: Variance thresholding is a feature selection method used to remove features with low variance. By calculating the variance of a feature, its degree of variation across the entire sample can be determined. If the variance of a feature is below a preset threshold, it can be considered a low-variance feature, which may lack a significant impact on the target variable.

[0069]

[0070] Where n represents the number of samples, and xi represents the value of the i-th sample. This represents the mean of the sample.

[0071] 3. Univariate analysis based on logistic regression: Univariate analysis based on logistic regression is used to evaluate the impact of a single factor on binary classification problems. It measures the contribution of a factor to the classification result by fitting a logistic regression model and calculating the coefficients (or weights) of the factors.

[0072]

[0073] Where logit(p) represents the log odds, p represents the probability of the classification result, β0 represents the intercept, and β1 to β... The coefficients representing factors, from x1 to x This indicates the value of the factor. In logistic regression, the sign and magnitude of the coefficient β represent the direction and degree of influence of the factor on the classification result.

[0074] (2) Image data analysis: Tumor region segmentation is performed on the time-series magnetic resonance imaging data, and radiomics handcrafted features and deep learning features based on visual transformers are extracted respectively; specifically including:

[0075] Image standardization: Due to differences in MR equipment parameters between hospitals and centers, acquired images may exhibit variations in resolution, sampling rate, and voxel count, as well as spatial misalignment or distortion, all of which can impact subsequent medical image analysis and processing. Therefore, image standardization is often necessary during medical image preprocessing. First, B-spline interpolation is used to address differences in sampling rate and pixel size, smoothing the image data while adjusting resolution. Then, 1x1x1 resampling unifies the MRI image sampling rate and pixel spacing to 1mm between pixels, homogenizing the image in three directions. Finally, the resampled images are processed...Z -Score standardization ensures consistency between data; the conversion formula is as follows:

[0076]

[0077] in The mean of the original data. The standard deviation of the original data. These are individual observations.

[0078] Tumor segmentation, using a pre-trained U-Net network to automatically segment tumor regions in MRI images: Currently, many rectal cancer studies using 2D layers have demonstrated effectiveness in prognostic assessment of rectal cancer. To reduce the time and resources wasted on manual annotation, this embodiment uses a pre-trained U-Net network to pre-segment the largest tumor area in 2D, followed by secondary correction by two radiologists (junior and intermediate). Due to significant differences in individual images, the image is segmented into 512×512 patches as input, with a batch size of 42 and a maximum layer of 30 feature maps. The U-Net network architecture is as follows. Figure 3 As shown.

[0079] In one downsampling operation, the following calculations were performed:

[0080]

[0081] In this operation, conv represents convolution, batch_norm represents batch normalization, relu represents the ReLU activation function, and max_pool represents pooling. This represents the input for this downsampling.

[0082] One upsampling operation performed the following calculations:

[0083]

[0084] Here, up_conv performs deconvolution, batch_norm performs batch normalization, relu represents the ReLU activation function, cat represents channel concatenation, and max_pool represents pooling. This represents the input for this upsampling. This represents the intermediate result of the corresponding downsampling convolution.

[0085] Image feature extraction is used to extract shape, histogram, and texture features from the segmented tumor region using the Pyradiomics package as the radiomics handcrafted features, and to extract high-dimensional features using a visual transformer as the deep learning features. Specifically, this includes...

[0086] Manual Feature Extraction in Radiomics: After tumor segmentation, three sets of handcrafted features—shape, histogram, and texture—were extracted from the three parametric MR images using the Pyradiomics 3.0.1 Python package. Subsequently, Spearman correlation analysis was employed to remove features with autocorrelation coefficients higher than 0.9, eliminating highly correlated features and reducing the impact of multicollinearity. The maximum correlation minimum redundancy (mRMR) algorithm and the Lasso algorithm were used for feature selection and dimensionality reduction to retain the features most influential on the prediction of the target variable, achieving feature dimensionality reduction and model optimization.

[0087] Spearman correlation analysis can be expressed by the following formula:

[0088]

[0089] Where rho represents the Spearman correlation coefficient, d represents the rank difference between the two variables, and n represents the sample size of the variable.

[0090] The core formula of the mRMR algorithm is as follows:

[0091]

[0092] Where mRMR represents the maximum relevance and minimum redundancy of a feature, and G(i, S) represents the information gain between feature i and dataset S. Representation of features Redundancy is measured by the information gain, calculated as the difference between the entropy of the dataset and the conditional entropy of a given feature. A higher information gain indicates a greater contribution of the feature to the classification or prediction of the dataset. Redundancy is also measured by the average correlation coefficient between features. Features with high redundancy can be predicted using other features and can be removed to reduce redundant information.

[0093] Deep Learning Feature Extraction in Imaging: This embodiment proposes to directly input the segmented 2D region of interest (ROI) of the tumor into the ViT network. ViT is a deep learning model based on the Transformer architecture, widely used in computer vision tasks such as image classification, object detection, and image generation. Compared to traditional CNN networks, ViT provides a novel approach, representing images as sequential data and utilizing the self-attention mechanism of the Transformer to learn global relationships within the images. ViT segments the input image, treating each segmented patch as an element in the sequence, and uses the Transformer architecture to learn the relationships between these patches. The penultimate layer is used as a deep learning feature for subsequent analysis. The Transformer network structure diagram is shown below. Figure 4 As shown.

[0094] The coding layer represents the operation of encoding each small block of the input image. The specific algorithm is as follows:

[0095] (14)

[0096] Then, L Transformer encoder modules are input for computation. The first code representing the entire image is then input into the multilayer perceptron classification layer and the Softmax activation function layer to finally obtain the depth risk score y.

[0097]

[0098] (3) Pathological data analysis: tumor cell nuclei were segmented from the pathological data, and the morphological features of tumor cell nuclei and the spatial distribution features of tumor cells based on graph convolutional networks and transformers were extracted.

[0099] A wealth of information features for analysis and diagnosis can be extracted from pathological images. To accurately extract the nuclear features and spatial distribution features of tumor cells, this embodiment designs a tumor cell segmentation network. First, different tissue cells in the pathological image are segmented to locate tumor cells. Then, a tumor cell spatial feature extractor based on GCN and Transformer is constructed. Compared with deep learning pathological analysis models that only use Multi-Instance Learning (MIL), it can provide more sufficient and clinically interpretable predictive evidence.

[0100] Before analyzing the WSI of rectal cancer, it is necessary to segment and identify the tumor cells to locate the nuclei. Because pathological images have high resolution and large size, manual annotation by doctors is too laborious. This embodiment aims to construct a more generalizable cell nucleus segmentation process using a small number of labeled samples. Figure 5 ).

[0101] Pathological data analysis includes:

[0102] The pathology image enhancement unit is used to enhance digital pathology images using generative adversarial networks.

[0103] A deep learning network with good segmentation performance requires a large amount of training data. However, when the number of samples is small, data augmentation methods are needed to expand the training dataset. In this embodiment, a method of training a Generative Adversarial Network (GAN) to generate images similar to the samples is adopted to achieve data augmentation.

[0104] The cell nucleus segmentation unit is used to locate and segment tumor cell nuclei in enhanced pathological images using a network based on a masked autoencoder pre-trained architecture. To deeply mine the depth information between different tissue cells in the pathological image, this embodiment adopts a MAE (Masked Auto Encoders) architecture for cell nucleus segmentation. The MAE architecture is a self-supervised learning method in computer vision. An augmented dataset is used as the MAE training input dataset to train an encoder based on the pathological image's own information. The encoder with its pre-trained parameters is used as the encoder for tumor cell nucleus segmentation. A decoder part of the segmentation network is designed based on Transformer and CNN. The loss function used by MAE is Mean Squared Error (MSE), as shown below:

[0105]

[0106] in Reconstruct the image for MAE. For the original image, the loss function used by the tumor cell nucleus segmentation network consists of the Dice loss function and the binary cross-entropy loss (CE), as shown below:

[0107]

[0108] in To predict the segmentation mask, This is the true mask for tumor cell nuclei.

[0109] The pathological feature extraction unit is used to calculate the nuclear area, entropy value and number ratio of the segmented tumor cell nuclei as the morphological features of the tumor cell nuclei, and to construct a cell feature map based on the location of the tumor cell nuclei. The map is then input into a network composed of a multi-layer graph convolutional network and a transformer module to extract the spatial distribution features of the tumor cells.

[0110] The aforementioned manual features of pathological cell nuclei do not include the spatial distribution features between cells. Therefore, this embodiment is based on the Graph Transformer pathological image spatial feature extraction network (…). Figure 6 Spatial feature extraction of tumor cells was performed.

[0111] GCN networks are commonly used for image spatial feature analysis, but they are more sensitive to the spatial feature distribution of small images, while they tend to oversmooth large images with too many nodes, affecting prediction performance. Transformer, as a deep learning model based on attention mechanisms, can capture global information of input image sequences, but its ability to learn local information features is weak. By combining GCN and Transformer, we can fully learn the spatial features between tumor cells in pathological images and the local information correlation features within each tumor cell. In WSI, cell features are extracted from all cells based on whether the nuclear mask exists in the same connected region and the image patch size, forming a cell feature map. Then The input is a four-layer GraphTransformer module. The output is subjected to mean pooling, then passes through a multilayer perceptron (MLP) layer and another GraphTransformer module. The output feature map is then subjected to dimensionality reduction using PCA principal component analysis to obtain the spatial features of tumor cells. Simultaneously, it passes through a linear layer to output a binary classification prediction value for high and low recurrence risk (WSI). The loss function used is the binary cross-entropy loss commonly used in binary classification tasks, as shown in the following formula:

[0112]

[0113] in The predicted value is 1 for high recurrence risk and 0 for low recurrence risk. This is the actual label value.

[0114] Through the aforementioned intelligent quantification process of digital pathological images, the pathological features, comprised of tumor cell nuclear features and tumor cell spatial features, are obtained and used for subsequent model construction. An additional predicted value is also obtained, used in subsequent interpretative analysis to describe the importance of pathological features in predicting the risk of recurrence.

[0115] The multimodal feature fusion and modeling module 103 receives all filtered feature vectors from the feature extraction module 102. These features are then concatenated and input into a fusion network based on a Transformer encoder architecture. This network learns the intrinsic relationships between features of different modalities and between features at different time points through its self-attention mechanism. Finally, through a fully connected layer and a Softmax activation function, it outputs a binary probability indicating whether the patient belongs to "high recurrence risk" or "low recurrence risk".

[0116] The multimodal feature fusion and modeling module includes:

[0117] Multi-time-series dataset establishment: This embodiment aims to establish a multi-time-series dataset, encompassing multimodal time-series data generated before neoadjuvant therapy, before radical surgery, and during follow-up. This primarily includes multi-parameter MRI imaging data, including T2WI, DWI, and DCE-MRI, as well as clinical demographic information such as age distribution, serological markers CEA, CA19-9, and CA-125 levels, treatment regimens, and medication use. Furthermore, it includes significant clinical events, such as bowel obstruction and emergency admissions, for subsequent analysis.

[0118] Longitudinal Model Construction: In this embodiment, the aforementioned multimodal data analysis and quantification methods are used to initially screen time-series clinical features and extract features from time-series image data. Then, the radiomics manual and deep learning features are input into the Transformer network to extract time-series image features. The time-series clinical features, time-series image features, and biopsy pathology features are combined and input into the Transformer network for fusion to construct the final prediction model.

[0119] Interpretability Analysis and Model Evaluation

[0120] (1) Interpretability analysis

[0121] To better understand the model and accelerate its clinical application, this embodiment aims to interpret the model based on Grad-CAM visualization images, using SHAP values ​​to reflect the contribution of model features, and combining expert prior knowledge sharing to evaluate the model, thereby improving its interpretability from different perspectives.

[0122] Grad-CAM Visualization: This embodiment proposes to integrate Grad-CAM into the ViT self-attention network for MRI image visualization. For the ViT network, each MRI slice is divided into many patches. Then, using the output and gradient of the last attention block of ViT, the contribution of each token to the clinical outcome is calculated. These contributions are then mapped back to the spatial location of the original image to form a heatmap. The specific steps are as follows:

[0123] First, starting from the last convolutional layer of the ViT model, we calculate the gradient with respect to the prediction result. Assuming the feature map of the last convolutional layer is A, the prediction result is y, and the loss function is L, we can obtain the gradient. Next, the gradients are spatially averaged to obtain the importance weights of each feature map. :

[0124]

[0125] Where A represents the size of the output of the last convolutional layer of the network, and Z represents the number of pixels in the feature map. ij The pixel values ​​at positions i and j representing the k-th feature map are weighted using the neuron importance weights obtained above. The features of the selected convolutional layer are weighted as shown in the following formula:

[0126]

[0127] Here, the ReLU function is the ReLU activation function. ∑ represents a weighted summation of all feature maps. Finally, the heatmap undergoes non-linear processing and interpolation to obtain a Grad-CAM map that matches the size of the input image, as shown below. Figure 7 As shown:

[0128] SHAP Interpretation of Feature Contribution: This embodiment intends to use the SHAP interpreter to calculate the specific contribution of features to a single sample and the overall population. The SHAP interpreter is a Python package developed in recent years that can be used to interpret the output of any machine learning model. Using the SHAP interpreter, we can calculate the contribution of each feature to the prediction of a single sample and the entire dataset. An illustration is shown below. Figure 8 As shown, in this model, each feature is considered a "contributor" and assigned a SHAP value, representing the importance of that feature in a given sample. The calculation formula is as follows:

[0129]

[0130] Assume the first The sample is , No. The first sample The features are The model's predicted value for this sample is The baseline of the entire model (usually the mean of the target variable for all samples) is... Based on the above formula, the contribution of each feature to a single sample and the contribution of the entire sample are finally obtained. In the image model, we intend to use the SHAP interpreter to obtain the individual contribution of handcrafted features and deep learning features on each parameter of T2WI, DWI, and DCE-MRI images to positive and negative samples (DFS positive or negative), as well as their contribution to the whole.

[0131] Regarding nuclear features, to comprehensively understand their contribution to the model, in addition to using SHAP values, the importance of relative morphological features was analyzed using an SVM linear model. First, tumor nuclear contours were extracted from tumor cells identified by the segmentation network, and nuclear size and morphology were calculated by averaging the values ​​from each WSI image. The nuclear size and shape of a single tumor cell were calculated using the following formula:

[0132]

[0133] (2) Model evaluation

[0134] Shared Decision Comparison and Evaluation: This embodiment aims to establish a model decision visualization interface based on the aforementioned Grad-CAM and SHAP interpretability studies. By sharing decisions between doctors and the model, the model's diagnostic effectiveness in assisting doctors is evaluated. Finally, the changes in predictive performance under the two models are compared to verify the model's actual clinical application effectiveness. The visualization interface is as follows: Figure 9 As shown. The specific model of shared decision-making is as follows: Doctors will use the visualization interface provided by the model, combined with clinical pathology and imaging information from multiple time points (the multi-time series information is consistent with the model's input data), to score the patient's prognostic risk from 1 to 5. Evaluation metrics include the model's area under the curve (AUROC), sensitivity, specificity, C-index, and Kaplan-Meier survival curve. Multi-time-point comparative evaluation: To verify the model's value in predicting the patient's long-term recurrence risk, this embodiment plans to evaluate the model at different time points. Specifically, before the start of follow-up, at time point t0, the data at time t0 will be used to construct a multimodal model, and its performance will be compared with the performance of models constructed from time series data added at subsequent time points t1 (i.e., the follow-up node at the sixth month after surgery) to t6 (i.e., the end of the three-year follow-up). By comparing the model performance at different time points, the predictive ability of the model for patient prognosis at different time periods can be evaluated, the model's sustained performance during follow-up can be understood, and its predictive effect can be assessed as a result of the addition of time series data. Evaluation metrics include the model's AUROC, sensitivity, specificity, C-index, and Kaplan-Meier survival curve.

[0135] This invention also provides a method for predicting the risk of postoperative recurrence of rectal cancer based on multimodal time-series data, comprising the following steps: S1: Constructing a cross-modal time-series dataset, the dataset including time-series clinical data, time-series magnetic resonance imaging data, and digital pathology images of stage III rectal cancer patients; S2: Preprocessing and feature filtering the time-series clinical data to obtain clinical features; S3: Segmenting the tumor regions in the time-series magnetic resonance imaging data, and extracting radiomics handcrafted features and deep learning features based on visual transformers, respectively; S4: Segmenting the tumor cell nuclei in the biopsy digital pathology images, and extracting tumor cell nucleus morphological features and tumor cell spatial distribution features based on graph convolutional networks and transformers; S5: Fusing all features extracted in steps S2 to S4, inputting them into a prediction model based on a transformer architecture for training and prediction, and outputting the patient's postoperative recurrence risk, etc.

[0136] The tumor region segmentation in step S3 specifically involves: using a pre-trained U-Net network for automatic initial segmentation, followed by manual correction by a radiologist.

[0137] The extraction of spatial distribution features of tumor cells in step S4 specifically includes: constructing a cell feature map based on the location of the segmented tumor cell nuclei, inputting the cell feature map into multiple graph convolution-transformer modules in sequence to model spatial relationships, and then obtaining the spatial distribution features of tumor cells after pooling and dimensionality reduction.

[0138] In step S5, the prediction model is trained using three-year disease-free survival as the clinical endpoint.

[0139] Figure 1 The corresponding method flow is shown, including steps S1 to S5. The specific implementation content corresponds to the functions of the above system modules, and will not be repeated here.

[0140] During model training, the presence or absence of recurrence (local recurrence or distant metastasis) three years post-surgery was used as the binary classification label. The cross-entropy loss function was employed, and the Adam optimizer was used for training on the internal training set, followed by hyperparameter tuning on the validation set.

[0141] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the steps of the above-described method.

[0142] In summary, this invention, through an innovative multimodal temporal data fusion and deep learning feature extraction framework, achieves a more accurate and dynamic assessment of the risk of recurrence after rectal cancer surgery, and has significant potential for clinical translation.

Claims

1. A system for predicting the risk of postoperative recurrence of rectal cancer based on multimodal time-series data, characterized in that, include: The data acquisition and preprocessing module is used to acquire cross-modal time-series datasets of stage III rectal cancer patients, which include: time-series clinical data, time-series magnetic resonance imaging data, and digital pathology images. The feature extraction module, connected to the data acquisition and preprocessing module, is used for: a) Filter the time-series clinical data to obtain clinical characteristics; b) The time-series magnetic resonance imaging data is segmented into tumor regions, and radiomics handcrafted features and deep learning features based on visual transformers are extracted respectively; c) Segment tumor cell nuclei in digital pathological images and extract morphological features of tumor cell nuclei and spatial distribution features of tumor cells based on graph convolutional networks and graph transformers; A multimodal feature fusion and modeling module, connected to the feature extraction module, is used to fuse the clinical features, the radiomics handcrafted features, the deep learning features, the tumor cell nuclear morphology features, and the tumor cell spatial distribution features, and input them into a prediction model based on a transformer architecture to output the patient's postoperative recurrence risk prediction results. The feature extraction module includes: The tumor segmentation unit is used to automatically segment tumor regions in magnetic resonance images using a pre-trained U-Net network. The image feature extraction unit is used to extract shape, histogram and texture features from the segmented tumor region as the radiomics handcrafted features using the Pyradiomics package, and to extract high-dimensional features as the deep learning features using a visual transformer. The pathology image enhancement unit is used to enhance digital pathology images using generative adversarial networks. The cell nucleus segmentation unit is used to locate and segment tumor cell nuclei in enhanced pathological images using a network based on a mask autoencoder pre-trained architecture. The pathological feature extraction unit is used to calculate the nuclear area, entropy value and number ratio from the segmented tumor cell nuclei as the morphological features of the tumor cell nuclei, and to construct a cell feature map based on the location of the tumor cell nuclei, which is then input into a network composed of a multi-layer graph convolutional network and a graph transformer module to extract the spatial distribution features of the tumor cells. The prediction model in the multimodal feature fusion and modeling module is a binary classification model based on a transformer encoder, which generates the final relapse risk score by associating and fusing features from different modalities and different time points.

2. The system according to claim 1, characterized in that, The time-series magnetic resonance imaging data includes at least T2-weighted imaging, diffusion-weighted imaging, and dynamic contrast-enhanced magnetic resonance imaging at multiple time points before neoadjuvant therapy, before radical surgery, and during postoperative follow-up.

3. A method for predicting the risk of postoperative recurrence of rectal cancer based on multimodal time-series data, characterized in that, Includes the following steps: S1: Construct a cross-modal time-series dataset, which includes time-series clinical data, time-series magnetic resonance imaging data, and digital pathology images of stage III rectal cancer patients; S2: Preprocess and feature-filter the time-series clinical data to obtain clinical features; S3: Perform tumor region segmentation on the time-series magnetic resonance imaging data, and extract radiomics handcrafted features and deep learning features based on visual transformers, respectively; S4: Segment the tumor cell nuclei in the digital pathological image and extract the morphological features of the tumor cell nuclei and the spatial distribution features of the tumor cells based on graph convolutional networks and graph transformers; S5: Fuse all the features extracted in steps S2 to S4, input them into the prediction model based on the transformer architecture for training and prediction, and output the patient's postoperative recurrence risk level. In step S3, the tumor region in the magnetic resonance image is automatically segmented using a pre-trained U-Net network; from the segmented tumor region, shape, histogram and texture features are extracted as the image omics handcrafted features using the Pyradiomics package, and high-dimensional features are extracted as the deep learning features using a visual transformer. In step S4, a generative adversarial network is used to augment the digital pathological image; a network based on a mask autoencoder pre-trained architecture is used to locate and segment tumor cell nuclei in the augmented pathological image; the nuclear area, entropy value, and number ratio of the segmented tumor cell nuclei are calculated as the morphological features of the tumor cell nuclei, and a cell feature map is constructed based on the location of the tumor cell nuclei. The map is then input into a network composed of a multi-layer graph convolutional network and a graph transformer module to extract the spatial distribution features of the tumor cells. In step S5, the prediction model is a binary classification model based on a transformer encoder, which generates the final relapse risk score by associating and fusing features from different modalities and different time points.

4. The method according to claim 3, characterized in that, The tumor region segmentation in step S3 specifically involves: using a pre-trained U-Net network for automatic initial segmentation, followed by manual correction by a radiologist.

5. The method according to claim 3, characterized in that, The extraction of spatial distribution features of tumor cells in step S4 specifically includes: constructing a cell feature map based on the location of the segmented tumor cell nuclei, inputting the cell feature map into multiple graph convolution-transformer modules in sequence to model spatial relationships, and then obtaining the spatial distribution features of tumor cells after pooling and dimensionality reduction.

6. The method according to claim 3, characterized in that, In step S5, the prediction model is trained using three-year disease-free survival as the clinical endpoint.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 3 to 6.

Images