Prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multi-mode model

Abstract

The invention provides a prostate cancer biochemical recurrence prediction system based on a deep learning image and clinical information multi-mode model. The prostate cancer biochemical recurrence prediction system comprises an input module, a preprocessing module, a feature selection module, a fusion module, a prediction module, a data management and storage module and a result output and decision support module. According to the method, bpMRI image data and clinical information of a patient are fully utilized, multi-mode information is fully fused through a deep learning technology, and accurate prediction of biochemical recurrence is achieved. And the robustness of the model is further enhanced by adopting 5-fold cross validation and a diversified data enhancement strategy. Experimental results show that the method can assist in making more accurate patient layering and personalized follow-up visit schemes, and potentially reduces the over-treatment and missed diagnosis risks. The invention provides an efficient and reliable AI solution for prostatic cancer prognosis evaluation, and provides powerful support for promoting precise medical treatment and making personalized treatment strategies.

Description

Technical Field

[0001] The present invention relates to the technical field of tumor recurrence prediction, and in particular to the prediction of biochemical recurrence of prostate cancer, and in particular to a prediction system for predicting the risk of biochemical recurrence of prostate cancer based on multimodal fusion images and clinical information based on deep learning. The deep learning method is applied to the image fusion and clinical information of prostate cancer patients to predict the biochemical recurrence of prostate cancer and perform risk stratification to formulate different follow-up plans. Background Art

[0002] Prostate cancer is the second most common cancer in men worldwide and the second leading cause of cancer death in men, placing a heavy burden on patients and society. It is estimated that in 2020 alone, there will be more than 1.4 million newly diagnosed cases of prostate cancer worldwide, accounting for 7.13% of all cancer incidence, and the number of prostate cancer-related deaths will reach 370,000. Currently, radical prostatectomy (RP) is one of the preferred treatments for prostate cancer patients and is also associated with good long-term prognosis. However, after undergoing radical prostatectomy, about one-third of prostate cancer patients will still experience postoperative biochemical recurrence at different time periods after surgery. Biochemical recurrence (BCR) of prostate cancer is defined as two consecutive serum prostate-specific antigen (PSA) values ​​> 0.2 ng / ml after radical treatment, which is an important sign of possible clinical recurrence of prostate cancer after surgery. BCR indicates that the disease is active and is more likely to have new metastases, such as bone metastasis and lymph node recurrence, which shortens the patient's survival time. Therefore, the prediction of BCR after radical prostatectomy is crucial for the patient's prognosis.

[0003] At present, preoperative assessment of BCR remains challenging because it can only be confirmed by pathological diagnosis. Traditionally, the pathological Gleason grading system is used to predict the prognosis of prostate cancer, but relying solely on the Gleason score cannot fully reflect the patient's prognostic risk, especially some patients with low Gleason scores may also experience biochemical recurrence in the short term. The CAPRA-S scoring system based on a large amount of clinical data combines indicators such as preoperative PSA level, Gleason score and clinical stage to provide a more comprehensive risk assessment. However, due to significant heterogeneity between individuals, single clinical data still has limitations in predicting biochemical recurrence. Currently, multiparametric magnetic resonance imaging (mpMRI) is widely used to predict BCR in prostate cancer. Studies have shown that factors such as Gleason score, tumor volume and metastasis assessed by mpMRI are significantly correlated with patient survival time and recurrence. The application of mpMRI in prostate cancer can clearly distinguish tumors of different risk levels and guide the selection of individualized and targeted treatment options, thereby significantly improving the patient's quality of life and prognosis.

[0004] Recent studies have also shown that dual-parameter magnetic resonance imaging (bpMRI) has the same effectiveness as mpMRI in the diagnosis of prostate cancer. With the rapid development of medical imaging technology and artificial intelligence (AI), dual-parameter magnetic resonance imaging (bpMRI) plays an increasingly important role in the diagnosis and prognosis of prostate cancer. bpMRI not only provides high-resolution anatomical images, but also reflects the functional and metabolic characteristics of tissues. However, due to the great heterogeneity between individuals, using clinical data alone or bpMRI imaging data alone to predict biochemical recurrence is not the optimal method.

[0005] Deep learning is a machine learning method that has emerged in the field of artificial intelligence. It uses multi-layer neural networks to learn complex representations of input data. Unlike traditional machine learning methods, deep learning does not require the selection of engineering features in advance. Instead, it improves the abstraction level of feature representation through layer-by-layer iterative learning, so that high-level feature representations are constructed layer by layer from low levels. Methods based on deep learning have been used to predict the prognosis of rectal cancer, the diagnosis and prognosis of lung cancer, and the prognostic gene prediction of gastric cancer, and have achieved good results. However, before the implementation of the present invention, a risk stratification prediction system for predicting biochemical recurrence of prostate cancer based on swin-transformer combined with clinical information and bpMRI imaging data has not been established. Summary of the invention

[0006] The purpose of the present invention is to overcome the above-mentioned defects and shortcomings in the prior art, and to provide an algorithm for predicting the risk of biochemical recurrence of prostate cancer based on a deep learning multimodal fusion of bpMRI images and clinical information, which integrates and analyzes the clinical information and magnetic resonance images of prostate cancer patients, and provides a prostate cancer biochemical recurrence prediction system based on a deep learning multimodal model of images and clinical information.

[0007] The present invention aims to provide a system for predicting the risk of biochemical recurrence of prostate cancer by integrating multimodal fusion images and clinical information based on deep learning. By fusing bpMRI data and clinical information, the system can predict the status of biochemical recurrence after RP in prostate cancer patients, and provide decision support for clinicians to formulate personalized postoperative follow-up and auxiliary treatment plans.

[0008] In order to achieve the above object, the specific technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention provides a prostate cancer biochemical recurrence prediction system based on a deep learning-based imaging image and clinical information multimodal model, including an input module, a preprocessing module, a feature selection module, a fusion module, a prediction module, a data management and storage module, and a result output and decision support module.

[0010] (1) Input module

[0011] The input module receives MRI images, complete follow-up information and clinical characteristic information based on certain rules. Specifically, the input module includes a patient selection module and an MRI acquisition and clinical characteristic information import module.

[0012] Among them, the patient selection module has an inclusion criteria submodule and an exclusion criteria submodule.

[0013] Inclusion criteria submodule: Patients who underwent MRI before radical prostatectomy and had high-quality MRI images and complete follow-up information but did not receive neoadjuvant therapy were selected for further evaluation.

[0014] Exclusion criteria submodule: Patients with missing key MRI sequences, poor image quality, neoadjuvant therapy, persistently elevated PSA, or incomplete clinical data were excluded from the next step of evaluation.

[0015] The MRI acquisition and clinical feature information import module includes an image acquisition submodule and a clinical information import submodule. The image acquisition submodule is constructed with T2WI, DWI and ROI as three-channel inputs, and a 3.0T MR scanner equipped with an abdominal phased array coil is used to perform prostate MRI scans, including T2WI and high b-value DWI sequences.

[0016] The clinical information import submodule imports clinical characteristic information from the hospital's inpatient system, including prostate-specific antigen PSA level, ISUP grade, pathological T stage, clinical T stage, and tumor volume.

[0017] (2) Preprocessing module

[0018] The preprocessing module includes an image preprocessing module for outlining the external contours of the MRI image and performing preliminary standardization processing, and a clinical feature preprocessing module for processing clinical feature information.

[0019] The image preprocessing module uses ITK-snap software and other custom functions to manually or automatically outline the external contours of prostate lesions on axial slices; the methods for preliminary standardization of images are as follows: including N4 deviation field correction, spatial registration, resampling, cropping, and intensity normalization to reduce interference and noise;

[0020] The clinical feature preprocessing module implements a data cleaning process for clinical feature information, standardizes the data format, handles missing values ​​such as deleting, filling, or using models to predict missing values, and identifies and corrects outliers to ensure the accuracy and reliability of the data set.

[0021] (3) Feature selection module

[0022] The feature selection module includes an image data selection module and a clinical feature selection module. The image data selection module is a deep learning model built based on swin transformer. The deep learning model uses RGB three-channel images composed of T2WI, DWI, and ADC as input to convert MRI images into Input into the Swin transformer image feature extraction module to obtain image features .

[0023] The clinical feature selection module analyzes the available clinical information and obtains clinical features through linear mapping. Specifically, the clinical feature selection module analyzes the available clinical information and selects clinical features with significant significance (p<0.05); then the clinical features are normalized to [0,1] by Min Max Scaler, and then the processed clinical features are extracted through linear mapping. .

[0024] Furthermore, univariate and multivariate analyses were performed on the available clinical information. Univariate analysis was used to evaluate the independent relationship between each clinical feature and BCR, and statistical tests such as t-test and chi-square test were used to evaluate each feature value. Multivariate analysis was used to evaluate the relationship between multiple clinical features and BCR while controlling for the influence of other variables, mainly using backward stepwise regression.

[0025] Clinical information Input into a linearly mapped clinical feature extraction module to obtain clinical features , which can effectively extract clinical variable information related to BCR status and compress feature dimensions through linear transformation for subsequent feature fusion:

[0026] .

[0027] (4) Fusion module

[0028] The fusion module extracts the image features and clinical features Input to the multimodal compact bilinear pool MCB for fusion to obtain the fused fusion features : .

[0029] The MCB method can capture the interactive information between features of different modalities and enhance the expressiveness of features through bilinear mapping, thereby generating a more representative fusion feature.

[0030] Specifically, the fusion module is selected from the MCB (Multimodal Compact Bilinear Pooling) method, which is a way to fuse multimodal feature vectors. The image feature vector and the text feature vector of the clinical feature are randomly projected into a high-dimensional space, and then the two vectors are effectively convolved using element-wise product in the fast Fourier transform space, which can better model the complex relationship between the two modalities and more effectively express the combination of multimodal features.

[0031] (5) Prediction module

[0032] The prediction module is a deep learning model built based on Swin Transformer, which obtains the fusion features Input to the classification head , calculate SoftMax and get the final output result , classification head It is composed of fully connected layers, which aims to map the extracted features to the probability distribution of each category, so as to output the BCR prediction probability: .

[0033] The prediction module is first trained, verified and evaluated in the early stage. The specific method is as follows:

[0034] a. Model training details:

[0035] (1) Batch size: 16;

[0036] (2) Training epochs: 100;

[0037] (3) Learning rate decay strategy: LambdaLR;

[0038] (4) Optimizer: SGD, stochastic gradient descent;

[0039] (5) Loss function: Weighted Cross-Entropy Loss, which is weighted according to BCR:NO-BCR at a ratio of 7:3 to deal with the existing class imbalance problem;

[0040] (6) Training method: 5-fold cross validation;

[0041] (7) Dataset division: Divide into training set and test set in a ratio of 4:1, and then continue to divide the training set and validation set in a ratio of 3:1 on the training set;

[0042] (8) Use the early stopping strategy: monitor the accuracy on the validation set. If there is no improvement for 20 consecutive epochs, stop training early to prevent overfitting.

[0043] Implement the models in version 2.0.1 of PyTorch and train them on an NVIDIA RTX 3070 GPU. Use an external validation set in real time to evaluate the performance of existing model predictions and use this as a basis for modification and parameter adjustment.

[0044] b. Model evaluation:

[0045] The models were evaluated using the 5-fold average AUC and other performance indicators (accuracy, sensitivity, specificity, and F1 score). The average AUC was calculated as the primary indicator, and the 95% CI was calculated by guiding the ROC curve using the bootstrap method 2000 times. In addition, we used Survival curves were used for time-to-event analysis of biochemical recurrence. Decision curves were also used to assess the overall net benefit of the model.

[0046] c. Model evaluation and optimization module

[0047] Use external validation sets in real time to evaluate the effectiveness of existing model predictions and use them as a basis for modification and parameter adjustment:

[0048] (c-1) Performance evaluation submodule: compare the prediction performance of deep learning models, clinical models and fusion models, and use ROC, DCA and calibration curves and various evaluation indicators for comprehensive display (including ACC, PPV, NPV, etc.);

[0049] (c-2) Model optimization: Provide recommended parameters based on the evaluation results, and incorporate new samples to continuously adjust and improve the model.

[0050] When making a prediction, the prediction model selects the three slices with the highest prediction probability among all tumor slices of a patient, averages their probabilities, and then uses them as the final predicted probability of biochemical recurrence of the case.

[0051] (6) The data management and storage module is the data storage part, including:

[0052] (a) Clinical data storage: storing patients’ clinical information and follow-up data;

[0053] (b) Image data storage: storing original MRI images and preprocessed images;

[0054] (c) Prediction result storage: save the prediction results and performance indicators of each model;

[0055] (d) Follow-up management submodule: The median follow-up time and the patient’s telephone number were recorded, patients who did not experience BCR or received additional treatment at the last follow-up were censored, and doctors were reminded of the need to follow up with the patients.

[0056] (7) Result output and decision support module, used to improve clinical intervention based on the results:

[0057] (7-1) Risk assessment submodule: Integrate the prediction results of each model to provide the patient's BCR risk assessment;

[0058] (7-2) Decision-making recommendation submodule: According to the risk assessment results, the patients are automatically classified into the close interval follow-up group or the wide interval follow-up group;

[0059] (a) Develop an individualized follow-up plan for the close interval follow-up group, including:

[0060] Follow-up was conducted every month within 1 year after surgery;

[0061] Follow-up every 3 months 2-3 years after surgery;

[0062] After 3 years, follow-up was conducted annually;

[0063] (b) Develop an individualized follow-up plan for the wide-interval follow-up group, including:

[0064] Follow-up was conducted every 6 months within 3 years after surgery;

[0065] After 3 years, the patients were followed up annually.

[0066] In a second aspect, the present invention provides a method for predicting the risk of biochemical recurrence of prostate cancer based on the above system, comprising the following steps:

[0067] (1) The input module includes bpMRI images, complete follow-up information, and clinical characteristics (prostate-specific antigen PSA level, ISUP grade, pathological T stage, clinical T stage, and tumor volume) of patients who underwent radical prostatectomy but did not receive neoadjuvant therapy and had high-quality MRI images and complete follow-up information before surgery;

[0068] (2) A preprocessing module that performs external contour delineation and preliminary standardization on MRI images, and a clinical feature preprocessing module that processes clinical feature information;

[0069] (3) The image data selection module in the feature selection module uses the RGB three-channel image composed of T2WI, DWI, and ADC as input, and converts the MRI image Input into the Swin transformer image feature extraction module to obtain image features The clinical feature selection module analyzes the available clinical information and selects the significant Then, these clinical characteristics are normalized to the maximum and minimum, and all clinical information is normalized to Then, the processed clinical features are extracted through linear mapping. ;

[0070] (4) The fusion module combines the image features after feature extraction and clinical features Input to the multimodal compact bilinear pool MCB for fusion to obtain the fused fusion features :

[0071] .

[0072] (5) The prediction module is a deep learning model built based on Swin Transformer. After model training and verification, the fusion features obtained are Input to the classification head , calculate SoftMax and get the final output result , classification head It is composed of fully connected layers, which aims to map the extracted features to the probability distribution of each category, so as to output the BCR prediction probability:

[0073] ;

[0074] The three sections with the highest prediction probabilities were selected from all tumor sections of each patient, and the average of the prediction probabilities of the selected sections was calculated as the final BCR prediction probability of the patient.

[0075] (6) The result output and decision support module classifies patients into a close interval follow-up group or a wide interval follow-up group based on the risk assessment results, and develops a personalized follow-up plan.

[0076] According to a third aspect of the present invention, there is provided a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned steps when executed by a processor.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] The present invention proposes an algorithm for predicting the risk of biochemical recurrence of prostate cancer by multimodal fusion of imaging images and clinical information. The algorithm achieves a comprehensive assessment of the risk of biochemical recurrence of prostate cancer after RP in patients by multimodally fusing the image information and key clinical information of bpMRI imaging data. The present invention adopts an advanced deep learning architecture to fully extract the characteristics of the patient's imaging data and multimodally fuses it with key clinical information such as PSA level and Gleason score. After rigorous experiments, this method has performed well in key performance indicators such as prediction accuracy and AUC.

[0079] The core advantage of the present invention is that it can make full use of the patient's bpMRI imaging data and clinical information. Through deep learning technology, the model can fully integrate multimodal information to achieve accurate prediction of biochemical recurrence. The 5-fold cross-validation and diversified data enhancement strategies adopted further enhance the robustness of the model. Experimental results show that this method can assist doctors in more accurate patient stratification and personalized follow-up plan formulation, potentially reducing the risk of overtreatment and missed diagnosis. Overall, the present invention provides an efficient and reliable AI solution for prostate cancer prognosis assessment, demonstrates great potential for application in clinical practice, and provides strong support for the advancement of precision medicine and personalized treatment strategy formulation.

[0080] This algorithm makes up for the limitation of the original prediction using a single data source, integrates multimodal information and uses the deep learning model swin-transformer, providing a reliable decision support tool for clinicians to develop personalized treatment plans and follow-up strategies. It realizes the stratified prediction of the biochemical recurrence risk of prostate cancer in patients, provides a reliable basis for predicting the risk of biochemical recurrence of prostate cancer, and helps clinicians decide on surgical strategies and evaluate the prognosis of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 The framework diagram of the prostate cancer biochemical recurrence prediction system based on deep learning multimodal model of imaging images and clinical information is shown;

[0082] Figure 2 A flow chart showing the prediction of biochemical recurrence in prostate cancer. DETAILED DESCRIPTION

[0083] The present invention will be described in detail below in conjunction with the embodiments and drawings, but the implementation of the present invention is not limited thereto.

[0084] The present invention aims to provide a system for predicting the risk of biochemical recurrence of prostate cancer based on deep learning-based multimodal fusion images and clinical information. By fusing bpMRI data and clinical information, the status of biochemical recurrence after radical prostatectomy (RP) in prostate cancer patients can be predicted.

[0085] like Figure 1 As shown, a prostate cancer biochemical recurrence prediction system 100 based on a deep learning-based imaging image and clinical information multimodal model includes an input module 1, a preprocessing module 2, a feature selection module 3, a fusion module 4, a prediction module 5, a data management and storage module 6, and a result output and decision support module 7. Among them, the preprocessing module 2 and the feature selection module 3 can be integrated into the bpMRI image data collection and processing subsystem and the clinical feature data collection and processing subsystem, respectively, to process the bpMRI data and the clinical information data, respectively.

[0086] 1. Input module

[0087] The input module 1 receives MRI images, complete follow-up information and clinical characteristic information. Specifically, the input module includes a patient selection module 11 and an MRI acquisition and clinical characteristic information import module 12. Among them, the patient selection module 11 has an inclusion standard submodule 111 and an exclusion standard submodule 112.

[0088] Inclusion criteria submodule 111 selected patients who underwent MRI before radical prostatectomy and had high-quality MRI images and complete follow-up information but did not receive neoadjuvant therapy for the next evaluation. Exclusion criteria submodule 112 excluded patients who were missing key MRI sequences, had poor image quality, had received neoadjuvant therapy, had persistently elevated PSA, or had incomplete clinical data from the next evaluation.

[0089] The MRI acquisition and clinical characteristic information import module 12 includes an image acquisition submodule 121 and a clinical information import submodule 122. The image acquisition submodule 121 is constructed with T2WI, DWI and ROI as three-channel inputs, and imports the bpMRI image obtained by prostate MRI scanning using a 3.0T MR scanner equipped with an abdominal phased array coil, and imports T2WI and high b-value DWI sequences at the same time; the clinical information import submodule 122 imports clinical characteristic information from the hospital's inpatient system, including prostate-specific antigen (PSA) level, ISUP grade, pathological T stage, clinical T stage and tumor volume.

[0090] 2. Preprocessing module and feature selection module

[0091] The preprocessing module 2 includes an image preprocessing module 21 and a clinical feature preprocessing module 22. The image preprocessing module 21 performs external contour delineation and preliminary standardization processing on the MRI image; the clinical feature preprocessing module 22 processes the clinical feature information; the feature selection module 3 includes an image data selection module 31 and a clinical feature selection module 32.

[0092] During actual operation, the image preprocessing module 21 and the image data selection module 31 can be integrated into the bpMRI image data collection and processing subsystem to complete the processing and feature extraction of the bpMRI image data; the clinical feature preprocessing module 22 and the clinical feature selection module 32 can be integrated into the clinical feature data collection and processing subsystem to complete the processing and feature extraction of the clinical feature data.

[0093] Specifically, the image preprocessing module 21 calls ITK-snap software and other custom functions to manually or automatically outline the outer contour of the prostate lesion on the axial slice; the method for preliminary standardization of the image is as follows: including N4 deviation field correction, spatial alignment, resampling, cropping and intensity normalization to reduce interference and noise.

[0094] The image data selection module 31 is a part of the deep learning model built based on the swin transformer. The deep learning model uses the RGB three-channel image composed of T2WI, DWI, and ADC as input to convert the MRI image Input it to get image features , and cropped the ROI area. A data enhancement strategy was used to enhance the robustness of the model, and then the swin transformer was used to extract the fused image features of the three sequences.

[0095] The clinical feature preprocessing module 22 implements a data cleaning process for the clinical feature information, standardizes the data format, processes missing values ​​such as deleting, filling or using a model to predict missing values, and identifies and corrects outliers to ensure the accuracy and reliability of the data set.

[0096] The clinical feature selection module 32 analyzes the available clinical information and obtains clinical features through linear mapping. Specifically, the clinical feature selection module analyzes the available clinical information and selects the significant Then, the clinical characteristics are normalized to the maximum and minimum scale (Min Max Scaler) to normalize all clinical information to Then, the processed clinical features are extracted through linear mapping. .

[0097] Specifically, univariate and multivariate analyses were used for the available clinical information. Univariate analysis was used to evaluate the independent relationship between each clinical feature and BCR, and statistical tests such as t-test and chi-square test were used to evaluate each feature value. Multivariate analysis was used to evaluate the relationship between multiple clinical features and BCR while controlling for the influence of other variables, mainly using backward stepwise regression.

[0098] Clinical information Input into a linearly mapped clinical feature extraction module to obtain clinical features It can effectively extract clinical variable information related to BCR status and compress feature dimensions through linear transformation for subsequent feature fusion.

[0099] 3. Fusion Module

[0100] Fusion module 4 extracts the image features and clinical features Input to the multimodal compact bilinear pool MCB for fusion to obtain the fused fusion features .

[0101] Multimodal Compact Bilinear Pooling (MCB) is a method of fusing multimodal feature vectors. It randomly projects the feature vectors of images and text into a high-dimensional space, and then effectively convolves the two vectors using element-wise product in the Fast Fourier Transform (FFT) space. This can better model the complex relationship between the two modalities and more effectively express the combination of multimodal features. and clinical features Input to the multimodal compact bilinear pool (MCB) for fusion to obtain the fused features The MCB method can capture the interactive information between different modal features and enhance the expressiveness of features through bilinear mapping, thereby generating a more representative fusion feature. .

[0102] 4. Prediction Module

[0103] Prediction module 5 is a deep learning model based on Swin Transformer, which obtains the fusion features Input to the classification head , calculate SoftMax and get the final output result , classification head It is composed of fully connected layers, which aims to map the extracted features to the probability distribution of each category, so as to output the BCR prediction probability: .

[0104] The prediction module is first trained, verified and evaluated in the early stage. The specific method is as follows:

[0105] (a) Model input: RGB three-channel image consisting of T2WI, DWI, and ADC after image preprocessing.

[0106] (b) Data augmentation strategy: center crop the image to pixels, random horizontal flip, random contrast and brightness adjustment, random rotation, random vertical flip, etc.

[0107] (c) Model training

[0108] Details include:

[0109] (1) Batch size: 16;

[0110] (2) Training epochs: 100;

[0111] (3) Learning rate decay strategy: LambdaLR;

[0112] (4) Optimizer: SGD (stochastic gradient descent);

[0113] (5) Loss function: Weighted Cross-Entropy Loss, which is weighted according to BCR:NO-BCR at a ratio of 7:3 to deal with the existing category imbalance problem.

[0114] (6) Training method: 5-fold cross validation.

[0115] (7) Dataset division: The dataset is divided into training set and test set in a ratio of 4:1, and then the training set is further divided into training set and validation set in a ratio of 3:1.

[0116] (8) An early stopping strategy is also used: the accuracy on the validation set is monitored, and if there is no improvement for 20 consecutive epochs, training is stopped early to prevent overfitting.

[0117] Implement the models in PyTorch (version 2.0.1) and train them on an NVIDIA RTX 3070 GPU.

[0118] (d) Model evaluation:

[0119] The models were evaluated using the 5-fold average AUC and other performance indicators (accuracy, sensitivity, specificity, and F1 score). The average AUC was calculated as the primary indicator, and the 95% CI was calculated by guiding the ROC curve using the bootstrap method 2000 times. In addition, we used Survival curves were used for time-to-event analysis of biochemical recurrence, and decision curves were used to evaluate the overall net benefit of the model. The results of the five-fold cross validation of the model of the present invention are shown in Table 1 below:

[0120] Table 1 Five-fold cross validation results of the model

[0121]

[0122] (e) Final model prediction:

[0123] The obtained fusion features Input to the classification head , calculate SoftMax and get the final output result The classification head consists of a fully connected layer, which aims to map the extracted features to the probability distribution of each category, thereby outputting the BCR prediction probability: .

[0124] (f) Model evaluation and optimization:

[0125] Use external validation sets in real time to evaluate the effectiveness of existing model predictions and use them as a basis for modification and parameter adjustment:

[0126] (f-1) Performance evaluation submodule: compare the prediction performance of deep learning models, clinical models and fusion models, and use ROC, DCA and calibration curves and various evaluation indicators for comprehensive display (including ACC, PPV, NPV, etc.);

[0127] (f-2) Model optimization: Provide recommended parameters based on the evaluation results, and incorporate new samples to continuously adjust and improve the model.

[0128] When making a prediction, the prediction model selects the three slices with the highest prediction probability among all tumor slices of a patient, averages their probabilities, and then uses them as the final predicted probability of biochemical recurrence of the case.

[0129] 5. Data management and storage module

[0130] The data management and storage module 6 is the data storage part of the system, including:

[0131] (a) Clinical data storage: storing patients’ clinical information and follow-up data;

[0132] (b) Image data storage: storing original MRI images and preprocessed images;

[0133] (c) Prediction result storage: save the prediction results and performance indicators of each model;

[0134] (d) Follow-up management submodule: The median follow-up time and the patient’s telephone number were recorded, patients who did not experience BCR or received additional treatment at the last follow-up were censored, and doctors were reminded of the need to follow up with the patients.

[0135] 6. Result output and decision support module

[0136] The result output and decision support module 7 is used to improve clinical intervention according to the results, including a risk assessment submodule and a decision suggestion submodule. Among them, the risk assessment submodule integrates the prediction results of each model and gives the patient's BCR risk assessment; the decision suggestion submodule automatically classifies the patient into a close interval follow-up group or a wide interval follow-up group according to the risk assessment results;

[0137] (a) Develop an individualized follow-up plan for the close interval follow-up group, including:

[0138] Follow-up was conducted every month within 1 year after surgery;

[0139] Follow-up every 3 months 2-3 years after surgery;

[0140] After 3 years, follow-up was conducted annually;

[0141] (b) Develop a personalized follow-up plan for the wide-interval follow-up group, including:

[0142] Follow-up was conducted every 6 months within 3 years after surgery;

[0143] After 3 years, the patients were followed up annually.

[0144] Figure 2 The flowchart of the system for predicting biochemical recurrence of prostate cancer is shown. The main steps are as follows:

[0145] (1) Analysis of clinical characteristics

[0146] Univariate and multivariate analyses were performed on the available clinical information; univariate analysis evaluated the independent relationship between each clinical feature and BCR, using statistical tests such as t-test and chi-square test to evaluate each feature value; multivariate analysis evaluated the relationship between multiple clinical features and BCR while controlling for the influence of other variables, mainly using backward stepwise regression method.

[0147] (2) Feature selection

[0148] Clinical information Input into a linearly mapped clinical feature extraction module to obtain clinical features . It can effectively extract clinical variable information related to BCR status and compress feature dimensions through linear transformation for subsequent feature fusion. PSA level, ISUP grade, pathological T stage, clinical T stage, tumor volume and PI-RADS score were selected as significantly related clinical features.

[0149] (3) Deep learning model development

[0150] Based on Swin Transformer, its training and verification are mainly based on MRI image data. The training and verification details are as described above. Using RGB three-channel images composed of T2WI, DWI, and ADC as input, the bpMRI image is input into the Swin transformer image feature extraction module to obtain image features: The ROI area was cropped, and a data enhancement strategy was used to enhance the robustness of the model. Then, the swin transformer was used to extract the fused image features of the three sequences. Finally, the fused features of the three sequences were sent to the fully connected (FC) layer and the SoftMax layer to obtain the biochemical recurrence prediction results based on bpMRI.

[0151] (4) Fusion model development

[0152] The proposed fusion model mainly consists of three parts: image feature extraction, clinical feature extraction and MCB feature fusion module. In order to fuse multimodal information, firstly, univariate and multivariate analysis of clinical information is performed to select the significant features. Then, these clinical characteristics are normalized to the maximum and minimum values ​​(Min MaxScaler) to normalize all clinical information to , and then the processed clinical features are extracted through the clinical feature extraction part.

[0153] Image feature extraction is based on the feature extraction part of the deep learning model based on Swin transformer that was previously established to extract the MRI imaging features of each patient. Multimodal Compact Bilinear Pooling (MCB) is a method of fusing multimodal feature vectors. It randomly projects the feature vectors of images and text into a high-dimensional space, and then effectively convolves the two vectors using element-wise product in the Fast Fourier Transform (FFT) space. This can better model the complex relationship between the two modalities and more effectively express the combination of multimodal features. and clinical features Input to the multimodal compact bilinear pool (MCB) for fusion to obtain the fused features The MCB method can capture the interactive information between different modal features and enhance the expressiveness of features through bilinear mapping, thereby generating a more representative fusion feature: .

[0154] Final model prediction:

[0155] The obtained fusion features Input to the classification head In the calculation , and get the final output result The classification head consists of a fully connected layer, which aims to map the extracted features to the probability distribution of each category, thereby outputting the BCR prediction probability.

[0156] The prediction performance of the system of the present invention and traditional clinical features was compared, see Table 2 below:

[0157] Table 2 Comparison of the predictive efficacy of the system of the present invention and traditional clinical features

[0158]

[0159] AUC (Area Under the Curve) refers to the area under the ROC curve and is a robust indicator for measuring the overall performance of the model. The higher the AUC, the stronger the model's ability to distinguish between positive and negative samples. The AUC value of the method of the present invention is 0.738, which is higher than the AUC values ​​of the traditional serum prostate-specific antigen (PSA), the Gleason Score system for prostate pathology, and the CAPRA-S scoring system, the most widely used prediction model in the world that integrates multiple clinical features. The system of the present invention has a significant improvement in predictive efficiency compared to other systems.

[0160] The present invention proposes an algorithm for predicting the risk of biochemical recurrence of prostate cancer by multimodal fusion of imaging images and clinical information. The algorithm achieves a comprehensive assessment of the risk of biochemical recurrence of prostate cancer after RP in patients by multimodally fusing the image information and key clinical information of bpMRI imaging data. The present invention adopts an advanced deep learning architecture to fully extract the characteristics of the patient's imaging data and multimodally fuses it with key clinical information such as PSA level and Gleason score. After rigorous experiments, this method has performed well in key performance indicators such as prediction accuracy and AUC.

[0161] The core advantage of the present invention is that it can make full use of the patient's bpMRI imaging data and clinical information. Through deep learning technology, the model can fully integrate multimodal information to achieve accurate prediction of biochemical recurrence. The 5-fold cross-validation and diversified data enhancement strategies adopted further enhance the robustness of the model. Experimental results show that this method can assist doctors in more accurate patient stratification and personalized follow-up plan formulation, potentially reducing the risk of overtreatment and missed diagnosis. Overall, the present invention provides an efficient and reliable AI solution for prostate cancer prognosis assessment, demonstrates great potential for application in clinical practice, and provides strong support for the advancement of precision medicine and personalized treatment strategy formulation.

[0162] The unexplained parts involved in the present invention are the same as the prior art or are implemented by the prior art. The applicant declares that the present invention illustrates the detailed method of the present invention through the above-mentioned embodiments, but the present invention is not limited to the above-mentioned detailed method, that is, it does not mean that the present invention must rely on the above-mentioned detailed method to be implemented. Those skilled in the art should understand that any improvement of the present invention, the equivalent replacement of the raw materials of the product of the present invention, the addition of auxiliary components, the selection of specific methods, etc., all fall within the protection scope and disclosure scope of the present invention.

Claims

1. A prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model, characterized in that: It includes input module, preprocessing module, feature selection module, fusion module, prediction module, data management and storage module, result output and decision support module. Wherein, the input module receives MRI images, complete follow-up information and clinical characteristic information based on certain rules; The preprocessing module includes an image preprocessing module and a clinical feature preprocessing module. The image preprocessing module performs external contour delineation and preliminary standardization processing on the MRI image, and the clinical feature preprocessing module processes clinical feature information. The feature selection module includes an image data selection module and a clinical feature selection module. The image data selection module is a deep learning model built based on swin transformer. The deep learning model uses RGB three-channel images composed of T2WI, DWI, and ADC as input to convert MRI images into Input it to get image features : ； The clinical feature selection module obtains clinical features by linear mapping ; The fusion module combines image features and clinical features Input to the multimodal compact bilinear pool MCB for fusion to obtain the fused fusion features : ； The prediction module is a deep learning model built based on Swin Transformer. Input to the classification head In the calculation, SoftMax is obtained to obtain the final output result. ; Classification Header It is composed of fully connected layers, which aims to map the extracted features to the probability distribution of each category, thereby outputting the predicted probability of prostate cancer: ； The result output and decision support module classifies patients into a close interval follow-up group or a wide interval follow-up group according to the risk assessment results, and formulates a personalized follow-up plan.

2. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 1, characterized in that: in, The input module selects patients who have undergone MRI examination before radical prostatectomy and have high-quality MRI images and complete follow-up information but have not received neoadjuvant therapy for the next step of evaluation, and excludes patients who lack key MRI sequences, have poor image quality, have received neoadjuvant therapy, have persistently elevated prostate-specific antigen (PSA) levels, or have incomplete clinical data from the next step of evaluation. The clinical characteristic information includes prostate specific antigen PSA level, ISUP grade, pathological T stage, clinical T stage and tumor volume.

3. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 2, characterized in that: in, The image preprocessing module calls ITK-snap software and other custom functions to manually or automatically outline the external contour of the prostate lesion on the axial slice; The image preprocessing module performs preliminary standardization processing on the image using the following methods: N4 deviation field correction, spatial registration, resampling, cropping and intensity normalization to reduce interference and noise; The clinical feature preprocessing module ensures the accuracy and reliability of the data set by implementing a data cleaning process, standardizing the data format, processing missing values, identifying and correcting outliers.

4. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 2, characterized in that: in, The clinical feature selection module performs maximum and minimum normalization on these clinical features, normalizing all clinical information to Then, the processed clinical features are extracted through linear mapping. .

5. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 1, characterized in that: in, The fusion module is selected from the MCB method, which randomly projects the image feature vector and the text feature vector of the clinical features into a high-dimensional space, and then expresses the combined multimodal features by effectively convolving the two vectors using element-wise product in the fast Fourier transform space.

6. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 1, characterized in that: in, The prediction module is first trained and verified in the early stage, and the specific method is as follows: (1) Batch size: 16; (2) Training rounds: 100; (3) Learning rate decay strategy: LambdaLR; (4) Optimizer: SGD, stochastic gradient descent; (5) Loss function: weighted cross entropy loss, weighted according to BCR: NO-BCR of 7:3 to deal with the problem of class imbalance; (6) Training method: 5-fold cross validation; (7) Dataset division: Divide into training set and test set in a ratio of 4:1, and then continue to divide the training set and validation set in a ratio of 3:1 on the training set; (8) Use early stopping strategy: monitor the accuracy on the validation set. If there is no improvement after 20 consecutive epochs, stop training early to prevent overfitting. Implement the models in version 2.0.1 of PyTorch and train them on an NVIDIA RTX 3070 GPU. Use an external validation set in real time to evaluate the performance of existing model predictions and use this as a basis for modification and parameter adjustment.

7. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 1, characterized in that: in, The prediction model selects the three sections with the highest prediction probabilities among all tumor sections of a patient, averages their probabilities, and then uses them as the final prediction probability of biochemical recurrence of the case.

8. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 1, characterized in that: in, The data management and storage module is used for clinical data storage, storage of original MRI images and preprocessed images, and storage of prediction results; a follow-up management submodule is provided in the data management and storage module to record the median follow-up time and the patient's telephone number, censor patients who did not develop BCR or receive additional treatment at the last follow-up, and remind doctors that they need to follow up on the patients.

9. The prostate cancer biochemical recurrence prediction system based on deep learning image and clinical information multimodal model according to claim 1, Features: The personalized follow-up plan developed by the close interval follow-up group includes: Follow-up was conducted every month within 1 year after surgery; Follow-up every 3 months 2-3 years after surgery; After 3 years, follow-up was conducted annually; The wide-gap follow-up group developed a personalized follow-up plan, including: Follow-up was conducted every 6 months within 3 years after surgery; After 3 years, the patients were followed up annually.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the function of the system as claimed in any one of claims 1 to 9 is realized.

Images