Multi-modal fusion and digital twinning driven liver cancer immunotherapy decision-making system and method

Abstract

The invention discloses a multi-modal fusion and digital twin driven liver cancer immunotherapy decision-making system and method, and relates to the technical field of medical image processing and computer vision, and the system comprises a data collection module which is used for obtaining a three-dimensional CT image, clinical variables, oncology characteristics and treatment information of a patient with non-resectable hepatocellular carcinoma; the image preprocessing module is used for carrying out registration, cutting, standardized resampling and enhancement processing on the multi-stage CT images to generate model input in a unified format; the multi-modal fusion prediction module is used for extracting image features by adopting three deep learning networks with complementary structures, fusing the image features with key prognostic variables and jointly predicting the total lifetime and the progression-free lifetime; the digital twinborn decision-making module is used for simulating survival tracks under various treatment paths based on an individualized virtual model and outputting optimal treatment recommendation; and the interactive application program is used for result display and recommendation generation. According to the invention, integration of multi-source heterogeneous data, survival result prediction and path simulation is realized.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, and more specifically, to a multimodal fusion and digital twin-driven decision-making system and method for liver cancer immunotherapy. It is applicable to prognostic prediction and individualized treatment decision support for patients with unresectable hepatocellular carcinoma undergoing combined immunotherapy, and falls within the application scope of artificial intelligence-assisted medical decision-making systems. Background Technology

[0002] Hepatocellular carcinoma (HCC) is a highly prevalent and deadly malignant tumor worldwide, especially for patients with unresectable, advanced-stage HCC, where prognostic assessment and treatment pathway selection present significant challenges in clinical practice. In recent years, immune checkpoint inhibitors (ICIs) combined with molecularly targeted therapy (MTT) or transarterial chemoembolization (TACE) combined with ICI+MTT regimens have become the main treatment strategies for patients with unresectable HCC. However, due to the heterogeneity of patients in tumor biology and treatment response, traditional assessment methods struggle to accurately predict individual survival prognosis, limiting the effective implementation of personalized treatment strategies.

[0003] Currently, commonly used clinical criteria for assessing response to solid tumors are mainly based on imaging findings and changes in tumor burden, which have limited predictive ability and are insufficient in identifying atypical response patterns specific to immunotherapy. Some radiomics methods attempt to extract high-dimensional image features from CT or MRI and construct survival prediction models, but most rely on only a single modality or a single model, failing to effectively integrate multi-source information such as clinical variables and treatment history, resulting in limited predictive performance and generalization ability. Furthermore, existing methods mostly employ static scoring mechanisms, lacking the ability to simulate dynamic changes in survival under different treatment pathways, leading to a lack of individualization and consistency in treatment recommendations.

[0004] In recent years, the rapid development of artificial intelligence and digital twin technologies has provided new opportunities for intelligent medical decision-making. Multimodal data fusion methods, by integrating heterogeneous information such as imaging, biochemical indicators, and treatment plans, can more comprehensively reflect the individual differences of patients. However, their current application in HCC immunotherapy is still focused on short-term efficacy assessment, lacking the ability to model long-term survival outcomes with precision. Meanwhile, although digital twin technology has been successfully applied in industry, it still faces challenges in medical scenarios, including the complexity of data fusion, the accuracy of individual modeling, and clinical interpretability. In particular, virtual simulation and personalized recommendations for HCC treatment pathways have not yet been achieved.

[0005] Therefore, there is an urgent need to develop an intelligent auxiliary decision-making system that integrates multimodal deep learning models and digital twin simulation mechanisms for survival prediction and treatment pathway selection in immunotherapy for patients with unresectable HCC, so as to improve the accuracy of prognostic assessment and the consistency of clinical decision-making and meet the core needs of precision medicine. Summary of the Invention

[0006] This invention provides a multimodal fusion and digital twin-driven decision-making system and method for hepatocellular carcinoma (HCC) immunotherapy, aiming to address the problems of insufficient accuracy in prognostic prediction, poor consistency in treatment decisions, and difficulties in optimizing individualized treatment pathways for HCC patients. The system and method integrate multi-source heterogeneous data, apply ensemble deep learning technology for survival prediction, and utilize digital twin technology to achieve dynamic simulation of treatment pathways, providing scientific decision support for clinical practice and applicable to precision medicine scenarios involving combined immunotherapy for HCC.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A decision-making system for liver cancer immunotherapy driven by multimodal fusion and digital twins includes:

[0009] The data acquisition module is used to collect patients' three-dimensional multi-phase enhanced CT images (including arterial phase, portal venous phase and delayed phase), as well as clinical variables (such as biochemical indicators, physical performance scores, virological status, etc.), oncological characteristics (such as number of lesions, tumor burden, portal vein tumor thrombus, etc.) and treatment information (including previous lines of treatment, type of treatment regimen, etc.).

[0010] The image preprocessing module is used to standardize the raw CT images. This module first uses the nnUNet model to automatically segment the liver for each phase of the image, and uses the portal venous phase as the registration reference, employing Elastix to register the arterial and delayed phase images to ensure spatial consistency across the three phases. Subsequently, the images are uniformly resampled to 1×1×1mm. 3 The images were taken at voxel resolution, and the contrast was enhanced by adaptive histogram equalization (CLAHE) and cropped to a standard size (128×128×128 voxels). The three images were stacked into a 4D tensor for use as input to subsequent deep learning models.

[0011] The multimodal fusion prediction module consists of three complementary deep learning networks: a supervised learning network based on the EfficientNet-B1 architecture, a semi-supervised autoencoder model with an encoder-bottleneck-decoder structure (combined with the DeepSurv survival prediction head), and a CNN-Transformer hybrid network combining local convolution and global Transformer mechanisms. These three networks extract image features at different structural depths in parallel and fuse them with key clinical variables selected using the LASSO method. The results are then input into a multimodal ensemble learning framework based on the random survival forest algorithm, outputting a combined vector representing the risk of overall survival (OS) and progression-free survival (PFS) for each individual.

[0012] The digital twin decision-making module initializes a virtual patient avatar using the survival risk score from the prediction module. The module dynamically simulates survival trajectories under two treatment pathways: ICI combined with MTT and transarterial chemoembolization (TACE) combined with ICI+MTT, calculating the median OS, median PFS, and survival probability changes for each pathway. Based on a preset survival difference threshold (e.g., 5% OS difference), the module compares the effects of different treatment regimens, selects the optimal pathway, and generates corresponding personalized treatment recommendations, overcoming the limitations of static scoring systems.

[0013] The interactive decision support platform presents predicted outcomes and treatment recommendations in a graphical format. This includes plotting OS and PFS survival curves, displaying confidence intervals, comparing the expected effects of different treatment pathways, and providing textual descriptions of treatment options. The application allows physicians to adjust input parameters in real time and dynamically update recommendation results, enhancing the flexibility and operability of clinical decision-making.

[0014] The decision-making method for liver cancer immunotherapy based on the above system includes the following steps:

[0015] Step 1: Obtain the patient's arterial phase, portal venous phase, and delayed phase CT images from the PACS system;

[0016] Step 2: Use the nnUNet model to automatically segment the liver in the three phases of the image. Using the portal venous phase as a reference, use Elastix to register the other phases and unify the image space.

[0017] Step 3: Resample the registered image to 1×1×1 mm voxels, apply CLAHE for contrast enhancement, and truncate the grayscale values ​​to -200 to 250 HU.

[0018] Step 4: Generate a 3D bounding box based on the liver mask and crop it to 128×128×128 voxels. Stack the three phase images to form a 4D tensor.

[0019] Step 5: Input the preprocessed image data into the following three types of deep neural networks to extract multi-level features and construct a survival risk representation;

[0020] Step 6: Integrate the above network output features and key clinical variables screened by LASSO, input them into the random survival forest, and output the survival risk vector;

[0021] Step 7: Construct a digital twin of the patient based on the survival vector to simulate the survival trajectory under the combined immune regimen and the combined TACE regimen. If the predicted difference is ≥5%, the one with the higher survival rate is recommended.

[0022] Step 8: Display the prediction results, survival curves, and recommended treatment plans through the interactive interface.

[0023] Beneficial effects:

[0024] First, this invention integrates multi-source heterogeneous data, including 3D enhanced CT images, clinical variables, oncological features, and treatment pathway information, through a data acquisition module. This overcomes the limitations of single-modal analysis commonly found in existing methods, providing a more comprehensive reflection of individual patient characteristics and laying a solid foundation for subsequent model predictions. Second, the multimodal fusion prediction module proposed in this invention employs an integrated deep learning strategy to jointly extract multidimensional features from images and clinical data, constructing a refined survival prediction model. This not only improves prognostic prediction accuracy but also compensates for the shortcomings of traditional standards (such as mRECIST) in identifying atypical responses to immunotherapy. Third, this invention introduces a digital twin decision-making module, constructing an individualized virtual model of the patient based on the aforementioned survival risk vector. This dynamically simulates the survival trajectory under ICI combined with MTT and TACE combined with ICI+MTT treatment pathways. By setting survival difference thresholds, the system achieves quantitative comparison and personalized recommendations between treatment pathways, overcoming the adaptive bottleneck of traditional static scoring mechanisms in dealing with patient heterogeneity and improving the scientific rigor and consistency of treatment recommendations. Fourth, this invention provides an interactive clinical decision support platform for visually displaying survival prediction results and recommended treatment pathways. It supports parameter adjustments, reassessments, and intervention simulations based on individual patient characteristics, enhancing the system's transparency, operability, and flexibility in real-world clinical environments. In summary, this invention exhibits good adaptability and scalability, suitable for HCC patient prognosis assessment, treatment pathway optimization, and stratified clinical trial design. It supports multi-center cohort data integration and personalized healthcare service deployment. Through in-depth multimodal data analysis and digital twin technology, this invention provides an intelligent and highly reliable decision support solution for precision medicine, particularly suitable for managing unresectable HCC with immunotherapy, demonstrating significant practical application value and clinical promotion potential. Attached Figure Description

[0025] Figure 1 This is a block diagram of the overall logic of the system of the present invention;

[0026] Figure 2 This is a flowchart of the image preprocessing module method in this invention;

[0027] Figure 3 This is a flowchart illustrating the workflow of the multimodal fusion prediction module in this invention.

[0028] Figure 4 This is a schematic diagram of the visual interactive decision support platform in this invention. Detailed Implementation

[0029] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. However, the following embodiments are only for explaining the present invention, and the scope of protection of the present invention should include all the contents of the claims. Moreover, through the description of the following embodiments, those skilled in the art can fully implement all the contents of the claims of the present invention.

[0030] This invention proposes a multimodal fusion and digital twin-driven decision-making system and method for liver cancer immunotherapy, specifically designed for predicting survival outcomes and recommending treatment pathways in patients with unresectable hepatocellular carcinoma after receiving combined immunotherapy. Based on real-world, multi-center clinical data, and combining imaging, clinical, pathological, and treatment information, this system achieves high-precision, highly interpretable, and dynamically adjustable intelligent decision support.

[0031] The system consists of five core modules: a data acquisition module, an image preprocessing module, a multimodal fusion prediction module, a digital twin decision-making module, and a visualization and interaction platform. Its overall system logic diagram is as follows: Figure 1 As shown, raw three-phase enhanced CT images are obtained from the PACS system and combined with structured clinical data from the EMR system. After standardization and feature extraction, the data are input into the deep learning prediction system. Digital twin technology is used to simulate the prognostic results under different treatment pathways. Finally, the recommended treatment plan and prediction results are displayed through a graphical interface.

[0032] In image preprocessing modules (such as...) Figure 2 To overcome the heterogeneity issues arising from different scanners and scanning protocols, this invention designs a standardized CT image processing workflow. First, three-phase enhanced CT images (arterial phase, portal venous phase, and delayed phase) are extracted from the PACS. Then, a pre-trained nnUNet model is used to automatically segment the liver in each phase, extracting the liver parenchyma region. Using the portal venous phase image as a spatial reference, Elastix software is used to perform three-dimensional non-rigid registration of the arterial and delayed phase images, achieving spatial alignment. Next, all images are spatially resampled to a 1×1×1 mm voxel resolution. Intensity images are interpolated using cubic interpolation, and mask images are interpolated using nearest neighbor interpolation. To further enhance image quality, the CLAHE algorithm is used for contrast enhancement, and the image HU values ​​are cropped to [-200, 250] to focus liver region information. Finally, a three-dimensional bounding box is constructed based on the segmentation mask, and the image is cropped to 128×128×128 voxels. The three-phase images are stacked to form a 4D tensor as model input.

[0033] The core of this invention is a multimodal fusion prediction module (such as...) Figure 3As shown in the diagram, its main objective is to comprehensively model the survival risk of patients after receiving immunotherapy by combining multi-source data. This module integrates four types of input data: ① Ensemble-DL depth features based on CT images; ② Clinical variables, including age, sex, alpha-fetoprotein level, liver function parameters, HBV infection status, and performance status; ③ Oncological characteristics, including BCLC stage, portal vein tumor thrombus, metastasis, number of tumors, maximum diameter, and tumor burden under the "up to seven" criteria; ④ Treatment information, including treatment regimen type, number of lines of treatment, and previous local treatment information. All variables are structured and then fed into the model for joint modeling.

[0034] To fully exploit the potential information in CT images, this invention constructs an Ensemble-DL module composed of three complementary 3D deep learning networks: First, a fully supervised network based on EfficientNet, employing an MBConv structure and SE attention mechanism, focusing on extracting local texture and structural features; second, a semi-supervised autoencoder network, whose structure includes an encoder, bottleneck layer, decoder, and DeepSurv survival prediction head, capable of simultaneously performing image reconstruction and survival risk estimation; and third, a CNN-Transformer hybrid network, which segments the image into axial, coronal, and sagittal views, uses 2D convolution to generate spatial embedding vectors, and combines this with a Transformer encoder to model global dependencies, thereby learning cross-sectional spatial structural information. All three networks employ a multi-task loss structure, predicting total survival and progression-free survival risk scores in parallel on two branches.

[0035] The risk vectors output by the three neural networks mentioned above are fed into a random survival forest ensemble model for fusion, generating a unified deep fusion image representation, named Ensemble-DL signature. To further improve the model's interpretability and generalization ability, this invention introduces the LASSO-Cox regression method to perform feature selection on all structured variables (clinical, tumor, and treatment information). The regularization parameter λ is optimized through 10-fold cross-validation to select a set of key variables significantly related to OS, which, together with the Ensemble-DL signature, form the final input vector. This vector is then input into the random survival forest model for survival risk modeling, outputting an accurate prognostic risk score.

[0036] The digital twin decision-making module is an extension of the MMF prediction module. Its goal is to achieve personalized treatment recommendations by simulating the differences in survival pathways under different treatment strategies (e.g., ICI+MTT vs. TACE+ICI+MTT). This module uses the risk vector output by the MMF system as the initial state to build a digital "patient clone" model. It then simulates the survival function under different pathways using a nonlinear Cox model or Bayesian inference methods. If the difference in predicted OS between two pathways exceeds a set threshold (e.g., 5%), the system automatically recommends the treatment pathway with higher survival benefit; if the difference is not significant, it recommends a plan with fewer side effects or lower economic costs, thus achieving clinically interpretable and selectable intelligent recommendations for multiple treatment options.

[0037] To support clinicians in operating and viewing results in real-world scenarios, this system is designed with... Figure 4 The platform is a visual, interactive application platform. It consists of three parts: an input parameter area, a prediction output area, and a result interpretation area. Users can manually input or import patient images and clinical data. The system automatically generates survival prediction results and recommended treatment pathways, and plots survival probability curves (including 95% confidence intervals) for different treatment plans. This helps physicians understand the key basis for the model's judgments, increasing clinical trust. All prediction results can be exported as structured reports, facilitating archiving in electronic medical record systems or use in multidisciplinary team (MDT) discussions.

[0038] The application results of this system in a multi-center cohort (Table 1) show that in both the multi-center external validation cohort (n=222) and the prospective cohort (n=106), the multimodal fusion system proposed in this invention significantly outperformed the comparative models in predicting both overall survival and progression-free survival. Specifically, in terms of overall survival prediction, the C-index of the multimodal fusion system was 0.77 (95% CI: 0.73-0.80) in the external testing cohort and reached 0.80 (95% CI: 0.75-0.85) in the prospective cohort, both higher than mRECIST (C-index of 0.55 and 0.60) and other deep learning-based single-network models, traditional clinical models, and ensemble learning models (highest being 0.72-0.73). Furthermore, the hazard ratios (HRs) of the multimodal fusion system in the two cohorts reached 2.36 (95% CI: 2.05–2.73) and 2.90 (95% CI: 2.22–3.79), respectively, demonstrating stronger risk discrimination ability. In terms of progression-free survival prediction, the C-index of the multimodal fusion system was 0.71 (95% CI: 0.68–0.74) in the external validation cohort and 0.74 (95% CI: 0.68–0.79) in the prospective cohort, also outperforming six comparative models, including mRECIST. Its corresponding HRs were 1.96 (95% CI: 1.72–2.22) and 1.90 (95% CI: 1.57–2.30), respectively, making it the best performing model among all models. Z-test results showed that the C-index differences between the multimodal fusion system and the other models were statistically significant (p < 0.05). This demonstrates that the MMF system, based on multimodal data fusion and integrated modeling strategies, not only performs well on training data, but also maintains good robustness and generalizability in independent multicenter datasets and real-world prospective clinical cohorts, making it valuable for deployment in various medical institutions.

[0039] Table 1. Performance comparison of the multimodal fusion system with six other prognostic models in predicting overall survival and progression-free survival.

[0040] Note: The values ​​in parentheses are 95% confidence intervals. "" indicates a comparison with the MMF system. The p-value was calculated using a two-tailed z-test.

[0041] In summary, this invention constructs an integrated prediction system that combines deep features of multimodal data with structured clinical information and introduces a digital twin dynamic simulation mechanism, achieving accurate prediction and personalized recommendations for the survival outcomes of patients undergoing immunotherapy for liver cancer. This system possesses high accuracy, strong robustness, good interpretability, and deployment flexibility, making it suitable for various scenarios such as clinical decision support, treatment pathway evaluation, and prospective trial design, demonstrating significant research value and practical application prospects.

[0042] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention.

Claims

1. A multimodal fusion and digital twin-driven decision-making system for liver cancer immunotherapy, characterized in that, include: The data acquisition module is used to acquire three-dimensional CT images, clinical variables, oncological features, and treatment information of patients with unresectable hepatocellular carcinoma. The image preprocessing module is used to standardize multi-phase CT images, including automatic liver segmentation, phase registration, image enhancement, intensity truncation, volume cropping, and 4D stacking. The multimodal fusion prediction module consists of three parallel deep learning networks: a supervised model based on the EfficientNet architecture, a semi-supervised autoencoder model with an encoder-decoder structure, and a convolution-Transformer hybrid model. These networks are used to extract image features and fuse them with key clinical variables to output a survival risk vector. The digital twin decision-making module is used to build a personalized virtual avatar, simulate the prognostic trajectory under different treatment options, and recommend the optimal treatment path; An interactive decision support platform for visually displaying prediction results and recommended solutions.

2. The multimodal fusion and digital twin-driven decision-making system for liver cancer immunotherapy according to claim 1, characterized in that, The image preprocessing module specifically includes: The nnUNet model was used to automatically segment the liver in arterial, portal venous, and delayed phase CT images. Using the portal venous phase as a reference, images of other phases were registered using Elastix; The image was resampled to a 1×1×1 mm voxel resolution, and the contrast was enhanced by applying adaptive histogram equalization CLAHE. The gray values ​​were truncated to -200 to 250 HU. The liver image was cropped to 128×128×128 voxels and stacked with three phases of CT images to form a 4D tensor.

3. The multimodal fusion and digital twin-driven decision-making system for liver cancer immunotherapy according to claim 1, characterized in that, The three deep learning networks of the multimodal fusion prediction module are as follows: First sub-network: A three-dimensional convolutional neural network based on the EfficientNet B1 architecture, integrating the MBConv module and channel attention mechanism, used for multi-task prediction of total survival and progress-free survival risk; The second sub-network is an autoencoder model with an encoder-bottleneck-decoder structure, which combines a DeepSurv prediction head and performs feature learning by jointly optimizing the image reconstruction loss and the Cox proportional risk loss. The third sub-network is a hybrid convolutional-transformer architecture that extracts local features through 3D convolution and then learns global spatial dependencies through a Transformer encoder.

4. The multimodal fusion and digital twin-driven decision-making system for liver cancer immunotherapy according to claim 1, characterized in that, The multimodal fusion prediction module further includes: The LASSO-Cox algorithm was used to screen key clinical variables and then fused with imaging features. Multimodal data are integrated using a random survival forest model to output a survival risk vector.

5. The multimodal fusion and digital twin-driven decision-making system for liver cancer immunotherapy according to claim 1, characterized in that, The digital twin decision-making module is specifically used for: Initialize the virtual patient model based on the survival risk vector; Survival trajectories under two pathways: simulated immunotherapy combined with targeted therapy and transarterial chemoembolization (TACE) combined with immunotherapy and targeted therapy. If the predicted difference is ≥5%, the option with a higher survival rate is recommended.

6. The multimodal fusion and digital twin-driven decision-making system for liver cancer immunotherapy according to claim 1, characterized in that, The interactive decision support platform includes: Survival curve visualization; Comparison of treatment options and display of confidence intervals; It supports clinicians in adjusting parameters and dynamically updating recommendation results.

7. A method for making decisions on immunotherapy for liver cancer that implements the system described in any one of claims 1-6, characterized in that, Includes the following steps: Step 1: Obtain the patient's arterial phase, portal venous phase, and delayed phase CT images from the system; Step 2: Use the nnUNet model to automatically segment the liver in the three phases of CT images. Using the portal venous phase as a reference, use Elastix to register images in other phases to unify the image space. Step 3: Resample the registered image to 1×1×1 mm voxels, apply CLAHE for contrast enhancement, and truncate the grayscale values ​​to -200 to 250 HU. Step 4: Generate a 3D bounding box based on the liver mask and crop it to 128×128×128 voxels. Stack the three phase CT images to form a 4D tensor. Step 5: Input the preprocessed image data into the deep neural network to extract multi-level features and construct a survival risk representation. Step 6: Integrate the above network output features and key clinical variables screened by the LASSO-Cox algorithm, input them into the random survival forest model, and output the survival risk vector; Step 7: Construct a digital twin of the patient based on the survival risk vector to simulate the survival trajectory under two pathways: immune combined with targeted therapy and transarterial chemoembolization (TACE) combined with immune targeted therapy. If the predicted difference is ≥5%, the one with the higher survival rate is recommended. Step 8: Display the prediction results, survival curves, and recommended treatment plans through the interactive interface.

8. The decision-making method for liver cancer immunotherapy according to claim 7, characterized in that, In step 5, the training of the deep neural network uses a multi-task loss function.

9. The decision-making method for liver cancer immunotherapy according to claim 7, characterized in that, Step 7: The simulation of the digital twin model is based on a nonlinear Cox model or a Bayesian inference method.

10. The decision-making method for liver cancer immunotherapy according to claim 7, characterized in that, In step 8, the interactive interface supports exporting structured reports for electronic medical record archiving or multidisciplinary consultations.

Images