A liver cancer postoperative recurrence prediction device based on a multi-modal transformer

Abstract

The application discloses a liver cancer postoperative recurrence prediction device based on a multi-modal Transformer. The device comprises a data acquisition module for acquiring multi-modal data of liver cancer patients; a multi-modal fusion prediction module for inputting the multi-modal data into a trained liver cancer multi-modal recurrence prediction model to output the probability of liver cancer postoperative recurrence. The model comprises a visual backbone network, a time series Transformer, a clinical feature encoding branch combined with a medical prior gate, and a cross-modal cross-attention fusion module. The application solves the core pain points in the prior art, such as excessive dependence on a single mode in liver cancer prediction, lack of medical prior guidance, and inability to process variable-length high-dimensional sequences, and realizes high-precision, strong-explainability intelligent prediction of liver cancer early recurrence risk 12 months after operation and auxiliary clinical decision-making.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of medical artificial intelligence and medical image analysis, specifically involving a liver cancer postoperative recurrence prediction device based on multimodal Transformer, which is particularly suitable for accurate prediction and engineering-assisted decision-making scenarios of early recurrence risk in hepatocellular carcinoma (HCC) patients 12 months after surgery. Background Technology

[0002] Hepatocellular carcinoma (HCC) is one of the leading causes of cancer-related deaths worldwide. Although radical hepatectomy is currently the preferred treatment for HCC, the extremely high recurrence rate (especially early recurrence within 12 months post-surgery) severely limits long-term survival benefits for patients. Accurate prediction of postoperative recurrence risk in liver cancer is of significant clinical engineering importance for developing personalized adjuvant therapy, optimizing follow-up strategies, and improving patient prognosis.

[0003] Currently, there are two main conventional technical solutions in the field of liver cancer recurrence detection: ① Traditional manual clinical assessment: This solution relies heavily on the subjective interpretation of patients' clinical test reports and medical images (such as enhanced MRI) by professional physicians. This solution is not only inefficient, but also suffers from subjective experience differences in human judgment, making it difficult to fully explore the high-dimensional heterogeneous information hidden in the imaging. ② Shallow machine learning or single-modal deep learning solutions: These typically only utilize clinical structured data (such as tumor size, age, etc.) to construct logistic regression or random forest models, or simply use 3D convolutional neural networks (CNNs) to identify medical images.

[0004] However, the aforementioned existing technologies still have significant limitations, specifically: ① Insufficient single-modal analysis capability: The biological behavior of liver cancer is extremely complex, and relying solely on clinical indicators or imaging alone cannot fully reflect the tumor characteristics. Existing single-modal networks cannot achieve deep coupling of heterogeneous data (high-dimensional continuous images and low-dimensional discrete parameters). ② Lack of constraints from prior medical knowledge: Existing multimodal solutions often perform simple feature concatenation, making the model susceptible to data noise interference and ignoring features that play a decisive role in medical consensus (such as microvascular invasion (MVI) and alpha-fetoprotein (AFP)), resulting in poor robustness and interpretability of the model in complex clinical environments. ③ Excessive computational overhead of network architecture: Patient MRIs contain multiple phases and variable-length slices (e.g., tens to hundreds of slices), and traditional 3D CNNs consume enormous amounts of memory, making end-to-end large-scale parallel training difficult and adapting to clinical deployment extremely challenging.

[0005] Therefore, existing technologies cannot effectively address the core pain points of feature fusion difficulties, weak interpretability, and high computational costs in the detection of postoperative recurrence of liver cancer. Summary of the Invention

[0006] To address the aforementioned technical challenges, this invention provides a liver cancer postoperative recurrence prediction device based on a multimodal Transformer. It aims to leverage the core capabilities of a large multimodal model to construct a standardized and highly accurate recurrence prediction framework. By deeply integrating prior medical knowledge into neural network gating, it achieves joint modeling of multi-phase MRI sequences and clinical multi-omics data, thus resolving the core pain points of existing technologies.

[0007] A multimodal Transformer-based device for predicting postoperative recurrence of liver cancer includes: The data acquisition and preprocessing module is used to acquire multimodal data of patients after hepatocellular carcinoma surgery. The multimodal data includes multiphase enhanced MRI image sequences and structured clinical data. The module performs resampling, standardization and normalization processing on the multiphase enhanced MRI image sequences and data cleaning and discretization processing on the structured clinical data. A multimodal fusion prediction module is used to run a pre-trained multimodal recurrence prediction model for liver cancer. This model uses a visual backbone network to extract slice-level feature sequences from preprocessed MRI image sequences at each axial position. Then, a Transformer temporal encoder is used to extract the spatial context relationships between continuous axial slice-level feature sequences to obtain global image tokens for each phase of the MRI image sequence. Finally, image summary features are obtained by aggregating slice-level feature sequences from MRI image sequences at the same phase. After aggregating image summary features from all phases of the patient, a global CLS fusion token is obtained. A clinical feature encoding network maps preprocessed structured clinical data to multidimensional clinical feature tokens. A feature importance gating module performs adaptive weighting of these tokens based on clinical priors to obtain weighted clinical tokens. A cross-modal Transformer fusion layer fuses the global image tokens, weighted clinical tokens, and global CLS fusion tokens from all phases to obtain a multimodal joint representation. This multimodal joint representation is input into a recurrence risk prediction network, ultimately outputting the predicted probability of postoperative recurrence risk for liver cancer. The interpretability analysis and decision support module is used to automatically trigger the SHAP interpretation algorithm to calculate the contribution of each structured clinical feature when the predicted probability reaches the high-risk warning threshold, and to start the 3D Grad-CAM algorithm to generate a three-dimensional lesion heat map, pushing the clinical feature contribution analysis and the imaging lesion heat map as visual evidence to clinicians. The training of the multimodal recurrence prediction model for liver cancer uses a total loss function, which is a weighted sum of cross-entropy loss and auxiliary regularization loss. The formula for calculating the auxiliary regularization loss is as follows:

[0008] in, The mean of gated activations for key features. The mean of the gated activations for non-critical features is the mean of the gated weights after activation mapping using the Sigmoid function, mathematically defined as:

[0009]

[0010] In the formula, For feature index set, A subset of key features is the number of key features, and the constant 0.5 is the maximum tolerance threshold for allowing the weights of non-key features to exceed the weights of key features.

[0011] Furthermore, the multimodal recurrence prediction model for liver cancer includes: A pre-trained ViT-B / 16 visual backbone network is used to extract each MRI image from the input multi-phase enhanced MRI image sequence and output a slice-level feature sequence. The temporal encoder consists of two multi-head attention Transformer Encoders with four attention heads and a dropout rate of 0.2. It includes independent first and second slice sequence encoding branches, which do not share parameters. These branches encode MRI image sequences with different numbers of slices. Before inputting the slice-level feature sequences into the slice sequence encoding network, a learnable one-dimensional positional code is added to the slice-level feature sequences to characterize the positional information of each axial slice in the spatial depth direction, resulting in a global image token for each phase of the MRI image sequence. Each slice-level feature sequence is arranged in axial spatial order within the same MRI image sequence to form a spatial-level slice feature sequence. A classification identifier (CLS) token is added to the starting position of the spatial-level slice feature sequence to aggregate the global information of the spatial-level slice feature sequences, obtaining the image summary features of each phase of the spatial-level slice feature sequence. The image summary features of all phases for each patient are merged into a global CLS fusion token. The clinical feature encoding module discretizes continuous variables in various clinical indicators in structured clinical data according to the clinical median or a preset medical threshold to obtain multi-level labels; based on the discretized labels, it constructs an independent embedding vocabulary for each clinical indicator, maps each clinical indicator to a multi-dimensional dense vector, and performs L2 normalization on the dense vector to obtain the clinical feature embedding matrix. The feature importance gating module is used to introduce prior clinical constraints for liver cancer, designating four core pathogenic indicators—tumor number, tumor size, microvascular invasion (MVI), and alpha-fetoprotein (AFP)—as critical features, while the rest are non-critical features. A learnable gated Logits vector is set for each clinical feature index i∈{1,2,...,26}. The gated Logits vector is initialized as follows:

[0012] The gating weights are obtained through the Sigmoid function. :

[0013] Gated weight tensor The dimension is , where F=26 is the number of clinical features; the gating weight tensor and the original clinical feature embedding matrix are multiplied element-wise along the feature dimension through a broadcast mechanism to obtain the weighted clinical token; A cross-modal Transformer fusion layer is used to concatenate the global image token, weighted clinical token, and global CLS fusion token of all phases along the sequence dimension to form a high-dimensional sequence. The cross-modal Transformer fusion layer contains two Transformer layers, eight attention heads, and a feedforward network dimension of 1024. It uses a self-attention mechanism to mine the deep correlation between image features and clinical features and outputs the multimodal joint representation corresponding to the global CLS fusion token. The prediction output layer, composed of a fully connected network, maps the multimodal joint representation to Logits of the recurrence probability after liver cancer surgery, and outputs a binary classification of the recurrence probability.

[0014] Furthermore, during the fine-tuning of the multimodal recurrence prediction model for liver cancer, the parameters of the Patch Embedding layer and the first 8 Transformer Block layers of the pre-trained ViT-B / 16 visual backbone network are frozen, and only the parameters of the last 4 layers are opened for fine-tuning; the temporal encoder, cross-modal Transformer fusion layer, and fully connected network adopt the base learning rate. Weight decay The embedding table and bias parameters of the feature importance gating module are grouped into an independent parameter group. The bias parameters are learnable Logits vectors with initial values ​​of 4.0 and -2.0, which are allowed to be updated during the backpropagation process of training, and the learning rate is increased to... The weight decay is set to 0; during training, non-critical features are randomly discarded with a 30% probability; the AdamW optimizer is used, the training epochs are 100, the batch size is set to 16, an early stopping strategy is adopted, and the area under the ROC curve on the validation set is monitored. If the AUC does not improve for 10 consecutive epochs, early stopping is triggered.

[0015] Furthermore, the multimodal data is preprocessed to generate training data as follows: For each slice in the multi-phase enhanced MRI image sequence, bilinear interpolation was used to uniformly resample to an image size of 224×224; the mean and standard deviation were calculated based on a single complete three-dimensional sequence of a single patient, Z-score normalization was performed, and pixel intensity was truncated within ±5 times the standard deviation range, and then linearly mapped to... Interval; the single-channel grayscale image is copied into three channels, and each channel is normalized according to the standard mean and variance of the ImageNet dataset; Features with a missing rate exceeding 20% ​​in the structured clinical data are removed, and the missing values ​​of the retained low-missing-rate features are processed using the global mode imputation method, thus converting the clinical data into discrete labels.

[0016] Furthermore, the pre-trained ViT-B / 16 visual backbone network segments the input multi-phase enhanced MRI images into 16×16 pixel image patches, and models the global dependencies between image patches through multi-layer Transformer Blocks, outputting slice-level feature sequences. Each patient was acquired with 20 multiphase enhanced MRI image sequences, including 4 36-slice axial slice sequences and 16 72-slice axial slice sequences; the first slice sequence coding branch network and the second slice sequence coding branch network of the temporal encoder were modeled independently and did not share parameters, so as to adaptively match the 36-slice and 72-slice axial slice sequences for temporal modeling respectively. The structured clinical data collected included 26 indicators, among which tumor number, tumor size, microvascular invasion, and alpha-fetoprotein were designated as key features. An independent embedding vocabulary was constructed for each indicator, with the vocabulary dimensions set to [missing value]. The original discretized labels are mapped to 256-dimensional dense vectors. In the cross-modal Transformer fusion layer, the sequence splicing order is: 20 global image tokens, 26 weighted clinical tokens, and 1 global CLS fusion token, with a sequence length of 47 and a uniform dimension of 256. The prediction output layer uses the Sigmoid activation function to output the early relapse probability.

[0017] Furthermore, it also includes model interpretability analysis and engineering-aided decision output steps: The trained model is encapsulated as a localized inference interface, which receives NIfTI format MRI images and structured clinical forms uploaded from the front end and outputs the recurrence probability. When the recurrence probability is ≥0.5, the decision interpretation module is triggered, and the SHAP algorithm is used to calculate the contribution of 26 clinical features to the prediction results and generate a feature importance ranking map. The 3D Grad-CAM algorithm is used to backpropagate the gradient to the last feature map of the cross-modal Transformer fusion layer or the visual backbone, and generate a three-dimensional heat map for each independent MRI sequence to accurately locate the focal areas in the MRI images that are related to the high-risk judgment. Clinical feature contribution maps and image heatmaps are pushed to the front end to assist clinicians in developing personalized intervention plans.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. Overcoming the bottleneck of single-modal accuracy and achieving deep coupling of heterogeneous data. By constructing a cross-modal attention mechanism, the barrier between 3D dense image features and 1D discrete clinical phenotypic data is broken down. The system is no longer limited to shallow feature stitching, but achieves multimodal joint representation in a high-level semantic space, significantly improving the AUC prediction accuracy and recall rate for early recurrence of liver cancer within 12 months.

[0019] 2. This invention innovatively introduces a medical prior gating mechanism to overcome the weakness of interpretability in "black box" models. The invention's unique feature importance gating combined with Importance Reg Loss forces deep learning models to prioritize clinically proven strongly correlated variables such as microvascular invasion (MVI) and alpha-fetoprotein (AFP). This not only prevents overfitting to massive amounts of noisy features but also allows the learned weights to be directly used in clinical SHAP attribution analysis, enhancing the algorithm's credibility and safety in real-world medical scenarios.

[0020] 3. A unique dimensionality reduction and aggregation network architecture significantly lowers the computing power threshold for industrial deployment. This invention employs a design of "frozen 2D backbone network for extracting slice features + sequence Transformer temporal aggregation," replacing the traditional memory-intensive 3D CNN network. This architecture can utilize a chunk inference strategy to achieve millisecond-level end-to-end inference of patient-level data containing thousands of slices on ordinary consumer-grade GPUs, providing substantial technical support for the large-scale deployment of clinical diagnostic software (CAD). Attached Figure Description

[0021] Figure 1 This is a flowchart of the overall method of the present invention; Figure 2 A schematic diagram of the architecture of a multimodal recurrence prediction model for liver cancer; Figure 3 This is a structural diagram of a feature importance-based gating module for medical priors. Figure 4 Heatmap for clinical attribution and interpretability analysis of the model (Grad-CAM); Figure 5 This is to distinguish the technical effects of the method of the present invention from those of the prior art. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Terminology Explanation: T1-weighted imaging (T1WI) is a type of magnetic resonance imaging (MRI) that acquires images by setting short repetition times (TR) and echo times (TE). In this sequence, the signal intensity of the image primarily reflects the longitudinal relaxation time between tissues. The difference in longitudinal magnetization vector recovery rate is due to the different rates of recovery of different tissues after being excited by radio frequency pulses. A short TR setting can effectively amplify this difference in recovery rate, thereby obtaining images with high anatomical contrast. In T1WI images, adipose tissue and tissues with high protein content usually show high signal, while tissues with high water content, such as water and cerebrospinal fluid, show low signal.

[0024] T2-weighted imaging (T2WI) is a type of magnetic resonance imaging (MRI) that acquires images by setting a longer repetition time (TR) and echo time (TE). In this sequence, the signal intensity of the image primarily reflects the transverse relaxation time between tissues. The difference in spin-spin interactions between different tissues is revealed by the long TE setting, which allows the differences in spin-spin interactions between tissues to be fully reflected during the lateral magnetization vector decay process. In T2WI images, pathological tissues with high water content (such as edema, inflammation, and tumor tissue) and cerebrospinal fluid usually show high signal intensity, while dense cortical bone, air, and calcified tissues show low signal intensity.

[0025] In medical imaging, the axial view (also known as the transverse section) is a cross-section that is scanned perpendicular to the long axis of the human body (the line connecting the head and the feet).

[0026] Axial slice sequence refers to a sequence in which 36 or 72 transverse scans of the liver are performed during a specific scanning phase, generating 36 or 72 consecutive axial slices, which are then combined to form a complete three-dimensional sequence.

[0027] Example This embodiment uses the prediction of early recurrence risk 12 months after surgery in hepatocellular carcinoma (HCC) patients as an application scenario, and fully implements the detection method described in this invention. Figure 1 As shown, a method for predicting postoperative recurrence of liver cancer based on multimodal Transformer is described, and the specific implementation steps are as follows: S1. Multimodal medical data acquisition and preprocessing Multimodal data of patients after hepatocellular carcinoma (HCC) surgery were collected using the hospital's PACS and electronic medical record (EMR) systems. Specifically, this included the following two parts of data: Medical imaging data: For each patient, 20 multi-phase enhanced MRI image sequences were acquired, specifically including 4 36-slice axial slice sequences and 16 72-slice axial slice sequences. The multi-phase sequences covered T1WI, T2WI, arterial phase (AP), portal venous phase (PVP), and delayed phase (DP). Each of the five different phases contained four different label images (label1: original image, label2: tumor area mask image, label3: peritumoral area mask image, label4: whole tumor mask image). The 36-slice axial slice sequences were primarily T2WI, while the 72-slice axial slice sequences primarily included T1WI, AP, PVP, and DP. The main difference between the two lies in slice thickness and scan resolution.

[0028] The acquired images underwent preprocessing as follows: First, bilinear interpolation was used to resample all slices to a uniform image size of 224×224 (the original resolution was 320×260). Second, to eliminate baseline differences in signal intensity caused by different patients and scanning devices, the mean and standard deviation were calculated based on a single complete 3D sequence of a single patient, Z-score normalization was performed, and pixel intensity was truncated to within ±5 times the standard deviation. Subsequently, pixel values ​​were linearly mapped to the [0,1] interval. Finally, to adapt to natural image pre-trained models (such as ViT), a channel duplication strategy was adopted, copying the normalized single-channel grayscale image three times in the channel dimension to generate a pseudo-color three-channel image (i.e., R=G=B=original grayscale value), and normalizing it according to the RGB mean and variance of ImageNet to match the distribution space of the underlying pre-trained parameters.

[0029] Structured clinical data: This embodiment collected 26 key indicators, covering continuous and discrete variables such as age, sex, number of tumors, tumor size, BCLC stage, tumor capsule, cirrhosis, microvascular invasion (MVI), satellite nodules, Edmondson-Steiner classification, HBsAg, albumin, total bilirubin, ALT, AST, platelets, hemoglobin, prothrombin time, alpha-fetoprotein (AFP), portal hypertension, hypertension, diabetes, operation time, intraoperative blood loss, intraoperative transfusion, and Child-Pugh classification. Among them, tumor number, tumor size, microvascular invasion (MVI), and alpha-fetoprotein (AFP) were set as four core key features.

[0030] Data cleaning and preprocessing were performed on the aforementioned key indicators. During this stage, a quality control interception strategy was first implemented: clinical features with a missing value rate exceeding 20% ​​were directly removed and excluded from subsequent modeling; only the retained features with low missing values ​​were imputed using the global mode method to ensure the statistical rationality of the data distribution. Secondly, different types of variables were labeled: for features such as gender, tumor size, and AFP, their original binary labels (0 and 1) were directly retained; for continuous indicators such as age, discretization was performed based on the clinical median or medical guidelines (e.g., dividing age into two levels with 55 years as the cutoff).

[0031] After the above data cleaning and preprocessing, 736 out of the 756 patients in the original dataset were successfully included. The final dataset was randomly divided into training and test sets in a 7:3 ratio.

[0032] S2. Construction of a multimodal recurrence prediction model for liver cancer and hyperparameter fine-tuning. This invention constructs the ViT-Multimodal-Recurrence network based on the PyTorch deep learning framework, such as... Figure 2 This system is used to fuse multi-phase MRI imaging sequences with structured clinical data to predict the risk of recurrence after HCC surgery. It specifically includes the following sub-modules: For the preprocessed multi-phase enhanced MRI image sequences in S1, this invention uses a pre-trained ViT-B / 16 model as the visual backbone network. To retain general image feature extraction capabilities and reduce computational consumption, the specific strategy in this embodiment is as follows: freeze the parameters of the ViT-B / 16 Patch Embedding layer and the first 8 Transformer Block layers, and only open the top 4 layers (Transformer Blocks 9 to 12) for fine-tuning. This operation preserves the model's ability to extract basic image textures, while allowing higher layers to focus on learning high-level lesion semantics specific to medical images. The input 224×224 resolution image is segmented into multiple 16×16 pixel image patches. After forward propagation through the backbone network, a set of features is extracted for each MRI image, and the feature sequence of each axial slice is output, thus obtaining the slice-level feature sequence.

[0033] To further extract the dynamic enhancement patterns of MRI image sequences in the temporal dimension (i.e., the enhancement characteristics of lesions in different phases) and model the spatial context relationships between consecutive axial slices within the same MRI image sequence, this invention introduces independent first slice sequence encoding branch networks and second slice sequence encoding branch networks. The two temporal branches do not share parameters and model independently to adapt to different resolution features. The Transformer's self-attention mechanism naturally supports variable-length sequence inputs, thus adapting to MRI image sequences with different numbers of slices. Before the MRI image sequence is input into the Encoder, a learnable one-dimensional positional encoding is added to the slice sequence to characterize the absolute physical depth of each slice in the axial direction. This encoding is used to adaptively match MRI image sequences containing 36 axial slices and MRI image sequences containing 72 axial slices, enabling the ViT-B / 16 model to recognize the sequential order and spatial depth relationships between different slices. Each independent slice sequence encoding branch network employs a two-layer multi-head attention Transformer Encoder (2 layers, 4 attention heads, and a Dropout rate of 0.2), with each layer having 4 attention heads. The multi-head mechanism enables the ViT-B / 16 model to capture dynamic information in different subspaces: some heads focus on the rapid local enhancement of lesions in the arterial phase, some focus on the overall contrast agent clearance in the delayed phase, and some extract the spatial continuity of tumor edges across slices. The first-layer encoder is used to perform preliminary aggregation of local spatiotemporal neighborhood features, and the second layer further extracts higher-order global temporal dynamic enhancement patterns. Each layer of slice-level feature sequences is composed of spatial-level slice feature sequences according to their spatial order on the axis. These sequences are input into the corresponding slice sequence encoding branch network to learn the global dependencies between adjacent and non-adjacent axial slices, resulting in global image tokens for each spatial-level slice feature sequence.

[0034] This invention adds a classification identifier (CLS Token) at the beginning of each spatial slice feature sequence to aggregate global information of each spatial slice feature sequence during the multi-head attention computation of the Transformer Encoder. This is achieved through the Transformer's internal multi-head self-attention mechanism. The CLSToken acts as a global aggregation node, absorbing effective lesion information from each slice through a query-key-value attention mechanism, thereby globally compressing and aggregating the dynamic features in the variable-length slice sequence. The final output CLSToken state serves as the high-dimensional image summary feature. High-dimensional image summary features are extracted from the CLS Token, ultimately transforming each patient's 20 MRI image sequences into 20 image feature vectors with a dimension of 256.

[0035] For the 26 collected structured clinical indicators (including discrete and continuous variables), this invention converts each indicator into a 256-dimensional clinical feature token. Since the clinical indicators include both discrete and continuous variables, this invention first classifies the continuous variables according to their clinical statistical distribution, unifying all indicators into three levels of labels: 0, 1, and 2. Therefore, this invention constructs an independent embedding vocabulary for each indicator, with the vocabulary dimensions set to... The original discrete labels (levels 0, 1, and 2) are mapped to 256-dimensional dense vectors, and L2 normalization is performed to constrain the feature representation space and prevent dominant bias caused by gradient magnitude differences between clinical features and high-dimensional imaging features during subsequent cross-modal fusion. The grading thresholds for continuous variables are independently determined based on clinical statistical distributions. Specifically, the `torch.nn.Embedding` module is called based on the PyTorch framework to construct a lookup table to map discrete labels to 256-dimensional dense vectors.

[0036] We further designed a Feature Importance Gate module to adaptively weight 26 clinical indicators: Let the feature index set be The key feature subset is K (containing four recognized core pathogenic indicators of liver cancer recurrence: tumor number, tumor size, microvascular invasion (MVI), and alpha-fetoprotein (AFP), which are the clinical prior constraints in the liver cancer recurrence prediction task). A learnable gated Logits vector is defined. To explicitly introduce medical prior knowledge in the early stages of model training, this embodiment initializes the gated Logits vector as follows:

[0037] Indicates the first The gated Logits value corresponding to each clinical feature.

[0038] No. The gated Logits values ​​corresponding to each clinical feature are used to generate a gated weight tensor using a Sigmoid activation function. :

[0039] like Figure 3 Gated weight tensor The dimension is Where F=26 represents the number of clinical features. The initial Logits bias +4.0 for key features, after Sigmoid mapping, results in a gating weight of approximately 0.98, while the weight for non-key features, after -2.0 mapping, is approximately 0.12. This allows the ViT-B / 16 model to focus more on key clinical factors such as tumor number, tumor size, microvascular invasion (MVI), and alpha-fetoprotein (AFP) during the initial training phase, while still allowing other non-key clinical features to be adaptively adjusted based on the real sample distribution during subsequent training.

[0040] Finally, based on the gated weight tensor Embedding matrix of original clinical features After weighting, we obtain the weighted clinical token:

[0041] In the formula, The original clinical feature embedding matrix (i.e., clinical feature tokens, with dimensions of 10 ... (B is the batch size, F=26 is the number of clinical features, and D=256 is the embedding dimension). This represents element-wise broadcast multiplication along the feature dimension.

[0042] This gating mechanism enables adaptive weighted control of information flows with different clinical characteristics, resulting in a weighted clinical token. .

[0043] During training, non-critical features are randomly discarded with a 30% probability, forcing the model to prioritize key prior indicators even when some clinical information is missing.

[0044] This invention employs a cross-modal Transformer fusion layer to achieve deep interaction between image and clinical features. The global image tokens of the aforementioned 20 MRI images, 26 gated and weighted clinical tokens, and one global CLS fusion token used for final classification are concatenated along the sequence dimension to form a high-dimensional sequence of length 47 (with a uniform dimension of 256). This sequence is then input into a cross-modal Transformer fusion layer (2 layers, 8 attention heads, and a feedforward network dimension of 1024), utilizing a self-attention mechanism to uncover the deep correlation between image and clinical features. The global CLS token features output by the fusion layer are treated as a multimodal joint representation, connected to a fully connected network (MLP) and the output layer, ultimately outputting the probability of early recurrence of liver cancer after surgery (Logits, binary classification).

[0045] To ensure efficient model convergence and excellent generalization ability, training is accelerated using distributed data parallelism (DDP) and mixed precision (AMP). The total loss function is a weighted sum of cross-entropy loss and auxiliary regularization loss, and the auxiliary loss is calculated as follows:

[0046] Here, the constant 0.5 represents the maximum tolerance threshold for non-critical feature weights to exceed critical feature weights. The mean of gated activations for key features. The mean of the gated activations for non-critical features is the mean of the gated weights after activation mapping using the Sigmoid function, mathematically defined as:

[0047]

[0048] In the formula, This represents the number of key features.

[0049] The ReLU function forces the average weight of key features to be significantly greater than that of non-key features. Penalize non-critical features with excessive weight, or This penalizes cases where the weights of key features are too small. The auxiliary loss participates in backpropagation with a coefficient of 1.0, aiming to constrain the gating module from deviating too far from medical priors, thus constraining the update of model parameters.

[0050] The optimizer uses AdamW and implements a differentiated learning rate strategy: the base learning rate is set for the core parameters of the visual backbone (ViT-B / 16), the Transformer encoder, and the fusion layer. Weight decay The gating module parameters (including the embedding table and bias terms, i.e., learnable logits vectors with initial values ​​of 4.0 and -2.0) are allowed to be updated during backpropagation of training. To enable adaptive fine-tuning based on the real data distribution even under the initial forced guidance of medical priors, these parameters are grouped into an independent parameter group, and the learning rate is increased to [missing value]. (10 times the backbone learning rate), while setting the weight decay to 0 to ensure that the gating parameters can freely jump out of the local optimum and fully fit the clinical prior distribution.

[0051] The training epochs are set to 100, the batch size is set to 16 based on the GPU memory, and an early stopping strategy (patience=10) is adopted. The main focus is on monitoring the area under the ROC curve (Validation AUC) on the validation set. If the AUC does not improve for 10 consecutive epochs, the early stopping mechanism is triggered, and the model weights with the best validation set AUC are saved to prevent overfitting.

[0052] Through the above-mentioned architectural design and fine-tuning strategy, this invention retains the powerful representation capabilities of the visual Transformer while deeply integrating prior knowledge of clinical medicine, achieving high-precision and highly interpretable prediction of liver cancer recurrence risk.

[0053] S3, Model Interpretability Analysis and Engineering Decision Support Output Utilizing the optimal model weights obtained after training, this invention encapsulates and constructs a localized high-performance inference interface (using the FastAPI framework) based on the Dify agent development framework or Python backend technology stack. The deployed system can receive multimodal patient data (including standard medical image NIfTI format files and structured clinical forms) uploaded from the front-end interface in real time, achieving millisecond-level low-latency response and rapidly outputting a quantitative prediction probability of postoperative recurrence risk in liver cancer.

[0054] To further enhance the reliability and practicality of the model in clinical applications, this invention designs and integrates a dual-driven decision interpretation module, the specific execution logic of which is as follows: Clinical feature attribution analysis: For high-risk warning cases with a predicted recurrence probability ≥0.5, the SHAP explanation algorithm (using the KernelExplainer kernel) is automatically triggered. This algorithm can deeply deconstruct the model's decision-making logic, accurately calculate and visualize the specific contribution (SHAP value) of 26 input clinical features (such as AFP, MVI, etc.) to the current high-risk judgment, and clearly reveal which clinical indicators dominate the model's warning decision.

[0055] Visualization and localization of lesions in imaging: The 3D Grad-CAM algorithm is launched simultaneously. This algorithm captures the gradient information on the last convolutional feature map of the model through a reverse automatic differentiation mechanism, generating a three-dimensional color heatmap. For example... Figure 4This heatmap can accurately focus on and highlight focal areas in MRI images that are strongly associated with a high risk of recurrence (such as tumor margins and suspected microvascular invasion areas), achieving "pixel-level" lesion localization. 3D Grad-CAM generates a corresponding three-dimensional heatmap for each independent MRI sequence. Since different phases (such as the arterial phase and the delayed phase) exhibit different enhancement patterns in lesions, independent generation helps clinicians switch between and view high-risk activation areas on specific phases in the front-end interface; if strict spatial rigid registration is subsequently performed, a weighted total heatmap can also be calculated for reference.

[0056] The aforementioned clinical feature contribution analysis and imaging lesion heatmaps will serve as traceable, visualized evidence, synchronously pushed to attending physicians in real time. Physicians can combine this AI-generated auxiliary evidence to comprehensively assess the patient's individual condition, thereby developing more scientific and precise subsequent clinical intervention plans. For example, deciding whether to implement enhanced radiofrequency ablation (RFA) treatment for high-risk patients, or developing personalized, shorter-cycle intensive follow-up strategies for extremely high-risk groups, ultimately achieving an intelligent upgrade of clinical decision support (CDSS). Figure 5 To compare the effectiveness of the method of this invention with existing machine learning methods in predicting recurrence after liver cancer surgery, this invention demonstrates stronger ability to distinguish between recurrence and non-recurrence (AUC) during the training phase, and more accurate identification of positive cases (recurrence) (high sensitivity), while more thorough exclusion of negative cases (non-recurrence) (high specificity). Even on unseen data, this invention maintains a high AUC and high sensitivity / specificity, indicating that the model is not only effective on the training set but also works stably on the validation set.

[0057] Variation Example To expand the scope of patent protection and prevent competitors from circumventing the technical solutions of this invention through alternative solutions, the following two types of variations are designed, taking into account the technical characteristics of the medical imaging field and the possible circumvention strategies of competitors. These variations cover model backbone replacement and regularization function deformation. All variations can achieve the purpose of this invention and do not deviate from the scope of protection of this invention.

[0058] Variation Example 1: Replacement of the Prediction Model Backbone Alternative approach: Replace the visual backbone ViT-B / 16 in S2 with other feature extraction networks, such as ResNet50, Swin Transformer, etc.; replace the classification activation function of the prediction head with a Softmax variant suitable for multi-class tasks or lifespan time-point prediction.

[0059] Note: This variation only involves a change in the basic operator; the core concepts of multimodal temporal aggregation and medical prior gating constraints remain unchanged, and it achieves the same technical effect as the preferred embodiment, thus still falling within the scope of protection of this invention. Furthermore, this substitution circumvents the limitation in the original claims that "only ViT-B / 16 is used as the visual backbone."

[0060] Variation Example 2: Transformation of the regularization function Alternative approach: Replace the Margin regularization loss function in S2, which is used to enforce the difference in importance between key and non-key features, with an L1 ranking loss or a cross-entropy constraint variant.

[0061] Note: This variation only involves an equivalent substitution of the regularization function form, which can still achieve the regulatory objective of forcing the weights of non-critical features to decay and the weights of critical features to increase, without affecting the core mechanism of multimodal temporal aggregation and gating constraints. Therefore, such equivalent technical substitutions still fall within the protection scope of this patent.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A device for predicting postoperative recurrence of liver cancer based on multimodal Transformer, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data of patients after hepatocellular carcinoma surgery. The multimodal data includes multiphase enhanced MRI image sequences and structured clinical data. The module performs resampling, standardization and normalization processing on the multiphase enhanced MRI image sequences and data cleaning and discretization processing on the structured clinical data. A multimodal fusion prediction module is used to run a pre-trained multimodal recurrence prediction model for liver cancer. This model uses a visual backbone network to extract slice-level feature sequences from preprocessed MRI image sequences at each axial position. Then, a Transformer temporal encoder is used to extract the spatial context relationships between continuous axial slice-level feature sequences to obtain global image tokens for each phase of the MRI image sequence. Finally, image summary features are obtained by aggregating slice-level feature sequences from MRI image sequences at the same phase. After aggregating image summary features from all phases of the patient, a global CLS fusion token is obtained. A clinical feature encoding network maps preprocessed structured clinical data to multidimensional clinical feature tokens. A feature importance gating module performs adaptive weighting of these tokens based on clinical priors to obtain weighted clinical tokens. A cross-modal Transformer fusion layer fuses the global image tokens, weighted clinical tokens, and global CLS fusion tokens from all phases to obtain a multimodal joint representation. This multimodal joint representation is input into a recurrence risk prediction network, ultimately outputting the predicted probability of postoperative recurrence risk for liver cancer. The interpretability analysis and decision support module is used to automatically trigger the SHAP interpretation algorithm to calculate the contribution of each structured clinical feature when the predicted probability reaches the high-risk warning threshold, and to start the 3D Grad-CAM algorithm to generate a three-dimensional lesion heat map, pushing the clinical feature contribution analysis and the imaging lesion heat map as visual evidence to clinicians. The training of the multimodal recurrence prediction model for liver cancer uses a total loss function, which is a weighted sum of cross-entropy loss and auxiliary regularization loss. The formula for calculating the auxiliary regularization loss is as follows: in, The mean of gated activations for key features. The mean of the gated activations for non-critical features is the mean of the gated weights after activation mapping using the Sigmoid function, mathematically defined as: In the formula, For feature index set, A subset of key features is the number of key features, and the constant 0.5 is the maximum tolerance threshold for allowing the weights of non-key features to exceed the weights of key features.

2. The liver cancer postoperative recurrence prediction device according to claim 1, characterized in that, The multimodal recurrence prediction model for liver cancer includes: A pre-trained ViT-B / 16 visual backbone network is used to extract each MRI image from the input multi-phase enhanced MRI image sequence and output a slice-level feature sequence. The temporal encoder consists of two multi-head attention Transformer Encoders with four attention heads and a dropout rate of 0.

2. It includes independent first and second slice sequence encoding branches, which do not share parameters. These branches encode MRI image sequences with different numbers of slices. Before inputting the slice-level feature sequences into the slice sequence encoding network, a learnable one-dimensional positional code is added to the slice-level feature sequences to characterize the positional information of each axial slice in the spatial depth direction, resulting in a global image token for each phase of the MRI image sequence. Each slice-level feature sequence is arranged in axial spatial order within the same MRI image sequence to form a spatial-level slice feature sequence. A classification identifier (CLS) token is added to the starting position of the spatial-level slice feature sequence to aggregate the global information of the spatial-level slice feature sequences, obtaining the image summary features of each phase of the spatial-level slice feature sequence. The image summary features of all phases for each patient are merged into a global CLS fusion token. The clinical feature encoding module discretizes continuous variables in various clinical indicators in structured clinical data according to the clinical median or a preset medical threshold to obtain multi-level labels; based on the discretized labels, it constructs an independent embedding vocabulary for each clinical indicator, maps each clinical indicator to a multi-dimensional dense vector, and performs L2 normalization on the dense vector to obtain the clinical feature embedding matrix. The feature importance gating module is used to introduce prior clinical constraints for liver cancer, designating four core pathogenic indicators—tumor number, tumor size, microvascular invasion (MVI), and alpha-fetoprotein (AFP)—as critical features, while the rest are non-critical features. A learnable gated Logits vector is set for each clinical feature index i∈{1,2,...,26}. The gated Logits vector is initialized as follows: The gating weights are obtained through the Sigmoid function. : Gated weight tensor The dimension is , where F=26 is the number of clinical features; the gating weight tensor and the original clinical feature embedding matrix are multiplied element-wise along the feature dimension through a broadcast mechanism to obtain the weighted clinical token; A cross-modal Transformer fusion layer is used to concatenate the global image token, weighted clinical token, and global CLS fusion token of all phases along the sequence dimension to form a high-dimensional sequence. The cross-modal Transformer fusion layer contains two Transformer layers, eight attention heads, and a feedforward network dimension of 1024. It uses a self-attention mechanism to mine the deep correlation between image features and clinical features and outputs the multimodal joint representation corresponding to the global CLS fusion token. The prediction output layer, composed of a fully connected network, maps the multimodal joint representation to Logits of the recurrence probability after liver cancer surgery, and outputs a binary classification of the recurrence probability.

3. The liver cancer postoperative recurrence prediction device according to claim 1, characterized in that, During the fine-tuning of the multimodal recurrence prediction model for liver cancer, the parameters of the Patch Embedding layer and the first 8 Transformer Block layers of the pre-trained ViT-B / 16 visual backbone network are frozen, and only the parameters of the last 4 Transformer Block layers are opened for fine-tuning; the temporal encoder, cross-modal Transformer fusion layer, and fully connected network adopt the base learning rate. Weight decay ; The embedding table and bias parameters of the feature importance gating module are grouped into an independent parameter group. The bias parameters are learnable Logits vectors with initial values ​​of 4.0 and -2.0, which are allowed to be updated during the backpropagation process of training, and the learning rate is increased to... The weight decay is set to 0; during training, non-critical features are randomly discarded with a 30% probability; the AdamW optimizer is used, the training epochs are 100, the batch size is set to 16, an early stopping strategy is adopted, and the area under the ROC curve on the validation set is monitored. If the AUC does not improve for 10 consecutive epochs, early stopping is triggered.

4. The liver cancer postoperative recurrence prediction device according to claim 1, characterized in that, The multimodal data is used to generate training data through the following preprocessing: For each slice in the multi-phase enhanced MRI image sequence, bilinear interpolation was used to uniformly resample to an image size of 224×224; the mean and standard deviation were calculated based on a single complete three-dimensional sequence of a single patient, Z-score normalization was performed, and pixel intensity was truncated within ±5 times the standard deviation range, and then linearly mapped to... Interval; the single-channel grayscale image is copied into three channels, and each channel is normalized according to the standard mean and variance of the ImageNet dataset; Features with a missing rate exceeding 20% ​​in the structured clinical data are removed, and the missing values ​​of the retained low-missing-rate features are imputed using the global mode method, thus converting the clinical data into discrete labels.

5. The liver cancer postoperative recurrence prediction device according to claim 1, characterized in that, The pre-trained ViT-B / 16 visual backbone network segments the input multi-phase enhanced MRI images into 16×16 pixel image patches, and models the global dependencies between image patches through multi-layer Transformer Blocks, outputting slice-level feature sequences. Each patient was acquired with 20 multiphase enhanced MRI image sequences, including 4 36-slice axial slice sequences and 16 72-slice axial slice sequences; the first slice sequence coding branch network and the second slice sequence coding branch network of the temporal encoder were modeled independently and did not share parameters, so as to adaptively match the 36-slice and 72-slice axial slice sequences for temporal modeling respectively. The structured clinical data collected included 26 indicators, among which tumor number, tumor size, microvascular invasion, and alpha-fetoprotein were designated as key features. An independent embedding vocabulary was constructed for each indicator, with the vocabulary dimensions set to [missing value]. The original discretized labels are mapped to 256-dimensional dense vectors. In the cross-modal Transformer fusion layer, the sequence splicing order is: 20 global image tokens, 26 weighted clinical tokens, and 1 global CLS fusion token, with a sequence length of 47 and a uniform dimension of 256. The prediction output layer uses the Sigmoid activation function to output the early relapse probability.

6. The liver cancer postoperative recurrence prediction device according to claim 1, characterized in that, It also includes model interpretability analysis and engineering-aided decision output steps: The trained model is encapsulated as a localized inference interface, which receives NIfTI format MRI images and structured clinical forms uploaded from the front end and outputs the recurrence probability. When the recurrence probability is ≥0.5, the decision interpretation module is triggered, and the SHAP algorithm is used to calculate the contribution of 26 clinical features to the prediction results and generate a feature importance ranking map. The 3D Grad-CAM algorithm is used to backpropagate the gradient to the last feature map of the cross-modal Transformer fusion layer or the visual backbone, and generate a three-dimensional heat map for each independent MRI sequence to accurately locate the focal areas in the MRI images that are related to the high-risk judgment. Clinical feature contribution maps and image heatmaps are pushed to the front end to assist clinicians in developing personalized intervention plans.

Images