Twin network-based multi-time sequence hepatocellular carcinoma treatment effect prediction system and storage medium

Abstract

A multi-temporal hepatocellular carcinoma efficacy prediction system and storage medium based on Siamese networks include a preprocessing module, a feature extraction module, a feature difference module, a temporal feature cross-attention module, a multi-scale fusion module, and a probability output module. The preprocessing module obtains multi-channel image data before and after treatment based on multi-phase image data from before and after treatment. The feature extraction module extracts pre-treatment multi-scale features from the pre-treatment multi-channel image data and post-treatment multi-scale features from the post-treatment multi-channel image data. The feature difference module calculates difference features based on the pre-treatment and post-treatment multi-scale features. The temporal feature cross-attention module calculates temporal cross-features based on the pre-treatment and post-treatment multi-scale features. The multi-scale fusion module fuses the difference features and temporal cross-features to obtain fused features. The probability output module obtains the treatment response probability based on the fused features.

Description

Technical Field

[0001] This application relates to the field of medical data processing technology, and more specifically to a multi-temporal hepatocellular carcinoma efficacy prediction system and storage medium based on twin networks. Background Technology

[0002] Globally, liver cancer ranks among the top malignant tumors in terms of incidence, and my country is also a major country in terms of disease burden. The liver is a vital metabolic organ, and surgical resection and interventional therapies can affect liver function. Furthermore, liver cancer is prone to vascular invasion and distant metastasis, resulting in a high recurrence rate. Predicting the treatment efficacy of liver cancer aims to achieve individualized treatment, allowing patients to receive more precise and safer treatment plans while avoiding the risks and waste of resources associated with ineffective treatments. The conditions of different liver cancer patients vary greatly, including tumor size, stage, liver function status, and genetic characteristics. Efficacy prediction can help determine a patient's potential response to specific treatments (such as surgery, targeted therapy, immunotherapy, and interventional therapy). Efficacy prediction can also assess treatment-related risks, helping doctors and patients prepare in advance. If the prediction indicates a high risk of treatment side effects, dosage adjustments or adjuvant medications can be made in advance to reduce the probability of complications. If the predicted prognosis is poor, palliative or supportive care can be initiated earlier, improving the patient's quality of life. Summary of the Invention

[0003] This application is proposed to address the aforementioned problems. According to one aspect of this application, a multi-temporal hepatocellular carcinoma efficacy prediction system based on Siamese networks is provided, comprising a preprocessing module, a feature extraction module, a feature difference module, a temporal feature cross-attention module, a multi-scale fusion module, and a probability output module.

[0004] The preprocessing module is used to obtain multi-channel image data before and after treatment based on multi-phase image data before and after treatment.

[0005] The feature extraction module is a Siamese network with shared parameters, which includes a three-stage model. Each stage model includes a cross-scale embedding layer and multiple attention modules, which generate first-scale features, second-scale features and third-scale features of multi-channel image data before and after treatment, respectively.

[0006] The feature difference module includes three stage difference modules. Each stage difference module calculates the difference between the post-treatment features and the pre-treatment features at the corresponding scale to obtain the first difference feature, the second difference feature, and the third difference feature.

[0007] The temporal feature cross-attention module includes three stages of temporal modules. Each stage of temporal module performs a linear transformation on the pre-treatment scale features to obtain linear transformation features. The product of the linear transformation features and the post-treatment scale features is added to the post-treatment scale features to obtain the temporal cross features of the corresponding scale.

[0008] The multi-scale fusion module fuses differential features and temporal cross features of the same scale to obtain fused features of the same scale, and fuses fused features of multiple scales to obtain the final fused features.

[0009] The probability output module obtains the treatment response probability based on the final fusion features.

[0010] In one embodiment of this application, the multi-phase imaging data includes arterial phase imaging data, portal venous phase imaging data, and delayed phase imaging data. The preprocessing module is used to obtain multi-channel imaging data before and after treatment based on the multi-phase imaging data before and after treatment, including:

[0011] Extract the region of interest from the arterial phase imaging data, portal venous phase imaging data, and delayed phase imaging data;

[0012] The regions of interest are stacked in the order of arterial phase, portal venous phase, and delayed phase to obtain multi-channel image data.

[0013] In one embodiment of this application, each stage model of the feature extraction module includes a cross-scale embedding layer and multiple attention modules:

[0014] The cross-scale embedding layer generates multi-scale embeddings based on the multi-channel image data;

[0015] The multiple attention modules process the multi-scale embedding to obtain pre-treatment scale features and post-treatment scale features.

[0016] In one embodiment of this application, the three-stage model of the feature extraction module includes a first-stage model, a second-stage model, and a third-stage model:

[0017] The first-stage model generates a pre-treatment first-scale feature based on the pre-treatment multi-channel image data, and generates a post-treatment first-scale feature based on the post-treatment multi-channel image data.

[0018] The second-stage model generates pre-treatment second-scale features based on the pre-treatment multi-channel image data, and generates post-treatment second-scale features based on the post-treatment multi-channel image data.

[0019] The third-stage model generates pre-treatment third-scale features based on the pre-treatment multi-channel imaging data and generates post-treatment third-scale features based on the post-treatment multi-channel imaging data.

[0020] In one embodiment of this application, the three-stage difference module of the feature difference module includes a first-stage difference module, a second-stage difference module, and a third-stage difference module:

[0021] The first-stage difference module calculates the difference between the first-scale feature after treatment and the first-scale feature before treatment to obtain the first difference feature;

[0022] The second-stage difference module calculates the difference between the post-treatment second-scale feature and the pre-treatment second-scale feature to obtain the second difference feature;

[0023] The third-stage difference module calculates the difference between the post-treatment third-scale feature and the pre-treatment third-scale feature to obtain the third difference feature.

[0024] In one embodiment of this application, the three-stage timing modules of the temporal feature cross-attention module include a first-stage timing module, a second-stage timing module, and a third-stage timing module:

[0025] The first-stage temporal module performs a linear transformation on the pre-treatment first-scale feature to obtain a first linear transformation feature, and adds the product of the first linear transformation feature and the post-treatment first-scale feature to the post-treatment first-scale feature to obtain a first temporal cross feature;

[0026] The second-stage temporal module performs a linear transformation on the pre-treatment second-scale feature to obtain a second linear transformation feature. The product of the second linear transformation feature and the post-treatment second-scale feature is added to the post-treatment second-scale feature to obtain the second temporal cross feature.

[0027] The third-stage temporal module performs a linear transformation on the pre-treatment third-scale feature to obtain a third linear transformation feature. The product of the third linear transformation feature and the post-treatment third-scale feature is added to the post-treatment third-scale feature to obtain the third temporal cross feature.

[0028] In one embodiment of this application, the multi-scale fusion module fuses differential features and temporal cross features of the same scale to obtain fused features of the same scale, and fuses fused features of multiple scales to obtain the final fused feature, including:

[0029] The first differential feature is fused with the first temporal cross feature to obtain the first fused feature;

[0030] The second difference feature is fused with the second temporal cross feature to obtain the second fused feature;

[0031] The third difference feature is fused with the third temporal cross feature to obtain the third fused feature;

[0032] The first fusion feature, the second fusion feature, and the third fusion feature are fused to obtain the final fusion feature.

[0033] According to another aspect of this application, a storage medium is provided that stores a computer program, which, when run, executes the aforementioned prediction system.

[0034] This application extracts multi-scale features from multi-channel image data through a feature extraction module, providing a more comprehensive data foundation for feature extraction; calculates the differences in multi-scale features before and after treatment through a feature difference module, accurately capturing subtle changes by calculating differences for features at different scales, avoiding simple difference; establishes a dynamic correlation between treatment and the time-series feature cross-attention module, enhancing the correlation between features and efficacy; and performs same-scale integration and cross-scale aggregation through a multi-scale fusion module, making the fused features better reflect the essence of efficacy, directly improving the decision-making ability of the prediction model and increasing prediction accuracy. Attached Figure Description

[0035] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0036] Figure 1 A system block diagram of a multi-temporal hepatocellular carcinoma efficacy prediction system based on a twin network according to an embodiment of this application is shown.

[0037] Figure 2 A flowchart is shown for a multi-temporal hepatocellular carcinoma efficacy prediction system based on a twin network according to an embodiment of this application;

[0038] Figure 3 This illustrates multi-phase image data diagrams according to embodiments of this application;

[0039] Figure 4 This diagram illustrates the calculation of timing crossover features according to an embodiment of this application.

[0040] Figure 5 A software schematic diagram of a multi-temporal hepatocellular carcinoma efficacy prediction system based on twin networks according to an embodiment of this application is shown.

[0041] Figure 6 This diagram illustrates a confusion matrix according to an embodiment of the present application.

[0042] Figure 7 The diagram shows the ROC curve of the training set according to an embodiment of this application;

[0043] Figure 8 This illustrates the ROC curves of the test set according to an embodiment of this application;

[0044] Figure 9 This illustrates a DCA curve of the training set according to an embodiment of this application;

[0045] Figure 10 This illustrates a DCA curve diagram of the test set according to an embodiment of this application;

[0046] Figure 11 This illustrates a training set calibration curve according to an embodiment of this application;

[0047] Figure 12 A calibration curve of the test set according to an embodiment of this application is shown.

[0048] Figure labels: Preprocessing module 110; Feature extraction module 120; Feature difference module 130; Temporal feature cross-attention module 140; Multi-scale fusion module 150; Probability output module 160. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of this application, and not all of the embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0050] First, refer to Figure 1 This describes a twin network-based multi-temporal hepatocellular carcinoma efficacy prediction system 100 used to implement embodiments of the present invention. Figure 1 A system block diagram of a multi-temporal hepatocellular carcinoma efficacy prediction system 100 based on Siamese networks according to an embodiment of this application is shown. The multi-temporal hepatocellular carcinoma efficacy prediction system 100 based on Siamese networks includes a preprocessing module 110, a feature extraction module 120, a feature difference module 130, a temporal feature cross-attention module 140, a multi-scale fusion module 150, and a probability output module 160.

[0051] Preprocessing module 110 is used to obtain multi-channel image data before and after treatment based on multi-phase image data before and after treatment; feature extraction module 120 is a Siamese network with shared parameters, including a three-stage model, each stage model including a cross-scale embedding layer and multiple attention modules, generating first-scale features, second-scale features and third-scale features of the multi-channel image data before and after treatment respectively; feature difference module 130 includes three-stage difference modules, each stage difference module calculates the difference between the post-treatment features and the pre-treatment features at the corresponding scale, obtaining the first difference feature, ... The second and third difference features; the temporal feature cross-attention module 140 includes three stages of temporal modules. Each stage of the temporal module performs a linear transformation on the pre-treatment scale features to obtain linear transformation features. The product of the linear transformation features and the post-treatment scale features is added to the post-treatment scale features to obtain the temporal cross features of the corresponding scale. The multi-scale fusion module 150 fuses the difference features and temporal cross features of the same scale to obtain fusion features of the same scale. The fusion features of multiple scales are fused to obtain the final fusion feature. The probability output module 160 obtains the treatment response probability based on the final fusion feature.

[0052] This application extracts multi-scale features from multi-channel image data through feature extraction module 120, providing a more comprehensive data foundation for feature extraction; calculates the differences in multi-scale features before and after treatment through feature difference module 130, accurately capturing subtle changes by calculating differences for features at different scales, avoiding simple difference; establishes a dynamic correlation between before and after treatment through temporal feature cross-attention module 140, improving the correlation between features and efficacy; and performs same-scale integration and cross-scale aggregation through multi-scale fusion module 150, making the fused features better reflect the essence of efficacy, directly improving the decision-making ability of the prediction model and increasing prediction accuracy.

[0053] In one embodiment, the multi-phase image data includes arterial phase image data, portal venous phase image data, and delayed phase image data. The preprocessing module 110 is used to obtain multi-channel image data before and after treatment based on the multi-phase image data before and after treatment, including: extracting regions of interest from the arterial phase image data, portal venous phase image data, and delayed phase image data; and stacking the regions of interest in the order of arterial phase, portal venous phase, and delayed phase to obtain multi-channel image data.

[0054] The dataset contains 178 HCC patients who received targeted / immunotherapy, and the data usage has been approved by the hospital ethics committee (approval number: 2025-RE-111). Each patient includes two sets of three-phase dynamic contrast-enhanced CT scans (arterial phase, portal venous phase, and delayed phase) before and after treatment. Patients were randomly assigned to the training and test sets in a 7:3 ratio. Efficacy labels were categorized by overall survival (OS) of all patients, with a pre- / post-0.5 classification: long OS: 89 patients; short OS: 89 patients. Figure 3 As shown, Figure 3 The training set includes images A, B, and C, representing pre-treatment image data, corresponding to the arterial phase, portal venous phase, and delayed phase, respectively. Images D, E, and F represent post-treatment image data, corresponding to the arterial phase, portal venous phase, and delayed phase, respectively. 1A, 1B, and 1C correspond to 1D, 1E, and 1F.

[0055] The preprocessing module 110 extracts regions of interest (ROIs) from the pre-treatment arterial phase, portal venous phase, and delayed phase image data, and stacks these ROIs in the order of arterial, portal venous, and delayed phases to obtain pre-treatment multi-channel image data. Similarly, the preprocessing module 110 extracts ROIs from the post-treatment arterial phase, portal venous phase, and delayed phase image data, and stacks these ROIs in the same order to obtain post-treatment multi-channel image data.

[0056] like Figure 5 As shown, the Mask input refers to the input of the mask file, which is a binary or multi-valued array file spatially aligned with the original CT image. The tumor region is automatically segmented using a UNet pre-trained model, and the largest lesion region is extracted. Regions of interest (ROIs) with the largest cross-section of the tumor are extracted from CT images in the arterial, portal venous, and delayed phases, based on radiologist annotations or automatic segmentation algorithms. A three-dimensional dilation algorithm is used to maintain the spatial consistency of ROIs across different phases, ensuring coverage of the entire lesion and surrounding infiltrative areas. The boundaries of the ROIs are appropriately expanded in three-dimensional space (x / y / z axes, corresponding to the transverse and longitudinal slices and slice thickness directions of the CT image). This avoids missing lesion details due to manual annotation deviations or algorithmic segmentation errors, while also covering surrounding infiltrative areas (such as micrometastases and edema zones), and ensuring a high degree of spatial matching in terms of location and extent among the ROIs in the arterial, portal venous, and delayed phases.

[0057] Three-channel data synthesis was performed, stacking the ROI cropping results from the three phases in the order of arterial phase (channel 1), portal venous phase (channel 2), and delayed phase (channel 3) to form a four-dimensional tensor structure of H×W×D×3, where D represents the number of slices along the long axis of the lesion. The data was uniformly resampled to a resolution of 1mm×1mm×1mm. The final dataset was constructed, and after all preprocessing was completed, the multi-channel image data was stored in a standardized format. Each sample consisted of a main sequence and its upper and lower auxiliary sequences, ensuring a consistent input format for the training and validation of the deep learning model.

[0058] Existing methods struggle to effectively capture the temporal evolution of tumors before and after treatment, lacking the ability to continuously analyze dynamic enhancement patterns during the arterial, portal venous, and delayed phases. Traditional single-timepoint assessments fail to reflect the dynamic process of treatment response, resulting in low accuracy in early efficacy prediction.

[0059] Different CT enhancement sequences (arterial phase / portal venous phase / delayed phase) contain complementary diagnostic information, but existing technologies mostly employ independent analysis strategies, which fail to maintain cross-sequence spatial consistency and make it difficult to uncover the synergistic diagnostic value between sequences. This application's multi-channel processing of multi-phase images, compared to single-phase data, adds information on lesion enhancement patterns in the portal venous and delayed phases, providing a more comprehensive data foundation for feature extraction and reducing prediction bias caused by missing information.

[0060] In one embodiment, each stage model of the feature extraction module 120 includes a cross-scale embedding layer and multiple attention modules: the cross-scale embedding layer generates multi-scale embeddings based on multi-channel image data; the multiple attention modules process the multi-scale embeddings to obtain pre-treatment scale features and post-treatment scale features.

[0061] To extract multi-channel features from multi-channel MRI images more efficiently, a feature extraction module 120 with a three-layer visual temporal coding mechanism is introduced to extract image information at different scales in the image.

[0062] Preprocessing the multi-channel data of each CT slice using convolutional layers helps standardize the input and reduce noise interference in subsequent layers. This step ensures that the data from each channel can be effectively integrated and used for preliminary feature extraction.

[0063] Each processed CT slice is segmented into fixed-size image patches using a cross-scale embedding layer, transforming them into initial feature representations that the model can process. This step allows the model to learn local texture and shape information from each small region.

[0064] like Figure 2As shown, the cross-scale embedding layer is a CEL (Cross-scale Embedding Layer). Located at the beginning of each stage, the CEL receives the output or input image from the previous stage as input. It performs convolution operations on the input using multiple convolutional kernels of different sizes, such as 4×4, 8×8, 16×16, and 32×32, thereby extracting features of different scales from the input. These features of different scales are then concatenated to form an embedding containing multi-scale information, enabling the self-attention module to receive multi-scale information and enhancing the model's expressive power.

[0065] Multiple attention modules can be composed of a Long-Short Distance Attention (LSDA) module and a Multilayer Perceptron (MLP). The LSDA further includes a Short-Distance Attention (SDA) module and a Long-Distance Attention (LDA) module. SDA establishes dependencies between adjacent embeddings, while LDA handles dependencies between distant embeddings. This design not only reduces computational burden but also preserves both small-scale and large-scale features in the embeddings. A dynamic position bias module plays a role in both SDA and LDA to obtain the positional representation of the embeddings. The MLP is used to further transform and map the features output by the attention modules, ultimately outputting feature representations for subsequent tasks.

[0066] A cross-scale attention mechanism is employed, utilizing a cross-scale embedding layer (CEL) and long-short distance attention (LSDA) to achieve collaborative perception of local details and global context. LSDA decomposes self-attention into short-range local dependency modeling and long-range global association capture. An efficient hierarchical processing architecture is employed, using an encoder-decoder structure. The encoder extracts multi-scale features through a two-stage attention layer, while the decoder performs cross-scale prediction fusion. Dynamic positional encoding introduces dynamic positional bias to achieve adaptive positional information modeling based on input size. This design significantly improves the model's ability to recognize multi-scale features in medical images while maintaining computational efficiency, making it particularly suitable for cross-phase analysis tasks in medical images such as CT scans.

[0067] In one embodiment, the feature extraction module 120 includes a three-stage model: a first-stage model, a second-stage model, and a third-stage model. The first-stage model generates pre-treatment first-scale features based on pre-treatment multi-channel image data and post-treatment first-scale features based on post-treatment multi-channel image data. The second-stage model generates pre-treatment second-scale features based on pre-treatment multi-channel image data and post-treatment second-scale features based on post-treatment multi-channel image data. The third-stage model generates pre-treatment third-scale features based on pre-treatment multi-channel image data and post-treatment third-scale features based on post-treatment multi-channel image data.

[0068] like Figure 2 As shown, the multi-scale feature output involves the feature extraction module 120 outputting feature maps at different scales at different stages of the model, including 1 / 2-scale, 1 / 4-scale, and 1 / 8-scale features. Each feature map is rich in information captured from a specific level. The final feature output is a set of high-dimensional feature vectors that integrate global and local information from the input multi-channel data, providing essential input features for subsequent tasks such as pathological analysis or treatment efficacy evaluation.

[0069] Existing models often limit feature extraction to a single scale (e.g., extracting only the macroscopic outline of the lesion or capturing only the microscopic texture). However, the evaluation of hepatocellular carcinoma treatment efficacy needs to consider both "global lesion morphological changes" (e.g., size, boundary) and "local detailed features" (e.g., internal necrosis areas, heterogeneous enhancement). Single-scale features can easily lead to "one-sided feature representation," affecting prediction accuracy. The feature extraction module 120 includes a cross-scale embedding layer and an attention module at each stage, generating "first / second / third scale features" (corresponding to three layers of features: "macro-meso-micro") to achieve full coverage of lesion information at different scales and avoid the limitations of a single scale.

[0070] like Figure 2 As shown, the feature extraction module 120 is a Siamese network. The Siamese network uses two branches with shared parameters to model the image data before and after treatment, respectively, and realizes the interaction and fusion of features through a cross-attention mechanism.

[0071] In one embodiment, the feature difference module 130 includes a multi-stage module: each stage module calculates the difference between the post-treatment scale features and the pre-treatment scale features to obtain the difference features.

[0072] Features were extracted from pre- and post-treatment image data using a twin network. Let the pre-treatment image features be... The imaging features after treatment are These features comprise feature sets at three different scales. and ,in These correspond to the three stages of multiple attention modules. For each scale... Calculate the difference features Differences in characteristics between post-treatment and pre-treatment:

[0073] ;

[0074] This difference calculation helps reveal the effects of treatment, especially in areas where local changes are significant.

[0075] In one embodiment, the feature difference module 130 has three stage difference modules, including a first stage difference module, a second stage difference module, and a third stage difference module: the first stage difference module calculates the difference between the first scale feature after treatment and the first scale feature before treatment to obtain the first difference feature; the second stage difference module calculates the difference between the second scale feature after treatment and the second scale feature before treatment to obtain the second difference feature; and the third stage difference module calculates the difference between the third scale feature after treatment and the third scale feature before treatment to obtain the third difference feature.

[0076] For the "features at each scale" (first / second / third scale features) output by the feature extraction module 120, the difference between "post-treatment and pre-treatment" (first / second / third difference features) is calculated respectively, realizing "precise quantification of differences at multiple scales" and directly focusing on the core changes related to efficacy.

[0077] In one embodiment, the temporal feature cross-attention module 140 includes a multi-stage module: each stage module performs a linear transformation on the pre-treatment scale features to obtain linearly transformed features, and adds the product of the linearly transformed features and the post-treatment scale features to the post-treatment scale features to obtain temporal cross features.

[0078] like Figure 4 As shown, let the initial input be... For pre-treatment scale features, a weight matrix is ​​used in the cross layer. and bias vector A linear transformation is performed on the pre-treatment dimensional features to obtain linearly transformed features. These linearly transformed features are then compared with the post-treatment dimensional features. After the Hadama product (element-wise product), plus post-treatment scale characteristics Temporal crossover features were obtained. The formula is expressed as:

[0079] ;

[0080] in, This represents the Hadamard product, used to perform feature crossover operations.

[0081] To enhance network stability and prevent overfitting, residual connections are introduced, where the input of each layer is directly added to the output. This design helps the model retain information from the original features after multiple layers of cross-connection, while also adding new cross-connected features. The temporal feature cross-connection attention module 140 achieves effective fusion of high-order features through multiple layers of recursive cross-connection. In practice, this significantly enhances the model's ability to capture complex patterns, especially when processing medical image data with significant before-and-after contrast differences.

[0082] In one embodiment, the three-stage temporal module of the temporal feature cross-attention module 140 includes a first-stage temporal module, a second-stage temporal module, and a third-stage temporal module: The first-stage temporal module performs a linear transformation on the pre-treatment first-scale feature to obtain a first linearly transformed feature, and adds the product of the first linearly transformed feature and the post-treatment first-scale feature to the post-treatment first-scale feature to obtain a first temporal cross-feature; The second-stage temporal module performs a linear transformation on the pre-treatment second-scale feature to obtain a second linearly transformed feature, and adds the product of the second linearly transformed feature and the post-treatment second-scale feature to the post-treatment second-scale feature to obtain a second temporal cross-feature; The third-stage temporal module performs a linear transformation on the pre-treatment third-scale feature to obtain a third linearly transformed feature, and adds the product of the third linearly transformed feature and the post-treatment third-scale feature to the post-treatment third-scale feature to obtain a third temporal cross-feature.

[0083] The core of efficacy prediction lies in analyzing lesion changes "before treatment - after treatment." However, traditional models often treat pre-treatment and post-treatment data as independent inputs, failing to establish a dynamic correlation between the two, thus failing to capture the "temporal dynamic information" related to efficacy. The temporal feature cross-attention module 140, through the operation of "linear transformation of pre-treatment scale features → product with post-treatment scale features → superposition of post-treatment features," forcibly establishes a dynamic correlation between pre-treatment and post-treatment features (such as the influence of pre-treatment lesion enhancement features on post-treatment enhancement changes), accurately capturing efficacy correlation information in the temporal dimension.

[0084] In one embodiment, the multi-scale fusion module 150 is used to fuse differential features and temporal cross features to obtain fused features, including: fusing a first differential feature and a first temporal cross feature to obtain a first fused feature; fusing a second differential feature and a second temporal cross feature to obtain a second fused feature; fusing a third differential feature and a third temporal cross feature to obtain a third fused feature; and fusing the first fused feature, the second fused feature, and the third fused feature to obtain a fused feature.

[0085] The multi-scale fusion module 150 will process and integrate the difference features and temporal cross features for each scale. For each scale s, there are difference features. and temporal cross features The fusion operation combines these two features into a unified feature vector, which is the fused feature at each scale. :

[0086] ;

[0087] This step merges feature information from different sources, providing a richer data foundation for the next step of in-depth analysis.

[0088] To enhance the expressive power of multi-scale features and extract more meaningful therapeutic efficacy indicators, feature fusion strategies can be employed. A weighted sum method can be used to integrate features from different scales. Obtain fusion features :

[0089] ;

[0090] in These are the weights of the fused features at each scale, which can be learned to optimize the model's performance.

[0091] A three-layer MLP prediction network is constructed as the probability output module 160. MLP stands for Multi-Layer Perceptron.

[0092] ;

[0093] The MLP prediction network consists of an input layer, hidden layers, and an output layer. Neurons between layers are connected by weights, and signals are transmitted unidirectionally from the input layer to the output layer (feedforward characteristic). For input layer output, For hidden layer output, For output layer output, , It is a non-linear activation function. , , , , , For the weight matrices and bias vectors of different layers, output This indicates the probability of a treatment response. For example... Figure 5 As shown, the probability output module 160 obtains the treatment response probability based on the fusion features. When the efficacy prediction probability is greater than or equal to 0.5, the treatment effect is considered good; when the efficacy prediction probability is less than 0.5, the treatment effect is considered poor.

[0094] The following metrics were used to comprehensively evaluate model performance: Accuracy, Precision, Recall, F1 Score, Sensitivity, Specificity, AUC, DCA, and calibration curve. Accuracy: Accuracy = (Number of correctly predicted samples) / (Total number of samples). Precision: Precision = True positives / (True positives + False positives). Recall: Recall = True positives / (True positives + False negatives). F1 Score: F1 = 2 × (Precision * Recall) / (Precision + Recall). Sensitivity: Sensitivity = True positives / (True positives + False negatives). Specificity: Specificity = True negatives / (True negatives + False positives). AUC: Area under the curve, used to measure the model's discriminative ability; an AUC between 0.80 and 0.90 indicates good overall model performance. PPV: Positive Predictive Value. NPV: Negative Predictive Value. DCA Curve: The DCA curve is used to evaluate the clinical net benefit of the model. Calibration Curve: The calibration curve is used to evaluate the comparison between the model's predicted probability and the true probability.

[0095] This model is compared with traditional methods. The traditional method is mRECIST (modified Response Evaluation Criteria in Solid Tumors), an evaluation system optimized based on the traditional RECIST standard. mRECIST is a modified standard for evaluating the efficacy of treatment in solid tumors, primarily used to assess the treatment effect of highly vascularized tumors. It mainly targets scenarios where tumors show significant necrosis after interventional vascular treatment, targeted therapy, etc., solving the problem of inaccurate judgment of "surviving tumor tissue" in traditional standards. Compared with the traditional method (mRECIST), this model shows superior performance on both the training and test sets, with a higher AUC than the traditional mRECIST method (training set 0.912 vs 0.851, test set 0.877 vs 0.840), demonstrating better stability and generalization ability. This model has better predictive performance for treatment effects. Specific comparisons are shown in Table 1 below:

[0096] .

[0097] like Figure 6 As shown, a set of confusion matrices is used to demonstrate the classification performance of the two methods (this model and mRECIST) on the training and test sets. The core objective is to evaluate the model's prediction accuracy for both "0" and "1" samples. Comparing this model and mRECIST: on the training set, this model is more accurate in identifying class "1" (55 vs 53); on the test set, this model has fewer errors in both classes than mRECIST, demonstrating better generalization performance.

[0098] like Figure 7 , 8 As shown in the figure, the ROC curve (Receiver Operating Characteristic) plot, where CI is the confidence interval. On the training set, the AUC of our model is 0.912, with a 95% confidence interval (95% CI) of 0.865–0.958; the AUC of mRECIST is 0.851, with a 95% CI of 0.785–0.918. On the training set, the AUC of our model is significantly higher than that of mRECIST, indicating that our model has a stronger ability to distinguish the research subjects in the training data. On the test set, the AUC of our model is 0.877, with a 95% CI of 0.757–0.996; the AUC of mRECIST is 0.840, with a 95% CI of 0.726–0.954. On the test set, the AUC of our model is still higher than that of mRECIST, indicating that our model maintains good generalization ability in new, unseen data, and its discrimination performance is superior to that of mRECIST.

[0099] like Figure 9 , 10 As shown in the Decision Curve Analysis (DCA) graph. In the training set: the net benefit curve of this model is significantly higher than mRECIST, all-treatment, and no-treatment within a relatively wide decision threshold range (approximately 0–0.8). This indicates that within this range, using this model to guide clinical decision-making (determining whether to treat) can achieve higher clinical net benefits than the existing standard (mRECIST) and extreme treatment strategies (all / no treatment). In the test set: the net benefit curve of this model is still higher than mRECIST, all-treatment, and no-treatment within a relatively wide decision threshold range (approximately 0–0.8). This demonstrates that the model can maintain good clinical net benefits on unseen datasets and has strong generalization ability. The net benefit of mRECIST is lower than that of this model, the net benefit of all-treatment drops sharply at high thresholds, and the net benefit of no-treatment is negligible.

[0100] like Figure 11 , 12As shown, the calibration curves are used. The training set is used for model fitting and training, and its calibration curve reflects the model's accuracy in predicting probabilities on the training data. The test set contains new data that the model has not seen before, and its calibration curve reflects the model's generalization ability (accuracy in predicting probabilities on new data). On the training set, both this model and mRECIST show good calibration results, indicating that both can learn the matching relationship between "predicted probability - actual positive rate" well on the training data. On the test set, both still maintain a certain level of calibration ability.

[0101] In the test set, this model outperformed the traditional model (mRECIST) in terms of AUC, accuracy, precision, DCA, and calibration curve, and this model showed good performance in predicting treatment effects.

[0102] This application designs a hierarchical multi-scale fusion module 150, which first fuses differential features and temporal cross features at the same scale (e.g., first differential feature + first temporal cross feature → first fused feature), and then further fuses fused features at different scales (first + second + third fused features → final fused feature). Through the logic of "integrating at the same scale first, and then aggregating across scales," feature confusion is avoided, and the synergistic value of multi-dimensional features is maximized. This makes the fused features more reflective of the essence of the therapeutic effect and directly improves the "decision-making ability" of the prediction model.

[0103] Technical feature optimization ultimately translates into higher predictive accuracy, enabling a more precise distinction between patients who are "treatment-responsive" (e.g., lesion shrinkage, necrosis) and those who are "treatment-ineffective / progressive" (e.g., lesion enlargement, new lesions), providing physicians with reliable efficacy assessment data. By accurately predicting efficacy, physicians can identify "treatment-ineffective" patients in advance and adjust treatment plans promptly (e.g., switching from interventional therapy to targeted therapy), avoiding side effects and wasted medical resources caused by ineffective treatment. Early and accurate efficacy prediction helps physicians develop follow-up management plans more tailored to the patient's condition (e.g., high-frequency follow-up monitoring or medication dosage adjustment), reducing the risk of disease progression and improving patient survival rates and quality of life. Ultimately, this achieves "precision, standardization, and clinicalization" of hepatocellular carcinoma efficacy prediction, combining technological innovation with clinical practical value.

[0104] This application's Siamese network-based multi-temporal hepatocellular carcinoma efficacy prediction system extracts multi-scale features from multi-channel image data through a feature extraction module, providing a more comprehensive data foundation for feature extraction; it calculates the differences in multi-scale features before and after treatment through a feature difference module, accurately capturing subtle changes by calculating differences for features at different scales, avoiding simple difference; it establishes a dynamic correlation between pre- and post-treatment through a temporal feature cross-attention module, enhancing the correlation between features and efficacy; and it performs same-scale integration and cross-scale aggregation through a multi-scale fusion module, making the fused features better reflect the essence of efficacy, directly improving the decision-making ability of the prediction model and increasing prediction accuracy.

[0105] Furthermore, this application also provides a storage medium storing a computer program thereon, which, when executed by a processor, causes the processor to perform the aforementioned twin-network-based multi-temporal hepatocellular carcinoma efficacy prediction system. The storage medium may, for example, include a smartphone memory card, a tablet computer storage component, a personal computer hard drive, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] To simplify this application and aid in understanding one or more of the various inventive aspects, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, the inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0108] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as a program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such a program implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0109] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A multi-temporal hepatocellular carcinoma treatment efficacy prediction system based on Siamese networks, characterized in that, It includes a preprocessing module, a feature extraction module, a feature differencing module, a temporal feature cross-attention module, a multi-scale fusion module, and a probability output module. The preprocessing module is used to obtain multi-channel image data before and after treatment based on multi-phase image data before and after treatment. The feature extraction module is a Siamese network with shared parameters, which includes a three-stage model. Each stage model includes a cross-scale embedding layer and multiple attention modules, which generate first-scale features, second-scale features and third-scale features of multi-channel image data before and after treatment, respectively. The feature difference module includes three stage difference modules. Each stage difference module calculates the difference between the post-treatment features and the pre-treatment features at the corresponding scale to obtain the first difference feature, the second difference feature, and the third difference feature. The temporal feature cross-attention module includes three stages of temporal modules. Each stage of temporal module performs a linear transformation on the pre-treatment scale features to obtain linear transformation features. The product of the linear transformation features and the post-treatment scale features is added to the post-treatment scale features to obtain the temporal cross features of the corresponding scale. The multi-scale fusion module fuses differential features and temporal cross features of the same scale to obtain fused features of the same scale, and fuses fused features of multiple scales to obtain the final fused features. The probability output module obtains the treatment response probability based on the final fusion features.

2. The multi-temporal hepatocellular carcinoma efficacy prediction system based on Siamese networks as described in claim 1, characterized in that, The multi-phase imaging data includes arterial phase imaging data, portal venous phase imaging data, and delayed phase imaging data. The preprocessing module is used to obtain multi-channel imaging data before and after treatment based on the multi-phase imaging data before and after treatment, including: Extract the region of interest from the arterial phase imaging data, portal venous phase imaging data, and delayed phase imaging data; The regions of interest are stacked in the order of arterial phase, portal venous phase, and delayed phase to obtain multi-channel image data.

3. The multi-temporal hepatocellular carcinoma efficacy prediction system based on Siamese networks as described in claim 1, characterized in that, Each stage of the feature extraction module includes a cross-scale embedding layer and multiple attention modules: The cross-scale embedding layer generates multi-scale embeddings based on the multi-channel image data; The multiple attention modules process the multi-scale embedding to obtain pre-treatment scale features and post-treatment scale features.

4. The multi-temporal hepatocellular carcinoma efficacy prediction system based on twin networks as described in claim 3, characterized in that, The feature extraction module has three stages: a first-stage model, a second-stage model, and a third-stage model. The first-stage model generates a pre-treatment first-scale feature based on the pre-treatment multi-channel image data, and generates a post-treatment first-scale feature based on the post-treatment multi-channel image data. The second-stage model generates pre-treatment second-scale features based on the pre-treatment multi-channel image data, and generates post-treatment second-scale features based on the post-treatment multi-channel image data. The third-stage model generates pre-treatment third-scale features based on the pre-treatment multi-channel imaging data and post-treatment third-scale features based on the post-treatment multi-channel imaging data.

5. The multi-temporal hepatocellular carcinoma efficacy prediction system based on Siamese networks as described in claim 4, characterized in that, The feature difference module comprises three stages: a first-stage difference module, a second-stage difference module, and a third-stage difference module. The first-stage difference module calculates the difference between the first-scale feature after treatment and the first-scale feature before treatment to obtain the first difference feature; The second-stage difference module calculates the difference between the post-treatment second-scale feature and the pre-treatment second-scale feature to obtain the second difference feature; The third-stage difference module calculates the difference between the post-treatment third-scale feature and the pre-treatment third-scale feature to obtain the third difference feature.

6. The multi-temporal hepatocellular carcinoma efficacy prediction system based on Siamese networks as described in claim 5, characterized in that, The three-stage temporal modules of the temporal feature cross-attention module include a first-stage temporal module, a second-stage temporal module, and a third-stage temporal module: The first-stage temporal module performs a linear transformation on the pre-treatment first-scale feature to obtain a first linear transformation feature, and adds the product of the first linear transformation feature and the post-treatment first-scale feature to the post-treatment first-scale feature to obtain a first temporal cross feature; The second-stage temporal module performs a linear transformation on the pre-treatment second-scale feature to obtain a second linear transformation feature. The product of the second linear transformation feature and the post-treatment second-scale feature is added to the post-treatment second-scale feature to obtain the second temporal cross feature. The third-stage temporal module performs a linear transformation on the pre-treatment third-scale feature to obtain a third linear transformation feature. The product of the third linear transformation feature and the post-treatment third-scale feature is added to the post-treatment third-scale feature to obtain the third temporal cross feature.

7. The multi-temporal hepatocellular carcinoma efficacy prediction system based on Siamese networks as described in claim 6, characterized in that, The multi-scale fusion module fuses differential features and temporal cross features at the same scale to obtain fused features at the same scale, and fuses fused features at multiple scales to obtain the final fused features, including: The first differential feature is fused with the first temporal cross feature to obtain the first fused feature; The second difference feature is fused with the second temporal cross feature to obtain the second fused feature; The third difference feature is fused with the third temporal cross feature to obtain the third fused feature; The first fusion feature, the second fusion feature, and the third fusion feature are fused to obtain the final fusion feature.

8. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed, performs the prediction system as described in any one of claims 1-7.

Images