A method and system for predicting an immune therapy response in a hepatocellular carcinoma patient

Abstract

The application relates to a method and system for predicting the immunotherapy response of a hepatocellular carcinoma patient, which comprises the following steps: obtaining the to-be-predicted CT image and to-be-predicted clinical data of the hepatocellular carcinoma patient, pre-processing the CT image and the clinical data, inputting the pre-processed CT image and the clinical data into a progressive fusion model, and obtaining the immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is obtained by training a training set; the training set comprises clinical data, intratumoral images and peritumoral images; the progressive fusion model uses a first layer fusion module to fuse the intratumoral images in the arterial phase and the intratumoral images in the portal phase in the to-be-predicted intratumoral images and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the to-be-predicted peritumoral images, a second layer splicing module is used to splice the fused intratumoral and peritumoral images and the to-be-predicted key clinical data, and a third layer classification module is used to classify and predict the spliced results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bioinformatics analysis, and particularly relates to a method and system for predicting the immunotherapy response of a hepatocellular carcinoma patient. BACKGROUND

[0002] Currently, surgical therapy is still considered as the mainstay for the treatment of hepatocellular carcinoma (HCC) patients. However, most HCC patients are diagnosed at an advanced stage, which severely limits the efficacy of surgical treatment. In recent years, the advent of immunotherapy has greatly changed the treatment pattern of cancer, and has attracted great attention in the field of HCC treatment due to its ability to effectively prolong the survival of HCC patients.

[0003] (1) Existing biomarkers have poor accuracy and are based on invasive procedures:

[0004] Due to the low objective response rate of less than 30% of current immunotherapy regimens, there is an urgent need to accurately select immunotherapy patients with high response rates. Previous studies have identified several potential immunotherapy biomarkers, including PD-L1 expression and tumor mutation burden. Although these markers show certain potential in application, they have poor predictive effects on HCC patients and are often based on invasive procedures such as pathological puncture, which has certain risks.

[0005] (2) The current mainstream convolutional neural network model has the disadvantage of being unable to capture long-range dependencies:

[0006] The emergence of medical image analysis provides a basis for non-invasive tumor prediction research. However, the deep learning architecture commonly used in the field of medical image analysis, convolutional neural network (CNN), has the disadvantage of being unable to capture long-range dependencies. In recent years, the emerging Vision transformers technology uses a self-attention mechanism that performs well in capturing long-range dependencies, such as extracting context information and non-local correlation information. Previous studies have also found that deep learning models based on Transformers are more stable than CNN models. Currently, deep learning models based on Transformers have gradually replaced CNN models and become the mainstream model in the field of computer vision. However, the application of Transformer technology in the field of medical image analysis has not been fully explored.

[0007] (3) Current image analysis research on hepatocellular carcinoma immunotherapy focuses on intratumoral images and does not involve peritumoral images:

[0008] Peritumoral imaging analysis is a unique field in medical imaging analysis, which is related to treatment response and immune microenvironment. In previous studies, researchers manually delineated intra-tumoral regions in baseline and follow-up images to extract features to establish predictive models for immunotherapy, however, they mostly ignored the relationship between peritumoral imaging and immunotherapy.

[0009] (4) The current commonly used post-fusion strategy is seriously affected by different modalities and different information densities:

[0010] The construction of deep learning models often involves the fusion of multi-modal data to extract and combine complementary information from different modalities, so as to make better decisions. Multi-modal data fusion is particularly important in the medical field, because findings in a single modality may have different interpretations in multi-modal fusion. However, current multi-modal studies mostly use a post-fusion multi-modal fusion strategy, which models the models trained for each modality separately. This method is seriously affected by different modalities and different information densities. SUMMARY

[0011] To solve the problems existing in the prior art, the purpose of the present application is to provide a prediction method and system for immunotherapy response of hepatocellular carcinoma patients, aiming to develop a multi-modal deep learning model to predict the response of HCC patients to immunotherapy by gradually fusing peritumoral and intratumoral imaging data from arterial and portal phases and clinical features.

[0012] To achieve the above-mentioned purpose, the present application provides the following scheme:

[0013] A prediction method for immunotherapy response of hepatocellular carcinoma patients, comprising:

[0014] Obtaining the predicted CT image and the predicted clinical data of the hepatocellular carcinoma patient, pre-processing the CT image and the clinical data, obtaining the predicted intratumoral and peritumoral imaging and the predicted key clinical data;

[0015] Inputting the predicted intratumoral and peritumoral imaging and the key clinical data into a progressive fusion model to obtain the immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is trained using a training set, and the training set comprises clinical data, intratumoral imaging and peritumoral imaging;

[0016] The progressive fusion model uses a first layer fusion module to fuse the arterial phase intratumoral imaging and the portal phase intratumoral imaging in the predicted intratumoral imaging, and the arterial phase peritumoral imaging and the portal phase peritumoral imaging in the predicted peritumoral imaging, a second layer splicing module to splice the fused intratumoral and peritumoral imaging and the predicted key clinical data, and a third layer classification module to classify and predict the spliced results.

[0017] Optionally, the acquiring the intratumor image and the peritumoral image to be predicted comprises:

[0018] Resampling the CT image, normalizing the resolution of the CT sequence to a target resolution, mapping the intensity value of the CT image to a preset interval, selecting a picture covering the peritumoral or tumor region, and acquiring the intratumor image and the peritumoral image to be predicted.

[0019] Optionally, the acquiring the key clinical data to be predicted comprises:

[0020] Standardizing the clinical data to be predicted, converting the clinical data to be predicted into a standard normal distribution, screening the clinical features related to the immunotherapy response, and acquiring the key clinical data to be predicted.

[0021] Optionally, the clinical data to be predicted comprises age, gender, body mass index, hepatitis B, hepatitis C, portal vein tumor thrombosis, alpha-fetoprotein, total bilirubin, albumin, prothrombin time, neutrophil-to-lymphocyte ratio, and Barcelona Clinic Liver Cancer (BCLC) stage.

[0022] Optionally, the training the progressive fusion model using the training set comprises:

[0023] Performing data enhancement processing on the intratumor image and the peritumoral image in the training set, and training the progressive fusion model using the training set after the data enhancement processing;

[0024] The data enhancement processing mode comprises random horizontal and vertical flipping, random translation, random scaling, and random rotation.

[0025] Optionally, the first layer fusion module comprises:

[0026] A first Conv layer is configured to use convolution to slide multiple times in the arterial phase intratumor image and the portal phase intratumor image in the intratumor image to be predicted and the arterial phase peritumoral image and the portal phase peritumoral image in the peritumoral image to be predicted, respectively, perform element-level multiplication and summation at each sliding position, generate a complete feature map, and acquire a first feature map;

[0027] A MobileNetv2 layer is configured to perform dimension increasing on the first feature map, perform convolution operation on the first feature map after the dimension increasing using a Dwise convolution with a preset step parameter, and perform dimension reduction to acquire a second feature map;

[0028] The first Mobile ViT block layer is configured to perform convolution operation on the second feature map and weighted combination, generate a new feature map, divide the new feature map into a plurality of patches, map each patch to a one-dimensional vector, and set position bias information of each patch, so as to perform global feature learning through a self-attention mechanism and obtain a third feature map.

[0029] Optionally, the MobileNetv2 layer comprises:

[0030] The first MV2 unit is configured to perform dimension increasing on the first feature map, perform convolution operation on the first feature map after dimension increasing by using Dwise convolution with a step parameter of 1, and perform dimension decreasing to obtain a first feature result.

[0031] The first MV2↓2 unit is configured to perform dimension increasing on the first feature result, perform convolution operation on the first feature result after dimension increasing by using Dwise convolution with a step parameter of 2, and perform dimension decreasing to obtain a second feature result.

[0032] The second MV2 unit is configured to perform dimension increasing on the second feature result, perform convolution operation on the second feature result after dimension increasing by using Dwise convolution with a step parameter of 1, and perform dimension decreasing to obtain a third feature result.

[0033] The third MV2 unit is configured to perform dimension increasing on the third feature result, perform convolution operation on the third feature result after dimension increasing by using Dwise convolution with a step parameter of 1, and perform dimension decreasing to obtain a fourth feature result.

[0034] The second MV2↓2 unit is configured to perform dimension increasing on the fourth feature result, perform convolution operation on the fourth feature result after dimension increasing by using Dwise convolution with a step parameter of 2, and perform dimension decreasing to obtain a second feature map.

[0035] Optionally, the second layer splicing module comprises:

[0036] The first MV2↓2 layer is configured to perform dimension increasing on the arterial phase intratumor image and the portal phase intratumor feature in the intratumor image to be predicted and the arterial phase peritumoral image and the portal phase peritumoral feature in the peritumoral image to be predicted in the third feature map respectively, perform convolution operation on the third feature map after dimension increasing by using Dwise convolution with a step parameter of 2, and perform dimension decreasing to obtain a fourth feature map.

[0037] The second Mobile ViT block layer is configured to perform convolution operation on the fourth feature map and weighted combination, generate a new feature map, divide the new feature map into a plurality of patches, map each patch to a one-dimensional vector, and set position bias information of each patch, so as to perform global feature learning through a self-attention mechanism and obtain a fifth feature map.

[0038] The second MV2↓2 layer is configured to perform dimensionality increasing on the fifth feature map, perform convolution operation on the fifth feature map after dimensionality increasing by using Dwise convolution with a step parameter of 2, and perform dimensionality reduction to obtain a sixth feature map.

[0039] The third Mobile ViT block layer is configured to perform convolution operation on the sixth feature map and weighted combination, generate a new feature map, divide the new feature map into a plurality of patches, map each patch to a one-dimensional vector, and set position bias information of each patch, so as to perform global feature learning through a self-attention mechanism and obtain a seventh feature map.

[0040] The second Conv layer is configured to generate a complete feature map by using convolution to slide on the seventh feature map multiple times, perform element-level multiplication and summation at each sliding position, and obtain an eighth feature map.

[0041] Optionally, the second layer splicing module further comprises:

[0042] The MLP layer comprises a plurality of fully connected layers and is configured to perform feature learning and integration on the to-be-predicted key clinical data.

[0043] To achieve the above object, the application further provides a prediction system for immunotherapy response of a hepatocellular carcinoma patient, comprising:

[0044] A data acquisition subsystem is configured to acquire to-be-predicted CT images and to-be-predicted clinical data of a hepatocellular carcinoma patient.

[0045] A data processing subsystem is configured to pre-process the CT images and the clinical data to obtain to-be-predicted intratumoral images, to-be-predicted peritumoral images, to-be-predicted key clinical data.

[0046] A treatment response prediction subsystem is configured to input the to-be-predicted intratumoral images, the to-be-predicted peritumoral images and the key clinical data into a progressive fusion model to obtain the immunotherapy response of the hepatocellular carcinoma patient.

[0047] The progressive fusion model utilizes a first layer fusion module to fuse the arteriophase intratumor image and the portal phase intratumor image in the to-be-predicted intratumor image and the arteriophase peritumoral image and the portal phase peritumoral image in the to-be-predicted peritumoral image, fuses the intratumor image and the peritumoral image after fusion with the to-be-predicted key clinical data through a second layer splicing module, and classifies and predicts the splicing result based on a third layer classification module.

[0048] The present application has the following advantages:

[0049] 1. Previous studies have identified several potential immunotherapy biomarkers, including PD-L1 expression and tumor mutational burden. Although these markers show some potential in application, they have poor predictive effects on HCC patients and are often based on invasive and invasive examinations such as pathological puncture. These examinations require high cost and may cause complications. Our design is based on medical imaging and clinical blood tests, which can be non-invasive prediction.

[0050] 2. Current image analysis research on hepatocellular carcinoma immunotherapy focuses on intratumor images and does not involve peritumoral images. Peritumoral image analysis is a unique field in medical imaging analysis, which is related to treatment response and immune microenvironment. In previous studies, researchers manually delineated intratumoral regions in baseline and follow-up images to extract features to establish an immunotherapy prediction model, but they mostly ignored the relationship between peritumoral images and immunotherapy. Our design also proves that the model (GIFT-IP) that combines intratumor images and peritumoral images is superior to the model (GIFT-I and GIFT-P) that uses intratumor or peritumoral images alone.

[0051] 3. Current deep learning models based on medical imaging are mostly built on convolutional neural network technology. However, convolutional neural networks have the disadvantage of not being able to capture long-range dependencies. The emerging transformer technology uses a self-attention mechanism that performs well in capturing long-range dependencies. Our research selects the MobileVIT architecture that combines transformer technology and CNN technology as the feature extraction backbone of the model to build the model

[0052] 4. The post-fusion strategy commonly used in current multi-modal research is severely affected by different modalities and different information densities. In our research, we introduce the progressive fusion strategy, which first forces the arteriophase and portal phase image data to interact and fuse, and then integrates peritumoral image features, intratumor image features, and clinical features for further fusion. The final generated model (GIFT-CIP) achieves good results in both the internal validation set and the external test set. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0054] Figure 1 A schematic diagram of collecting patient images and clinical data according to an embodiment of the present application;

[0055] Figure 2 A workflow diagram for lesion segmentation, model construction and downstream analysis;

[0056] Figure 3 A schematic diagram of GIFT-CIP model training according to an embodiment of the present application;

[0057] Figure 4 A schematic diagram of training progressive fusion models with different training sets according to an embodiment of the present application; wherein, Figure 4 a is a schematic diagram of GIFT-IP model training, Figure 4 b is a schematic diagram of GIFT-CP model training, Figure 4 c is a schematic diagram of GIFT-P model training, Figure 4 d is a schematic diagram of GIFT-CI model training, Figure 4 e is a schematic diagram of GIFT-I model training;

[0058] Figure 5 A schematic diagram of evaluating the ability of the model to predict immunotherapy response according to an embodiment of the present application; wherein, Figure 5 a is a ROC curve diagram for predicting the ability of immunotherapy response using the training group, Figure 5 b is a ROC curve diagram for predicting the ability of immunotherapy response using the internal validation group, Figure 5 c is a ROC curve diagram for predicting the ability of immunotherapy response using the external test group;

[0059] Figure 6 A Kaplan-Meier curve for progression-free survival and overall survival based on the GIFT-CIP model according to an embodiment of the present application; wherein, Figure 6 a is a Kaplan-Meier curve for progression-free survival of the training group, Figure 6 b is a Kaplan-Meier curve for progression-free survival of the internal validation group, Figure 6 c is a Kaplan-Meier curve for progression-free survival of the external test group, Figure 6 d is a Kaplan-Meier curve for overall survival of the training group, Figure 6e is the Kaplan-Meier curve of the total survival of the internal validation group, Figure 6 f is the Kaplan-Meier curve of the total survival of the external test group;

[0060] Figure 7 is the decision analysis curve of the embodiment of the application; wherein, Figure 7 a is the decision analysis curve in the training group, Figure 7 b is the decision analysis curve in the internal validation group, Figure 7 c is the decision analysis curve in the external test group;

[0061] Figure 8 is the difference schematic diagram of the MobileNetv2 module of the embodiment of the application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the application will be apparently and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the application.

[0063] In order to make the above objectives, characteristics and advantages of the application more apparent, comprehensible and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0064] As Figure 2 shown, the embodiment discloses a prediction method for immunotherapy response of a hepatocellular carcinoma patient, comprising: acquiring to-be-predicted CT images and to-be-predicted clinical data of the hepatocellular carcinoma patient, pre-processing the CT images and the clinical data, acquiring to-be-predicted intratumoral and peritumoral images and to-be-predicted key clinical data; inputting the to-be-predicted intratumoral and peritumoral images and the key clinical data into a progressive fusion model to acquire the immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is obtained by training a training set, the training set comprising: clinical data, intratumoral images and peritumoral images; the progressive fusion model fuses the intratumoral images in the arterial phase and the intratumoral images in the portal phase in the to-be-predicted intratumoral images and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the to-be-predicted peritumoral images by using a first layer fusion module, performs feature splicing on the fused intratumoral and peritumoral images and the to-be-predicted key clinical data by using a second layer splicing module, and performs classification prediction on the splicing result based on a third layer classification module.

[0065] The gradual fusion model (GIFT) uses a first layer fusion module to fuse the arterial phase intratumoral image and the portal phase intratumoral image in the to-be-predicted intratumoral image, and the arterial phase peritumoral image and the portal phase peritumoral image in the to-be-predicted peritumoral image, a second layer splicing module is used to splice the features of the fused intratumoral and peritumoral images and the to-be-predicted key clinical data, and a third layer classification module is used to classify and predict the spliced results. The GIFT model selects the MobileVIT architecture proposed in the International Learning Representation Conference in 2022 as the feature extraction backbone of the model. The first layer is in the order of Conv (convolution layer), MV2, MV2↓2, MV2, MV2, MV2↓2, Mobile ViT block (L=2). The second layer is in the order of MV2↓2, Mobile ViT block (L=4), MV2↓2, Mobile ViT block (L=3). Unlike image information, clinical features are extracted by an MLP (multilayer perceptron) module. The third layer is also composed of an MLP module, and finally classification is performed. The MV2, Mobile ViT block, and MLP module are introduced as follows:

[0066] Conv (convolution layer): a deep learning layer that performs convolution operation. Convolution operation is a conventional and basic operation method. Convolution operation refers to that a convolution kernel slides on an input image, performs element-level multiplication and summation at each position to generate an output value, and this process is repeated multiple times to generate a complete feature map.

[0067] MV2 (MobileNetv2 module): first use 1×1 convolution to increase dimension, then use 3×3 Depthwise (Dwise) convolution, and finally use 1×1 convolution to reduce dimension. RELU6 activation function is used. The layers in the deep learning model are linear function fitting, and it is inevitable to have linear characteristics. After adding a nonlinear activation function unit, when the output layer of the linear function passes through the nonlinear activation unit, the output presents a nonlinear change. Through multiple layers of fitting, nonlinear modeling of the input can be completed, and a feature combination function can also be performed. RELU6 is a conventional non-saturated activation function. MobileNetv2 module is divided into two types, one MV2 (Dwise convolution operation step parameter is 1), and one MV2↓2 (Dwise convolution operation step parameter is 2). The difference between the two MobileNetv2 modules is shown in Figure 8 .

[0068] Mobile ViT block: composed of three sub-modules, local representation, global representations and fusion. First, through the local representation module, first 3x3 convolution operation and then pointwise convolution operation. Pointwise convolution operation is also a common convolution operation, which is very similar to the conventional convolution operation. Its convolution kernel size is 1x1xM, M is the channel number of the last layer. So the convolution operation here will combine the feature map in the depth direction in the last step to generate a new feature map. Then enter the global representation module, which undergoes three steps including unfolding, standard visual Transformer and folding. The standard visual Transformer first divides the input picture into a patch (small block), then maps each patch to a one-dimensional vector through linear transformation, then adds position bias information, and then calculates through L Transformer modules. Finally, output through a fully connected layer. The Transformer module is a basic deep learning module that learns global features through self-attention mechanism. The fully connected layer is a basic component of neural networks. Finally, through the fusion module, first through a 1x1 convolution, then the feature map is concatenated with the input of the Mobile ViT block, and the features are fused through the convolution layer to output.

[0069] MLP (multilayer perceptron) module: composed of multiple fully connected layers. The fully connected layer is a basic component of neural networks. In a layer, each neuron is connected to all neurons in the previous layer, and each connection has its own weight. The fully connected layer has the functions of learning features, integrating information and classifying regression.

[0070] Further, obtaining the to-be-predicted intratumoral and peritumoral image includes: resampling the CT image, normalizing the resolution of the CT sequence to a target resolution, and mapping the intensity value of the CT image to a preset interval, selecting a picture covering the peritumor or tumor region, and obtaining the to-be-predicted intratumoral and peritumoral image.

[0071] Further, obtaining the to-be-predicted key clinical data includes: standardizing the to-be-predicted clinical data, converting the to-be-predicted clinical data to a standard normal distribution, screening clinical features related to immunotherapy response, and obtaining the to-be-predicted key clinical data.

[0072] Specifically, the imaging and clinical data of patients were collected from the research center: patients were excluded according to the following criteria: (a) previously received local treatment, including ablation or transcatheter arterial chemoembolization, (b) imaging or pathology excluded the possibility of HCC, (c) no contrast-enhanced computed tomography (CECT) imaging within 1 month before immunotherapy, (d) received less than two cycles of immunotherapy, (e) no measurable target lesions at baseline, (f) presence of other malignancies.

[0073] Imaging follow-up examinations were scheduled approximately every 8-12 weeks. Immunotherapy response was evaluated between baseline and the first follow-up. Immunotherapy response was determined according to RECIST 1.1. Progressive disease (PD) was defined as an increase of at least 20% in the sum of the longest diameters of target lesions or the appearance of one or more new lesions.

[0074] Clinical data collection: age, gender, body mass index (BMI), hepatitis B, hepatitis C, portal vein tumor thrombus (PVTT), alpha-fetoprotein (AFP, ng / ml), total bilirubin (TBil, μmol / L), albumin (ALB, g / L), prothrombin time (PT, s), neutrophil-to-lymphocyte ratio (NLR), Barcelona Clinic Liver Cancer (BCLC) stage.

[0075] CT image data collection: arterial and portal phase CT images of the screened patients. The CT images of all patients were input into the ITK-SNAP software (version 3.6). The region of interest (ROI) of the CT images was manually segmented from the largest tumor area sequence and the nearest two sequences of the arterial and portal phase CT images, respectively. The intratumoral area was expanded by 10 mm to obtain the peritumoral area.

[0076] Clinical feature and CT image data preprocessing: the clinical features were standardized using the Z-score method, which converts the original clinical features to a standard normal distribution. The LASSO logistic regression method was used to screen the key clinical features related to immunotherapy response.

[0077] The obtained image data is resampled using a linear interpolation algorithm, normalizing the resolution of the CT sequence to 1 mm / pixel, and mapping the intensity values of the images to (0, 1). Pictures of 224x224 pixels covering the entire peritumoral or tumor region are selected. Each picture set includes image pictures of the intratumoral and peritumoral at arterial and portal phases of the same patient at the same level, including an intratumoral picture at arterial phase, a peritumoral picture at arterial phase, an intratumoral picture at portal phase and a peritumoral picture at portal phase of the same patient at the same level. The present application extracts 627 picture sets from 209 patients in the training cohort, 270 picture sets from 90 patients in the internal validation cohort, and 255 picture sets from 85 patients in the external test cohort. Real-time data augmentation techniques are used when training. Each picture has a 20% chance of data augmentation. Data augmentation methods include random horizontal and vertical flipping, random translation (0-30 pixels), random scaling (0-30%) and random rotation. This enhancement is only performed in the training cohort.

[0078] Further, the clinical data to be predicted includes: age, gender, body mass index, hepatitis B, hepatitis C, portal tumor thrombosis, alpha-fetoprotein, total bilirubin, albumin, prothrombin time, neutrophil-to-lymphocyte ratio and Barcelona Clinic Liver Cancer (BCLC) stage.

[0079] Further, training the progressive fusion model using the training set comprises: performing data augmentation processing on the intratumoral image and the peritumoral image in the training set, and training the progressive fusion model using the training set after the data augmentation processing; wherein the data augmentation processing method comprises: random horizontal and vertical flipping, random translation, random scaling and random rotation.

[0080] Specifically, the Transformers-based progressive fusion model is established according to the above-mentioned clinical features and CT multi-modal data: the model is developed using Python (version 3.11) and PyTorch framework (version 2.0.1). The MobileVIT architecture containing MobileNetv2 (MV2) module and MobileVit module is selected as the feature extraction backbone of the model, and the Transformers-based progressive fusion model (GIFT) is built by combining the multi-modal fusion strategy of progressive fusion. The training data is divided into three modalities including clinical data (C), intratumoral image (I) and peritumoral image (P). Six modal combinations for training the model are designed: CIP, IP, CP, P, CI and I. The trained models are GIFT-CIP, GIFT-IP, GIFT-CP, GIFT-P, GIFT-CI and GIFT-I respectively; wherein the model structure is as follows: Figure 3 and Figure 4 a-e.

[0081] As an example of the GIFT-CIP model Figure 3 ), the model consists of three layers: the first layer, the second layer, and the classification layer. At the end of the first layer, the first fusion is performed between the arterial phase and the portal phase image data (including the fusion of the intra-tumor image in the arterial phase and the portal phase, and the fusion of the peri-tumor image in the arterial phase and the portal phase). This fusion is performed using weighted summation (weighting factor 0.5) to facilitate the full interactive fusion of image features and blood supply features. The second fusion is performed at the end of the second layer. The peri-tumor image features, intra-tumor image features, and clinical features are integrated using feature concatenation, and then input to the classification layer for interactive fusion.

[0082] Because the distribution of PD and non-PD patients in the training group is uneven, over-sampling is used to evenly and randomly distribute the patches in each category to improve model accuracy. In the model training process, stochastic gradient descent (SGD) is used for optimization, with a global learning rate of 0.00001 (without decay). Cross-Entropy Loss is selected as the loss function.

[0083] Obtaining the immune therapy response of hepatocellular carcinoma patients includes:

[0084] The pre-processed image data and clinical data are input into the model, and the model's prediction value is mapped to (0, 1) by Sigmoid. The prediction value can approximately represent the probability of progression after immunotherapy. According to the prediction value and the cutoff value, the patients are divided into a high-risk group or a low-risk group.

[0085] If the patient is divided into the high-risk group, the patient is more likely to progress after immunotherapy. This suggests that clinicians can use TACE, HAIC, and other local treatments and targeted drug treatments or receive immunotherapy combined with local treatment, targeted treatment, and other treatments to achieve better prognosis.

[0086] Model accuracy evaluation and model validation:

[0087] The performance of the multi-modal machine learning model is evaluated by drawing the ROC curve, and the model is validated in the training group, internal validation group, and external test group. The "RMDA" package is used to draw the decision curve analysis (DCA) curve to evaluate the clinical utility of the model.

[0088] The maximum Youden index is calculated based on the ROC curve of the training group to obtain the critical value, and then the high-risk group and the low-risk group are divided. The Kaplan-Meier curve is drawn to compare the differences in progression-free survival and overall survival between the high-risk group and the low-risk group.

[0089] The embodiment also discloses a prediction system for immunotherapy response of a hepatocellular carcinoma patient, comprising: a data acquisition subsystem for acquiring to-be-predicted CT images and to-be-predicted clinical data of the hepatocellular carcinoma patient; a data processing subsystem for preprocessing the CT images and the clinical data, acquiring to-be-predicted intratumoral and peritumoral images and to-be-predicted key clinical data; a treatment response prediction subsystem for inputting the to-be-predicted intratumoral and peritumoral images and the key clinical data into a progressive fusion model to acquire the immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is obtained by training a training set, the training set comprising: clinical data, intratumoral images and peritumoral images; the progressive fusion model uses a first layer fusion module to fuse the intratumoral images in the arterial phase and the intratumoral images in the portal phase in the to-be-predicted intratumoral images and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the to-be-predicted peritumoral images, performs feature splicing on the fused intratumoral and peritumoral images and the to-be-predicted key clinical data through a second layer splicing module, and performs classification prediction on the splicing result based on a third layer classification module.

[0090] The embodiment also discloses screening of patients: the present retrospective study is approved by the institutional review board and the ethics committee of Nanfang Hospital (NFH), and the subjects waive informed consent due to the retrospective nature of the study. The workflow is as shown in Figure 1 The present application includes the following hospitals: Nanfang Hospital (May 2018-September 2022), the First Affiliated Hospital of Nanchang University (March 2019-January 2022), the Hospital of Integrated Traditional Chinese and Western Medicine of Southern Medical University (October 2019-May 2023), Jiangxi Cancer Hospital (April 2019-July 2021), and the First Affiliated Hospital of Nanhua University (January 2021-December 2022). Patients are screened according to the inclusion and exclusion criteria, and the specific screening process is shown in Figure 1 .

[0091] The patients screened from Nanfang Hospital are randomly divided into a training set of 209 patients and an internal validation set of 90 patients at a ratio of 7:3. The remaining 85 patients from the other four centers are included in the external validation group. The clinical characteristics of the participants are shown in Table 2. After LASSO logistic regression algorithm, age, NLR, TBil, ALB and PT are selected as key clinical features.

[0092] Model accuracy evaluation and model validation: the receiver operating characteristic (ROC) curves of the GIFT-CIP, GIFT-IP, GIFT-CP, GIFT-P, GIFT-CI and GIFT-I models are shown in Figure 5a-c, Area Under the Curve (AUC), true positive rate, true negative rate, false positive rate, false negative rate of each model, see Table 1. Based on the above indicators, the GIFT-CIP model performs best.

[0093] Based on the maximum Youden index, the optimal cutoff value is 0.56 using the AUC of the training cohort of the GIFT-CIP model. Further, the high-risk group and the low-risk group are divided based on the Kaplan-Meier curve of the GIFT-CIP model, as shown in Figure 6 a-f. In the training group, internal validation group and external test group, the high-risk group and the low-risk group showed significant differences in progression-free survival (p<0.05); in terms of overall survival, in addition to the internal validation group, the training group and the external test group also showed significant differences (p<0.05).

[0094] Decision analysis (DCA) curve of each model, as shown in Figure 7 a-c. The DCA curve is a method of evaluating whether a clinical decision is feasible by considering patient risk and benefit, i.e. whether the clinical decision made using the model in actual clinical practice will do more good than harm. The GIFT-CIP model performs best in the DCA curve.

[0095] In the research of the present application, there is a significant correlation between the arterial phase and the portal phase computed tomography (CT) images, while the correlation between clinical features and intra-tumor and peritumor imaging is relatively small. Progressive fusion as a multi-modal data fusion strategy forces the model to consider the correlation between specific modalities, which can first fuse data with high correlation, and then fuse data with low correlation at a later level. This feature makes the progressive fusion strategy very suitable for the application scenario of the current research. In previous studies, models constructed using the progressive fusion strategy are superior to models using other fusion strategies in some application scenarios.

[0096] Table 1: Prediction performance of the model

[0097]

[0098]

[0099] Wherein, AUC, Area Under the Curve; TPR, true positive rate; TNR, true negative rate; FPR, false positive rate; FNR, false negative rate.

[0100] Table 2: Baseline table of patient clinical characteristics

[0101]

[0102]

[0103] The above embodiments are only used to describe the preferred modes of the present application, and are not used to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements of the technical solutions of the present application made by those skilled in the art shall fall within the protection scope of the present application.

Claims

1. A method of predicting an immunotherapeutic response in a hepatocellular carcinoma patient, characterized in that, The application relates to a method for predicting the immunotherapy response of a hepatocellular carcinoma patient. The method comprises the following steps: acquiring the CT image and the clinical data to be predicted of a hepatocellular carcinoma patient, and preprocessing the CT image and the clinical data to acquire the intratumoral and peritumoral image to be predicted and the key clinical data to be predicted; inputting the intratumoral and peritumoral image to be predicted and the key clinical data into a progressive fusion model to acquire the immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is trained by using a training set, and the training set comprises clinical data, intratumoral image and peritumoral image; the first layer fusion module is used for fusing the intratumoral image to be predicted in the arterial phase and the intratumoral image to be predicted in the portal phase and the peritumoral image to be predicted in the arterial phase and the peritumoral image to be predicted in the portal phase, the second layer splicing module is used for splicing the fused intratumoral and peritumoral image and the key clinical data to be predicted, and the third layer classification module is used for classifying and predicting the splicing result; the first layer fusion module comprises: a first Conv layer, which is used for sliding on the intratumoral image to be predicted in the arterial phase, the intratumoral image to be predicted in the portal phase, the peritumoral image to be predicted in the arterial phase and the peritumoral image to be predicted in the portal phase by using convolution multiple times, performing element-level multiplication and summation at each sliding position, generating complete feature mapping and acquiring a first feature map; a MobileNetv2 layer, which is used for dimensioning the first feature map, performing convolution operation on the dimensioned first feature map by using Dwise convolution with a preset step parameter, and dimensioning to acquire a second feature map; a first Mobile ViT block layer, which is used for performing convolution operation and weighted combination on the second feature map to generate a new feature map, dividing the new feature map into a plurality of blocks, mapping each block to a one-dimensional vector, setting the position bias information of each block, learning global features through a self-attention mechanism to acquire a third feature map; the second layer splicing module comprises: a first MV2↓2 layer, which is used for dimensioning the arterial phase intratumoral image to be predicted and the portal phase intratumoral feature to be predicted in the third feature map and the arterial phase peritumoral image to be predicted and the portal phase peritumoral feature to be predicted, performing convolution operation on the dimensioned third feature map by using Dwise convolution with a step parameter of 2, and dimensioning to acquire a fourth feature map; a second Mobile ViT block layer, which is used for performing convolution operation and weighted combination on the fourth feature map to generate a new feature map, dividing the new feature map into a plurality of blocks, mapping each block to a one-dimensional vector, setting the position bias information of each block, learning global features through a self-attention mechanism to acquire a fifth feature map; a second MV2↓2 layer, which is used for dimensioning the fifth feature map, performing convolution operation on the dimensioned fifth feature map by using Dwise convolution with a step parameter of 2, and dimensioning to acquire a sixth feature map. The third Mobile ViT block layer is configured to perform convolution operation on the sixth feature map and weighted combination, generate a new feature map, divide the new feature map into a plurality of patches, map each patch to a one-dimensional vector, and set position bias information of each patch, so as to perform global feature learning through a self-attention mechanism and obtain a seventh feature map; The second Conv layer is configured to slide on the seventh feature map multiple times by using convolution respectively, perform element-level product and summation at each sliding position, generate a complete feature map, and obtain an eighth feature map; The second layer splicing module further includes: The MLP layer includes a plurality of fully connected layers, and is configured to perform feature learning and integration on the to-be-predicted key clinical data.

2. The method of predicting the immunotherapeutic response of a patient with hepatocellular carcinoma according to claim 1, characterized in that, The to-be-predicted intratumoral and peritumoral images are obtained by: Resampling the CT images, normalizing the resolution of the CT sequence to a target resolution, mapping the intensity value of the CT images to a preset interval, selecting pictures covering the peritumoral or tumor region, and obtaining the to-be-predicted intratumoral and peritumoral images.

3. The method of predicting the immunotherapeutic response of a patient with hepatocellular carcinoma according to claim 1, characterized in that, The to-be-predicted key clinical data are obtained by: Standardizing the to-be-predicted clinical data, converting the to-be-predicted clinical data into a standard normal distribution, screening clinical features related to immunotherapy response, and obtaining the to-be-predicted key clinical data.

4. The method of predicting the immunotherapeutic response of a patient with hepatocellular carcinoma according to claim 1, characterized in that, The to-be-predicted clinical data include: age, gender, body mass index, hepatitis B, hepatitis C, portal vein tumor thrombus, alpha-fetoprotein, total bilirubin, albumin, prothrombin time, neutrophil-to-lymphocyte ratio, and Barcelona clinical liver cancer stage.

5. The method of predicting the immunotherapeutic response of a patient with hepatocellular carcinoma according to claim 1, characterized in that, The progressive fusion model is trained by using the training set, and the training includes: Data enhancement processing is performed on the intratumoral images and the peritumoral images in the training set, and the progressive fusion model is trained by using the training set after the data enhancement processing; The data enhancement processing mode includes: random horizontal and vertical flipping, random translation, random scaling, and random rotation.

6. The method of predicting the immunotherapeutic response of a patient with hepatocellular carcinoma according to claim 1, characterized in that, The MobileNetv2 layer includes: The first MV2 unit is configured to perform dimension increasing on the first feature map, perform convolution operation on the first feature map after the dimension increasing by using Dwise convolution with a step parameter of 1, and perform dimension decreasing to obtain a first feature result; The first MV2↓2 unit is configured to perform dimension increasing on the first feature result, perform convolution operation on the first feature result after the dimension increasing by using Dwise convolution with a step parameter of 2, and perform dimension decreasing to obtain a second feature result; The second MV2 unit is configured to perform dimension increasing on the second feature result, perform convolution operation on the second feature result after the dimension increasing by using Dwise convolution with a step parameter of 1, and perform dimension decreasing to obtain a third feature result; The third MV2 unit is configured to perform dimension increasing on the third feature result, perform convolution operation on the third feature result after the dimension increasing by using Dwise convolution with a step parameter of 1, and perform dimension decreasing to obtain a fourth feature result; A second MV2↓2 unit is configured to up-sample the fourth feature result, perform convolution operation on the up-sampled fourth feature result by using Dwise convolution with a step size of 2, and down-sample the result to obtain a second feature map.

7. A system for predicting the immunotherapeutic response of a patient with hepatocellular carcinoma, implemented by means of the method according to any one of claims 1 to 6, characterized in that, The application relates to a method for predicting the immunotherapy response of a hepatocellular carcinoma patient. The data acquisition subsystem is configured to acquire CT images and clinical data of the hepatocellular carcinoma patient to be predicted. The data processing subsystem is configured to preprocess the CT images and the clinical data to obtain intra-tumor images and peritumoral images to be predicted and key clinical data to be predicted. The treatment response prediction subsystem is configured to input the intra-tumor images and the peritumoral images to be predicted and the key clinical data to be predicted into a progressive fusion model to obtain the immunotherapy response of the hepatocellular carcinoma patient. The progressive fusion model is trained by using a training set, and the training set comprises clinical data, intra-tumor images and peritumoral images. The progressive fusion model is configured to fuse the intra-tumor images to be predicted in the arterial phase and the intra-tumor images to be predicted in the portal phase and the peritumoral images to be predicted in the arterial phase and the peritumoral images to be predicted in the portal phase by using a first layer fusion module, perform feature splicing on the fused intra-tumor images and the peritumoral images and the key clinical data to be predicted by using a second layer splicing module, and perform classification prediction on the spliced result by using a third layer classification module.

Images