Prediction method and system for immunotherapy response of hepatocellular carcinoma patient

Abstract

The invention relates to a hepatocellular carcinoma patient immunotherapy response prediction method and system, and the method comprises the steps: obtaining a to-be-predicted CT image and to-be-predicted clinical data of a hepatocellular carcinoma patient, carrying out the preprocessing of the CT image and the clinical data, inputting the preprocessed CT image and clinical data into a progressive fusion model, and carrying out the prediction of the immunotherapy response of the hepatocellular carcinoma patient. Obtaining an immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is obtained by training through a training set, and the training set comprises clinical data, intratumoral images and peritumoral images; according to the progressive fusion model, a first-layer fusion module is used for fusing an arterial phase intratumoral image and a portal phase intratumoral image in an intratumoral image to be predicted and fusing an arterial phase peritumoral image and a portal phase peritumoral image in a peritumoral image to be predicted; and performing feature splicing on the fused intratumoral and peritumoral images and to-be-predicted key clinical data through a second-layer splicing module, and performing classification prediction on a splicing result based on a third-layer classification module.

Description

Technical Field

[0001] The present invention relates to the technical field of bioinformatics analysis, and in particular to a method and system for predicting immunotherapy response of patients with hepatocellular carcinoma. Background Art

[0002] Currently, surgical treatment is still regarded as the mainstay of treatment for hepatocellular carcinoma (HCC), but most HCC patients are diagnosed only in the late stage, which severely limits the efficacy of surgical treatment. In recent years, the advent of immunotherapy has greatly changed the treatment landscape of cancer. It has attracted great attention in the field of HCC treatment because it can effectively prolong the survival of HCC patients.

[0003] (1) Existing biomarkers have poor accuracy and are based on invasive tests:

[0004] Since the objective response rate of current immunotherapy regimens is less than 30%, there is an urgent need to accurately select patients with high response rates for immunotherapy. Previous studies have identified several potential immunotherapy biomarkers, including PD-L1 expression and tumor mutation burden. Although these markers have shown certain potential in application, their predictive effect on HCC patients is poor and these biomarkers are often based on invasive examinations such as pathological punctures, which have certain risks.

[0005] (2) The current mainstream convolutional neural network model has the disadvantage of not being able to capture long-range dependencies:

[0006] The emergence of medical image analysis has provided a foundation for non-invasive tumor prediction research. However, the convolutional neural network (CNN), a deep learning architecture commonly used in the field of medical image analysis, has the disadvantage of not being able to capture long-range dependencies. In recent years, the self-attention mechanism used by the emerging vision transformer technology has performed well in capturing long-range dependencies, such as extracting contextual information and non-local correlation information. Previous studies have also found that Transformer-based deep learning models are more stable than CNN models. At present, Transformer-based deep learning models have gradually replaced CNN models and become the mainstream models in the field of computer vision. However, the application of Transformer technology in the field of medical image analysis has not been fully explored.

[0007] (3) At present, imaging analysis studies of immunotherapy for hepatocellular carcinoma focus on intratumoral imaging but do not involve peritumoral imaging:

[0008] Peritumoral imaging analysis is a unique area in medical imaging analysis that is related to treatment response and immune microenvironment. In previous studies, researchers manually delineated intratumoral regions in baseline and follow-up images to extract features to build immunotherapy prediction models, but they mostly ignored the relationship between peritumoral imaging and immunotherapy.

[0009] (4) The commonly used post-fusion strategy is easily affected by the different information densities of different modalities:

[0010] The construction of deep learning models often involves the fusion of multimodal data to extract and combine complementary information from different modalities to make better decisions. The fusion of multimodal data is particularly important in the medical field, because findings in a single modality may have different interpretations in multimodal fusion. However, most current multimodal studies have adopted a post-fusion multimodal fusion strategy, which aggregates models trained separately for each modality for modeling. This approach is severely affected by the different information densities of different modalities. Summary of the invention

[0011] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method and system for predicting the immunotherapy response of patients with hepatocellular carcinoma, aiming to develop a multimodal deep learning model, which integrates the peritumoral and intratumoral imaging data and clinical characteristics from the arterial and portal phases through a progressive fusion strategy to predict the response of HCC patients to immunotherapy.

[0012] To achieve the above object, the present invention provides the following solutions:

[0013] A method for predicting immunotherapy response in patients with hepatocellular carcinoma, comprising:

[0014] Obtaining CT images and clinical data to be predicted of patients with hepatocellular carcinoma, preprocessing the CT images and clinical data, and obtaining intra-tumor and peritumoral images and key clinical data to be predicted;

[0015] Inputting the intratumoral and peritumoral images to be predicted and key clinical data into a progressive fusion model to obtain the immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is trained using a training set, and the training set includes: clinical data, intratumoral images and peritumoral images;

[0016] The progressive fusion model uses the first-layer fusion module to fuse the intratumor images in the arterial phase and the intratumor images in the portal phase in the intratumor images to be predicted, and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, and uses the second-layer splicing module to perform feature splicing on the fused intratumor and peritumoral images and the key clinical data to be predicted, and classifies and predicts the splicing results based on the third-layer classification module.

[0017] Optionally, obtaining the intra-tumor and peritumoral images to be predicted includes:

[0018] The CT image is resampled, the resolution of the CT sequence is normalized to the target resolution, and the intensity value of the CT image is mapped to a preset interval, and a picture covering the peritumoral or tumor area is selected to obtain the intratumoral and peritumoral images to be predicted.

[0019] Optionally, obtaining the key clinical data to be predicted includes:

[0020] The clinical data to be predicted are standardized, converted into a standard normal distribution, and clinical features related to immunotherapy response are screened to obtain the key clinical data to be predicted.

[0021] Optionally, the clinical data to be predicted include: age, gender, body mass index, hepatitis B, hepatitis C, portal vein tumor thrombosis, alpha-fetoprotein, total bilirubin, albumin, prothrombin time, neutrophil-to-lymphocyte ratio and Barcelona Clinic Liver Cancer staging.

[0022] Optionally, using the training set to train and obtain the progressive fusion model includes:

[0023] Performing data enhancement processing on the intratumor images and peritumoral images in the training set, and using the training set after the data enhancement processing to train and obtain the progressive fusion model;

[0024] Among them, data enhancement processing methods include: random horizontal and vertical flipping, random translation, random scaling and random rotation.

[0025] Optionally, the first-layer fusion module includes:

[0026] The first Conv layer is used to use convolution to slide multiple times on the intratumor images in the arterial phase and the intratumor images in the portal phase in the intratumor images to be predicted, and on the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, perform element-wise multiplication and sum at each sliding position, generate a complete feature map, and obtain a first feature map;

[0027] The MobileNetv2 layer is used to increase the dimension of the first feature map, perform a convolution operation on the first feature map after the dimension increase using a Dwise convolution with a preset step size parameter, and perform a dimension reduction to obtain a second feature map;

[0028] The first Mobile ViT block layer is used to perform convolution operation and weighted combination on the second feature map to generate a new feature map, divide the new feature map into a number of blocks, map each block to a one-dimensional vector, and set the position bias information of each block, so as to perform global feature learning through a self-attention mechanism to obtain a third feature map.

[0029] Optionally, the MobileNetv2 layer includes:

[0030] A first MV2 unit is used to increase the dimension of the first feature map, perform a convolution operation on the first feature map after the dimension increase using a Dwise convolution with a step size parameter of 1, and perform dimension reduction to obtain a first feature result;

[0031] A first MV2↓2 unit is used to increase the dimension of the first feature result, perform a convolution operation on the first feature result after the dimension increase using a Dwise convolution with a step size parameter of 2, and perform dimension reduction to obtain a second feature result;

[0032] The second MV2 unit is used to increase the dimension of the second feature result, perform a convolution operation on the second feature result after the dimension increase using a Dwise convolution with a step size parameter of 1, and perform dimension reduction to obtain a third feature result;

[0033] A third MV2 unit is used to increase the dimension of the third feature result, perform a convolution operation on the increased dimension third feature result using a Dwise convolution with a step size parameter of 1, and perform dimensionality reduction to obtain a fourth feature result;

[0034] The second MV2↓2 unit is used to increase the dimension of the fourth feature result, perform a convolution operation on the increased dimension fourth feature result using a Dwise convolution with a step parameter of 2, and perform dimensionality reduction to obtain a second feature map.

[0035] Optionally, the second layer splicing module includes:

[0036] The first MV2↓2 layer is used to respectively increase the dimension of the intratumoral image and the portal vein phase intratumoral features in the intratumoral image to be predicted and the peritumoral image and the portal vein phase peritumoral features in the peritumoral image to be predicted in the third feature map, perform a convolution operation on the increased-dimensional third feature map using a Dwise convolution with a step size parameter of 2, and perform dimensionality reduction to obtain a fourth feature map;

[0037] The second Mobile ViT block layer is used to perform convolution operation and weighted combination on the fourth feature map to generate a new feature map, divide the new feature map into a plurality of blocks, map each block to a one-dimensional vector, and set the position bias information of each block, so as to perform global feature learning through a self-attention mechanism to obtain a fifth feature map;

[0038] The second MV2↓2 layer is used to increase the dimension of the fifth feature map, perform a convolution operation on the increased dimension of the fifth feature map using a Dwise convolution with a step size parameter of 2, and perform dimensionality reduction to obtain a sixth feature map;

[0039] The third Mobile ViT block layer is used to perform convolution operation and weighted combination on the sixth feature map to generate a new feature map, divide the new feature map into a plurality of blocks, map each block to a one-dimensional vector, and set the position bias information of each block, so as to perform global feature learning through a self-attention mechanism to obtain a seventh feature map;

[0040] The second Conv layer is used to slide on the seventh feature map multiple times by using convolution, perform element-level multiplication and sum at each sliding position, generate a complete feature map, and obtain the eighth feature map.

[0041] Optionally, the second layer splicing module further includes:

[0042] The MLP layer includes several fully connected layers, which are used to perform feature learning and integration on the key clinical data to be predicted.

[0043] To achieve the above object, the present invention also provides a prediction system for immunotherapy response of hepatocellular carcinoma patients, comprising:

[0044] A data acquisition subsystem, used to acquire CT images and clinical data to be predicted of patients with hepatocellular carcinoma;

[0045] A data processing subsystem, used for preprocessing the CT images and clinical data to obtain intratumoral and peritumoral images to be predicted and key clinical data to be predicted;

[0046] The treatment response prediction subsystem inputs the intratumoral and peritumoral images to be predicted and key clinical data into a progressive fusion model to obtain the immunotherapy response of the hepatocellular carcinoma patient; the progressive fusion model is trained using a training set, and the training set includes: clinical data, intratumoral images and peritumoral images;

[0047] The progressive fusion model uses the first-layer fusion module to fuse the intratumor images in the arterial phase and the intratumor images in the portal phase in the intratumor images to be predicted, and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, and uses the second-layer splicing module to perform feature splicing on the fused intratumor and peritumoral images and the key clinical data to be predicted, and classifies and predicts the splicing results based on the third-layer classification module.

[0048] The beneficial effects of the present invention are:

[0049] 1. Previous studies have identified several potential immunotherapy biomarkers, including PD-L1 expression and tumor mutation burden. Although these markers have shown some potential in application, the predictive effect on HCC patients is poor and these biomarkers are often based on invasive examinations such as pathological puncture. These examinations require high costs and may cause complications. Our design is based on medical imaging and clinical blood tests, which can perform non-invasive predictions.

[0050] 2. Currently, imaging analysis studies of immunotherapy for hepatocellular carcinoma focus on intratumoral images without involving peritumoral images. Peritumoral imaging analysis is a unique field in medical imaging analysis, which is related to treatment response and immune microenvironment. In previous studies, researchers manually delineated intratumoral regions in baseline and follow-up images to extract features to establish immunotherapy prediction models, but they mostly ignored the relationship between peritumoral images and immunotherapy. Our design also demonstrated that the model that integrates intratumoral and peritumoral images (GIFT-IP) is superior to the models with single modality of intratumoral or peritumoral images (GIFT-I and GIFT-P).

[0051] 3. Currently, most deep learning models for medical images are based on convolutional neural network technology. However, convolutional neural networks have the disadvantage of not being able to capture long-range dependencies. The self-attention mechanism used by the emerging transformer technology performs well in capturing long-range dependencies. Our research selected the MobileVIT architecture that combines transformer technology with CNN technology as the feature extraction backbone of the model to build the model.

[0052] 4. The post-fusion strategy commonly used in multimodal research is easily affected by the different information densities of different modalities. In this study, we introduced a progressive fusion strategy. The model first forced the arterial phase and portal venous phase image data to be interactively fused, and then further integrated the peritumoral image features, intratumoral image features and clinical features. The final generated model (GIFT-CIP) achieved good results in both the internal validation set and the external test set. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A schematic diagram of collecting patient images and clinical data according to an embodiment of the present invention;

[0055] Figure 2 Workflow diagram for lesion segmentation, model building, and downstream analysis;

[0056] Figure 3 A schematic diagram of GIFT-CIP model training according to an embodiment of the present invention;

[0057] Figure 4 Schematic diagram of training a progressive fusion model with different training sets according to an embodiment of the present invention; wherein, Figure 4 a is a schematic diagram of GIFT-IP model training. Figure 4 b is a schematic diagram of GIFT-CP model training. Figure 4 c is a schematic diagram of GIFT-P model training. Figure 4 d is a schematic diagram of GIFT-CI model training. Figure 4 e is a schematic diagram of GIFT-I model training;

[0058] Figure 5 This is a schematic diagram of the ability of the evaluation model of an embodiment of the present invention to predict immunotherapy response; wherein, Figure 5 a is the ROC curve of the ability to predict immunotherapy response using the training group, Figure 5 b is the ROC curve of the ability to predict immunotherapy response using the internal validation group, Figure 5 c is the ROC curve of the ability to predict immunotherapy response using an external test group;

[0059] Figure 6 is the Kaplan-Meier curve of progression-free survival and overall survival based on the GIFT-CIP model in an embodiment of the present invention; wherein, Figure 6 a is the Kaplan-Meier curve of progression-free survival in the training group. Figure 6 b is the Kaplan-Meier curve of progression-free survival in the internal validation group. Figure 6 c is the Kaplan-Meier curve of progression-free survival in the external test group. Figure 6 d is the Kaplan-Meier curve of overall survival of the training group, Figure 6e is the Kaplan-Meier curve of overall survival of the internal validation group. Figure 6 f is the Kaplan-Meier curve of overall survival of the external test group;

[0060] Figure 7 is a decision analysis curve of an embodiment of the present invention; wherein, Figure 7 a is the decision analysis curve in the training group, Figure 7 b is the decision analysis curve in the internal validation group, Figure 7 c is the decision analysis curve in the external test group;

[0061] Figure 8 Schematic diagram of the difference of the MobileNetv2 module in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] like Figure 2 As shown, the present embodiment discloses a method for predicting the immunotherapy response of patients with hepatocellular carcinoma, including: obtaining CT images to be predicted and clinical data to be predicted of patients with hepatocellular carcinoma, preprocessing the CT images and clinical data, obtaining intratumoral and peritumoral images to be predicted and key clinical data to be predicted; inputting the intratumoral and peritumoral images to be predicted and the key clinical data to be predicted into a progressive fusion model to obtain the immunotherapy response of patients with hepatocellular carcinoma; the progressive fusion model is trained using a training set, and the training set includes: clinical data, intratumoral images and peritumoral images; the progressive fusion model uses a first-layer fusion module to fuse the intratumoral images in the arterial phase with the intratumoral images in the portal phase in the intratumoral images to be predicted, and between the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, and features are spliced ​​on the fused intratumoral and peritumoral images and the key clinical data to be predicted through a second-layer splicing module, and classification and prediction are performed on the splicing results based on the third-layer classification module.

[0065] The gradual fusion transformer-based model (GIFT) uses the first-layer fusion module to fuse the intratumor images in the arterial phase and the portal phase in the intratumor images to be predicted, and the peritumoral images in the arterial phase and the portal phase in the peritumoral images to be predicted. The second-layer splicing module performs feature splicing on the fused intratumor and peritumoral images and the key clinical data to be predicted, and the splicing results are classified and predicted based on the third-layer classification module. The GIFT model uses the MobileVIT architecture proposed in the 2022 International Conference on Learning Representations as the feature extraction backbone of the model. The first layer is Conv (convolutional layer), MV2, MV2↓2, MV2, MV2, MV2↓2, Mobile ViT block (L=2) in order. The second layer is MV2↓2, Mobile ViT block (L=4), MV2↓2, Mobile ViT block (L=3) in order. Unlike image information, clinical features are extracted through an MLP (multilayerperceptron) module. The third layer is also composed of an MLP module, and finally classified. The introduction of MV2, Mobile ViT block, and MLP module is as follows:

[0066] Conv (convolutional layer): A deep learning layer that performs convolution operations. Convolution operations are a common and basic operation method. Convolution operations refer to sliding the convolution kernel on the input image, performing element-wise multiplication and summing at each position to generate an output value. This process is repeated many times to generate a complete feature map.

[0067] MV2 (MobileNetv2 module): First use 1×1 convolution to achieve dimensionality increase, then use 3×3Depthwise (Dwise) convolution, and finally use 1×1 convolution to achieve dimensionality reduction. Use RELU6 activation function. The layers in the deep learning model are all linear function fittings, and linear characteristics are inevitable. After adding the nonlinear activation function unit, when the output layer of the linear function passes through the nonlinear activation unit, its output presents a nonlinear change. In this way, after multiple layers of fitting, the nonlinear modeling operation of the input can be completed, and it can also play a role of feature combination. RELU6 is a conventional non-saturated activation function. MobileNetv2 modules are divided into two types, one is MV2 (the step size parameter is 1 when performing Dwise convolution operations), and the other is MV2↓2 (the step size parameter is 2 when performing Dwise convolution operations). The difference between the two MobileNetv2 modules can be seen in Figure 8 .

[0068] Mobile ViT block: It consists of three sub-modules: local representation module, global representation module and fusion module. First, through the local representation module, a 3x3 convolution operation and a Pointwise convolution operation are performed. Pointwise convolution operation is also a common convolution operation. It is very similar to the conventional convolution operation. The size of its convolution kernel is 1×1×M, where M is the number of channels in the previous layer. Therefore, the convolution operation here will perform a weighted combination of the feature map of the previous step in the depth direction to generate a new feature map. Then enter the global representation module, which undergoes three steps including unfolding, standard visual transformer and folding. The standard visual transformer first divides the input image into patches, then maps each patch to a one-dimensional vector through linear changes, then adds the position bias information, and then calculates through L transformer modules, and finally outputs through a fully connected layer. The Transformer module is a basic and commonly used deep learning module that performs global feature learning through the self-attention mechanism. The fully connected layer is a basic component in the neural network. Finally, through the fusion module, a 1x1 convolution is first performed, and then the feature map is concatenated with the input of the Mobile ViT block, and the feature fusion is output through the convolution layer.

[0069] MLP (multilayer perceptron) module: It consists of multiple fully connected layers. Fully connected layers are a basic component in neural networks. In a layer, each neuron is connected to all neurons in the previous layer, and each connection has its own weight. Fully connected layers have the functions of learning features, integrating information, and classification and regression.

[0070] Furthermore, obtaining the intratumoral and peritumoral images to be predicted includes: resampling the CT images, normalizing the resolution of the CT sequence to the target resolution, and mapping the intensity value of the CT image to a preset interval, selecting an image covering the peritumoral or tumor area, and obtaining the intratumoral and peritumoral images to be predicted.

[0071] Furthermore, obtaining the key clinical data to be predicted includes: standardizing the clinical data to be predicted, converting the clinical data to be predicted into a standard normal distribution, screening clinical features related to immunotherapy response, and obtaining the key clinical data to be predicted.

[0072] Specifically, the imaging and clinical data of the patients were collected from the research center: patients were excluded according to the following criteria: (a) previous local treatment, including ablation or transcatheter arterial chemoembolization, (b) the possibility of HCC was excluded by imaging or pathology, (c) no contrast-enhanced computed tomography (CECT) imaging within 1 month before immunotherapy, (d) less than two cycles of immunotherapy, (e) no measurable target lesions at baseline, and (f) the presence of other malignancies.

[0073] Follow-up imaging examinations were scheduled approximately every 8-12 weeks. Immunotherapy response was assessed between baseline and first follow-up. Immunotherapy response was determined according to RECIST 1.1. Progressive disease (PD) was defined as at least a 20% increase in the sum of the longest diameters of target lesions or the appearance of one or more new lesions.

[0074] Clinical data collection: age, gender, body mass index (BMI), hepatitis B, hepatitis C, portal vein tumor thrombosis (PVTT), alpha-fetoprotein (AFP, ng / ml), total bilirubin (TBil, μmol / L), albumin (ALB, g / L), prothrombin time (PT, s), neutrophil-to-lymphocyte ratio (NLR), and Barcelona Clinic Liver Cancer (BCLC) staging.

[0075] CT image data collection: Arterial and portal venous phase CT images of the selected patients. The CT images of all patients were input into ITK-SNAP software (version 3.6). The region of interest (ROI) of the CT image was manually segmented from the largest tumor region sequence and the two nearest sequences in the arterial and portal venous phase CT images. The intratumoral region was expanded by 10 mm to obtain the peritumoral region.

[0076] Clinical characteristics and CT image data preprocessing: Clinical characteristics were standardized using the Z-score method, which converted the original clinical characteristics into a standard normal distribution. LASSO logistic regression was used to screen key clinical characteristics associated with immunotherapy response.

[0077] The obtained image data is resampled by linear interpolation algorithm, the resolution of CT sequence is normalized to 1mm / pixel, and the intensity value of the image is mapped to (0,1). A 224×224 pixel image covering the entire peritumoral or tumor area is selected. Each image set includes intratumoral and peritumoral images of the same patient at the same level in the arterial phase and portal phase, including an intratumoral image in the arterial phase, an arterial phase peritumoral image, a portal phase intratumoral image and a portal phase peritumoral image of the same patient at the same level. The present invention extracts 627 image sets from 209 patients in the training cohort, 270 image sets from 90 patients in the internal validation cohort, and 255 image sets from 85 patients in the external test cohort. During training, real-time data enhancement technology is used. Each image has a 20% chance of data enhancement. Data enhancement methods include random horizontal and vertical flipping, random translation (0-30 pixels), random scaling (0-30%) and random rotation. This enhancement is only done in the training cohort.

[0078] Furthermore, the clinical data to be predicted include: age, sex, body mass index, hepatitis B, hepatitis C, portal vein tumor thrombosis, alpha-fetoprotein, total bilirubin, albumin, prothrombin time, neutrophil-to-lymphocyte ratio and Barcelona Clinic Liver Cancer stage.

[0079] Furthermore, using the training set to train the progressive fusion model includes: performing data enhancement processing on the intratumor images and peritumoral images in the training set, and using the training set after the data enhancement processing to train the progressive fusion model; wherein the data enhancement processing method includes: random horizontal and vertical flipping, random translation, random scaling and random rotation.

[0080] Specifically, a progressive fusion model based on Transformers was established according to the above clinical characteristics and multimodal data of CT: the model was developed using Python (version 3.11) and PyTorch framework (version 2.0.1). The MobileVIT architecture including the MobileNetv2 (MV2) module and the MobileVit module was selected as the feature extraction backbone of the model, and combined with the multimodal fusion strategy of progressive fusion, a progressive fusion model based on Transformers (GIFT) was built. The training data is divided into three modalities, including clinical data (C), intratumoral images (I) and peritumoral images (P). Six modality combinations for training models were designed: CIP, IP, CP, P, CI and I. The trained models are GIFT-CIP, GIFT-IP, GIFT-CP, GIFT-P, GIFT-CI and GIFT-I, respectively; among them, the model structure, such as Figure 3 and Figure 4 ae.

[0081] Take the GIFT-CIP model as an example ( Figure 3 ), the model consists of three layers: the first layer, the second layer and the classification layer. At the end of the first layer, the first fusion is performed between the arterial phase and the portal venous phase image data (including the fusion of the arterial phase and the portal phase of the intratumoral image, and the fusion of the arterial phase and the portal phase of the peritumoral image). This fusion is performed by weighted summation (weighting coefficient 0.5) to promote the full interactive fusion of image features and blood supply features. The second fusion is performed at the end of the second layer. The peritumoral image features, intratumoral image features and clinical features are integrated by feature concatenation (concat), and then input into the classification layer for interactive fusion.

[0082] Since PD and non-PD patients are unevenly distributed in the training group, the patches are uniformly and randomly distributed in each category by oversampling to improve the model accuracy. Stochastic gradient descent (SGD) is used for optimization during model training, and the global learning rate is 0.00001 (no decay). Cross-Entropy Loss is selected as the loss function.

[0083] Obtaining a response to immunotherapy in patients with hepatocellular carcinoma involves:

[0084] The preprocessed imaging data and clinical data are input into the model, and the predicted value of the model is mapped to (0, 1) by Sigmoid, which can approximately represent the probability of progression after immunotherapy. Patients are divided into high-risk group or low-risk group according to the predicted value and cutoff value.

[0085] If patients are classified as high-risk groups, they may have a higher chance of progression after immunotherapy. This suggests that high-risk patients can receive local treatments such as TACE and HAIC and targeted drug treatments, or receive immunotherapy combined with other treatments such as local treatments and targeted treatments to achieve better prognosis.

[0086] Model accuracy assessment and model validation:

[0087] The performance of the multimodal machine learning model was evaluated by drawing ROC curves, and the model was validated in the training group, internal validation group, and external test group. The decision curve analysis (DCA) curve was drawn using the "RMDA" package to evaluate the clinical practicality of the model.

[0088] The maximum Youden index was calculated based on the ROC curve of the training group to obtain the critical value, and then the high-risk group and low-risk group were divided. The Kaplan-Meier curve was drawn to compare the differences in progression-free survival and overall survival between the high-risk group and the low-risk group.

[0089] The present embodiment also discloses a prediction system for immunotherapy response of hepatocellular carcinoma patients, including: a data acquisition subsystem, used to obtain CT images and clinical data to be predicted of hepatocellular carcinoma patients; a data processing subsystem, used to pre-process CT images and clinical data, and obtain intratumoral and peritumoral images to be predicted and key clinical data to be predicted; a treatment response prediction subsystem, which inputs the intratumoral and peritumoral images and key clinical data to be predicted into a progressive fusion model to obtain the immunotherapy response of hepatocellular carcinoma patients; the progressive fusion model is obtained by training with a training set, and the training set includes: clinical data, intratumoral images and peritumoral images; the progressive fusion model uses a first-layer fusion module to fuse the intratumoral images in the arterial phase and the intratumoral images in the portal phase in the intratumoral images to be predicted, and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, and the second-layer splicing module is used to feature splice the fused intratumoral and peritumoral images and the key clinical data to be predicted, and the splicing results are classified and predicted based on the third-layer classification module.

[0090] This example also discloses the screening of patients: This retrospective study was approved by the Institutional Review Board and Ethics Committee of Nanfang Hospital (NFH). Due to the retrospective nature of the study, the subjects waived informed consent. Figure 1 As shown. The present invention incorporates patients who have received treatment at Nanfang Hospital (May 2018 to September 2022), the First Affiliated Hospital of Nanchang University (March 2019 to January 2022), the Hospital of Integrated Traditional Chinese and Western Medicine of Southern Medical University (October 2019 to May 2023), Jiangxi Cancer Hospital (April 2019-July 2021), and the First Affiliated Hospital of South China University of Technology (January 2021-December 2022). Patients were screened according to the inclusion and exclusion criteria, and the specific screening process was as follows: Figure 1 .

[0091] The patients screened from Nanfang Hospital were randomly divided into a training set of 209 and an internal validation set of 90 in a ratio of 7:3. The remaining four centers screened 85 patients for the external validation group. The clinical characteristics of the participants are shown in Table 2. After the LASSO logistic regression algorithm, age, NLR, TBil, ALB, and PT were screened as key clinical characteristics.

[0092] Model accuracy evaluation and model validation: Receiver operating characteristic (ROC) curves of GIFT-CIP, GIFT-IP, GIFT-CP, GIFT-P, GIFT-CI and GIFT-I models, such as Figure 5ac, the area under the curve (AUC), true positive rate, true negative rate, false positive rate, and false negative rate of each model are shown in Table 1. Based on the above indicators, the GIFT-CIP model performed best.

[0093] Based on the maximum Youden index, the optimal cutoff value (cutoff value) was determined using the training cohort AUC of the GIFT-CIP model, which was 0.56. The high-risk group and low-risk group were then divided based on the Kaplan-Meier curve of the GIFT-CIP model, as shown in Figure 6 af. In the training group, internal validation group, and external test group, the high-risk group and the low-risk group showed significant differences in progression-free survival (p<0.05); in terms of overall survival, except for the internal validation group, the training group and the external test group also showed significant differences (p<0.05).

[0094] The decision analysis (DCA) curves of each model are as follows: Figure 7 ac. The DCA curve is a method for evaluating whether clinical decisions are feasible by considering the risks and benefits of patients, that is, whether the clinical decisions made using this model in actual clinical practice will have more benefits than harms. The GIFT-CIP model performed best in the DCA curve.

[0095] In the present study, there was a significant correlation between arterial and portal venous phase computed tomography (CT) images, whereas the correlation between clinical features and intra-tumoral and peritumoral imaging was small. Progressive fusion, as a multimodal data fusion strategy, forces the model to consider the correlation between specific modalities, and can fuse highly correlated data first, and then fuse less correlated data at a later level. This feature makes the progressive fusion strategy very suitable for the application scenario of the current study. In previous studies, models built using the progressive fusion strategy outperformed models using other fusion strategies in some application scenarios.

[0096] Table 1: Prediction performance of the model

[0097]

[0098]

[0099] Among them, AUC, Area Under the Curve; TPR, true positive rate, true positive rate; TNR, true negative rate, true negative rate; FPR, false positive rate, false positive rate; FNR, false negative rate, false negative rate.

[0100] Table 2: Baseline clinical characteristics of patients

[0101]

[0102]

[0103] The above embodiments are only descriptions of the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for predicting immunotherapy response in patients with hepatocellular carcinoma, characterized in that: include: Obtaining CT images and clinical data to be predicted of patients with hepatocellular carcinoma, preprocessing the CT images and clinical data, and obtaining intra-tumor and peritumoral images and key clinical data to be predicted; Inputting the intratumoral and peritumoral images to be predicted and key clinical data into a progressive fusion model to obtain the immunotherapy response of the hepatocellular carcinoma patient; The progressive fusion model is obtained by training with a training set, wherein the training set includes: clinical data, intratumoral images and peritumoral images; The progressive fusion model uses the first-layer fusion module to fuse the intratumor images in the arterial phase and the intratumor images in the portal phase in the intratumor images to be predicted, and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, and uses the second-layer splicing module to perform feature splicing on the fused intratumor and peritumoral images and the key clinical data to be predicted, and classifies and predicts the splicing results based on the third-layer classification module.

2. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 1, characterized in that: Acquiring the intra-tumor and peritumoral images to be predicted includes: The CT image is resampled, the resolution of the CT sequence is normalized to the target resolution, and the intensity value of the CT image is mapped to a preset interval, and a picture covering the peritumoral or tumor area is selected to obtain the intratumoral and peritumoral images to be predicted.

3. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 1, characterized in that: Acquiring the key clinical data to be predicted includes: The clinical data to be predicted are standardized, converted into a standard normal distribution, and clinical features related to immunotherapy response are screened to obtain the key clinical data to be predicted.

4. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 1, characterized in that: The clinical data to be predicted include: age, gender, body mass index, hepatitis B, hepatitis C, portal vein tumor thrombosis, alpha-fetoprotein, total bilirubin, albumin, prothrombin time, neutrophil-to-lymphocyte ratio and Barcelona Clinic Liver Cancer staging.

5. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 1, characterized in that: Using the training set to train and obtain the progressive fusion model includes: Performing data enhancement processing on the intratumor images and peritumoral images in the training set, and using the training set after the data enhancement processing to train and obtain the progressive fusion model; Among them, data enhancement processing methods include: random horizontal and vertical flipping, random translation, random scaling and random rotation.

6. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 1, characterized in that: The first layer fusion module includes: The first Conv layer is used to use convolution to slide multiple times on the intratumor images in the arterial phase and the intratumor images in the portal phase in the intratumor images to be predicted, and on the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, perform element-wise multiplication and sum at each sliding position, generate a complete feature map, and obtain a first feature map; The MobileNetv2 layer is used to increase the dimension of the first feature map, perform a convolution operation on the increased-dimensional first feature map using a Dwise convolution with a preset step size parameter, and perform dimensionality reduction to obtain a second feature map; The first Mobile ViT block layer is used to perform convolution operation and weighted combination on the second feature map to generate a new feature map, divide the new feature map into a number of blocks, map each block to a one-dimensional vector, and set the position bias information of each block, so as to perform global feature learning through a self-attention mechanism to obtain a third feature map.

7. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 6, characterized in that: The MobileNetv2 layer includes: A first MV2 unit is used to increase the dimension of the first feature map, perform a convolution operation on the first feature map after the dimension increase using a Dwise convolution with a step size parameter of 1, and perform dimension reduction to obtain a first feature result; A first MV2↓2 unit is used to increase the dimension of the first feature result, perform a convolution operation on the first feature result after the dimension increase using a Dwise convolution with a step size parameter of 2, and perform dimension reduction to obtain a second feature result; The second MV2 unit is used to increase the dimension of the second feature result, perform a convolution operation on the second feature result after the dimension increase using a Dwise convolution with a step size parameter of 1, and perform dimension reduction to obtain a third feature result; A third MV2 unit is used to increase the dimension of the third feature result, perform a convolution operation on the increased dimension third feature result using a Dwise convolution with a step size parameter of 1, and perform dimensionality reduction to obtain a fourth feature result; The second MV2↓2 unit is used to increase the dimension of the fourth feature result, perform a convolution operation on the increased dimension fourth feature result using a Dwise convolution with a step parameter of 2, and perform dimensionality reduction to obtain a second feature map.

8. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 1, characterized in that: The second layer splicing module includes: The first MV2↓2 layer is used to respectively increase the dimension of the intratumoral image and the portal vein phase intratumoral features in the intratumoral image to be predicted and the peritumoral image and the portal vein phase peritumoral features in the peritumoral image to be predicted in the third feature map, perform a convolution operation on the increased-dimensional third feature map using a Dwise convolution with a step size parameter of 2, and perform dimensionality reduction to obtain a fourth feature map; The second Mobile ViT block layer is used to perform convolution operation and weighted combination on the fourth feature map to generate a new feature map, divide the new feature map into a plurality of blocks, map each block to a one-dimensional vector, and set the position bias information of each block, so as to perform global feature learning through a self-attention mechanism to obtain a fifth feature map; The second MV2↓2 layer is used to increase the dimension of the fifth feature map, perform a convolution operation on the increased dimension of the fifth feature map using a Dwise convolution with a step size parameter of 2, and perform dimensionality reduction to obtain a sixth feature map; The third Mobile ViT block layer is used to perform convolution operation and weighted combination on the sixth feature map to generate a new feature map, divide the new feature map into a plurality of blocks, map each block to a one-dimensional vector, and set the position bias information of each block, so as to perform global feature learning through a self-attention mechanism to obtain a seventh feature map; The second Conv layer is used to slide on the seventh feature map multiple times by using convolution, perform element-level multiplication and sum at each sliding position, generate a complete feature map, and obtain the eighth feature map.

9. The method for predicting immunotherapy response in patients with hepatocellular carcinoma according to claim 1, characterized in that: The second layer splicing module also includes: The MLP layer includes several fully connected layers, which are used to perform feature learning and integration on the key clinical data to be predicted.

10. A prediction system for immunotherapy response in patients with hepatocellular carcinoma, characterized in that: include: A data acquisition subsystem, used to acquire CT images and clinical data to be predicted of patients with hepatocellular carcinoma; A data processing subsystem, used for preprocessing the CT images and clinical data to obtain intratumoral and peritumoral images to be predicted and key clinical data to be predicted; A treatment response prediction subsystem, which inputs the intratumoral and peritumoral images to be predicted and key clinical data into a progressive fusion model to obtain the immunotherapy response of the hepatocellular carcinoma patient; The progressive fusion model is obtained by training with a training set, wherein the training set includes: clinical data, intratumoral images and peritumoral images; The progressive fusion model uses the first-layer fusion module to fuse the intratumor images in the arterial phase and the intratumor images in the portal phase in the intratumor images to be predicted, and the peritumoral images in the arterial phase and the peritumoral images in the portal phase in the peritumoral images to be predicted, and uses the second-layer splicing module to perform feature splicing on the fused intratumor and peritumoral images and the key clinical data to be predicted, and classifies and predicts the splicing results based on the third-layer classification module.

Images