Multi-mode hepatocellular carcinoma recurrence risk prediction method, prediction system and application

Abstract

The invention relates to the technical field of artificial intelligence precision medical treatment, in particular to a multi-mode hepatocellular carcinoma recurrence risk prediction method and system and application, and the prediction method comprises the steps of multi-mode data acquisition, data standardization preprocessing, model training and recurrence risk prediction. The method comprises the following steps: firstly, collecting multi-modal data of a patient, wherein the multi-modal data comprises preoperative data, postoperative data and a recurrence risk label; secondly, carrying out standardized preprocessing on the collected multi-modal data; and updating parameters of the pre-operation model and the post-operation model by using an adaptive moment estimation algorithm to obtain a trained multi-modal hepatocellular carcinoma recurrence risk prediction model which comprises the pre-operation model and the post-operation model and can be used for performing recurrence risk prediction before and after an operation respectively. According to the method, the deep learning technology is used, the multi-modal data before and after the operation can be respectively utilized, the recurrence risk of the HCC patient can be accurately predicted before and after the operation, and a targeted treatment scheme is formulated for the HCC patient.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence precision medicine technology, and specifically to a multimodal hepatocellular carcinoma recurrence risk prediction method, prediction system and application. Background Art

[0002] Hepatocellular carcinoma (HCC) is one of the malignant tumors with the highest mortality rate. Liver resection is the first-line treatment for early-stage HCC recommended by domestic and international guidelines and is the most important radical treatment for HCC. However, the recurrence rate of HCC after surgery is high, with a recurrence rate of up to 70% within 5 years, which greatly limits the long-term benefits of liver resection. Most HCC recurrences occur within 2 years after surgery, accounting for about 70% of all postoperative recurrences, which are called early recurrences. Early recurrence is associated with a worse prognosis, especially recurrent tumors that exceed the Milan criteria at the time of early recurrence, which are usually difficult to receive radical treatment and have a significantly worse prognosis. Accurately identifying these patients with HCC can enable targeted selection of treatment options and improve patient survival benefits.

[0003] Previous studies on hepatocellular carcinoma recurrence have mainly focused on the time of tumor recurrence, such as whether early recurrence occurs. And most recurrence prediction models are built based on single-modal data, mainly clinical indicators. However, simply focusing on the time of recurrence cannot accurately stratify the prognosis of recurrent patients, because patients who can receive radical treatment again after early recurrence usually have a better prognosis. Due to the significant individual heterogeneity of hepatocellular carcinoma, stratifying hepatocellular carcinoma patients simply by clinical indicators often has poor prediction results and is difficult to meet clinical needs.

[0004] Dynamic enhanced magnetic resonance imaging (MRI) plays an important role in the monitoring, diagnosis, and treatment evaluation of HCC and is widely used in clinical work. Pathological histology is known as the 'gold standard' in the medical cancer diagnosis and treatment process, and plays a vital role in the diagnosis and prognosis of HCC. These two common medical data in the clinical diagnosis and treatment of HCC are helpful in judging the recurrence of HCC. Some studies have used preoperative MRI images or postoperative digital pathological sections (Whole Slide Image, WSI) to predict postoperative recurrence and prognosis of HCC, but they are also based only on single-modality data of the tumor and cannot fully reflect the characteristics of the tumor, and the prediction effect is still not ideal. At present, there is an urgent need for a method that can make full use of medical data of different modalities to accurately identify HCC patients at high risk of recurrence.

[0005] With the rapid development of deep learning technology, neural networks have been widely used in the field of clinical medicine. They can deeply mine image data and structured data, realize the fusion of data of different modalities, and predict and evaluate various major diseases. Based on deep learning technology, by making full use of and integrating the medical data available in clinical practice, it is expected to perform more accurate recurrence risk stratification for highly heterogeneous HCC patients to achieve the goal of individualized treatment. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a multimodal hepatocellular carcinoma recurrence risk prediction method, prediction system and application. The method uses deep learning technology and can use preoperative and postoperative multimodal data to accurately predict the recurrence risk of HCC patients before and after surgery, and formulate targeted treatment plans for HCC patients.

[0007] In order to solve the above technical problems, the present invention is implemented by the following technical solutions:

[0008] In a first aspect, the present invention provides a multimodal hepatocellular carcinoma recurrence risk prediction method, wherein the recurrence risk prediction is based on constructing a multimodal hepatocellular carcinoma recurrence risk prediction model and predicting the preoperative and postoperative recurrence risks based on the multimodal hepatocellular carcinoma recurrence risk prediction model;

[0009] The construction of a multimodal hepatocellular carcinoma recurrence risk prediction model comprises:

[0010] S1. Multimodal data acquisition: preoperative data, postoperative data and recurrence risk labels are collected separately, wherein the preoperative data include preoperative MRI images and preoperative key clinical indicators of patients with hepatocellular carcinoma, and the postoperative data include postoperative digital pathological sections and postoperative key clinical indicators;

[0011] S2, data standardization preprocessing: including parallel preprocessing of three different modalities of data, namely, preoperative MRI images collected before and after surgery, postoperative digital pathological sections, and key clinical indicators;

[0012] S3. Model training: Use the adaptive moment estimation algorithm to update the parameters of the preoperative model and the postoperative model to obtain a trained multimodal hepatocellular carcinoma recurrence risk prediction model.

[0013] In the above scheme, the present invention realizes a comprehensive assessment of the risk of recurrence of hepatocellular carcinoma by integrating multimodal data such as preoperative MRI images, preoperative key clinical indicators, postoperative digital pathological sections, and postoperative key clinical indicators. This fusion of multimodal data can more comprehensively reflect the characteristics of the tumor and the overall condition of the patient, thereby improving the accuracy and reliability of the prediction of the risk of recurrence. The method provided by the present invention can not only predict the risk of recurrence of hepatocellular carcinoma patients before surgery, but also make more accurate predictions after surgery. This dual prediction mechanism can provide doctors with more comprehensive information, thereby guiding patients to accept more personalized treatment plans. For example, for patients predicted to be at high risk before surgery, preventive measures can be considered to reduce the risk of postoperative recurrence; for patients predicted to be at high risk after surgery, more active postoperative monitoring and adjuvant treatment can be performed.

[0014] As a further technical solution of the present invention, in step S1, the preoperative MRI image is a dynamic enhanced MRI image collected from a patient with hepatocellular carcinoma within one month before surgery, including arterial phase, portal venous phase, and delayed phase images;

[0015] Screening and collection of key clinical indicators before and after surgery: Single factor regression was used to screen key clinical indicators related to the risk of recurrence, and key clinical indicators before and after surgery were collected and sorted;

[0016] The postoperative digital pathology section collection: collecting hematoxylin and eosin (HE) stained sections after liver resection of patients with hepatocellular carcinoma, selecting the HE sections with the largest tumor tissue area for digital pathology section scanning, and obtaining postoperative digital pathology sections;

[0017] The recurrence risk signature is a retrospective collection of the recurrence risk signatures of cases that meet the inclusion and exclusion criteria, and is defined as whether recurrent hepatocellular carcinoma exceeding the Milan criteria occurs within two years after surgery.

[0018] As a further technical solution of the present invention, in step S2, preoperative MRI image preprocessing is performed: for each patient, three sequence images of the arterial phase, portal venous phase, and delayed phase in the dynamic enhanced MRI are selected, and the image with the largest tumor area in each sequence image is selected to delineate the tumor area; a minimum bounding box containing the complete tumor area is delineated, and after the delineation is completed, the bounding box is expanded outward by a certain number of pixels to obtain the complete tumor boundary, and the image size is resampled and the pixel value is standardized;

[0019] Preprocessing of key clinical indicators before and after surgery: one-hot label processing for categorical variables and Z-score standardization for numerical variables;

[0020] Postoperative digital pathology section preprocessing: OTSU foreground segmentation was performed on the digital pathology sections to obtain tissue area images, and then block processing, resampling and pixel value standardization were performed.

[0021] As a further technical solution of the present invention, in step S3, preoperative model training is performed: the traditional CNN model ResNet50 is used as a feature extractor to extract features of the preoperative MRI images after standardized preprocessing, and an image feature vector is obtained. A multi-layer perceptron is used to perform linear transformation on the preprocessed preoperative key clinical indicator data to obtain a preoperative clinical feature vector. The image-clinical fusion vector is obtained by vector splicing and input into the preoperative classification network to obtain a recurrence risk prediction result.

[0022] As a further technical solution of the present invention, in step S3, postoperative model training: use the traditional CNN model ResNet50 after removing the classification head as a feature extractor to extract features from the preoperative MRI image after standardized preprocessing to obtain an image feature vector; use the self-supervised pre-trained general pathology basic model to extract features from all the blocks generated after the digital pathology slice segmentation to obtain the pathological feature vector of each block in the digital pathology slice; obtain the image-pathology fusion vector based on the Transformer cross-attention mechanism; use a multi-layer perceptron to perform linear transformation on the preprocessed complete preoperative and postoperative key clinical indicator data to obtain a complete clinical feature vector; use the vector splicing method to splice the image-pathology fusion vector and the complete clinical feature vector to obtain the final image-pathology-clinical fusion vector, which is input into the postoperative classification network to obtain the recurrence risk prediction result.

[0023] In the above scheme, the present invention uses deep learning technology, such as CNN (convolutional neural network) and Transformer models, to extract and fuse features of multimodal data, thereby achieving a rapid and efficient prediction of the risk of recurrence of hepatocellular carcinoma. The application of this technology improves the accuracy of prediction and provides more convenience for patients and doctors.

[0024] In a second aspect, the present invention provides a multimodal hepatocellular carcinoma recurrence risk prediction system, which is used to implement the multimodal hepatocellular carcinoma recurrence risk prediction method, comprising:

[0025] A multimodal data acquisition module, used to respectively collect preoperative data and postoperative data and recurrence risk labels, wherein the preoperative data includes preoperative MRI images and preoperative key clinical indicators of patients with hepatocellular carcinoma, and the postoperative data includes postoperative digital pathological sections and postoperative key clinical indicators;

[0026] The data standardization preprocessing module is used for parallel preprocessing of three different modalities of data, including preoperative MRI images, digital pathological sections, and key clinical indicators collected before and after surgery;

[0027] A model training module, used to update the parameters of the preoperative model and the postoperative model using an adaptive moment estimation algorithm to obtain a trained multimodal hepatocellular carcinoma recurrence risk prediction model; the multimodal hepatocellular carcinoma recurrence risk prediction model includes a preoperative model and a postoperative model, which are used to make predictions before and after the operation respectively;

[0028] The recurrence risk prediction module is used to predict the preoperative and postoperative recurrence risks respectively using the trained multimodal hepatocellular carcinoma recurrence risk prediction model.

[0029] Among them, the preoperative and postoperative models can be used independently to accurately predict the recurrence risk of HCC patients before and after surgery. The preoperative model inputs the arterial phase, portal venous phase, and delayed phase images of preoperative dynamic enhanced MRI, as well as the preoperative key clinical indicators after screening, to obtain the preoperative recurrence risk prediction results. The postoperative model inputs the WSI obtained from the postoperative pathological HE slice scan, the arterial phase, portal venous phase, and delayed phase images of preoperative dynamic enhanced MRI, and the complete post-screening clinical indicators, to obtain a more accurate recurrence risk prediction result than before surgery.

[0030] In a third aspect, the present invention provides an application of a multimodal hepatocellular carcinoma recurrence prediction system in predicting the recurrence risk of early hepatocellular carcinoma patients.

[0031] In a fourth aspect, the present invention provides an application of a multimodal hepatocellular carcinoma recurrence prediction system in accurate prognostic stratification and guiding personalized treatment plans for early hepatocellular carcinoma patients.

[0032] The method, system and application of the present invention have broad application prospects in the prediction of recurrence risk of hepatocellular carcinoma, accurate prognosis stratification and guidance of personalized treatment plans. By accurately predicting the risk of recurrence, doctors can formulate more reasonable and effective treatment plans for patients, thereby improving the survival rate and quality of life of patients. In addition, the method can also provide new ideas and methods for the prevention and early detection of hepatocellular carcinoma, which has significant social value.

[0033] Among them, for a patient with hepatocellular carcinoma who meets the surgical indications, after routine preoperative MRI imaging examinations and surgical examinations, preoperative MRI images and key clinical indicators are collected and input into the preoperative model to obtain the preoperative recurrence risk prediction results. For patients predicted as low risk by the preoperative model, radical liver resection should be performed as recommended by the guidelines; but for patients predicted as high risk by the preoperative model, preventive measures such as preoperative neoadjuvant therapy or extended liver resection should be considered to reduce the risk of postoperative recurrence. After the patient has undergone radical liver resection, the surgical pathology sections obtained should be used to scan and obtain digital pathology sections, and complete key clinical indicators should be collected and input into the postoperative model together with the preoperative images to obtain the postoperative recurrence risk prediction results. For patients predicted as low risk by the postoperative model, routine postoperative monitoring should be performed according to the follow-up plan recommended by the guidelines; for patients predicted as high risk by the postoperative model, more active postoperative recurrence and monitoring should be performed, and participation in relevant clinical trials of postoperative adjuvant therapy should be considered.

[0034] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal hepatocellular carcinoma recurrence risk prediction method.

[0035] In a sixth aspect, the present invention provides a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the multimodal hepatocellular carcinoma recurrence risk prediction method is implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention proposes a multimodal hepatocellular carcinoma recurrence risk prediction method and prediction system, which fully integrates the common imaging, pathological image data and clinical structured data in clinical diagnosis and treatment based on deep learning technology; fully interprets the tumor from two different levels: macroscopic overall tumor imaging and clinical information and microscopic tissue pathology information, and has a significantly better prediction effect than single-modal data, achieving accurate stratification of HCC patients. The recurrence risk prediction system of the present invention is closely integrated with clinical applications, and the sequential prediction results of the dual models before and after surgery can provide clinicians and patients with recurrence risk assessment and specific treatment plan recommendations, which has a high clinical translation application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of the multimodal recurrence risk prediction system.

[0039] Figure 2 This is the model structure diagram of the preoperative model in the recurrence prediction system.

[0040] Figure 3This is the model structure diagram of the postoperative model in the recurrence prediction system.

[0041] Figure 4 Flowchart for dual-model sequential prediction of the recurrence prediction system in clinical application scenarios. DETAILED DESCRIPTION

[0042] The specific implementation of the present invention is described below in conjunction with the accompanying drawings to facilitate the understanding of the present invention by those skilled in the art. Obviously, the present invention is not limited to the scope of the specific implementation. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are non-creative work, and all inventions and creations using the invention concept of this law are protected.

[0043] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0044] Example 1: Multimodal hepatocellular carcinoma recurrence risk prediction method based on deep learning

[0045] like Figure 1 As shown, first, multimodal data of HCC patients are collected, including preoperative MRI images, clinical indicators, and postoperative WSI images; secondly, the collected multimodal data are preprocessed in a standardized manner; the preprocessed multimodal training data is used to train the recurrence risk prediction system, and the recurrence risk prediction system includes a preoperative model and a postoperative model: both use parallel deep learning models to extract features from data of different modalities, respectively, to obtain feature vectors of different modalities; the preoperative model uses a vector splicing method to obtain an image-clinical fusion vector, and the postoperative model uses a cross-attention mechanism to obtain an image-pathology fusion vector, and then splices the clinical vector; and the fusion vector is input into the classification network to obtain the final recurrence risk prediction result. Finally, the recurrence risk of preoperative and postoperative hepatocellular carcinoma is predicted based on the trained recurrence risk prediction system. The specific steps are as follows:

[0046] Step 1: Multimodal data collection: For the preoperative model and postoperative model, collect preoperative data, postoperative data and the corresponding patient's recurrence risk label.

[0047] 1.1 Screening of key clinical indicators: Univariate logistic regression was used to screen key clinical indicators related to recurrence risk, and clinical indicators related to recurrence risk results were selected. The key clinical indicators after screening included clinical indicators available before surgery: patient age (years), tumor size (cm), platelet count (PLT) in blood routine examination (*10 9 / L), neutrophil-to-lymphocyte ratio (NLR), serum alkaline phosphatase (U / L), γ-glutamyl transferase (U / L), whether cirrhosis occurs, whether serum alpha-fetoprotein is greater than 400 (ng / ML), Barcelona stage (stage 0 or A), of which the latter three are categorical variables. Clinical indicators available after surgery: 4 categorical indicators, whether extensive hepatectomy was performed, whether microvascular invasion exists, whether satellite lesions exist, and whether the tumor is poorly differentiated.

[0048] 1.2 Preoperative multimodal data collection: Dynamic enhanced MRI images (including arterial phase, portal venous phase, and delayed phase images) of HCC patients within one month before surgery should be collected. Images of qualified quality should be selected as much as possible without obvious artifacts or blurring. Key clinical indicators before surgery should also be collected and sorted.

[0049] 1.3 Complete multimodal data acquisition after surgery: Hematoxylin and eosin (HE) stained sections were collected from HCC patients after liver resection. Professional pathologists selected the HE sections with the largest tumor tissue area and used a digital scanner to scan the digital pathology sections under a 20x or 40x objective lens to obtain WSI; and complete key clinical indicators were collected.

[0050] 1.4 Collection of recurrence risk labels: Obtain the latest follow-up information of the patient after surgery through the hospital medical record system or telephone follow-up, confirm whether recurrence occurs after surgery, the time of recurrence, the size, number and location of the tumor at the time of recurrence, and determine whether the first recurrence of hepatocellular carcinoma that exceeds the Milan standard (Milan standard: the diameter of a single tumor does not exceed 5CM or the number of tumors with more than 3 and the maximum diameter does not exceed 3CM, there is no large blood vessel invasion, and there is no lymph node or extrahepatic metastasis) occurs within two years after surgery. If it occurs, it is judged as a high-risk recurrence risk, if not, it is judged as a low-risk recurrence risk.

[0051] Step 2: Data standardization preprocessing: Figure 2 As shown in the figure, it includes parallel preprocessing of three different modal data: MRI images, WSI, and clinical indicators. Specifically, it includes the following steps:

[0052] 2.1 MRI image preprocessing: For each patient, three sequence images of the arterial phase, portal venous phase, and delayed phase in dynamic enhanced MRI were selected, and the image with the largest tumor area in each sequence image was selected to delineate the tumor area. After obtaining the original MRI data, the Simple-ITK package was used to convert the format from the original DICOM format to the NFITI format. Professional imaging physicians used annotation software to outline the minimum bounding box containing the complete tumor area. After the delineation was completed, the bounding box was expanded outward by 10 pixels to obtain the complete tumor boundary. The image containing the tumor and the tumor boundary was intercepted from the original image according to the expanded bounding box, resampled to a standardized size of 224 pixels * 224 pixels, and the single-channel grayscale image was copied into a three-channel RGB image format. Each patient obtained 3 (224, 224, 3)-dimensional RGB images. The pixel values ​​are standardized according to the mean value mean = [0.485, 0.456, 0.406] and the standard deviation std = [0.229, 0.224, 0.225].

[0053] 2.2 Preprocessing of key clinical indicators: Categorical variables in clinical indicators (whether cirrhosis occurred, whether serum alpha-fetoprotein was greater than 400 ng / mL, Barcelona stage grade, whether extensive liver resection was performed, whether microvascular invasion occurred, whether satellite lesions occurred, and whether the tumor was poorly differentiated) were processed with one-hot labels, and numerical variables (patient age (years), tumor size (cm), platelet count (PLT) in blood routine examination (*10 9 / L), neutrophil-to-lymphocyte ratio (NLR), serum alkaline phosphatase (U / L), and γ-glutamyl transferase (U / L) were standardized using the Z-score method, and both the training set and validation set were standardized using the mean and standard deviation in the training data.

[0054] 2.3WSI preprocessing: Convert the WSI in a special storage format into a digital pathology format (SVS or TIFF, etc.) commonly used for digital pathology image processing. Perform OTSU foreground segmentation on the WSI to obtain tissue area images, and use tiled sliding segmentation to obtain non-overlapping patches of size 256*256um. Each patch is then resampled to 224 pixels*224 pixels, and the pixel values ​​are standardized according to the mean value mean=[0.485, 0.456, 0.406] and the standard deviation std=[0.229, 0.224, 0.225].

[0055] Step 3: Model training: The recurrence risk prediction system includes a preoperative model and a postoperative model. The preoperative model is trained first, and then the postoperative model is trained.

[0056] 3.1 As Figure 2 As shown, the preoperative model includes a feature extraction module, a feature fusion module, and a classification network. The feature extraction module includes a ResNet50 model for preoperative imaging data and an MLP model for structured clinical data; for the preoperative model, a vector splicing method is used to obtain an image-clinical fusion vector; the image-clinical fusion vector obtained by the preoperative model is input into the preoperative classification network to obtain the preoperative recurrence risk prediction result. The specific steps are as follows:

[0057] A1 uses the traditional convolutional neural network model ResNet50 to extract features from three MRI images at different times after standardized preprocessing, specifically including: using the OpenCV package to read the three RGB images converted in step 1, using the ResNet50 model after removing the classification head as a feature extractor to extract features from the three MRI images at different times after standardized preprocessing, and obtaining a 3*2048-dimensional image feature vector. Use MLP to perform linear transformation on the preprocessed clinical data to obtain the preoperative clinical feature vector, specifically including: firstly, perform a fully connected layer linear transformation on the preprocessed 12-dimensional data to obtain a 32-dimensional intermediate vector, then pass through the ReLU activation function layer and the dropout layer (dropout rate = 0.1), and finally perform a fully connected layer linear transformation to obtain a 64-dimensional preoperative clinical feature vector.

[0058] For the preoperative model in A2, the 3*2048-dimensional image feature vector obtained after processing by the ResNet50 model was flattened to obtain a 6144-dimensional intermediate vector, and then the dimension was gradually reduced through two consecutive composite layers (including linear layer, activation function and Dropout layer), first to 2048 dimensions and then to 256 dimensions to obtain the final image feature vector. The 256-dimensional image feature vector was concatenated with the 64-dimensional preoperative clinical feature vector obtained in A1 to obtain a 320-dimensional image-clinical fusion vector.

[0059] The image-clinical fusion vector obtained by the A3 preoperative model is input into the preoperative classification network to obtain the preoperative recurrence risk prediction result. The preoperative classification network includes a linear layer, which maps the 320-dimensional fusion vector to a 1-dimensional prediction value and uses sigmoid normalization to obtain the final recurrence risk prediction result (between 0 and 1). If the recurrence risk prediction result is greater than 0.5, it is predicted to be a high-risk recurrence model.

[0060] A4 model training uses a binary cross entropy loss function to calculate the loss between the model prediction result and the actual recurrence risk label; the Adaptive Moment Estimation (Adam) algorithm is used to minimize the loss function; the hyperparameters and corresponding model parameters that perform best in the internal validation set are used as the final training results of the preoperative model.

[0061] 3.2 The input of the postoperative model is the data of preoperative MRI images, clinical and pathological modalities. Compared with the preoperative model, it also includes pathological images and some clinical indicators that can only be obtained after surgery. Figure 3 As shown, the postoperative model includes a feature extraction module, a multimodal feature fusion module, and a classification network. The feature extraction module includes ResNet50 for processing image data, MLP for processing clinical data, and a general feature extractor for pathological data; the postoperative model uses a Transformer-based cross-attention mechanism to obtain an image-pathology fusion vector, and uses a multi-layer perceptron to perform linear transformation on the pre-processed complete preoperative and postoperative key clinical indicator data to obtain a complete clinical feature vector; the image-pathology fusion vector and the complete clinical feature vector are spliced ​​using a vector splicing method to obtain the final image-pathology-clinical fusion vector; the final image-pathology-clinical fusion vector is input into the postoperative classification network to obtain a more accurate binary classification prediction result for recurrence risk. The specific steps are as follows:

[0062] The parameters of the feature extractor module of B1 ResNet50 are consistent with the preoperative model, and no parameter update is performed. The image feature vector of 2048*3 dimensions is also obtained. The public universal pathology basic model based on self-supervised pre-training, such as UNI, is used to extract features from all patches generated after WSI segmentation, and obtain the feature vector of each patch in WSI. Each patch obtains a 1024-dimensional vector, and the feature vectors of all patches in each WSI are stored separately in an h5 file, containing N (N is the number of patches in each WSI)*1024-dimensional vectors. MLP is used to linearly transform the preprocessed complete preoperative and postoperative key clinical data to obtain a complete clinical feature vector, which specifically includes first performing a fully connected layer linear transformation on the preprocessed 20-dimensional data to obtain a 32-dimensional intermediate vector, then passing through a ReLU activation function layer and a dropout layer (dropout rate = 0.1), and finally a fully connected layer linear transformation to obtain a 64-dimensional clinical feature vector.

[0063] The postoperative model of B2 uses the Transformer-based cross-attention mechanism to obtain the image-pathology fusion vector, which includes: firstly, linearly transforming and reducing the dimension of the image and pathology feature vectors after feature extraction in B1, respectively obtaining intermediate vectors of 256*3 and 256*N, and concatenating the two intermediate vectors to obtain a fusion vector of (3+N)*256; using a linear layer to split the fusion vector into three vectors of (3+N)*128 dimensions, namely, Query, Key, and Value, and calculating the dot product between Query and Key to obtain the attention scores between different parts; applying the Softmax function to the attention scores to obtain the attention weights. Using the attention weights, the Value vectors are weighted and summed to obtain the output of the image and pathology parts. The outputs of these two parts are concatenated in the feature dimension to form a multimodal vector. Finally, these multimodal vectors are further refined through layer normalization and a feed-forward network to obtain a 256-dimensional image-pathology fusion vector. Then, the vector splicing method is used to splice the fusion vector and the 64-dimensional clinical feature vector obtained by B1 to obtain the final image-pathology-clinical fusion vector.

[0064] The B3 postoperative model inputs the final image-pathology-clinical fusion vector into the postoperative classification network to obtain a more accurate binary prediction result of recurrence risk. The postoperative classification network includes linear layer 1, ReLU activation function layer and linear layer 2. Linear layer 1 maps the 320-dimensional fusion vector to a 64-dimensional intermediate vector. After the ReLU activation function, the 64-dimensional intermediate vector passes through linear layer 2 to obtain a 1-dimensional prediction value, which is normalized using sigmoid to obtain the final recurrence risk prediction result. If the recurrence risk prediction result is greater than 0.5, it is predicted to be a high-risk recurrence pattern.

[0065] B4 model training uses the binary cross entropy loss function to calculate the loss between the model prediction results and the actual recurrence risk label; the Adaptive Moment Estimation (Adam) algorithm is used to minimize the loss function; the hyperparameters and corresponding model parameters that perform best in the internal validation set are used as the final training results of the postoperative model.

[0066] Step 4: Recurrence risk prediction: Based on the trained hepatocellular carcinoma recurrence risk prediction system, the recurrence risk can be predicted before and after surgery.

[0067] Example 2: Multimodal hepatocellular carcinoma recurrence risk prediction system based on deep learning

[0068] A multimodal hepatocellular carcinoma recurrence risk prediction system, used to implement the multimodal hepatocellular carcinoma recurrence risk prediction method in Example 1, comprising:

[0069] A multimodal data acquisition module, used to respectively collect preoperative data and postoperative data and recurrence risk labels, wherein the preoperative data includes preoperative MRI images and preoperative key clinical indicators of patients with hepatocellular carcinoma, and the postoperative data includes postoperative digital pathological sections and postoperative key clinical indicators;

[0070] Data standardization preprocessing module, used for parallel preprocessing of three different modalities of data including MRI images collected before and after surgery, digital pathology sections, and key clinical indicators;

[0071] A model training module, used to update the parameters of the preoperative model and the postoperative model using an adaptive moment estimation algorithm to obtain a trained multimodal hepatocellular carcinoma recurrence risk prediction model; the multimodal hepatocellular carcinoma recurrence risk prediction model includes a preoperative model and a postoperative model, which are used to make predictions before and after the operation respectively;

[0072] The recurrence risk prediction module is used to predict the preoperative and postoperative recurrence risks respectively using the trained multimodal hepatocellular carcinoma recurrence risk prediction model.

[0073] Table 1 shows the predictive performance results of the preoperative model and the postoperative model in the multimodal recurrence risk prediction system. Both the preoperative model and the postoperative model were trained in the same training set. The model parameters with the best performance in the internal validation set were selected as the final training results, and the predictive performance was evaluated in the external test set. The preoperative model and the postoperative model in this prediction system were trained based on multimodal data. In the external test set, the area under the receiver operating characteristic curve (AUC) was 0.74 and 0.83, respectively. The experimental results show that the preoperative model and the postoperative model trained based on multimodal data have achieved excellent predictive performance, and the postoperative model can achieve relatively more accurate predictions.

[0074] Table 1: Predictive performance of the multimodal recurrence risk prediction system

[0075]

[0076] Table 2 shows the comparison results of the predictive effectiveness of the preoperative model and the preoperative single-modality prediction model in this multimodal recurrence risk prediction system. The single-image model refers to the use of only preoperative MRI image data; the single clinical model (preoperative) only uses the selected key preoperative clinical indicators for training. Similarly, the two single-modality models were trained and verified in the same data set. The preoperative model performed better than the two single-modality models in the external test set, and the AUC was improved by 0.04 and 0.03 respectively. The implementation results show that the preoperative model based on multimodal data has better prediction effect than the general single-modality prediction model.

[0077] Table 2: Predictive performance of preoperative models compared with single-modality prediction models

[0078]

[0079]

[0080] Table 3 shows the prediction performance comparison results of the postoperative model and the postoperative single / dual modality prediction model in this multimodal recurrence risk prediction system. The imaging-pathology model refers to a model that only uses imaging and pathology data but not clinical data; the single pathology model only uses pathology image data; the single clinical model (postoperative) only uses all the key clinical indicators after screening for training. The postoperative model performed better than the two single modality models in the external test set, with AUCs increased by 0.03 and 0.11 respectively. The postoperative model also performed slightly better than the imaging-pathology model in the internal validation set. The experimental results show that the postoperative model based on multimodal data has better prediction effect than the general single modality prediction model.

[0081] Table 3: Predictive performance of postoperative models compared with single-modality prediction models

[0082]

[0083] In summary, the present invention significantly improves the accuracy and reliability of predicting the risk of recurrence of hepatocellular carcinoma by integrating multimodal data, applying deep learning technology, and providing dual predictions before and after surgery. It provides doctors with more comprehensive and accurate information to guide patients to accept personalized treatment plans, and has significant clinical application value and social value.

[0084] Example 3: Clinical application of a multimodal hepatocellular carcinoma recurrence prediction system based on deep learning

[0085] The recurrence prediction system we built has both preoperative and postoperative models. The preoperative model only requires preoperative imaging data and clinical data, and can predict the risk of recurrence for patients with early hepatocellular carcinoma before surgery. The postoperative model further combines the digital pathological images obtained after surgery and postoperative clinical information to make a more accurate prediction of the risk of recurrence. The preoperative model has a high sensitivity and can identify as many patients as possible who are truly at high risk of recurrence before surgery to prevent them from undergoing surgical resection without survival benefit. The postoperative model has a higher overall predictive efficacy and a higher specificity, ensuring that patients predicted to be low risk are indeed at low risk of recurrence.

[0086] like Figure 4 As shown in the figure, for a patient with hepatocellular carcinoma who meets the surgical indications, after routine preoperative MRI imaging examinations and surgical examinations, preoperative MRI images and key clinical indicators are collected and input into the preoperative model to obtain the preoperative recurrence risk prediction results. For patients predicted as low risk by the preoperative model, radical liver resection should be performed as recommended by the guidelines; however, for patients predicted as high risk by the preoperative model, preventive measures such as preoperative neoadjuvant therapy or extended liver resection should be considered to reduce the risk of postoperative recurrence. After the patient has undergone radical liver resection, the surgical pathology sections obtained should be used to scan and obtain digital pathology sections, and complete key clinical indicators should be collected and input into the postoperative model together with the preoperative images to obtain the postoperative recurrence risk prediction results. For patients predicted as low risk by the postoperative model, routine postoperative monitoring should be performed according to the follow-up plan recommended by the guidelines; for patients predicted as high risk by the postoperative model, more active postoperative recurrence and monitoring should be performed, and participation in relevant clinical trials of postoperative adjuvant therapy should be considered.

[0087] In summary, the recurrence risk prediction system of the present invention is closely integrated with clinical applications and can provide recurrence risk assessment and treatment plan guidance for clinicians and patients.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0089] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A multimodal hepatocellular carcinoma recurrence risk prediction method, characterized in that: The recurrence risk prediction is based on constructing a multimodal hepatocellular carcinoma recurrence risk prediction model and predicting the preoperative and postoperative recurrence risks based on the multimodal hepatocellular carcinoma recurrence risk prediction model; The construction of a multimodal hepatocellular carcinoma recurrence risk prediction model comprises: S1. Multimodal data acquisition: preoperative data, postoperative data and recurrence risk labels are collected separately, wherein the preoperative data include preoperative MRI images and preoperative key clinical indicators of patients with hepatocellular carcinoma, and the postoperative data include postoperative digital pathological sections and postoperative key clinical indicators; S2, data standardization preprocessing: including parallel preprocessing of three different modalities of data, namely, preoperative MRI images collected before and after surgery, postoperative digital pathological sections, and key clinical indicators; S3. Model training: Use the adaptive moment estimation algorithm to update the parameters of the preoperative model and the postoperative model to obtain a trained multimodal hepatocellular carcinoma recurrence risk prediction model.

2. The multimodal hepatocellular carcinoma recurrence risk prediction method according to claim 1, characterized in that: In step S1, the preoperative MRI images are dynamic enhanced MRI images collected from patients with hepatocellular carcinoma within one month before surgery, including arterial phase, portal venous phase, and delayed phase images; Screening and collection of key clinical indicators before and after surgery: Single factor regression was used to screen key clinical indicators related to the risk of recurrence, and key clinical indicators before and after surgery were collected and sorted; The collection of postoperative digital pathological sections: collecting hematoxylin & eosin (HE) stained sections after liver resection of patients with hepatocellular carcinoma, selecting the HE sections with the largest tumor tissue area for digital pathological section scanning, and obtaining postoperative digital pathological sections; The recurrence risk signature is a retrospective collection of the recurrence risk signatures of cases that meet the inclusion and exclusion criteria, and is defined as whether recurrent hepatocellular carcinoma exceeding the Milan criteria occurs within two years after surgery.

3. The multimodal hepatocellular carcinoma recurrence risk prediction method according to claim 1, characterized in that: In step S2, preoperative MRI image preprocessing is performed: for each patient, three sequence images of the arterial phase, portal venous phase, and delayed phase in the dynamic enhanced MRI are selected, and the image with the largest tumor area in each sequence image is selected to delineate the tumor area; the minimum bounding box containing the complete tumor area is delineated, and after the delineation is completed, the bounding box is expanded outward by a certain number of pixels to obtain the complete tumor boundary, and the image size is resampled and the pixel value is standardized; Preprocessing of key clinical indicators before and after surgery: one-hot label processing for categorical variables and Z-score standardization for numerical variables; Postoperative digital pathology section preprocessing: OTSU foreground segmentation was performed on the digital pathology sections to obtain tissue area images, and then block processing, resampling and pixel value standardization were performed.

4. The multimodal hepatocellular carcinoma recurrence risk prediction method according to claim 1, characterized in that: In step S3, preoperative model training: the traditional CNN model ResNet50 is used as a feature extractor to extract features from the standardized preoperative MRI images after preprocessing to obtain image feature vectors, and a multi-layer perceptron is used to perform linear transformation on the preprocessed preoperative key clinical indicator data to obtain preoperative clinical feature vectors. The image-clinical fusion vector is obtained by vector splicing and input into the preoperative classification network to obtain the recurrence risk prediction result.

5. The multimodal hepatocellular carcinoma recurrence risk prediction method according to claim 1, characterized in that: In step S3, postoperative model training: the traditional CNN model ResNet50 is used as a feature extractor after removing the classification head to extract features from the preoperative MRI images after standardized preprocessing to obtain image feature vectors; the self-supervised pre-trained general pathology basic model is used to extract features from all the blocks generated after the digital pathology slice segmentation to obtain the pathological feature vectors of each block in the digital pathology slice; the image-pathology fusion vector is obtained based on the Transformer cross-attention mechanism; the preprocessed complete preoperative and postoperative key clinical indicator data is linearly transformed using a multi-layer perceptron to obtain a complete clinical feature vector; the image-pathology fusion vector and the complete clinical feature vector are spliced ​​using the vector splicing method to obtain the final image-pathology-clinical fusion vector, which is input into the postoperative classification network to obtain the recurrence risk prediction result.

6. A multimodal hepatocellular carcinoma recurrence risk prediction system, characterized in that: The method for predicting the risk of recurrence of hepatocellular carcinoma using the multimodal method according to any one of claims 1 to 5 comprises: A multimodal data acquisition module, used to respectively collect preoperative data and postoperative data and recurrence risk labels, wherein the preoperative data includes preoperative MRI images and preoperative key clinical indicators of patients with hepatocellular carcinoma, and the postoperative data includes postoperative digital pathological sections and postoperative key clinical indicators; The data standardization preprocessing module is used for parallel preprocessing of three different modalities of data, including preoperative MRI images collected before and after surgery, postoperative digital pathology slices, and key clinical indicators; A model training module, used to update the parameters of the preoperative model and the postoperative model using an adaptive moment estimation algorithm to obtain a trained multimodal hepatocellular carcinoma recurrence risk prediction model; the multimodal hepatocellular carcinoma recurrence risk prediction model includes a preoperative model and a postoperative model, which are used to make predictions before and after the operation respectively; The recurrence risk prediction module is used to predict the preoperative and postoperative recurrence risks respectively using the trained multimodal hepatocellular carcinoma recurrence risk prediction model.

7. An application of the multimodal hepatocellular carcinoma recurrence prediction system as claimed in claim 6 in predicting the recurrence risk of early hepatocellular carcinoma patients.

8. An application of the multimodal hepatocellular carcinoma recurrence prediction system as described in claim 6 in accurate prognostic stratification and guiding personalized treatment plans for early hepatocellular carcinoma patients.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the multimodal hepatocellular carcinoma recurrence risk prediction method according to any one of claims 1 to 5 is implemented.

10. A computer device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the multimodal hepatocellular carcinoma recurrence risk prediction method according to any one of claims 1 to 5 is implemented.

Images