A fully automatic liver cancer recurrence risk prediction system and device based on whole liver imaging and deep learning

Abstract

The present invention discloses a fully automatic liver cancer recurrence risk prediction system and device based on whole liver imaging and deep learning, including: an enhanced CT liver automatic segmentation module for obtaining a whole liver mask based on an enhanced CT image; an enhanced CT liver cancer automatic segmentation module for obtaining a liver cancer mask based on an enhanced CT image and a whole liver mask; a recurrence risk prediction module based on whole liver imaging and deep learning for obtaining a whole liver region of interest based on an enhanced CT image and a whole liver mask, and combining the liver cancer mask as a common input, predicting a risk score, and generating a significance map related to the recurrence risk. The present invention fully mines important recurrence risk-related information in the liver parenchyma inside and outside the tumor by modeling and analyzing the whole liver imaging region including the tumor. The system performance exceeds that of traditional clinical models and tumor-based models, and has the advantages of being fully automatic and non-invasive, which helps to improve the personalized management of liver cancer patients.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and clinical decision support, and in particular relates to a fully automatic liver cancer recurrence risk prediction system and device based on whole liver imaging and deep learning. Background Art

[0002] Hepatocellular carcinoma is the most common pathological type of liver cancer. Preventing the recurrence and metastasis of hepatocellular carcinoma and prolonging survival are the key goals pursued in the field of liver cancer treatment and are also one of the focuses of current clinical research. Liver resection is the main means for patients with hepatocellular carcinoma to achieve long-term survival benefits. Preoperative prediction of the risk of recurrence and death in patients with hepatocellular carcinoma is an important prerequisite for risk stratification and selection of the best treatment plan, which can maximize the clinical benefit of patients. However, there is currently no consensus on the best tool for predicting the risk of postoperative recurrence of hepatocellular carcinoma in the preoperative scenario. Due to the advantages of non-invasiveness, easy access and easy handling, the mainstream preoperative prediction model is based on medical records and / or images such as enhanced computed tomography (CT) collected during routine clinical diagnosis and treatment. In recent years, many studies have shown that models based on enhanced CT radiomics have the ability to predict recurrence-free survival and overall survival of hepatocellular carcinoma before surgery, and the model that combines radiomics fingerprints and clinical characteristics performs better (Ji GW, Zhu FP, Xu Q, et al. Machine-learning analysis of contrast-enhanced CT radiomics predicts recurrence of hepatocellular carcinoma after resection: A multi-institutional study [J]. EBioMedicine, 2019, 50: 156-65.), (Wang F, Chen Q, Zhang Y, et al. CT-Based Radiomics forthe Recurrence Prediction of Hepatocellular Carcinoma After SurgicalResection [J]. J Hepatocell Carcinoma, 2022, 9: 453-65.), but the accuracy of the model still needs to be improved.

[0003] In the patent document with authorization announcement number CN113571194B, a method and system for predicting the long-term prognosis of hepatocellular carcinoma based on postoperative residual liver imaging omics are disclosed. The method includes obtaining enhanced CT arterial phase, portal phase and delayed phase images after partial liver resection for hepatocellular carcinoma, manually segmenting the residual liver area layer by layer, extracting imaging omics features in the residual liver, screening and reducing the dimensionality of the imaging omics features, and using the Cox proportional risk model to construct a patient postoperative recurrence risk prediction model.

[0004] The technology disclosed in the existing literature has the following shortcomings:

[0005] First, the method requires a lot of manual work and is not easy to popularize in busy clinical practice. Second, the method uses a tumor-based method, which only extracts the image features of the tumor and peritumoral area in the enhanced CT image for model construction, and cannot fully capture the features of the tumor and non-tumor liver tissue that are closely related to the risk of recurrence. Third, the tumor-based method cannot fully characterize the lesion characteristics of all types of liver cancer. For example, in actual operation, it is impossible to accurately outline the boundaries of diffusely distributed or multiple tumors or tumors with blurred boundaries, and therefore it is impossible to extract features from them. Third, hepatocellular carcinoma is often accompanied by chronic liver diseases such as liver fibrosis, cirrhosis, chronic viral hepatitis, alcoholic hepatitis, and non-alcoholic fatty liver disease. These concomitant diseases can weaken the liver's reserve function and affect the postoperative prognosis of hepatocellular carcinoma patients (Berardi G, Morise Z, Sposito C, et al. Development of a nomogram to predict outcome after liver resection for hepatocellular carcinoma in Child-Pugh B cirrhosis [J]. J Hepatol, 2020, 72(1): 75-84.). Information on liver tissue lesions other than tumors is ignored in tumor-based analyses. Although Child-Pugh grade, cirrhosis status and other clinical characteristics are often used to reflect liver function levels, the latest BCLC guidelines point out that there is insufficient evidence-based medicine to show that Child-Pugh grade is closely related to the prognosis of hepatocellular carcinoma after surgery (Reig M, Forner A, Rimola J, et al. BCLC strategy for prognosis prediction and treatment recommendation: The 2022 update [J]. JHepatol, 2022, 76(3): 681-93.), and qualitative cirrhosis status indicators are obviously unable to fully characterize the overall characteristics of the liver. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the prior art and propose a fully automatic liver cancer recurrence risk prediction system and device based on whole liver imaging and deep learning. The present invention aims to solve the following three key technical problems: how to construct a high-precision fully automatic enhanced CT image liver automatic segmentation model; how to construct a highly accurate fully automatic enhanced CT image liver cancer automatic segmentation model; how to model the recurrence risk from the whole liver region of interest image, and realize whole liver image analysis to comprehensively extract the features related to prognosis in the tumor itself and the liver background. After solving the above three problems one by one, the present invention finally integrates the above three modules into a set of convenient and accurate preoperative hepatocellular carcinoma resection postoperative recurrence risk prediction system suitable for promotion to clinical practice, assisting clinicians to accurately risk stratify patients, select appropriate surgical candidates and formulate personalized treatment plans, and ultimately achieve an improvement in the overall survival rate and quality of life of patients.

[0007] The objective of the present invention is achieved through the following technical solution: a fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning, the system comprising:

[0008] Enhanced CT liver automatic segmentation module, used to obtain the whole liver mask based on enhanced CT arterial phase and portal phase images;

[0009] Enhanced CT liver cancer automatic segmentation module, used to obtain liver cancer mask based on enhanced CT arterial phase and portal phase images and whole liver mask;

[0010] The recurrence risk prediction module based on whole liver imaging and deep learning is used to obtain the whole liver region of interest in the arterial phase and portal phase based on the enhanced CT arterial phase and portal phase images and the whole liver mask, and combine the liver cancer mask as a common input, and extract features from the feature maps of multiple resolutions generated in sequence by the convolutional neural network. The feature maps of the arterial phase and portal phase extracted each time are respectively subjected to phase-phase feature fusion and output scale-specific feature vectors. The feature vectors of different scales are collectively referred to as multi-scale features. After the phase-phase feature fusion, new arterial phase and portal phase feature maps are generated for the next layer of convolution. The arterial phase and portal phase feature maps generated by the last layer of convolution are spliced ​​and fused to obtain the main feature vector. The main feature vector is spliced ​​with the multi-scale feature and input into multiple layers of full connection to obtain the whole liver image feature, which is then input into another fully connected layer to predict and output the recurrence risk score. The higher the recurrence risk score, the higher the recurrence risk.

[0011] Furthermore, the enhanced CT arterial phase and portal phase images are the most recent enhanced CT arterial phase and portal phase images acquired within one month before liver resection.

[0012] Furthermore, the model in the enhanced CT liver automatic segmentation module is obtained by retraining multiple liver segmentation models using enhanced CT images; these models are used to segment the liver area in the enhanced CT arterial phase and portal phase images, respectively, and generate corresponding whole liver masks; then, the results of each model are integrated through the voting method to determine the final liver label of each pixel.

[0013] Furthermore, the enhanced CT liver cancer automatic segmentation module takes enhanced CT arterial phase and portal phase images and whole liver mask as input. The enhanced CT liver cancer automatic segmentation model in this module expands the single-phase attention module in PA-ResSeg into a dual-phase attention module, which is used to fuse the arterial phase and portal phase information; finally, the enhanced CT liver cancer automatic segmentation module can simultaneously and accurately segment the liver cancer areas in the arterial phase and portal phase images, and generate a liver cancer mask.

[0014] Furthermore, the recurrence risk prediction module based on whole liver imaging and deep learning preprocesses the arterial phase and portal phase images, and the specific process is as follows:

[0015] For the enhanced CT images, the arterial phase and portal phase images are preprocessed respectively, and the preprocessing process is as follows: resampling the image and the corresponding whole liver mask to a uniform resolution; adjusting the grayscale value in the image according to a fixed window width and window position; multiplying the image pixel value by the whole liver mask point-to-point to generate a whole liver region of interest; calculating the centroid of the whole liver region of interest in the arterial phase and portal phase respectively, and aligning the whole liver region of interest in the arterial phase and portal phase and the liver cancer mask in the arterial phase and portal phase based on the centroid; intercepting the whole liver region of interest and the liver cancer mask based on a fixed bounding box as the whole liver image input of the recurrence risk prediction model.

[0016] Furthermore, the recurrence risk prediction module based on whole liver imaging and deep learning uses a deep convolutional neural network to predict the recurrence risk, as follows:

[0017] The architecture of the deep convolutional neural network includes a feature extraction backbone, an inter-phase attention unit, a multi-scale feature extraction unit, and a recurrence risk score output unit in sequence;

[0018] The feature extraction backbone is designed to achieve feature extraction based on whole liver images, and includes two parallel neural network branches with exactly the same structure, namely, the arterial phase branch and the portal phase branch; the inputs of the arterial phase branch and the portal phase branch are the whole liver region of interest and liver cancer mask generated in the first stage in the arterial phase, and the whole liver region of interest and liver cancer mask in the portal phase, respectively; each branch starts with a layer of three-dimensional convolution, and then connects four convolution blocks in sequence, each convolution block corresponds to a resolution layer, and includes a convolution downsampling layer, a convolution layer, a batch normalization layer, a linear rectification function ReLU activation layer and a residual connection in sequence; the outputs of the two branches are spliced ​​in the channel dimension and then connected to the convolution layer and the fully connected layer, and finally a 64-dimensional backbone feature vector is output;

[0019] In each resolution layer of the feature extraction trunk, the feature maps output by the previous resolution layer of the arterial phase and portal phase branches are respectively input into the phase-to-phase attention unit to complete the phase-to-phase feature fusion, and then new arterial phase and portal phase feature maps are respectively generated and then used as the input of the next resolution layer of the feature extraction trunk. The arterial phase and portal phase feature maps generated by the last phase-to-phase attention unit are spliced ​​and input into the new convolution block to obtain the trunk feature vector;

[0020] In each non-highest resolution layer of the feature extraction trunk, the feature map output by the previous resolution layer of the arterial phase and portal phase branches will be simultaneously input into the multi-scale feature extraction unit to extract and fuse the information from the arterial phase and portal phase, so as to output the scale-specific feature vector of the current resolution layer; specifically, the multi-scale feature extraction unit first performs maximum pooling processing on the feature maps output by the previous resolution layer of the arterial phase and portal phase branches, and the maximum pooling operations of different resolution layers all use a unified target size; the feature maps of the arterial phase and portal phase after maximum pooling are respectively subjected to feature conversion through convolution layers, and then spliced ​​in the channel dimension, and then input into another convolution layer to realize the fusion of the dual-phase information; finally, the fused feature map is flattened and input into two fully connected layers, and a sixty-four-dimensional scale-specific feature vector is output; the feature extraction trunk includes a total of four non-highest resolution layers, and finally four sixty-four-dimensional scale-specific feature vectors are output;

[0021] The trunk feature vector obtained by the feature extraction trunk is concatenated with the four sixty-four-dimensional scale-specific feature vectors of the multi-scale feature extraction unit, and is sequentially input into three fully connected layers to finally output a recurrence risk score, which represents the logarithm of the ratio of the individual recurrence risk to the baseline recurrence risk. A higher recurrence risk score reflects a higher recurrence risk for the patient.

[0022] Furthermore, the loss function in the recurrence risk prediction module based on whole liver imaging and deep learning adopts a negative logarithmic partial likelihood function; the recurrence risk score output by the recurrence risk prediction unit is input into the loss function to obtain the recurrence-free survival loss.

[0023] Furthermore, the system also includes a recurrence risk prediction model interpretation module, which is used to display the relationship between each pixel point in the patient's enhanced CT image and the recurrence risk; the recurrence risk prediction model interpretation module outputs a class activation map based on the Grad-CAM algorithm, highlights the correlation between each pixel point in the whole liver image area and the recurrence risk, and assists doctors in understanding the biological mechanism related to recurrence by visualizing the image features that lead to increased recurrence risk, thereby making reliable clinical decisions.

[0024] In the second aspect, the present invention also provides a fully automatic liver cancer recurrence risk prediction device based on whole liver imaging and deep learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning.

[0025] In a third aspect, the present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning.

[0026] Beneficial effects of the present invention:

[0027] (1) It can perform high-precision automatic liver segmentation in fully automatic enhanced CT images.

[0028] (2) It can perform the most advanced fully automatic segmentation of liver cancer in enhanced CT images.

[0029] (3) The recurrence risk prediction model based on deep convolutional neural network can comprehensively extract the imaging features of the whole liver including the tumor that are related to the recurrence risk of hepatocellular carcinoma resection.

[0030] (4) The recurrence risk prediction model based on deep convolutional neural network ultimately outputs a recurrence risk score.

[0031] (5) The recurrence risk prediction model interpretation module visualizes the operating mechanism of the model to assist clinicians in understanding and appropriately using the recurrence risk prediction model based on deep convolutional neural networks.

[0032] (6) The fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning provided by the present invention is fully automated, does not require manual intervention by doctors, and is suitable for seamless integration into clinical practice processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described below are only partial schematic diagrams of the present invention. For ordinary technicians in this field and related fields, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 This is a flow chart of the fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning disclosed in the present invention.

[0035] Figure 2 It is a model diagram in the enhanced CT liver automatic segmentation module disclosed in the present invention.

[0036] Figure 3 It is a model diagram in the enhanced CT liver cancer automatic segmentation module disclosed in the present invention.

[0037] Figure 4 This is a model diagram of the recurrence risk prediction module based on whole liver imaging and deep learning disclosed in the present invention.

[0038] Figure 5 This is a performance comparison between the fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning disclosed in the present invention and the traditional clinical model and the tumor-based model.

[0039] Figure 6 This is a structural diagram of a fully automatic liver cancer recurrence risk prediction device based on whole liver imaging and deep learning provided by the present invention. DETAILED DESCRIPTION

[0040] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings and examples. The accompanying drawings and examples are only provided to illustrate the implementation modes of the present invention and do not limit the present invention in any form.

[0041] 1. Research subjects

[0042] The subjects of this example were 795 HCC patients who underwent liver resection, and the exclusion criteria were as follows:

[0043] (1) Non-primary HCC;

[0044] (2) received more than one hepatectomy;

[0045] (3) received any one or more anticancer treatments before surgery;

[0046] (4) Previous occurrence of other tumors;

[0047] (5) Accompanied by other tumors;

[0048] (6) The quality of enhanced CT images is not high;

[0049] 2. Research Methods

[0050] All research subjects were randomly divided into a model development set (N=608) and a test set (N=187) at a ratio of 4:1. The process of the fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning disclosed in the present invention is as follows: Figure 1 As shown, the system includes an enhanced CT liver automatic segmentation module, an enhanced CT liver cancer automatic segmentation module, a recurrence risk prediction module based on whole liver imaging and deep learning, and a recurrence risk prediction model interpretation module.

[0051] (1) Enhanced CT liver automatic segmentation module

[0052] The enhanced CT liver automatic segmentation module is used to automatically and accurately segment the liver in the enhanced CT arterial phase and portal phase images to obtain a whole liver mask. Further, the whole liver region of interest can be obtained based on the whole liver mask and the enhanced CT image. The enhanced CT liver automatic segmentation module includes an enhanced CT liver automatic segmentation integrated model, the model architecture is as follows Figure 2 400 samples were randomly selected from the model development set, and the liver contour and tumor contour were annotated semi-automatically. Four published liver segmentation models based on deep learning were trained separately, namely model 1: nnU-Net, model 2: PraNet, model 3: UNet 3+, and model 4: Attention U-Net. Finally, the voting method was used to integrate the above four models into the final enhanced CT liver automatic segmentation model. The model diagram is shown in Figure 2. The specific process is as follows:

[0053] Each individual model in the enhanced CT liver automatic segmentation integrated model is used to segment the liver area in the arterial phase and portal phase images respectively, and the whole liver and liver cancer masks of the individual models are obtained, wherein the values ​​of the pixels belonging to the background, liver, and liver cancer are marked as 0, 1, and 2, respectively. Since liver cancer belongs to the liver, the values ​​of the pixels where the liver cancer is located are all changed to 1, thereby obtaining the final whole liver mask of each individual model. The whole liver segmentation probability map output by each individual model in the enhanced CT liver automatic segmentation integrated model is binarized to 0 or 1 by rounding, wherein the pixel value belonging to the liver is 1, and the pixel value belonging to the background is 0, thereby obtaining the whole liver mask of the individual model. The whole liver masks of all individual models in the enhanced CT liver automatic segmentation integrated model are merged, and the final liver label of each pixel is determined by voting, that is, the principle of minority obeys majority is followed, thereby obtaining the final whole liver masks of the enhanced CT arterial phase and portal phase, respectively. The arterial phase and portal phase images of the enhanced CT of the patient are point-to-point multiplied with the whole liver mask obtained by the enhanced CT liver automatic segmentation model to obtain the whole liver region of interest in the arterial phase and portal phase of the enhanced CT, respectively.

[0054] (2) Enhanced CT liver cancer automatic segmentation module

[0055] The enhanced CT liver cancer automatic segmentation module is used to automatically segment liver tumors in enhanced CT arterial phase and portal phase images to obtain liver cancer masks. The enhanced CT liver cancer automatic segmentation module includes an enhanced CT liver cancer automatic segmentation model, the model architecture is as follows Figure 3 As shown. Based on the published deep learning-based public liver cancer segmentation model PA-ResSeg (Xu Y, CaiM, Lin L, et al. PA-ResSeg: A phase attention residual network for liver tumor segmentation from multiphase CT images[J]. Medical Physics, 2021, 48(7): 3752-3766.), the single-phase attention unit is expanded to a dual-phase attention unit to form an enhanced CT liver cancer automatic segmentation model biPA-ResSeg suitable for dual-phase. The biPA-ResSeg model is trained using the dataset used to develop the enhanced CT liver automatic segmentation model mentioned above. The workflow is as follows:

[0056] The whole liver region of interest in the enhanced CT arterial phase and portal phase is simultaneously input into the enhanced CT liver cancer automatic segmentation model, and the liver cancer segmentation probability map in the arterial phase and portal phase is simultaneously output, wherein the closer the probability value of each pixel in the probability map is to 1, the greater the possibility that it is liver cancer, and the closer it is to 0, the greater the possibility that it is background. The liver cancer segmentation probability map is binarized to 0 or 1 by rounding, wherein the pixel value belonging to liver cancer is 1, and the pixel value belonging to the background is 0, thereby obtaining a liver cancer mask.

[0057] (3) Recurrence risk prediction module based on whole liver imaging and deep learning

[0058] The whole liver imaging and deep learning recurrence risk prediction module sequentially includes a first stage of data preprocessing and a second stage of recurrence risk prediction based on a deep convolutional neural network.

[0059] The process of data preprocessing in the first stage is as follows:

[0060] For the enhanced CT images, the arterial phase and portal phase images are preprocessed respectively, and the preprocessing includes resampling the image and the corresponding whole liver mask to the median resolution calculated based on the training set, that is, 5mm×0.684mm×0.684mm. The grayscale value in the image is adjusted according to the fixed window width (350HU) and window level (75HU). The image is multiplied point-to-point with the whole liver mask to obtain the whole liver region of interest. The centroid of the whole liver region of interest in the arterial phase and portal phase is obtained respectively, and the arterial phase, portal phase whole liver region of interest, arterial phase and portal phase liver cancer mask are registered with the centroid. Based on the training set, the maximum size bounding box of the liver in three-dimensional space is calculated to be 48×352×480, and the coordinate axes of the bounding box are respectively the z-axis (indicating the head to foot direction), the y-axis (indicating the front to back direction), and the x-axis (indicating the left to right direction). The bounding box is used to intercept the whole liver region of interest and the liver cancer mask as the whole liver image input of the recurrence risk prediction model.

[0061] The second stage of the recurrence risk prediction module based on whole liver imaging and deep learning is the recurrence risk prediction based on deep convolutional neural network. The architecture of the neural network LiverSurv includes a feature extraction backbone, an interphase attention unit, a multi-scale feature extraction unit, and a recurrence risk score output unit. The model diagram is as follows: Figure 4 , the workflow is as follows:

[0062] The purpose of the feature extraction backbone is to realize feature extraction based on whole liver images. It contains two parallel neural network branches with exactly the same structure, referred to as the arterial phase branch and the portal phase branch, which use the whole liver images of the arterial phase and portal phase generated in the first stage as inputs respectively. The arterial phase branch or the portal phase branch starts with a layer of three-dimensional convolution, followed by four convolution blocks in sequence. Each convolution block corresponds to a resolution layer, which is composed of a convolution-based downsampling layer, a normal convolution layer, a batch normalization layer, an activation layer, and a residual connection. The outputs of the last layer of the two branches are spliced ​​in the channel dimension, followed by a convolution layer and a fully connected layer, and finally a 64-dimensional backbone feature vector is output.

[0063] At each resolution layer of the feature extractor, the feature maps output by the previous resolution layer of the arterial phase branch and the portal phase branch are respectively input into the phase-by-phase attention unit for phase-by-phase fusion, and then new arterial phase and portal phase feature maps are respectively generated as the input of the next resolution layer of the feature extractor. The arterial phase and portal phase feature maps generated by the last phase-by-phase attention unit are spliced ​​to obtain the trunk feature vector. The structure of the phase-by-phase attention unit is consistent with the published Multimodal Transfer Module (MMTM) (Joze HRV, Shaban A, Iuzzolino ML, et al. MMTM: Multimodal transfer module for CNNfusion[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 13289-13299.). The only difference is that the inputs A and B of MMTM are replaced by the arterial phase and portal phase feature maps respectively.

[0064] In each non-highest resolution layer of the feature extraction trunk, the feature map output by the previous resolution layer of the arterial phase and portal phase branches will be simultaneously input into the multi-scale feature extraction unit to extract and fuse the information from the arterial phase and portal phase, so as to output the scale-specific feature vector of the current resolution layer; specifically, the multi-scale feature extraction unit first performs maximum pooling processing on the feature maps output by the previous resolution layer of the arterial phase and portal phase branches, and the maximum pooling operations of different resolution layers all use a unified target size; the feature maps of the arterial phase and portal phase after maximum pooling are respectively subjected to feature conversion through convolution layers, and then spliced ​​in the channel dimension, and then input into another convolution layer to realize the fusion of the dual-phase information; finally, the fused feature map is flattened and input into two fully connected layers, and a sixty-four-dimensional scale-specific feature vector is output; the feature extraction trunk includes a total of four non-highest resolution layers, and finally four sixty-four-dimensional scale-specific feature vectors are output;

[0065] The scale-specific feature vectors of all resolution layers are concatenated with the trunk feature vector output by the feature extraction trunk and input into a multilayer perceptron. The penultimate layer of the multilayer perceptron allows the input of a one-dimensional clinical feature vector, and finally outputs a recurrence risk score, which represents the logarithm of the ratio of an individual's recurrence risk to a baseline recurrence risk. A higher value reflects a higher risk of recurrence for the patient.

[0066] The key points of the training strategy of the recurrence risk prediction model based on whole liver imaging and deep learning are as follows:

[0067] (a) The model development set was randomly divided into four folds. Three of the folds were used as training sets each time, and the remaining fold was used as an internal validation set. Four models were obtained after taking turns using the four folds. The four models were integrated to obtain the final whole liver imaging and deep learning recurrence risk prediction model. The integration method was to perform Z-score conversion on the recurrence risk score predicted by each model and then average them as the final recurrence risk score of the whole liver imaging and deep learning recurrence risk prediction model.

[0068] (b) During the model training process, the whole liver region of interest in the arterial and portal phases of enhanced CT was subjected to consistent random scaling, random flipping, random brightness multiplication transformation, and random Gaussian blur operations at the same time, in order to expand the training sample data set.

[0069] (c) The loss function in the recurrence risk prediction module based on whole liver imaging and deep learning adopts a negative logarithmic partial likelihood function. The recurrence risk score output by the recurrence risk prediction unit is input into the loss function to obtain the recurrence-free survival loss.

[0070] (4) Recurrence risk prediction model interpretation module

[0071] The recurrence risk prediction model interpretation module is designed to assist medical staff in understanding the working mechanism of the recurrence risk prediction model based on whole liver imaging and deep learning. The output of the recurrence risk prediction model interpretation module is a class activation map, which shows the focus of the model during the model reasoning process, that is, what kind of image features the model is based on to calculate the patient's recurrence risk score. The Grad-CAM method is used in the recurrence risk prediction model interpretation module. The weights of the first layer of convolution are obtained through back propagation, and the weights of each channel are multiplied by the value of the corresponding feature map, and the final class activation map is obtained by point-to-point addition.

[0072] 3. Experimental results

[0073] (1) Enhance the performance of CT liver automatic segmentation model

[0074] Table 1 shows the liver segmentation performance comparison of the integrated enhanced CT liver automatic segmentation model and its basic models on the test set. The results show that the comprehensive accuracy of the integrated enhanced CT liver automatic segmentation model is better than that of the four basic models.

[0075] Table 1. Performance comparison of the integrated enhanced CT liver automatic segmentation model and its basic model

[0076]

[0077] (2) Improving the performance of CT liver cancer automatic segmentation model

[0078] Table 2 shows the performance of the enhanced CT liver cancer automatic segmentation model on the test set. The results show that the model has acceptable performance in liver cancer segmentation in the enhanced CT arterial phase and portal phase.

[0079] Table 2. Enhanced CT liver cancer automatic segmentation model performance

[0080]

[0081] (3) Performance comparison between the recurrence risk prediction system based on whole liver imaging and deep learning and the traditional model

[0082] like Figure 5 As shown in the results, the fully automatic recurrence risk prediction system based on whole liver imaging and deep learning is significantly better than the traditional clinical model that uses clinical features without manual operation as input, and the model that combines the risk score output by the system with these clinical features has the best performance.

[0083] like Figure 6 As shown in the figure, the fully automatic recurrence risk prediction system based on whole liver imaging and deep learning is superior to the other three traditional models based on tumor regions of interest, namely, the model based on tumor semantic features (including the maximum diameter of the tumor and the number of tumors) (tumor semantic model), the model based on tumor imaging genomics features (tumor imaging genomics model), and the deep convolutional neural network based on tumor regions of interest (tumor deep learning model).

[0084] 4. Experimental conclusion

[0085] In summary, the present invention discloses for the first time a fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning, which can segment the liver from preoperative enhanced CT images to obtain the whole liver region of interest, and can segment liver cancer from preoperative enhanced CT images to obtain liver cancer masks, and then extract whole liver information including tumors based on whole liver imaging, liver cancer masks and deep learning to predict the risk of recurrence after liver cancer resection. The performance of the fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning in predicting recurrence risk is significantly better than traditional clinical models and models based on tumor regions of interest. In addition, the fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning has the advantages of being fully automatic and non-invasive, and is expected to be seamlessly embedded in the clinical practice process to help improve the individualized management of liver cancer patients.

[0086] Corresponding to the aforementioned embodiment of a fully automatic liver cancer recurrence risk prediction method based on whole liver imaging and deep learning, the present invention also provides an embodiment of a fully automatic liver cancer recurrence risk prediction device based on whole liver imaging and deep learning.

[0087] See also Figure 6An embodiment of the present invention provides a fully automatic liver cancer recurrence risk prediction device based on whole liver imaging and deep learning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a fully automatic liver cancer recurrence risk prediction method based on whole liver imaging and deep learning in the above embodiment.

[0088] The embodiment of the fully automatic liver cancer recurrence risk prediction device based on whole liver imaging and deep learning provided by the present invention can be applied to any device with data processing capabilities, and the any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 6 As shown in FIG. 1 , a hardware structure diagram of any device with data processing capability is provided in which a fully automatic liver cancer recurrence risk prediction device based on whole liver imaging and deep learning is located. Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0089] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0090] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0091] An embodiment of the present invention also provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a fully automatic liver cancer recurrence risk prediction method based on whole liver imaging and deep learning in the above embodiment is implemented.

[0092] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0093] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the fully automatic liver cancer recurrence risk prediction method based on whole liver imaging and deep learning.

[0094] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning, characterized in that: The system includes: An enhanced CT liver automatic segmentation module is used to obtain a whole liver mask based on enhanced CT arterial phase and portal phase images; the models in the enhanced CT liver automatic segmentation module are retrained by using enhanced CT images to train multiple publicly published liver segmentation models, namely model 1: nnU-Net, model 2: PraNet, model 3: UNet 3+, model 4: Attention U-Net; these models are used to segment the liver area in the enhanced CT arterial phase and portal phase images, and generate corresponding whole liver masks; then, the results of each model are integrated by voting method to determine the final liver label of each pixel; An enhanced CT liver cancer automatic segmentation module is used to obtain a liver cancer mask based on enhanced CT arterial phase and portal phase images and a whole liver mask; the enhanced CT liver cancer automatic segmentation module takes enhanced CT arterial phase and portal phase images and a whole liver mask as input, and the enhanced CT liver cancer automatic segmentation model in the module is to expand the single-phase attention module in PA-ResSeg into a dual-phase attention module, which is used to fuse arterial phase and portal phase information; finally, the enhanced CT liver cancer automatic segmentation module can accurately segment the liver cancer area in the arterial phase and portal phase images at the same time, and generate a liver cancer mask; The recurrence risk prediction module based on whole liver imaging and deep learning is used to obtain the whole liver region of interest in the arterial phase and portal phase based on the enhanced CT arterial phase and portal phase images and the whole liver mask, and combine the liver cancer mask as a common input, and extract features from the feature maps of multiple resolutions generated in sequence by the convolutional neural network. The feature maps of the arterial phase and portal phase extracted each time are respectively subjected to phase-by-phase feature fusion and output scale-specific feature vectors. The feature vectors of different scales are collectively referred to as multi-scale features. After the phase-by-phase feature fusion, new arterial phase and portal phase feature maps are generated for the next layer of convolution. The arterial phase and portal phase feature maps generated by the last layer of convolution are spliced ​​and fused to obtain the main feature vector. The main feature vector is spliced ​​with the multi-scale feature and input into multiple layers of full connection to obtain the whole liver image feature, and then input into a fully connected layer to predict and output the recurrence risk score. The higher the recurrence risk score, the higher the recurrence risk. The recurrence risk prediction module based on whole liver imaging and deep learning uses a deep convolutional neural network to predict the recurrence risk, as follows: The architecture of the deep convolutional neural network includes a feature extraction backbone, an inter-phase attention unit, a multi-scale feature extraction unit, and a recurrence risk score output unit in sequence; The feature extraction backbone is designed to achieve feature extraction based on whole liver images, and includes two parallel neural network branches with exactly the same structure, namely, the arterial phase branch and the portal phase branch; the inputs of the arterial phase branch and the portal phase branch are the whole liver region of interest and liver cancer mask generated in the first stage in the arterial phase, and the whole liver region of interest and liver cancer mask in the portal phase, respectively; each branch starts with a layer of three-dimensional convolution, and then connects four convolution blocks in sequence, each convolution block corresponds to a resolution layer, and includes a convolution downsampling layer, a convolution layer, a batch normalization layer, a linear rectification function ReLU activation layer and a residual connection in sequence; the outputs of the two branches are spliced ​​in the channel dimension and then connected to the convolution layer and the fully connected layer, and finally a 64-dimensional backbone feature vector is output; In each resolution layer of the feature extraction trunk, the feature maps output by the previous resolution layer of the arterial phase and portal phase branches are respectively input into the phase-to-phase attention unit to complete the phase-to-phase feature fusion, and then new arterial phase and portal phase feature maps are respectively generated and then used as the input of the next resolution layer of the feature extraction trunk. The arterial phase and portal phase feature maps generated by the last phase-to-phase attention unit are spliced ​​and input into the new convolution block to obtain the trunk feature vector; In each non-highest resolution layer of the feature extraction trunk, the feature map output by the previous resolution layer of the arterial phase and portal phase branches will be simultaneously input into the multi-scale feature extraction unit to extract and fuse the information from the arterial phase and portal phase, so as to output the scale-specific feature vector of the current resolution layer; specifically, the multi-scale feature extraction unit first performs maximum pooling processing on the feature maps output by the previous resolution layer of the arterial phase and portal phase branches, and the maximum pooling operations of different resolution layers all use a unified target size; the feature maps of the arterial phase and portal phase after maximum pooling are respectively subjected to feature conversion through convolution layers, and then spliced ​​in the channel dimension, and then input into another convolution layer to realize the fusion of the dual-phase information; finally, the fused feature map is flattened and input into two fully connected layers, and a sixty-four-dimensional scale-specific feature vector is output; the feature extraction trunk includes a total of four non-highest resolution layers, and finally four sixty-four-dimensional scale-specific feature vectors are output; The trunk feature vector obtained by the feature extraction trunk is concatenated with the four sixty-four-dimensional scale-specific feature vectors of the multi-scale feature extraction unit, and is sequentially input into three fully connected layers to finally output a recurrence risk score, which represents the logarithm of the ratio of the individual recurrence risk to the baseline recurrence risk. A higher recurrence risk score reflects a higher recurrence risk for the patient.

2. According to claim 1, a fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning is characterized in that: The enhanced CT arterial phase and portal phase images are the most recent enhanced CT arterial phase and portal phase images acquired within one month before liver resection.

3. The fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning according to claim 1, characterized in that: The recurrence risk prediction module based on whole liver imaging and deep learning preprocesses the arterial phase and portal phase images. The specific process is as follows: For the enhanced CT image, the arterial phase and portal phase images are preprocessed respectively, and the preprocessing process is sequentially resampling the image and the corresponding whole liver mask to a uniform resolution; adjusting the grayscale value in the image according to a fixed window width and window position; multiplying the image pixel value by the whole liver mask point-to-point to generate a whole liver region of interest; calculating the centroid of the whole liver region of interest in the arterial phase and portal phase respectively, and registering the whole liver region of interest in the arterial phase and portal phase and the liver cancer mask in the arterial phase and portal phase based on the centroid; The whole liver region of interest and liver cancer mask were captured based on a fixed bounding box and used as the whole liver image input for the recurrence risk prediction model.

4. The fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning according to claim 1, characterized in that: The loss function in the recurrence risk prediction module based on whole liver imaging and deep learning adopts a negative logarithmic partial likelihood function; the recurrence risk score output by the recurrence risk prediction unit is input into the loss function to obtain the recurrence-free survival loss.

5. The fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning according to claim 1, characterized in that: The system also includes a recurrence risk prediction model interpretation module, which is used to display the relationship between each pixel point in the patient's enhanced CT image and the recurrence risk; the recurrence risk prediction model interpretation module is based on the Grad-CAM algorithm output class activation map, highlighting the correlation between each pixel point in the whole liver image area and the recurrence risk, and assisting doctors in understanding the biological mechanism related to recurrence by visualizing the image features that lead to increased recurrence risk, so as to make reliable clinical decisions.

6. A fully automatic liver cancer recurrence risk prediction device based on whole liver imaging and deep learning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it runs a fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by the processor, it runs a fully automatic liver cancer recurrence risk prediction system based on whole liver imaging and deep learning as described in any one of claims 1 to 5.

Images