Method, equipment and medium for pathological omics analysis of hepatocellular carcinoma based on HE images

Abstract

The embodiment of the present invention discloses a method, device and medium for pathological omics analysis of hepatocellular carcinoma based on HE images. The method comprises the following steps: obtaining a HE-stained slice, scanning the HE-stained slice, and obtaining a WSI-HE image; preprocessing the WSI-HE image to obtain a WSI-HE feature image; performing patch segmentation on the WSI-HE feature image to identify the feature representation of the WSI-HE feature image; performing multi-instance learning on the feature representation of the WSI-HE feature image to obtain a pathological group category prediction result of the HE-stained slice; and grouping the corresponding patients of the HE-stained slice into hepatocellular carcinoma pathological groups according to the pathological group category prediction result. The present invention calculates the prediction value by performing pathological omics analysis on the HE image of the hepatocellular carcinoma lesion, and provides the recommended information of triple therapy accordingly, so as to provide a more accurate and valuable decision reference for clinical workers.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method, device and medium for pathological omics analysis of hepatocellular carcinoma based on HE images. Background Art

[0002] Primary liver cancer is a malignant tumor that is widely distributed worldwide, with high morbidity and mortality. Due to the rapid progression of liver cancer and atypical early symptoms, a large number of patients are diagnosed in the middle and late stages when they first visit the doctor. At this time, they often have vascular invasion or distant metastasis, missing the best time for radical treatment such as surgical resection or liver transplantation. The pathological classification of primary liver cancer includes hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma, and mixed liver cancer, among which hepatocellular carcinoma is the main focus of liver cancer case research and treatment.

[0003] Hepatic Arterial Infusion Chemotherapy (HAIC) is a recognized and effective local treatment for hepatocellular carcinoma. It can significantly reduce the adverse reactions of systemic chemotherapy drugs while maintaining the local concentration of chemotherapy drugs in the tumor. In addition, tyrosine kinase inhibitors (TKIs) and immune checkpoint inhibitors (ICIs) are standard treatment options for patients with advanced hepatocellular carcinoma and have been proven to prolong patient survival and improve quality of life. The combination of TKIs and ICIs (referred to as "triple therapy" in this application) on the basis of HAIC can further significantly improve the objective response rate of patients with hepatocellular carcinoma and bring survival benefits to patients.

[0004] However, clinical results show that not all patients with hepatocellular carcinoma can achieve a good treatment response when receiving triple therapy. A considerable proportion of patients still have progressive tumors after receiving triple therapy. At this time, it may be too late to consider other systemic treatment measures, which seriously affects the patient's prognosis and quality of life. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method, device and medium for pathological omics analysis of hepatocellular carcinoma based on HE images.

[0006] The first aspect of the present invention provides a method for pathological omics analysis of hepatocellular carcinoma based on HE images, comprising the following steps:

[0007] Acquiring a HE-stained section, and scanning the HE-stained section to obtain a WSI-HE image;

[0008] Preprocessing the WSI-HE image to obtain a WSI-HE characteristic image;

[0009] Performing patch segmentation on the WSI-HE feature image to identify feature representations of the WSI-HE feature image;

[0010] Performing multi-instance learning on the feature representation of the WSI-HE feature image to obtain a pathological group category prediction result of the HE-stained section;

[0011] According to the pathological group category prediction results, the corresponding patients of the HE-stained sections were grouped into hepatocellular carcinoma pathological groups.

[0012] Furthermore, the HE-stained sections are obtained, specifically comprising the following steps:

[0013] Patients with hepatocellular carcinoma were selected from the patient medical record database based on their historical treatment regimens;

[0014] Collecting HE-stained sections of the hepatocellular carcinoma patients obtained by screening;

[0015] The HE-stained sections were marked according to the patient's treatment response as documented in the historical treatment plan.

[0016] Further, the preprocessing of the WSI-HE image to obtain the WSI-HE characteristic image specifically includes the following steps:

[0017] Converting the WSI-HE image into a preset scalable storage format;

[0018] Screening the WSI-HE images according to preset image screening conditions, removing the WSI-HE images that do not meet the preset image screening conditions, and naming the WSI-HE images that meet the preset image screening conditions according to a unified naming rule; the preset image screening conditions include clarity, contrast, and focus of the WSI-HE images;

[0019] The named WSI-HE images are stored uniformly as feature images.

[0020] Further, performing patch segmentation on the WSI-HE characteristic image and identifying the characteristic representation of the WSI-HE characteristic image specifically includes the following steps:

[0021] Determine a ROI region in the WSI-HE feature image;

[0022] Performing non-overlapping sampling on the ROI area in the WSI-HE feature image to obtain multiple patch blocks;

[0023] The patch blocks are screened according to preset block screening conditions, and WSI-HE images that do not meet the preset block screening conditions are removed; the block screening conditions include stains, defects, and out-of-focus;

[0024] The patch blocks that meet the block screening conditions are input into the feature extraction model, and the feature representation in the patch blocks is extracted by the feature extraction model.

[0025] Furthermore, the feature extraction model is a ResNet50 model, which is obtained by the following steps:

[0026] Design a ResNet50 model that includes convolutional layers, pooling layers, and fully connected classification layers, and synchronize the ResNet50 model to the cloud;

[0027] Use the Imagenet database to pre-train the cloud-based ResNet50 model to obtain a trained cloud-based ResNet50 model;

[0028] Migrate the trained cloud-based ResNet50 model parameters to the local ResNet50 model;

[0029] The fully connected classification layer in the local ResNet50 model is removed, and the convolutional layer and pooling layer are retained for feature extraction.

[0030] Furthermore, the feature representation of the WSI-HE feature image is subjected to multi-instance learning to obtain the pathological group category prediction result of the HE stained section, which is achieved through a multi-instance learning model and specifically includes the following steps:

[0031] Flattening the feature representation of the WSI-HE feature image into a one-dimensional vector;

[0032] The feature representation of the one-dimensional vector is concatenated to obtain the instance feature matrix;

[0033] The instance feature matrix is ​​input into a multi-instance learning model to obtain a prediction result as a pathological group category prediction result of the HE-stained section.

[0034] Furthermore, the multi-instance learning model is an EfficientNet-B5 model, which is obtained by the following steps:

[0035] Design an EfficientNet-B5 model that includes convolutional layers, pooling layers, and fully connected classification layers, and synchronize the EfficientNet-B5 model to the cloud;

[0036] Use the Imagenet database to pre-train the cloud-based EfficientNet-B5 model to obtain a trained cloud-based EfficientNet-B5 model;

[0037] Migrate the trained cloud-based EfficientNet-B5 model parameters to the local EfficientNet-B5 model;

[0038] The fully connected layer in the EfficientNet-B5 model is replaced by a preset fully connected layer; the preset fully connected layer includes two neurons, which are used to output positive prediction results and negative prediction results respectively.

[0039] Furthermore, the method of grouping the corresponding patients of the HE-stained sections into hepatocellular carcinoma pathological groups according to the pathological group category prediction result specifically comprises the following steps:

[0040] Determine the PRPS value of the patient according to the prediction result; the PRPS value is used to characterize the expected treatment response of the patient to hepatocellular carcinoma treatment;

[0041] The PRPS values ​​of the patients were compared with the preset PRPS threshold, and patients with values ​​above the PRPS threshold and those not above the PRPS threshold were divided into different groups;

[0042] The patient's subsequent treatment plan is determined according to the group into which the patient is assigned.

[0043] A second aspect of the present invention discloses an electronic device, comprising a processor and a memory;

[0044] The memory is used to store programs;

[0045] The processor executes the program to implement the HE image-based hepatocellular carcinoma pathological omics analysis method.

[0046] A third aspect of the present invention discloses a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the HE image-based hepatocellular carcinoma pathological omics analysis method.

[0047] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0048] The embodiments of the present invention have the following beneficial effects: The present invention provides a method, device and medium for pathological omics analysis of hepatocellular carcinoma based on HE images, which can comprehensively extract the morphological characteristics of tumors by using pathological omics technology, taking into account tumor heterogeneity, while the existing technology may not fully consider the complexity of tumors. The present invention defines the tumor area in the WSI-HE image as a key region of interest (ROI), and uses an efficient manual annotation method to create a basic knowledge base, which not only simplifies the analysis process, but also significantly reduces the computational burden of high-resolution histopathological analysis. The pathological omics model of the present invention is trained on clinical data from different hospitals, and uses pre-trained deep learning models (such as ResNet50 and EfficientNet-B5) to extract and analyze pathological images through transfer learning, and has passed internal and external verification, proving its wide application potential and high prediction accuracy in a variety of clinical environments. The implementation of the present invention is expected to provide new scientific tools for the individualized treatment of hepatocellular carcinoma, promote the further development of precision medicine, and bring more personalized and effective treatment plans to patients.

[0049] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 It is a schematic diagram of the basic process of the hepatocellular carcinoma pathological omics analysis method based on HE images of the present invention.

[0052] Figure 2 It is a flow chart of the present invention for performing patch segmentation on a feature image and extracting feature representation.

[0053] Figure 3 It is a flow chart of the present invention for performing multi-instance learning on feature representation.

[0054] Figure 4 It is a schematic diagram of the ROC curve and confusion matrix when the training set used by the model in a specific embodiment of the present invention is the overall test set.

[0055] Figure 5 It is a schematic diagram of the ROC curve and confusion matrix when the training set used by the model in a specific embodiment of the present invention is an internal test set.

[0056] Figure 6 It is a schematic diagram of the ROC curve and confusion matrix when the training set used by the model in a specific embodiment of the present invention is an external test set.

[0057] Figure 7 It is a structural schematic diagram of an electronic device of the present invention.

[0058] Figure 8 It is a schematic diagram of the structure of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] Histopathological images contain rich tumor genetic information and can provide a large number of pathological features, such as tumor tissue structure, cell morphology, tumor microenvironment, immune cell infiltration, tumor differentiation, thick beam type, etc. These features are closely related to the efficacy of tumor patients. However, the characteristics reflected by the above-mentioned technical methods are relatively single and cannot fully reflect the characteristics of tumor heterogeneity. With the rapid development of pathology and bioinformatics, pathological genomics has become an important branch in cancer research and clinical diagnosis. Whole Slide Imaging (WSI) technology digitizes pathological sections, greatly expanding the scope of pathological feature extraction, including but not limited to key parameters such as color, morphology, and texture. Furthermore, combined with deep learning algorithms such as Convolutional Neural Networks (CNN), pathological areas can be analyzed more accurately, quickly and reliably, which not only improves the accuracy of pathological diagnosis, but also provides new perspectives and tools for pathological research. However, due to the extremely high resolution of digital pathology images, the size of the entire WSI may be very large, containing hundreds of millions of pixels, which exceeds the processing capacity of general computers or graphics processing units (GPUs), so processing the entire image may face some challenges. In addition, factors such as differences in pathology slice preparation processes, scanner performance and configuration differences, and reagent performance differences in real scenarios also affect the quality of pathology images and analysis results.

[0061] To date, no research has successfully developed a pathomic model specifically for predicting tumor response in patients with hepatocellular carcinoma receiving triple therapy (i.e., the combined use of HAIC, TKIs, and ICIs). Therefore, the present invention aims to study and create an innovative method for predicting treatment response based on a pathomic model, which has efficient computing performance, can run under limited computing power and data cache space conditions, and exhibits good generalization.

[0062] like Figure 1 As shown, the first embodiment of the present invention provides a method for pathological omics analysis of hepatocellular carcinoma based on HE images, comprising the following steps:

[0063] Step 1: Obtain HE-stained sections, scan the HE-stained sections, and obtain WSI-HE images;

[0064] Step 2: Preprocess the WSI-HE image to obtain the WSI-HE feature image;

[0065] Step 3: Perform patch segmentation on the WSI-HE feature image and identify the feature representation of the WSI-HE feature image;

[0066] Step 4: Perform multi-instance learning on the feature representation of the WSI-HE feature image to obtain the pathological group category prediction results of the HE-stained sections;

[0067] Step 5: According to the pathological group category prediction results, the corresponding patients of the HE-stained sections are grouped into hepatocellular carcinoma pathological groups.

[0068] Step 1: Obtain HE-stained sections and scan them to obtain WSI-HE images. Step 1 is mainly used to collect hematoxylin-eosin (HE)-stained sections of tumor tissue puncture biopsies from patients with hepatocellular carcinoma, providing a basis for subsequent digital processing and analysis.

[0069] In a specific embodiment of the present invention, Step 1 involves the following detailed operations:

[0070] Step 1.1: Screen patients with hepatocellular carcinoma from the patient medical record database based on their historical treatment regimens.

[0071] The embodiment of the present invention mainly screens patients with hepatocellular carcinoma who have received triple therapy and meet specific conditions. The screening conditions include: (1) patients with hepatocellular carcinoma who are determined by a multidisciplinary team to be unsuitable for surgical resection or patients with early hepatocellular carcinoma who have a high risk of postoperative recurrence; (2) patients who use triple therapy as the initial treatment plan; (3) patients who have completed at least two cycles of hepatic artery infusion chemotherapy; (4) patients who have undergone at least one enhanced computed tomography or magnetic resonance imaging examination before treatment; (5) after completing 2 to 3 cycles of hepatic artery infusion chemotherapy, patients need to undergo an upper abdominal enhanced computed tomography or magnetic resonance imaging review to evaluate the efficacy; (6) patients have at least one measurable intrahepatic lesion; (7) patients' liver function (Child-Pugh) grade is A or B; (8) patients' Eastern Cooperative Oncology Group performance status score (ECOG PS) is less than 2; (9) patients have no other malignant tumors, severe cardiopulmonary diseases, or major diseases that affect survival; (10) patients' follow-up data are complete.

[0072] After the screening is completed, the embodiment of the present invention randomly selects 70% of the patients as the training set to train the model used subsequently; the remaining 30% of the patient data is used as the test set. At the same time, these test sets are further divided into internal test sets and external test sets according to the different hospitals from which the data comes. The use of all patient data in the embodiment of the present invention has been reviewed and approved by the ethics committees of the two hospitals to ensure the compliance of the research and the protection of patient privacy. During the implementation of the invention, patient data is strictly kept confidential to prevent the leakage of patient privacy.

[0073] Step 1.2: Collect HE-stained sections of the screened hepatocellular carcinoma patients.

[0074] After the screening is completed, the embodiment of the present invention collects HE-stained pathological sections obtained from the hepatocellular carcinoma puncture biopsy of the screened patients. Each section is checked to exclude sections with unclear diagnosis (no hepatocellular carcinoma cells on the slide) or quality problems (such as tissue folds or tears).

[0075] Step 1.3: Mark the HE-stained sections according to the patient's treatment response recorded in the historical treatment plan.

[0076] In the embodiment of the present invention, data labeling is performed based on the optimal treatment response of the patient's tumor lesions during triple therapy. Specifically, the evaluation of tumor treatment response adopts a modified solid tumor efficacy evaluation standard, mainly relying on enhanced CT images. In enhanced CT images, the blood vessels of tumor lesions are usually richer than normal tissues. Therefore, after the injection of contrast agent, the active lesions will show more obvious enhancement phenomenon, namely "intratumor arterial enhancement", which can be observed by brightness changes on CT images. Specifically, the following steps are included: (1) using enhanced CT images, identifying and observing the intratumoral arterial enhancement of tumor lesions, and measuring and recording the longest diameter of active lesions; (2) by comparing the changes in the diameter of active lesions before and after treatment, the tumor response is divided into complete remission (the intratumoral arterial enhancement of the lesion completely disappears, that is, there is no viable tumor lesion), partial remission (the diameter of the active lesion decreases by at least 30%), disease stabilization (the reduction in the diameter of the active lesion does not meet the criteria for partial remission, and the increase does not meet the criteria for disease progression) and disease progression (the diameter of the active lesion increases by at least 20%, or new arterial phase enhancing lesions appear); (3) according to the above criteria, the HE-stained sections of patients with complete remission or partial remission are marked as "responsive", and the HE-stained sections of patients with stable disease or disease progression are marked as "unresponsive".

[0077] Step 1.4: Scan the HE-stained sections to obtain WSI-HE images.

[0078] In the embodiment of the present invention, a high-quality full-slice digital scanner is used to scan all HE slices, and the image resolution is set to a ×40 objective lens to obtain a high-definition pathological slice image.

[0079] Step 2: Preprocess the WSI-HE image to obtain the WSI-HE feature image. Step 2 is mainly used to preprocess the pathological image acquired by WSI to optimize the image quality and prepare for feature extraction.

[0080] In a specific embodiment of the present invention, Step 2 includes the following detailed operations:

[0081] Step 2.1: Convert the WSI-HE image to the preset scalable storage format.

[0082] The embodiment of the present invention uniformly converts all scanned images into an expandable SVS format for storage.

[0083] Step 2.2: Filter the WSI-HE images according to the preset image screening conditions, remove the WSI-HE images that do not meet the preset image screening conditions, and name the WSI-HE images that meet the preset image screening conditions according to the unified naming rules; the preset image screening conditions include the clarity, contrast and focus of the WSI-HE images.

[0084] In the embodiment of the present invention, a strict quality control process is performed on all scanned images, and the quality control indicators include clarity, contrast, focus, etc., to ensure that the clarity, contrast and focus of each image meet the standards to meet the needs of subsequent analysis.

[0085] Step 2.3: Store the named WSI-HE images as feature images.

[0086] In the embodiment of the present invention, all scanned images are stored in a secure database according to a unified naming rule.

[0087] Step 3: Perform patch segmentation on the WSI-HE feature image and identify the feature representation of the WSI-HE feature image. Step 3 is mainly used to perform accurate image segmentation technology, extract small tissue areas, namely "image patches", from the region of interest (ROI) marked by the doctor, and perform feature representation on them to capture the microscopic characteristics of tumor tissue.

[0088] In a specific embodiment of the present invention, reference Figure 2 , Step 3 includes the following detailed operations:

[0089] Step 3.1: Determine the ROI area in the WSI-HE feature image.

[0090] In the embodiment of the present invention, the determination of the region of interest is performed by an experienced pathologist based on professional knowledge, using the Automated Slide Analysis Platform software (ASAP) v1.9.0 software to roughly outline the tumor area to identify the region of interest (ROI). The purpose of this step is to obtain valuable data features from the whole slice image.

[0091] Step 3.2: Perform non-overlapping sampling on the ROI area in the WSI-HE feature image to obtain multiple patch blocks.

[0092] In the embodiment of the present invention, the OpenSlide library is used to sample the WSI in a non-overlapping manner according to a sliding window size of 512×512 pixels in the ROI area at a magnification of 400 times to generate a series of patches. Non-overlapping sampling means that when selecting ROI on an image, there is no overlap between each ROI area, that is, there are no shared pixels between adjacent ROIs. Compared with overlapping sampling, non-overlapping sampling can reduce the amount of sampling calculations and running time, and also avoid the problem of pixel merging in overlapping areas.

[0093] Step 3.3: Filter the patch blocks according to the preset block filtering conditions, and remove the WSI-HE images that do not meet the preset block filtering conditions.

[0094] In the embodiment of the present invention, screening the patch blocks refers to excluding patches in the background area that do not contain any tissue cells and patches whose images are blurred due to out-of-focus, stains, etc., to ensure the quality of the selected patches.

[0095] Step 3.4: Input the patch blocks that meet the block screening conditions into the feature extraction model, and extract the feature representation in the patch blocks through the feature extraction model.

[0096] The feature extraction model used in the present invention is a ResNet50 model, which performs forward propagation through the basic model to extract the corresponding feature vector representation. The ResNet50 model is specifically trained by the following steps.

[0097] Step 3.4.1: Design a ResNet50 model including convolutional layers, pooling layers, and fully connected classification layers, and synchronize the ResNet50 model to the cloud;

[0098] Step 3.4.2: Use the Imagenet database to pre-train the cloud-based ResNet50 model to obtain a trained cloud-based ResNet50 model;

[0099] Step 3.4.3: Migrate the trained cloud-based ResNet50 model parameters to the local ResNet50 model;

[0100] Step 3.4.4: Remove the fully connected classification layer in the local ResNet50 model, and keep the convolutional layer and pooling layer for feature extraction.

[0101] The embodiment of the present invention uses Imagenet, a large image database, to pre-train the ResNet50 model used. After the training is completed in the cloud, the trained model parameters are migrated to the local model through transfer learning for fast parameter initialization. Since the local model is used to extract the feature representation of the patch block, it is necessary to remove the fully connected classification layer in the local model so that the output of the model is the feature vector representation of the patch block rather than the classification result, which is convenient for subsequent modeling.

[0102] Step 4: Perform multi-instance learning on the feature representation of the WSI-HE feature image to obtain the pathological group category prediction results of the HE-stained sections. Step 4 is mainly used to build a multi-instance learning model based on WSI data and use a deep learning algorithm to conduct in-depth analysis of pathological features to improve the generalization ability and prediction accuracy of the model.

[0103] In a specific embodiment of the present invention, Figure 3 As shown, Step 4 includes the following detailed operations:

[0104] Step 4.1: Flatten the feature representation of the WSI-HE feature image into a one-dimensional vector.

[0105] In the embodiment of the present invention, each WSI is regarded as a "bag" containing n slice instances. The label of the entire WSI is the bag label (including labels such as "responsive" and "unresponsive"). The feature representation of the patch extracted in Step 3 is flattened into a one-dimensional vector for subsequent splicing.

[0106] Step 4.2: Concatenate the feature representations of the one-dimensional vectors to obtain the instance feature matrix.

[0107] In the embodiment of the present invention, one-dimensional feature representations are concatenated, so that each "bag" can be represented as an instance feature matrix of n rows, where each row represents the feature of an instance.

[0108] Step 4.3: Input the instance feature matrix into the multi-instance learning model and obtain the prediction result as the pathological group category prediction result of the HE stained section.

[0109] The multi-instance learning model used in the present invention is the EfficientNet-B5 model, whose optimizer is Adam, the learning rate is set to 0.01, the loss function is set to binary crossentropy, the training cycle is 3000 times and the model with the best performance is saved. The EfficientNet-B5 model is trained specifically by the following steps.

[0110] Step 4.3.1: Design an EfficientNet-B5 model that includes convolutional layers, pooling layers, and fully connected classification layers, and synchronize the EfficientNet-B5 model to the cloud;

[0111] Step 4.3.2: Use the Imagenet database to pre-train the cloud-based EfficientNet-B5 model to obtain a trained cloud-based EfficientNet-B5 model;

[0112] Step 4.3.3: Migrate the trained cloud-based EfficientNet-B5 model parameters to the local EfficientNet-B5 model;

[0113] Step 4.3.4: Use the preset fully connected layer to replace the migrated fully connected layer in the EfficientNet-B5 model; the preset fully connected layer includes two neurons, which are used to output positive prediction results and negative prediction results respectively.

[0114] The EfficientNet-B5 model of the embodiment of the present invention is also trained based on the Imagenet database and transfer learning. After the model is trained, the last fully connected classification layer of the pre-trained model is removed, and only the convolution layer and the pooling layer are retained for feature extraction. A fully connected layer containing two neurons is added, which is used to predict the prediction results of "responsive" and "unresponsive" to triple therapy. The significance of step 4.3.4 is: (1) Adaptive adjustment: By removing the last fully connected layer of the pre-trained model and adding a fully connected layer containing two neurons, the model can better adapt to the new task requirements. This is because the last layer of the pre-trained model is designed for the original task (image classification on Imagenet), while the newly added fully connected layer of the present invention is directly modeled for the new task (predicting "responsive" and "unresponsive"), so that the model can output specific prediction results related to the task. (2) Avoid overfitting: Directly using the pre-trained model on a new task may lead to overfitting, especially when the amount of data for the new task is much smaller than that of the pre-trained task. By adjusting the model structure, the risk of overfitting can be reduced. (3) Improved interpretability: Through the specific output layer design, the model's prediction results can be explained more clearly. That is, the outputs of two neurons can directly correspond to two different treatment responses, making the results easier to understand and explain.

[0115] Step 5: According to the prediction results of the pathological group category, the corresponding patients of the HE-stained sections are grouped into hepatocellular carcinoma pathological groups. Step 5 is mainly used to group liver cancer patients by comparing the predicted value output by the model with the preset threshold to distinguish the responding group and the non-responding group of triple therapy.

[0116] In a specific embodiment of the present invention, Step 5 includes the following detailed operations:

[0117] Step 5.1: Determine the patient's PRPS value based on the prediction results; the PRPS value is used to characterize the patient's expected treatment response to hepatocellular carcinoma treatment.

[0118] The PRPS (Post-treatment risk prediction score) value is used to characterize the patient's final prognostic risk prediction. The PRPS value is a continuous value ranging from 0 to 1. When the PRPS value is closer to 0, it means that the model predicts that the patient is more likely to respond to triple therapy; on the contrary, when the PRPS value is closer to 1, it means that the model predicts that the patient may not respond to triple therapy.

[0119] Step 5.2: Compare the patient's PRPS value with the preset PRPS threshold, and patients with values ​​above the PRPS threshold and those not above the PRPS threshold are divided into different groups.

[0120] For the convenience of clinical application, the present invention calculates a PRPS value for each patient through a multi-instance model, and uses this to stratify patients. According to ROC curve analysis, it is determined that the optimal threshold for this binary classification problem is about 0.532. This threshold reflects the point at which the best balance is achieved between the true positive rate and the false positive rate to optimize the sensitivity and specificity of the prediction. The PRPS value of each patient is compared with the preset threshold of 0.532 to determine its efficacy prediction group.

[0121] Step 5.3: Determine the patient's subsequent treatment plan based on the group into which the patient is assigned.

[0122] Specifically, if the PRPS value is lower than 0.532, the patient is classified as the "triple therapy response group", which means that triple therapy is expected to achieve positive therapeutic effects on the patient. The doctor can decide to implement triple therapy based on this prediction result. If the PRPS value is higher than or equal to 0.532, the patient is classified as the "triple therapy non-response group", which means that triple therapy is expected to have a poor effect on the patient. The doctor can consider choosing other treatment options based on this prediction result.

[0123] Below is a multi-instance learning model that was successfully established and verified based on case data of 194 hepatocellular carcinoma patients and corresponding HE slice images. The experimental results show that the model exhibits high prediction accuracy on the overall test set, internal test set, and external test set.

[0124] Specific evaluation indicators of model performance include: (1) ROC curve: shows the true positive rate (sensitivity) and false positive rate (1-specificity) of the model at the threshold set by the present invention. (2) Area under the ROC curve: measures the ability of the model to distinguish different efficacy prediction groups. The AUC value of a perfect classifier is 1, while the AUC value of a model without discrimination is close to 0.5. (3) Confusion matrix: provides the prediction accuracy of the model on different tumor response categories, including true positive rate, false positive rate, true negative rate and false negative rate. (4) Accuracy: indicates the percentage of correct predictions (including true positives and true negatives) of the model to all predictions. (5) Positive prediction rate: indicates the proportion of samples that are truly responsive among all samples predicted to be responsive. (6) Negative prediction rate: indicates the proportion of samples that are truly non-responsive among all samples predicted to be non-responsive. (7) Sensitivity: indicates the ability of the model to correctly identify responsive patients. (8) Specificity: indicates the ability of the model to correctly identify non-responsive patients. (9) F1 score is the harmonic mean of sensitivity and specificity, providing a single indicator of model accuracy.

[0125] The model's prediction results are obtained through Figure 4 , Figure 5 and Figure 6 A visual display was performed. Figure 4 The ROC curve and confusion matrix on the entire test set are shown; Figure 5 The ROC curve and confusion matrix on the internal test set are shown; Figure 6 The ROC curve and confusion matrix on the external test set are shown. Figure 4-Figure 6 The data can be obtained in Table 1 as follows:

[0126] Table 1

[0127]

[0128] Combined with Table 1, we can intuitively see the performance of the model on different test sets. These results not only prove the high prediction accuracy of the model, but also demonstrate its stability and generalization ability on different data sets. The model of the present invention provides a scientific and reliable tool for predicting the efficacy of triple therapy in patients with hepatocellular carcinoma, which helps guide clinical treatment decisions.

[0129] Figure 7: is a schematic diagram of the structure of the electronic device proposed in the second embodiment of the present invention. In this embodiment, the memory stores program instructions for implementing the hepatocellular carcinoma pathology group analysis method based on HE images in any of the above embodiments. The processor is used to execute the program instructions stored in the memory to perform hepatocellular carcinoma pathology group analysis based on HE images. Among them, the processor can also be called a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0130] The contents of the method in the first embodiment of the present invention are all applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0131] Figure 8 It is a schematic diagram of the structure of the computer-readable storage medium of the third embodiment of the present invention. The computer-readable storage medium of the fourth embodiment of the present invention stores program instructions that can implement the above-mentioned hepatocellular carcinoma pathology group analysis method based on HE images, wherein the program instructions can be stored in the above-mentioned storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0132] The contents of the method in the first embodiment of the present invention are all applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0133] This embodiment further provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above-mentioned related steps to implement the hepatocellular carcinoma pathological omics analysis method based on HE images provided in the above embodiment.

[0134] Those skilled in the art will appreciate that the modules in the devices in the embodiments of the present invention may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments of the present invention may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the corresponding claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the corresponding claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0135] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0136] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0137] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to method embodiments, the relevant parts can refer to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed in multiple places, such as nodes of a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying creative labor.

[0138] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0140] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0141] In the embodiments of the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present invention may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context in the specific embodiment.

[0142] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations of the present invention, and those of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention. Those skilled in the art will readily come to think of other embodiments of the present invention after considering the specification and practicing the present invention. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention. The specification and embodiments are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

Claims

1. A pathological omics analysis method for hepatocellular carcinoma based on HE images, characterized in that: The following steps are involved: Acquiring a HE-stained section, and scanning the HE-stained section to obtain a WSI-HE image; Preprocessing the WSI-HE image to obtain a WSI-HE characteristic image; Performing patch segmentation on the WSI-HE feature image to identify feature representations of the WSI-HE feature image; Performing multi-instance learning on the feature representation of the WSI-HE feature image to obtain a pathological group category prediction result of the HE-stained section; According to the pathological group category prediction results, the corresponding patients of the HE-stained sections are grouped into hepatocellular carcinoma pathological groups; The method of performing multi-instance learning on the feature representation of the WSI-HE feature image to obtain the pathological group category prediction result of the HE stained section is implemented by a multi-instance learning model, and specifically includes the following steps: Flattening the feature representation of the WSI-HE feature image into a one-dimensional vector; The feature representation of the one-dimensional vector is concatenated to obtain the instance feature matrix; Inputting the instance feature matrix into a multi-instance learning model to obtain a prediction result as a pathological group category prediction result of the HE-stained section; The multi-instance learning model is an EfficientNet-B5 model, which is obtained by the following steps: Design an EfficientNet-B5 model that includes convolutional layers, pooling layers, and fully connected classification layers, and synchronize the EfficientNet-B5 model to the cloud; Use the Imagenet database to pre-train the cloud-based EfficientNet-B5 model to obtain a trained cloud-based EfficientNet-B5 model; Migrate the trained cloud-based EfficientNet-B5 model parameters to the local EfficientNet-B5 model; The fully connected layer in the EfficientNet-B5 model is replaced by a preset fully connected layer; the preset fully connected layer includes two neurons, which are used to output a positive prediction result and a negative prediction result respectively; The step of grouping the corresponding patients of the HE-stained sections into hepatocellular carcinoma pathology groups according to the pathology group category prediction result specifically comprises the following steps: Determine the PRPS value of the patient according to the prediction result; the PRPS value is used to characterize the expected treatment response of the patient to hepatocellular carcinoma treatment; The PRPS values ​​of the patients were compared with the preset PRPS threshold, and patients with values ​​above the PRPS threshold and those not above the PRPS threshold were divided into different groups; The patient's subsequent treatment plan is determined based on the group into which the patient is assigned; a value above the PRPS threshold indicates that the patient has no response to triple therapy, and a value below the PRPS threshold indicates that the patient has responded to triple therapy;.

2. The method for pathological omics analysis of hepatocellular carcinoma based on HE images according to claim 1, characterized in that: The step of obtaining HE-stained sections specifically comprises the following steps: Patients with hepatocellular carcinoma were selected from the patient medical record database based on their historical treatment regimens; Collecting HE-stained sections of the hepatocellular carcinoma patients obtained by screening; The HE-stained sections were marked according to the patient's treatment response as documented in the historical treatment plan.

3. The method for pathological omics analysis of hepatocellular carcinoma based on HE images according to claim 2, characterized in that: The preprocessing of the WSI-HE image to obtain the WSI-HE characteristic image specifically includes the following steps: Converting the WSI-HE image into a preset scalable storage format; Screening the WSI-HE images according to preset image screening conditions, removing the WSI-HE images that do not meet the preset image screening conditions, and naming the WSI-HE images that meet the preset image screening conditions according to a unified naming rule; the preset image screening conditions include clarity, contrast, and focus of the WSI-HE images; The named WSI-HE images are stored uniformly as feature images.

4. The method for pathological omics analysis of hepatocellular carcinoma based on HE images according to claim 1, characterized in that: The performing patch segmentation on the WSI-HE characteristic image and identifying the characteristic representation of the WSI-HE characteristic image specifically comprises the following steps: Determine a ROI region in the WSI-HE feature image; Performing non-overlapping sampling on the ROI area in the WSI-HE feature image to obtain multiple patch blocks; The patch blocks are screened according to preset block screening conditions, and WSI-HE images that do not meet the preset block screening conditions are removed; the block screening conditions include stains, defects, and out-of-focus; The patch blocks that meet the block screening conditions are input into the feature extraction model, and the feature representation in the patch blocks is extracted by the feature extraction model.

5. The method for pathological omics analysis of hepatocellular carcinoma based on HE images according to claim 4, characterized in that: The feature extraction model is a ResNet50 model, which is obtained by the following steps: Design a ResNet50 model that includes convolutional layers, pooling layers, and fully connected classification layers, and synchronize the ResNet50 model to the cloud; Use the Imagenet database to pre-train the cloud-based ResNet50 model to obtain a trained cloud-based ResNet50 model; Migrate the trained cloud-based ResNet50 model parameters to the local ResNet50 model; The fully connected classification layer in the local ResNet50 model is removed, and the convolutional layer and pooling layer are retained for feature extraction.

6. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement a pathological omics analysis method for hepatocellular carcinoma based on HE images according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement a hepatocellular carcinoma pathological omics analysis method based on HE images according to any one of claims 1 to 5.

Images