Liver cancer analysis method, system and equipment based on multi-modal deep learning and medium

Abstract

The invention relates to the technical field of liver cancer analysis, in particular to a liver cancer analysis method, system and equipment based on multi-modal deep learning and a medium. The method comprises the following steps: acquiring pathological data, transcriptome data, methylation data and clinical data of a target patient, and respectively preprocessing to obtain target pathological data, target transcriptome data, target methylation data and target clinical data; respectively carrying out feature extraction to obtain pathological features, transcriptome features, methylation features and clinical features; performing feature fusion on the pathological features, the transcriptome features, the methylation features and the clinical features to obtain liver cancer analysis features; extracting lipid metabolism related marker characteristics of the target patient, and performing liver cancer comprehensive analysis on the target patient according to the lipid metabolism related marker characteristics and the liver cancer analysis characteristics. According to the application, the tumor characteristics of the target patient can be comprehensively reflected, the accuracy of auxiliary diagnosis is improved, and accurate prognosis prediction is provided for the target patient.

Description

Background Art

[0002] With the advancement of medical imaging technology and genomics, the diagnosis and prognosis prediction of liver tumors are increasingly dependent on multi-dimensional data analysis. Although medical images such as CT and MRI provide important information on tumor morphology, it is still challenging to distinguish between benign and malignant tumors and predict prognosis only through imaging data. Liver biopsy is still used as the gold standard for the diagnosis of benign and malignant liver tumors in clinical practice, but as an invasive examination, its clinical application has certain limitations. Summary of the invention

[0003] The present application provides a liver cancer analysis method, system, device and medium based on multimodal deep learning, which can fuse data and features from different modalities into a unified feature representation, and ultimately perform classification or prognosis prediction, thereby realizing the synergy of multimodal features.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a liver cancer analysis method based on multimodal deep learning is provided, the method comprising:

[0006] Obtain pathological data, transcriptome data, methylation data, and clinical data of target patients;

[0007] Preprocessing the pathological data, the transcriptome data, the methylation data, and the clinical data respectively to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data;

[0008] Performing feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data;

[0009] Performing feature fusion on the pathological features, the transcriptome features, the methylation features, and the clinical features to obtain liver cancer analysis features of the target patient;

[0010] The lipid metabolism-related marker features of the target patient are extracted based on the methylation data, and a comprehensive liver cancer analysis is performed on the target patient based on the lipid metabolism-related marker features and the liver cancer analysis features.

[0011] In one embodiment of the present application, based on the above scheme, the pathological data is a full-slice image, and the target pathological data is obtained by the following steps:

[0012] Performing full-slice image segmentation on the pathological data to obtain a plurality of image blocks;

[0013] Screening out target image blocks whose cell tissue area ratio is higher than a preset threshold value from each of the image blocks;

[0014] Adjusting the image parameters of each of the target image blocks so as to adjust the image parameters to a preset reference value;

[0015] After the image parameters of each of the target image blocks are adjusted to the reference value, generating the target pathology data according to each of the target image blocks;

[0016] The image parameters include brightness, contrast, gamma correction and saturation.

[0017] In one embodiment of the present application, based on the above scheme, the pathological characteristics are obtained by the following steps:

[0018] Perform feature extraction on the target pathological data through a ResNet152 model to obtain the pathological features of the target patient;

[0019] The model weights of the ResNet152 model are obtained by the following steps:

[0020] A cross entropy loss function is used to perform a probability difference distribution of model prediction on the target pathology data, and a back propagation algorithm is used to calculate the gradient of the loss of the probability difference distribution relative to the ResNet152 model parameters;

[0021] The model weights are obtained by using an SGD optimizer according to the gradient calculation.

[0022] In one embodiment of the present application, based on the above scheme, the transcriptome features and the methylation features are obtained by the following steps:

[0023] Using a unified manifold approximation and projection method to perform feature mapping on the target transcriptome data and the target methylation data in a high-dimensional space, respectively, to obtain transcriptome mapping data and methylation mapping data in a low-dimensional space;

[0024] determining the transcriptome signature based on the transcriptome mapping data, and determining the methylation signature based on the methylation mapping data;

[0025] The unified manifold approximation and projection method is specifically the following steps:

[0026] S1, calculates the neighboring probability of data points in high-dimensional space;

[0027] Among them, S1 can calculate the neighboring probability of any two data points in the target transcriptome data or the target methylation data by the following formula:

[0028]

[0029] In the formula, dist(x i ,x j ) represents the Euclidean distance between data point i and data point j; ρ i is the nearest neighbor distance of data point i; σ i is the preset scale parameter, p ij is the proximity probability of any two data points.

[0030] S2, calculates the similarity of data points in low-dimensional space;

[0031] Wherein, S2 can calculate the similarity of any two data points in the transcriptome mapping data or the methylation mapping data by the following formula:

[0032]

[0033] In the formula, y i and j are the positions of data points i and j in the low-dimensional space respectively; a and b are hyperparameters, q ij is the similarity between any two data points;

[0034] S3, the loss function of the mapping between high-dimensional space and low-dimensional space is determined by the following formula:

[0035]

[0036] Wherein, L is the loss function.

[0037] In one embodiment of the present application, based on the above scheme, the liver cancer analysis characteristics of the target patient are obtained by the following steps:

[0038] Inputting the pathological features, the transcriptome features, the methylation features, and the clinical features into a pre-set multimodal fusion model to obtain the liver cancer analysis features output by the multimodal fusion model;

[0039] The mathematical expression of the multimodal fusion model is as follows:

[0040]

[0041] In the formula, is the liver cancer analysis feature output by the multimodal fusion model, W class is a weight matrix corresponding to the target pathological data, the target transcriptome data, the target methylation data and the target clinical data. LayerNorm represents layer normalization, f1, f2, f3 and f4 are the pathological features, the transcriptome features, the methylation features and the clinical features respectively.

[0042] In one embodiment of the present application, based on the above scheme, the comprehensive liver cancer analysis of the target patient according to the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics includes:

[0043] Liver cancer is diagnosed and prognosis is predicted for the target patient based on the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics.

[0044] According to one aspect of an embodiment of the present application, a liver cancer analysis system based on multimodal deep learning is provided, the system comprising:

[0045] An input unit, used to obtain pathological data, transcriptome data, methylation data and clinical data of target patients;

[0046] a preprocessing unit, configured to preprocess the pathological data, the transcriptome data, the methylation data, and the clinical data, respectively, to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data;

[0047] a hierarchical processing unit, for performing feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data;

[0048] A fusion processing unit is used to fuse the pathological features, the transcriptome features, the methylation features and the clinical features to obtain the liver cancer analysis features of the target patient.

[0049] An analysis unit extracts lipid metabolism-related marker features of the target patient based on the methylation data, and performs a comprehensive liver cancer analysis on the target patient according to the lipid metabolism-related marker features and the liver cancer analysis features.

[0050] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program includes executable instructions. When the executable instructions are executed by a processor, the method described in the above embodiment is implemented.

[0051] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions of the processors, wherein when the executable instructions are executed by the one or more processors, the one or more processors implement the methods described in the above embodiments.

[0052] Beneficial effects of the present application: Compared with existing liver cancer diagnosis technologies, traditional methods often rely only on imaging data such as CT or MRI, which can easily lead to missed diagnosis or misdiagnosis. The present application integrates multiple information sources such as pathological images, transcriptomes, methylation and clinical data, which significantly improves the accuracy of diagnosis. The present application achieves a more comprehensive understanding of tumors through feature fusion of multimodal data.

[0053] The present application obtains the liver cancer analysis characteristics of the target patient by fusing the pathological characteristics, the transcriptome characteristics, the methylation characteristics and the clinical characteristics, and extracts the lipid metabolism-related marker characteristics of the target patient based on the methylation data to conduct a comprehensive liver cancer analysis on the patient, which can more comprehensively reflect the tumor characteristics of the target patient, improve the accuracy of auxiliary diagnosis, provide clinicians with more comprehensive and in-depth pathological information, and provide accurate prognosis predictions for target patients.

[0054] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0056] Figure 1 It is a flowchart of a liver cancer analysis method based on multimodal deep learning according to an embodiment of the present application;

[0057] Figure 2 This is a logical principle diagram of a liver cancer analysis system based on multimodal deep learning according to an embodiment of the present application;

[0058] Figure 3is a block diagram of a liver cancer analysis system based on multimodal deep learning according to an embodiment of the present application;

[0059] Figure 4 It is a schematic diagram of the system structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0061] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the present application.

[0062] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or micro-control node devices.

[0063] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0064] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0065] The following is a detailed description of the technical background of the embodiments of the present application and the direction of the lipid metabolism-related markers studied in the embodiments of the present application:

[0066] Liver cancer is a global health challenge, and its incidence is growing worldwide. It is estimated that by 2025, one million people will be affected by liver cancer each year, and hepatocellular carcinoma (HCC) is the most common form of liver cancer, accounting for 90% of cases. Liver cancer is one of the most common malignant tumors in my country. More than 50% of new cases of liver cancer worldwide occur in my country each year, and its incidence and mortality rates remain high. Hepatitis B virus (HBV), hepatitis C virus (HCV), cirrhosis, metabolic syndrome, etc. are generally considered to be predisposing factors for hepatocellular carcinoma. Therefore, accurate diagnosis of liver cancer is particularly important to improve the 5-year survival rate of liver cancer patients. Unfortunately, liver cancer is not sensitive to chemotherapy, so surgical treatment is the first choice for liver cancer patients. However, since most patients with hepatocellular carcinoma are already in the advanced stage when diagnosed, they can only receive systemic treatment. In recent years, systemic treatment for liver cancer patients has evolved from single-drug targeted therapy (sorafenib and lenvatinib) to checkpoint inhibitors plus targeted therapy (PD-L1 monoclonal antibody, atezolizumab combined with VEGF antagonist bevacizumab). Although significant progress has been made in the clinical treatment of liver cancer, only a small number of patients can obtain lasting clinical benefits. In view of the current medical situation in my country, the development of a new diagnostic system is of great significance to the growing demand for liver cancer diagnosis.

[0067] The occurrence and development of liver cancer is a multifactorial, multi-stage process. The liver is the central organ of lipid metabolism, and the occurrence and development of its tumors may depend on energy metabolism processes including lipogenesis and conversion. Lipid metabolism refers to the processes of synthesis, decomposition, transport and storage of lipids in organisms. Abnormal lipid metabolism is usually manifested as abnormal increase or decrease in blood lipid levels, including imbalances in total cholesterol, triglycerides, low-density lipoprotein (LDL) and high-density lipoprotein (HDL). More and more epidemiological and clinical studies have shown that abnormal lipid metabolism is associated with an increased risk of liver cancer. For example, a high-cholesterol diet may be a risk factor for the development of liver cancer, while hypocholesterolemia may increase the risk of liver cancer. Abnormal lipid metabolism is not only related to the occurrence of liver cancer, but may also affect the progression and metastasis of liver cancer.

[0068] Studies have shown that tumor cells can adapt to their rapid proliferation needs by changing lipid metabolism, including the synthesis, oxidation and esterification of fatty acids. In addition, changes in lipid metabolism in the tumor microenvironment may also promote tumor angiogenesis, immune escape and metastasis. Although studies have revealed the connection between abnormal lipid metabolism and liver cancer, the specific mechanism is still not completely clear. In-depth research on the mechanism of action of abnormal lipid metabolism in liver cancer is of great significance for early diagnosis, risk assessment and the formulation of treatment strategies. In addition, understanding how abnormal lipid metabolism affects the biological behavior and clinical treatment of liver cancer, and exploring the pathological manifestations and molecular markers related to lipid metabolism in various subtypes of liver cancer will help develop new prevention and clinical treatment methods.

[0069] In recent years, deep learning technology has made significant progress in the field of medical image analysis. However, the field of pathological image analysis still faces challenges, mainly due to the complex tissue and texture features of pathological images, as well as the subtle differences between different pathological types. Although convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been initially applied in CT image feature extraction and tumor prognosis prediction, these methods mainly focus on single-modality data and do not comprehensively observe and combine the analysis of multi-type data of human tumors. Therefore, the scalability, reliability, and accuracy of tumor recognition and prognosis need to be further improved. Therefore, multimodal deep learning models that can effectively integrate information from different modalities can better simulate human perception and cognitive processes, thereby achieving a more comprehensive and in-depth understanding. Therefore, multimodal deep learning models have become a key cross-integration research field in medicine. In addition to medical imaging and genomic data, pathological and transcriptional data can also reveal information about the internal microstructure and gene expression of tumors to further improve diagnosis and prognosis. However, deep learning models based on clinical pathological images combined with multimodal deep learning models have not found solutions for actual clinical applications. As the cost of various molecular analyses decreases and deep learning techniques advance, this approach may become more cost-effective in the future and facilitate the clinical application of related technologies.

[0070] This application will focus on integrating liver cancer data of different modalities, combining the multi-factorial effects of lipid metabolism on liver cancer, and ultimately using multimodal deep learning technology to establish a liver cancer diagnosis classification, prognosis prediction, and lipid metabolism-related marker expression prediction system based on pathological images, providing a practical diagnostic model for clinical use. The system is expected to be integrated into the clinical process to reduce the workload of pathologists and clinicians, while promoting the learning and training of inexperienced pathologists to support the diagnosis and treatment of liver cancer.

[0071] With the advancement of deep learning technology in the medical field, artificial intelligence (AI) technologies, including radiomics and deep learning methods, have emerged as a unique opportunity to improve the scope of clinical care for hepatocellular carcinoma by predicting biological features and prognosis in the field of medical imaging.

[0072] In the field of radiomics, the diagnostic technology of liver cancer has become increasingly mature. At present, diagnostic methods based on medical imaging technologies such as CT and MRI have been widely used. Tietz E et al. constructed a multivariate logistic regression model based on CT images to predict new liver cancer in high-risk patients with cirrhosis and showed good efficacy, further confirming the important role of radiomics in the diagnosis of liver cancer. However, although radiomics can provide objective and quantitative diagnostic information, due to the existence of some benign solid lesions, misdiagnosis or missed diagnosis may occur due to the limitations of imaging examinations. In order to overcome these limitations, researchers actively explored the integration of features extracted by deep learning into radiomics models. Gan Fuwen et al. achieved preoperative prediction of liver cancer by combining imaging features extracted by high-throughput T2-weighted imaging with parameter fine-tuning of the EfficientNet-B7 model and deep labels constructed by LightGBM. This study not only enriched the connotation of radiomics, but also provided new ideas for the precise diagnosis of liver cancer.

[0073] Histopathological image examination is the gold standard for oncological diagnosis. Its images show the cells and tissue structure of tissue sections, which can be used to determine cell types, tissue structures, the number and morphology of abnormal cells, and evaluate the specific situation of liver cancer. In recent years, with the rapid development of artificial intelligence technology, it has become possible to use digital whole slide images (WSI) for histopathological classification, which has greatly improved the diagnostic efficiency and accuracy of liver cancer. Shao Runhua and others used convolutional neural networks to conduct in-depth research on pathological images of liver cancer, including diagnostic algorithms in three aspects: liver tumor detection, image segmentation, and preoperative prediction. On May 20, 2024, Nakatsuka T's team proposed a deep learning model that predicts the development of liver cancer based on HE-stained HCC pathological images. It is reported that the model has shown remarkable performance in prediction, with an accuracy rate of up to 81.0% and an AUC value of 0.80. However, although this achievement is already considerable, there is still room for improvement. To this end, we consider fusing multiple data modalities to construct a multimodal deep learning model in order to further improve the accuracy and reliability of liver cancer pathological diagnosis. This is not only the main direction of this study, but also an important development trend in the field of liver cancer diagnosis in the future.

[0074] At present, the application of artificial intelligence in the analysis of tumor pathological tissue markers has achieved remarkable results. Saha et al. proposed a Her2Net model based on deep learning, which can automatically evaluate HER-2 staining of breast cancer. This model not only achieved high accuracy, but also had a false positive rate of only 6.84%. This achievement fully demonstrated the great potential of AI in tumor pathological diagnosis. AI-assisted tumor pathological diagnosis has the advantages of automation, high efficiency and repeatability, especially in the quantitative interpretation of tissue pathological markers, which helps to improve the objectivity and accuracy of tumor companion diagnosis. This also provides a new idea for this study, that is, to conduct a deeper analysis and diagnosis of tumors by integrating multi-model and multi-omics information.

[0075] Serum markers are the most convenient and non-invasive diagnostic methods in clinical applications. Among the many tumor markers, alpha-fetoprotein (AFP), alpha-fetoprotein isomers (AFP-L3%), and abnormal prothrombin (DCP) are called the "three liver cancer items". At present, the liver cancer triple test is a commonly used serum marker combination for early diagnosis of liver cancer in the world. Studies have shown that the combined detection of DCP, AFP and AFP-L3% can increase the detection rate of liver cancer to 85.9%-94.57%, which can basically meet the needs of liver cancer diagnosis. However, due to methodological limitations, AFP-L3% detection is still a difficult point in the three liver cancer tests. The main problem is that the operation process is cumbersome and time-consuming, and the test results need to be multiplied by the dilution multiple, which requires a lot of manpower, which not only increases the cost of detection, but also limits its wide application in clinical practice. To this end, Moldogazieva NT et al. proposed that high-throughput proteomics analysis technology could be combined with artificial intelligence algorithms and predictive models to further explore new HCC candidate biomarkers, improve the sensitivity and specificity of HCC detection, and provide strong support for predicting treatment responses.

[0076] With the advancement of medical imaging technology and genomics, the diagnosis and prognosis prediction of liver tumors are increasingly dependent on multi-dimensional data analysis. Although medical images such as CT and MRI provide important information on tumor morphology, it is still challenging to distinguish between benign and malignant tumors and predict prognosis only through imaging data. Liver biopsy is still used as the gold standard for the diagnosis of benign and malignant liver tumors in clinical practice, but as an invasive examination, its clinical application has certain limitations. In recent years, deep learning has made significant breakthroughs in the field of medical image analysis, but in the field of pathological images, the classification research on benign and malignant liver tumors is still insufficient, mainly due to the complex tissue structure and texture characteristics of pathological images, as well as the subtle differences between different pathological types. Therefore, combining data from multiple modalities for comprehensive analysis has become a key research direction. In addition to medical imaging omics, omics data such as pathological omics and transcriptomics can also reveal the microstructure and gene expression information inside the tumor, improving the diagnosis and prognosis prediction effect.

[0077] At present, deep learning has been initially applied in the diagnosis and prognosis prediction of other tumors, such as convolutional neural networks (CNN) for CT image feature extraction and classification, recurrent neural networks (RNN) for time series medical data modeling to predict tumor prognosis, and we will also use deep learning technology to study liver tumors. However, the current tumor deep learning prediction scheme mainly focuses on single-modality data and fails to fully utilize the correlation and complementarity between multi-modal data. Therefore, there is still room for improvement in the accurate identification of benign and malignant tumors and prognosis prediction.

[0078] The purpose of the present application is to establish a liver cancer diagnosis classification, prognosis prediction and lipid metabolism-related marker expression prediction system based on pathological images, including: an input unit for obtaining the patient's pathological data, transcriptome data, methylation data and clinical data; a preprocessing unit for preprocessing the pathological data, transcriptome data, methylation data and clinical data respectively; a hierarchical processing unit for inputting the various preprocessed data into a multimodal prediction model (multimodal fusion model) that integrates a pathological model, a transcriptome model, a methylation model and a clinical model to predict the pathological results (pathological features), transcriptome results (transcriptome features), methylation results (methylation features) and clinical results (clinical features); an analysis unit for obtaining the final tumor classification, prognosis prediction and lipid metabolism marker expression prediction results through pathological results (pathological features), transcriptome results (transcriptome features), methylation results (methylation features) and clinical results (clinical features).

[0079] This application will use multimodal deep learning technology to effectively integrate various types of tumor data, more comprehensively reflect the characteristics of tumors, more effectively assist clinical diagnosis, and provide doctors with more comprehensive and in-depth pathological information, which can be widely used in clinical work. In later research, by appropriately adjusting the model structure and data processing flow, it can also be applied to other medical fields, such as disease diagnosis, drug development, gene editing and other different medical tasks and scenarios.

[0080] The following is a detailed explanation of the terms used in the embodiments of the present application:

[0081] 1. Multimodal fusion model: A deep learning model that combines different types of data (such as pathological data, transcriptome data, methylation data, and clinical data). By fusing data from multiple modalities, it can fully simulate the biological characteristics of liver cancer and provide more accurate information for clinical diagnosis and prognosis.

[0082] 2. Pathological data (whole slide image): In medical diagnosis, high-resolution images obtained through pathological sections can reflect the microscopic characteristics of tissue and cell structures. Whole Slide Image (WSI) is a standard format for pathological image analysis and contains complete information about tumor tissue.

[0083] 3. Transcriptome data: Gene expression data obtained through RNA sequencing technology, which is used to analyze the transcription level of genes in cells or tissues, revealing the biological characteristics of liver cancer and its response to treatment.

[0084] 4. Methylation data: DNA methylation is a gene regulation method that affects gene expression by changing the methylation status on DNA. Methylation data is usually obtained through methylation microarray technology, which is used to analyze the genomic modification information of tumors and reflect the epigenetic characteristics in the development of liver cancer.

[0085] 5. Clinical data: Patient information recorded during liver cancer diagnosis and treatment, usually including the patient's age, gender, medical history, cirrhosis status, etc., can provide a personalized basis for tumor risk assessment and treatment plans.

[0086] 6. Lipid metabolism: The synthesis, decomposition and transport process of lipids in organisms is closely related to the occurrence of liver cancer.

[0087] 7. Feature weighting: A technique that calculates the weights of different modal data through the Softmax algorithm and applies the weights to the features to ensure that the importance of each modality is reasonably considered during multimodal data fusion.

[0088] 8. Layer Normalization: A standardization technique used to adjust the distribution of features in the neural network layer to prevent the feature differences between different modalities from being too large, thereby stabilizing the model training process.

[0089] 9. Data cleaning: During the data processing process, the noise, missing values ​​and outliers in the data set are processed to improve the data quality and the training effect of the model.

[0090] 10. Data augmentation: Generate additional training data through random cropping, flipping, color jittering, etc., so as to enhance the generalization ability of the model and reduce overfitting.

[0091] 11. Convolutional Neural Network (CNN): A deep learning model for image processing that extracts spatial features of images through convolution operations and is widely used in medical image analysis.

[0092] 12. Cross-Entropy Loss Function: A commonly used loss function that measures the difference between the predicted probability distribution of the model output and the true label to help optimize the classification performance of the model.

[0093] 13. Pretraining: The process of training a model on a large-scale dataset so that the model can learn common features and thus be more effectively trained for specific tasks in application scenarios with small sample data.

[0094] 14. Hyperparameter optimization: The process of adjusting parameters such as learning rate and regularization coefficient during model training to achieve the best performance of the model.

[0095] 15.UMAP (Uniform Manifold Approximation and Projection): A nonlinear dimensionality reduction technique that optimizes data layout and maintains the topological structure of high-dimensional data, making it easier to observe data distribution in low-dimensional space. It is used to analyze high-dimensional molecular data such as transcriptome and methylation data.

[0096] 16. Fully Connected Layer: A layer in a neural network that connects each input node to each output node. It is usually used as the last layer for classification tasks to complete the classification output of the model.

[0097] 17.Softmax: A mathematical function used for multi-classification tasks, often used in the output layer of a neural network. The Softmax function converts a set of real numbers into a probability distribution, where the probability value of each category is between 0 and 1, and the sum of the probabilities of all categories is 1.

[0098] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0099] According to one aspect of the present application, a liver cancer analysis method based on multimodal deep learning is provided. Figure 1 The flowchart of the liver cancer analysis method based on multimodal deep learning according to the embodiment of the present application is shown. The liver cancer analysis method based on multimodal deep learning includes at least steps 110 to 150, which are described in detail as follows:

[0100] In step 110, pathological data, transcriptome data, methylation data, and clinical data of the target patient are obtained.

[0101] Specifically, the multimodal data mentioned in the embodiments of the present application include at least pathological data, transcriptome data, methylation data and clinical data. The application integrates multiple information sources such as pathological images, transcriptomes, methylation and clinical data, which significantly improves the accuracy of diagnosis. The present application achieves a more comprehensive understanding of tumors through feature fusion of multimodal data.

[0102] In step 120, the pathological data, the transcriptome data, the methylation data and the clinical data are preprocessed respectively to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data and target clinical data corresponding to the clinical data.

[0103] In one embodiment of the present application, the pathological data is a full-slice image, and the target pathological data is obtained by the following steps:

[0104] Performing full-slice image segmentation on the pathological data to obtain a plurality of image blocks;

[0105] Screening out target image blocks whose cell tissue area ratio is higher than a preset threshold value from each of the image blocks;

[0106] Adjusting the image parameters of each of the target image blocks so as to adjust the image parameters to a preset reference value;

[0107] After the image parameters of each of the target image blocks are adjusted to the reference value, generating the target pathology data according to each of the target image blocks;

[0108] The image parameters include brightness, contrast, gamma correction and saturation.

[0109] This application provides a pathological model, a transcriptome model, a methylation model and a clinical model, and then a multimodal prediction model obtained by fusing the pathological model, the transcriptome model, the methylation model and the clinical model. The following is a detailed explanation of the pathological model, the transcriptome model, the methylation model and the clinical model:

[0110] Specifically, pathological data, including whole slide images (WSI), plays a vital role in disease diagnosis, research and treatment. WSI has great requirements for computing resources for deep learning due to its huge amount of data and high resolution. In order to make more effective use of these images, we will cut the original image and decompose it into image blocks (patches) with lower pixel values, which can reduce the complexity of our data processing. After the initial cutting, the patches are filtered and used for data enhancement. By using various image processing operations, the patches are finally randomly cropped into image blocks with lower pixel values. Then the areas with low tissue area ratio are further eliminated, and the elimination step is operated by a skilled pathologist to remove the low-quality areas caused by HE staining technology factors. The embodiment of the present application can enhance the generalization of the pathological model by inputting the pathological data into the pathological model, reduce its overfitting to the training data, and improve the adaptability and robustness of the pathological model to the changes in the real image.

[0111] For pathological data, relevant preprocessing of the image data is required. By individually adjusting specific parameters of each image block, such as brightness, contrast, gamma correction and saturation, the brightness, contrast, gamma correction and saturation distribution of the image are adjusted to be close to the preset baseline value, that is, the pre-set average value. The next stage of the processing will involve checking the preprocessing method of the input image to match the input requirements of the deep learning model (multimodal fusion model). In particular, each channel of the image needs to be normalized.

[0112] The following is a detailed description of the preprocessing of the transcriptome data, the methylation data and the clinical data:

[0113] The transcriptome data was cleaned and preprocessed. At the same time, the methylation data was cleaned, organized, and quality controlled on the same platform. During the model training process, we will standardize it to eliminate the influence of sample differences and gene length and convert it into a format suitable for model training.

[0114] Clinical data are statistical indicators presented in the form of text descriptions or charts to evaluate the possible outcomes of disease development, as well as the effectiveness of patient treatment and survival prospects. They usually include survival time, disease recurrence, recovery status, etc. They can assist doctors in developing more personalized treatment plans, help to gain a deeper understanding of the natural course of the disease, influencing factors, etc., and provide a basis for new treatment methods and drug development. In addition, clinical prognosis data also include patient conditions such as age, gender, and race, which will also affect the patient's tumor prediction to a certain extent. In actual applications, clinical prognosis data can be either a doctor's text description of the patient's prognosis or related diagnosis and treatment data recorded in a tabular form. Regarding clinical prognosis data preprocessing, we will perform missing value processing and outlier processing to ensure the integrity and accuracy of the data.

[0115] In step 130, feature extraction is performed on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data.

[0116] In one embodiment of the present application, the pathological features are obtained by the following steps:

[0117] Perform feature extraction on the target pathological data through a ResNet152 model to obtain the pathological features of the target patient;

[0118] The model weights of the ResNet152 model are obtained by the following steps:

[0119] A cross entropy loss function is used to perform a probability difference distribution of model prediction on the target pathology data, and a back propagation algorithm is used to calculate the gradient of the loss of the probability difference distribution relative to the ResNet152 model parameters;

[0120] The model weights are obtained by using an SGD optimizer according to the gradient calculation.

[0121] Specifically, regarding the construction of the pathological model, deep learning network models such as deep convolutional neural network (CNN), recurrent neural network (RNN) or graph neural network (GNN) can be used as the basic image model to adapt to different data structures and features. This application will select the ResNet152 model as the basic model of the pathological model. As a convolutional neural network model, ResNet152 can effectively extract key features in the image through pre-training on large-scale image data sets. These features have important reference value for tumor diagnosis.

[0122] The construction and training of the image model include: using the pre-trained ResNet152 model as a feature extractor for specific pathological image classification tasks. To ensure that the model output matches the classification requirements, we replace the last fully connected layer of the model with a new fully connected layer with n output nodes based on actual considerations to adapt to specific pathological image classification tasks.

[0123] During the training process, the parameters of the model were fine-tuned for the tumor diagnosis task to improve its adaptability to the task and output tumor labels and their corresponding probabilities. The cross-entropy loss function was used to measure the difference between the probability distribution predicted by the model and the true label, and the gradient of the loss relative to the model parameters was calculated by the back-propagation algorithm. The SGD optimizer was used to update the model weights according to the calculated gradient, where the learning rate was initially set to 0.001 and decayed by 0.1 after every a epoch. During the training of b epochs, the learning rate was continuously adjusted to optimize the model performance. An epoch represents the process in which the entire training dataset is traversed and learned by the model once. An epoch is the process in which the algorithm completes a full forward pass and backward pass for all training samples.

[0124] In one embodiment of the present application, the transcriptome features and the methylation features are obtained by the following steps:

[0125] Using a unified manifold approximation and projection method to perform feature mapping on the target transcriptome data and the target methylation data in a high-dimensional space, respectively, to obtain transcriptome mapping data and methylation mapping data in a low-dimensional space;

[0126] determining the transcriptome signature based on the transcriptome mapping data, and determining the methylation signature based on the methylation mapping data;

[0127] The unified manifold approximation and projection method is specifically the following steps:

[0128] S1, calculates the neighboring probability of data points in high-dimensional space;

[0129] Among them, S1 can calculate the neighboring probability of any two data points in the target transcriptome data or the target methylation data by the following formula:

[0130]

[0131] In the formula, dist(x i ,x j ) represents the Euclidean distance between data point i and data point j; ρ iis the nearest neighbor distance of data point i; σ i is the preset scale parameter, p ij is the proximity probability of any two data points.

[0132] S2, calculates the similarity of data points in low-dimensional space;

[0133] Wherein, S2 can calculate the similarity of any two data points in the transcriptome mapping data or the methylation mapping data by the following formula:

[0134]

[0135] In the formula, y i and j are the positions of data points i and j in the low-dimensional space respectively; a and b are hyperparameters, q ij is the similarity between any two data points;

[0136] S3, the loss function of the mapping between high-dimensional space and low-dimensional space is determined by the following formula:

[0137]

[0138] Wherein, L is the loss function.

[0139] In this application, a comprehensive method is used to analyze transcriptome data and methylation microarray data (methylation data) to monitor tumor progression and evaluate treatment efficacy. By comparing RNA transcriptome data of tumor tissue and normal tissue, biomarkers with diagnostic value can be screened out.

[0140] To ensure the accuracy and consistency of the data, the transcriptome data was fully cleaned and preprocessed. At the same time, the methylation microarray data was cleaned, organized, and quality controlled using the same platform. During the model training process, we will standardize it to eliminate the influence of sample differences and gene length and convert it into a format suitable for model training.

[0141] After data preprocessing, we used statistical methods and bioinformatics analysis, combined with the characteristics of lipid metabolism-related genes, to identify key genes for liver tumor classification. Changes in the expression levels of these genes may be closely related to specific biological processes or signaling pathways in tumor development, which is crucial to improving model prediction accuracy and model interpretability. We extracted these genes and their copy numbers and included them in a new list for subsequent training.

[0142] Before model building, the goal is to evaluate the independent impact of a single data pattern on classification performance. To this end, we chose UMAP (Uniform Manifold Approximation and Projection), a recently developed nonlinear dimensionality reduction technique that uses local manifold approximation and fuzzy simplicial set representation (local fuzzy simplicial set representation) to construct the topological structure of high-dimensional data. The UMAP algorithm aims to maintain the topological properties of the dataset while reducing the data dimension, minimizing the cross entropy between high-dimensional and low-dimensional topological representations by optimizing the data layout. We used the UMAP package to reduce the dimensionality of transcriptome data and methylation microarray data, and then mapped the data to a low-dimensional space to observe their distribution, thereby gaining a more comprehensive understanding of the molecular characteristics and treatment response of tumors. Its main steps and formulas are as follows:

[0143] ① Calculation of neighboring probability in high-dimensional space

[0144] For any two data points i and j, UMAP first calculates the probability of their neighbor relationship in high-dimensional space. This probability p ij The calculation method is:

[0145]

[0146] In the formula, dist(x i ,x j ) represents the Euclidean distance between data point i and data point j; ρ i

[0147] is the nearest neighbor distance of data point i; σ i is the preset scale parameter, p ij

[0148] is the proximity probability of any two data points.

[0149] ②Similarity calculation in low-dimensional space

[0150] In low-dimensional space, UMAP uses a formula similar to t-SNE to calculate the similarity q between two data points i and j. ij :

[0151]

[0152] In the formula, y i and j

[0153] are the positions of data points i and j in the low-dimensional space respectively; a and b are hyperparameters, q ij

[0154] is the similarity between any two data points.

[0155] ③Loss function

[0156] The UMAP loss function optimizes the low-dimensional representation by minimizing the difference in similarity between high-dimensional and low-dimensional spaces. The goal is to minimize the cross entropy loss:

[0157]

[0158] This loss function ensures that neighboring points in high-dimensional space also remain nearby in low-dimensional space, thereby preserving the local structure and topology of the data.

[0159] Both transcriptomics models and methylation models will consist of multiple fully connected layers to capture the patterns in gene expression data and the mathematical relationships (non-linear relationships) between variables.

[0160] In one embodiment of the present application, regarding the clinical data model

[0161] Clinical data are statistical indicators presented in the form of text descriptions or charts to evaluate the possible outcomes of disease development, as well as the effectiveness of patient treatment and survival prospects. They usually include survival time, disease recurrence, recovery status, etc. They can assist doctors in developing more personalized treatment plans, help to gain a deeper understanding of the natural course of the disease, influencing factors, etc., and provide a basis for new treatment methods and drug development. In addition, clinical prognostic data also include patient conditions such as age, gender, and race, which will also affect the patient's tumor prediction to a certain extent. In actual applications, clinical prognostic data can be either a doctor's text description of the patient's prognosis or related diagnosis and treatment data recorded in a tabular form.

[0162] Regarding clinical prognosis data preprocessing, missing value processing and outlier processing are performed to ensure the integrity and accuracy of the data. Key information in clinical prognosis data is extracted. Key information has a greater impact on tumor prediction and therefore needs to be processed separately.

[0163] Clinical input data contains both numerical and categorical features. In the clinical model, an embedding layer is used to concatenate categories to form a single input vector. Next, feature extraction and dimensionality reduction are performed on the input vector to introduce nonlinearity and reduce overfitting. The clinical model also exists in the form of a deep learning model that takes clinical input data as input and outputs tumor labels and their corresponding probabilities. During the model construction process, the network structure and parameters are also adjusted to improve the performance and robustness of the model.

[0164] In step 140, the pathological features, the transcriptome features, the methylation features, and the clinical features are fused to obtain liver cancer analysis features of the target patient.

[0165] In one embodiment of the present application, the liver cancer analysis characteristics of the target patient are obtained by the following steps:

[0166] Inputting the pathological features, the transcriptome features, the methylation features, and the clinical features into a pre-set multimodal fusion model to obtain the liver cancer analysis features output by the multimodal fusion model;

[0167] The mathematical expression of the multimodal fusion model is as follows:

[0168]

[0169] In the formula, is the liver cancer analysis feature output by the multimodal fusion model, W class is a weight matrix corresponding to the target pathological data, the target transcriptome data, the target methylation data and the target clinical data. LayerNorm represents layer normalization, f1, f2, f3 and f4 are the pathological features, the transcriptome features, the methylation features and the clinical features respectively.

[0170] Specifically, in order to build a deep learning model (multimodal prediction model) that can accurately diagnose tumors, this application adopts a multi-information fusion strategy to solve the heterogeneity of multimodal data.

[0171] The initial step involves creating a feature fusion layer, which is used to fuse feature vectors (128 dimensions for each source) from four different sources (pathology data, transcriptome data, methylation data, clinical data) into a low-dimensional space. In order to facilitate the effective integration of these features, a feature weighting layer is introduced. The softmax algorithm ensures that the sum of the weights is 1. In the forward propagation phase of the model, weights are first assigned to each feature, and then these weights are multiplied with the corresponding features to achieve feature weighting. Subsequently, the weighted features are combined into a single feature tensor, which provides a comprehensive feature representation for subsequent fusion and classification. In addition, a layer normalization layer is added to normalize the features, thereby further stabilizing the training process. Finally, the fused features are mapped to the final classification output through a classifier layer (linear layer), and the number of classifier output nodes matches the number of task categories.

[0172] The mathematical expression of the multimodal prediction model is shown below.

[0173]

[0174] Represents the output of the multimodal prediction model. W classRepresents the weight matrix of the output of the pathological model, transcriptional model, methylation model and clinical model. LayerNorm represents layer normalization. f1, f2, f3 and f4 represent pathological features, RNA sequencing result features (transcriptome features), methylation microarray data features (methylation features) and clinical features, respectively.

[0175] z i Represents the original output of the i-th category. Through the Softmax function, the model can derive the probability of each category and make classification decisions.

[0176] In the training phase, the above-constructed fusion model is trained as a whole and its performance is continuously improved. After sufficient training and optimization, the deep learning fusion model will have a higher diagnostic ability, and this trained model can be used to predict patient samples in practical applications.

[0177] During the prediction stage, the relevant data corresponding to each patient (sample) is input into the multimodal tumor prediction model. The prediction model will generate corresponding tumor labels based on the knowledge and rules it has learned, thereby providing doctors with a powerful auxiliary tool.

[0178] In step 150, the lipid metabolism-related marker features of the target patient are extracted based on the methylation data, and a comprehensive liver cancer analysis is performed on the target patient based on the lipid metabolism-related marker features and the liver cancer analysis features.

[0179] In one embodiment of the present application, the comprehensive liver cancer analysis of the target patient according to the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics includes:

[0180] Liver cancer is diagnosed and prognosis is predicted for the target patient based on the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics.

[0181] Specifically, the multimodal prediction model of the present application also includes a lipid metabolism feature prediction module.

[0182] This application introduces a lipid metabolism feature prediction module into the multimodal prediction model to reveal the relationship between abnormal lipid metabolism and the occurrence and development of the disease in liver cancer patients. This module is based on lipid metabolism-related genes in transcriptome data, and through in-depth feature extraction and analysis, it predicts the expression level of lipid metabolism-related markers to provide auxiliary decision support for personalized treatment. The technical implementation is as follows:

[0183] Identification and feature extraction of lipid metabolism related genes:

[0184] In the preprocessing stage of transcriptome data, bioinformatics methods are used to identify lipid metabolism genes related to the development of liver cancer. These genes include key genes in metabolic pathways such as lipid synthesis, decomposition, and transport, such as cholesterol synthesis genes and fatty acid synthase. The screened lipid metabolism gene data are further cleaned and standardized to ensure the consistency and reliability of the input data. At the same time, dimensionality reduction techniques (such as UMAP) are used to extract the most representative lipid metabolism features to reduce noise interference and improve the stability of the model.

[0185] Feature prediction model design:

[0186] The lipid metabolism feature prediction module adopts a multi-layer neural network architecture, in which the input layer receives the processed lipid metabolism gene expression data. The hidden layer transforms the features through a nonlinear activation function to extract potential metabolic feature patterns. The expression of various lipid metabolism markers is predicted through the Softmax layer, and the model outputs the expression probability of lipid metabolism-related markers. The final output probability distribution can reflect the abnormal degree of lipid metabolism in patients and provide an intuitive reference for the metabolic status in clinic.

[0187] Prediction of abnormal lipid metabolism status and personalized decision support:

[0188] This module combines the expression characteristics of lipid metabolism genes to accurately identify liver cancer patients with abnormal lipid metabolism. Such patients may have different prognoses and treatment needs. The model's prediction results can help doctors identify high-risk patients and develop more precise treatment plans for them. For example, in patients with abnormal lipid metabolism, specific lipid metabolism inhibitors or nutritional regulation may become potential treatment strategies. The model's prediction results provide a basis for the selection of such treatment options and can achieve more personalized treatment interventions in clinical practice.

[0189] 1) Multimodal deep learning fusion model (multimodal prediction model): This application proposes a multimodal deep learning fusion model that achieves accurate diagnosis and prognosis prediction of liver cancer by integrating pathological data, transcriptome data, methylation data and clinical data. The model further optimizes the fusion effect through feature weighting and layer normalization, providing higher accuracy and reliability for liver cancer diagnosis.

[0190] (2) UMAP dimensionality reduction mechanism: This application uses the UMAP (Uniform Manifold Approximation and Projection) algorithm to reduce the dimensionality of high-dimensional data, retaining the local topological structure of multimodal data. Through UMAP dimensionality reduction, each modal data can still maintain its tight distribution of features after dimensionality reduction, so that the model can efficiently capture the potential feature relationship of the data in a low-dimensional space. This mechanism reduces data redundancy and effectively reduces computational complexity, providing a more concise and information-rich feature representation for subsequent feature fusion and classification.

[0191] (3) Feature weighting mechanism: This application introduces a feature weighting mechanism and uses the Softmax algorithm to calculate the weight of each modal feature to ensure that the relative importance of features can be effectively considered when fusing data from different modalities, thereby improving the overall performance of the model.

[0192] (4) Prediction of lipid metabolism-related characteristics: In combination with the multifactorial effects of lipid metabolism on liver cancer, this application uses a model to predict the expression of lipid metabolism-related markers, providing a new research perspective on the role of abnormal lipid metabolism in liver cancer and facilitating clinical precision treatment.

[0193] (5) Flexibility of model structure: The system has a flexible structure that can adjust the model’s modal input according to actual needs. It is suitable for various types of tumor diagnosis and other medical tasks, which improves the versatility and scalability of the model.

[0194] In summary, the multimodal prediction model can provide doctors with diagnostic suggestions and prognosis prediction results in real time, helping doctors to better formulate treatment plans and make decisions. Doctors can combine the prediction results of the multimodal prediction model with the patient's clinical information to formulate a more accurate treatment plan. At the same time, the multimodal prediction model can also be used for early screening and prevention of tumors to improve the quality of life and health of the population. Combining the multimodal prediction model with the clinical workflow provides doctors with a convenient, fast and accurate diagnostic support system. This integration method has the potential to improve doctors' work efficiency and enable the multimodal prediction model to better adapt to actual clinical needs.

[0195] This application will establish a liver cancer diagnosis classification, prognosis prediction, and lipid metabolism-related marker expression prediction system based on pathological images. In response to the problem of single data in traditional diagnostic methods, deep learning technology is used to effectively integrate pathological genomics, transcriptomics, methylation, and clinical prognosis data, which can more comprehensively reflect the characteristics of the tumor, and is expected to improve the accuracy of auxiliary diagnosis and provide clinicians with more comprehensive and in-depth pathological information. In view of the heterogeneity between multimodal data, technical means are used to eliminate differences and conflicts, ensure the effectiveness and stability of the model, and adaptively adjust different modal data. Technical means are used to accelerate model convergence, expand training data sets, and improve the accuracy and generalization ability of multimodal prediction models, so that multimodal prediction models can process data of multiple modalities at the same time, and have stronger feature extraction and classification capabilities. Combining multimodal deep learning models with clinical workflows can provide doctors with real-time diagnostic suggestions and prognosis prediction results, which can adapt to the needs of different clinical scenarios.

[0196] The multimodal deep learning model (multimodal prediction model) of this application has important application value in clinical liver cancer management. By combining pathological data, transcriptome data, methylation data and clinical data, the model can provide doctors with more accurate and comprehensive liver cancer diagnosis information. Especially in the early diagnosis of liver cancer, the model uses the complementarity of multimodal data to improve the sensitivity and specificity of detection and reduce the possibility of misdiagnosis and missed diagnosis. For prognosis prediction, the model can provide detailed prognostic information through multimodal data analysis to assist doctors in formulating more personalized treatment plans.

[0197] In addition, this application also includes a predictive function for lipid metabolism characteristics, which provides a new tool for studying the relationship between abnormal lipid metabolism and the development of liver cancer. This function not only helps to identify potential high-risk patients, but also provides a reference for treatment plans targeting lipid metabolism. This multimodal deep learning model has strong clinical promotion potential and can be embedded in the hospital's clinical workflow to help doctors diagnose and manage liver cancer patients more accurately and efficiently in actual work. By providing high-quality auxiliary diagnostic information, this application not only reduces the workload of doctors, but also supports the refined development of clinical liver cancer management and improves patients' survival rate and quality of life.

[0198] Figure 2 This is a logic diagram of a liver cancer analysis system based on multimodal deep learning in an embodiment of the present application. Figure 3 This is a block diagram of a liver cancer analysis system 300 based on multimodal deep learning according to an embodiment of the present application. According to an embodiment of the present application, a liver cancer analysis system 300 based on multimodal deep learning includes: an input unit 301, a preprocessing unit 302, a hierarchical processing unit 303, a fusion processing unit 304, and an analysis unit 305.

[0199] An input unit 301 is used to obtain pathological data, transcriptome data, methylation data and clinical data of a target patient;

[0200] A preprocessing unit 302 is used to preprocess the pathological data, the transcriptome data, the methylation data and the clinical data respectively to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data and target clinical data corresponding to the clinical data;

[0201] A hierarchical processing unit 303 is used to perform feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data;

[0202] The fusion processing unit 304 is used to fuse the pathological features, the transcriptome features, the methylation features and the clinical features to obtain the liver cancer analysis features of the target patient.

[0203] The analysis unit 305 extracts the lipid metabolism-related marker features of the target patient based on the methylation data, and performs a comprehensive liver cancer analysis on the target patient according to the lipid metabolism-related marker features and the liver cancer analysis features.

[0204] As another aspect, the present application further provides a computer-readable storage medium on which a program product capable of implementing the method provided above in this specification is stored. In some possible implementations, various aspects of the present application may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present application described in the above "Embodiment Method" section of this specification.

[0205] According to the embodiment of the present application, the program product for implementing the above method can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0206] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0207] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0208] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0209] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0210] As another aspect, the present application also provides an electronic device capable of implementing the above method.

[0211] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as a "circuit", "module" or "system".

[0212] Refer to the following Figure 4 The electronic device 400 according to this embodiment of the present application is described. Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0213] like Figure 4As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include but are not limited to: at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410).

[0214] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 executes the steps described in the above “Example Method” section of this specification according to various exemplary implementations of the present application.

[0215] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0216] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0217] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller node, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0218] The electronic device 400 may also communicate with one or more external devices 1200 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 450. In addition, the electronic device 400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0219] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0220] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0221] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A liver cancer analysis method based on multimodal deep learning, characterized in that: The method comprises: Obtain pathological data, transcriptome data, methylation data, and clinical data of target patients; Preprocessing the pathological data, the transcriptome data, the methylation data, and the clinical data respectively to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data; Performing feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data; Performing feature fusion on the pathological features, the transcriptome features, the methylation features, and the clinical features to obtain liver cancer analysis features of the target patient; The lipid metabolism-related marker features of the target patient are extracted based on the methylation data, and a comprehensive liver cancer analysis is performed on the target patient based on the lipid metabolism-related marker features and the liver cancer analysis features.

2. The liver cancer analysis method based on multimodal deep learning according to claim 1, characterized in that: The pathological data is a full-slice image, and the target pathological data is obtained by the following steps: Performing full-slice image segmentation on the pathological data to obtain a plurality of image blocks; Screening out target image blocks whose cell tissue area ratio is higher than a preset threshold value from each of the image blocks; Adjusting the image parameters of each of the target image blocks so as to adjust the image parameters to a preset reference value; After the image parameters of each of the target image blocks are adjusted to the reference value, generating the target pathology data according to each of the target image blocks; The image parameters include brightness, contrast, gamma correction and saturation.

3. The liver cancer analysis method based on multimodal deep learning according to claim 2, characterized in that: The pathological characteristics are obtained by the following steps: Perform feature extraction on the target pathological data through a ResNet152 model to obtain the pathological features of the target patient; The model weights of the ResNet152 model are obtained by the following steps: A cross entropy loss function is used to perform a probability difference distribution of model prediction on the target pathology data, and a back propagation algorithm is used to calculate the gradient of the loss of the probability difference distribution relative to the ResNet152 model parameters; The model weights are obtained by using an SGD optimizer according to the gradient calculation.

4. The liver cancer analysis method based on multimodal deep learning according to claim 3, characterized in that: The transcriptome features and the methylation features are obtained by the following steps: Using a unified manifold approximation and projection method to perform feature mapping on the target transcriptome data and the target methylation data in a high-dimensional space, respectively, to obtain transcriptome mapping data and methylation mapping data in a low-dimensional space; determining the transcriptome signature based on the transcriptome mapping data, and determining the methylation signature based on the methylation mapping data; The unified manifold approximation and projection method is specifically the following steps: S1, calculates the neighboring probability of data points in high-dimensional space; Among them, S1 can calculate the neighboring probability of any two data points in the target transcriptome data or the target methylation data by the following formula: In the formula, dist(x i ,x j ) represents the Euclidean distance between data point i and data point j; is the nearest neighbor distance of data point i; σ i is the preset scale parameter, p ij is the proximity probability of any two data points. S2, calculates the similarity of data points in low-dimensional space; Wherein, S2 can calculate the similarity of any two data points in the transcriptome mapping data or the methylation mapping data by the following formula: In the formula, y i and j are the positions of data points i and j in the low-dimensional space respectively; a and b are hyperparameters, q ij is the similarity between any two data points; S3, the loss function of the mapping between high-dimensional space and low-dimensional space is determined by the following formula: Wherein, L is the loss function.

5. The liver cancer analysis method based on multimodal deep learning according to claim 4, characterized in that: The liver cancer analysis characteristics of the target patient are obtained by the following steps: Inputting the pathological features, the transcriptome features, the methylation features, and the clinical features into a pre-set multimodal fusion model to obtain the liver cancer analysis features output by the multimodal fusion model; The mathematical expression of the multimodal fusion model is as follows: In the formula, is the liver cancer analysis feature output by the multimodal fusion model, W class is a weight matrix corresponding to the target pathological data, the target transcriptome data, the target methylation data and the target clinical data, LayerNorm represents layer normalization, f1, f2, f3 and f4 are the pathological characteristics, the transcriptome characteristics, the methylation characteristics and the clinical characteristics, respectively.

6. The liver cancer analysis method based on multimodal deep learning according to claim 5, characterized in that: The comprehensive liver cancer analysis of the target patient according to the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics comprises: Liver cancer is diagnosed and prognosis is predicted for the target patient based on the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics.

7. A liver cancer analysis system based on multimodal deep learning, characterized in that: The system comprises: An input unit, used to obtain pathological data, transcriptome data, methylation data and clinical data of target patients; a preprocessing unit, configured to preprocess the pathological data, the transcriptome data, the methylation data, and the clinical data, respectively, to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data; a hierarchical processing unit, for performing feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data; A fusion processing unit is used to fuse the pathological features, the transcriptome features, the methylation features and the clinical features to obtain the liver cancer analysis features of the target patient. An analysis unit extracts lipid metabolism-related marker features of the target patient based on the methylation data, and performs a comprehensive liver cancer analysis on the target patient according to the lipid metabolism-related marker features and the liver cancer analysis features.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the operations performed by the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories, wherein the one or more memories store at least one program code, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the method according to any one of claims 1 to 6.

Images