Liver cancer analysis method, system, equipment and medium based on multimodal deep learning

Abstract

The present application relates to the technical field of liver cancer analysis, and in particular, to a liver cancer analysis method, system, device and medium based on multimodal deep learning. The method comprises: obtaining the pathological data, transcriptome data, methylation data and clinical data of the target patient, and performing preprocessing respectively to obtain target pathological data, target transcriptome data, target methylation data and target clinical data; performing feature extraction respectively to obtain pathological features, transcriptome features, methylation features and clinical features; performing feature fusion on the pathological features, transcriptome features, methylation features and clinical features to obtain liver cancer analysis features; extracting lipid metabolism-related marker features of the target patient, and performing a comprehensive liver cancer analysis on the target patient based on the lipid metabolism-related marker features and liver cancer analysis features. The present application can comprehensively reflect the tumor characteristics of the target patient, improve the accuracy of auxiliary diagnosis, and provide accurate prognosis prediction for the target patient.

Description

Technical Field

[0001] The present application relates to the technical field of liver cancer analysis, and in particular, to a liver cancer analysis method, system, device and medium based on multimodal deep learning. Background Art

[0002] With advances in medical imaging and genomics, the diagnosis and prognosis of liver tumors are increasingly reliant on multidimensional data analysis. While medical imaging techniques such as CT and MRI provide important information on tumor morphology, distinguishing between benign and malignant tumors and predicting prognosis based solely on imaging data remains challenging. Liver biopsy remains the gold standard for diagnosing benign and malignant liver tumors, but as an invasive procedure, its clinical application has certain limitations. Summary of the Invention

[0003] The present application provides a liver cancer analysis method, system, device and medium based on multimodal deep learning, which can fuse data and features from different modalities into a unified feature representation, and ultimately perform classification or prognosis prediction, thereby realizing the synergistic effect of multimodal features.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a liver cancer analysis method based on multimodal deep learning is provided, the method comprising:

[0006] Obtain pathological data, transcriptome data, methylation data, and clinical data of target patients;

[0007] Preprocessing the pathological data, the transcriptome data, the methylation data, and the clinical data respectively to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data;

[0008] performing feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data;

[0009] Performing feature fusion on the pathological features, the transcriptome features, the methylation features, and the clinical features to obtain liver cancer analysis features of the target patient;

[0010] The lipid metabolism-related marker features of the target patient are extracted based on the methylation data, and a comprehensive liver cancer analysis is performed on the target patient based on the lipid metabolism-related marker features and the liver cancer analysis features.

[0011] In one embodiment of the present application, based on the above solution, the pathological data is a full-slice image, and the target pathological data is obtained by the following steps:

[0012] Performing full-slice image segmentation on the pathological data to obtain a plurality of image blocks;

[0013] Screening out target image blocks whose cell tissue area ratio is higher than a preset threshold value from each of the image blocks;

[0014] Adjusting the image parameters of each target image block to adjust the image parameters to preset reference values;

[0015] After the image parameters of each target image block are adjusted to the reference value, generating the target pathology data according to each target image block;

[0016] The image parameters include brightness, contrast, gamma correction and saturation.

[0017] In one embodiment of the present application, based on the above scheme, the pathological characteristics are obtained by the following steps:

[0018] Performing feature extraction on the target pathological data using a ResNet152 model to obtain the pathological features of the target patient;

[0019] The model weights of the ResNet152 model are obtained by the following steps:

[0020] A cross entropy loss function is used to perform model prediction of the target pathology data using a probability difference distribution, and a back propagation algorithm is used to calculate the gradient of the loss of the probability difference distribution relative to the ResNet152 model parameters;

[0021] The model weights are obtained by using an SGD optimizer according to the gradient calculation.

[0022] In one embodiment of the present application, based on the above scheme, the transcriptome features and the methylation features are obtained by the following steps:

[0023] Using a unified manifold approximation and projection method to perform feature mapping on the target transcriptome data and the target methylation data in a high-dimensional space, respectively, to obtain transcriptome mapping data and methylation mapping data in a low-dimensional space;

[0024] determining the transcriptome signature based on the transcriptome mapping data, and determining the methylation signature based on the methylation mapping data;

[0025] The unified manifold approximation and projection method is specifically as follows:

[0026] S1, calculates the neighboring probability of data points in high-dimensional space;

[0027] Wherein, S1 can calculate the proximity probability of any two data points in the target transcriptome data or the target methylation data by the following formula:

[0028]

[0029] Where, dist(x i ,x j ) represents the Euclidean distance between data point i and data point j; ρ i is the nearest neighbor distance of data point i; σ i is the preset scale parameter, p ij is the proximity probability of any two data points.

[0030] S2, calculates the similarity of data points in low-dimensional space;

[0031] Wherein, S2 can calculate the similarity of any two data points in the transcriptome mapping data or the methylation mapping data by the following formula:

[0032]

[0033] Where y i and y j are the positions of data points i and j in the low-dimensional space respectively; a and b are hyperparameters, q ij is the similarity between any two data points;

[0034] S3, determines the loss function of the mapping between high-dimensional space and low-dimensional space by the following formula:

[0035]

[0036] Wherein, L is the loss function.

[0037] In one embodiment of the present application, based on the above scheme, the liver cancer analysis characteristics of the target patient are obtained by the following steps:

[0038] Inputting the pathological features, the transcriptome features, the methylation features, and the clinical features into a pre-set multimodal fusion model to obtain the liver cancer analysis features output by the multimodal fusion model;

[0039] The mathematical expression of the multimodal fusion model is as follows:

[0040]

[0041] Where, is the liver cancer analysis feature output by the multimodal fusion model, W class is a weight matrix corresponding to the target pathological data, the target transcriptomic data, the target methylation data, and the target clinical data. LayerNorm represents layer normalization, f1, f2, f3, and f4 are the pathological features, the transcriptomic features, the methylation features, and the clinical features, respectively.

[0042] In one embodiment of the present application, based on the aforementioned scheme, the comprehensive liver cancer analysis of the target patient according to the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics includes:

[0043] Liver cancer is diagnosed and prognosis is predicted for the target patient based on the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics.

[0044] According to one aspect of an embodiment of the present application, a liver cancer analysis system based on multimodal deep learning is provided, the system comprising:

[0045] An input unit, used to obtain pathological data, transcriptome data, methylation data, and clinical data of target patients;

[0046] a preprocessing unit, configured to preprocess the pathological data, the transcriptome data, the methylation data, and the clinical data, respectively, to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data;

[0047] a hierarchical processing unit, configured to perform feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data;

[0048] A fusion processing unit is used to fuse the pathological features, the transcriptome features, the methylation features and the clinical features to obtain the liver cancer analysis features of the target patient.

[0049] An analysis unit extracts lipid metabolism-related marker features of the target patient based on the methylation data, and performs a comprehensive liver cancer analysis on the target patient according to the lipid metabolism-related marker features and the liver cancer analysis features.

[0050] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program includes executable instructions. When the executable instructions are executed by a processor, the method described in the above embodiment is implemented.

[0051] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions of the processors, wherein when the executable instructions are executed by the one or more processors, the one or more processors implement the method described in the above embodiments.

[0052] Beneficial effects of this application: Compared with existing liver cancer diagnosis technologies, traditional methods often rely solely on imaging data such as CT or MRI, which can easily lead to missed diagnoses or misdiagnoses. This application integrates multiple information sources such as pathological images, transcriptomes, methylation and clinical data, significantly improving the accuracy of diagnosis. This application achieves a more comprehensive understanding of tumors through feature fusion of multimodal data.

[0053] This application obtains the liver cancer analysis characteristics of the target patient by fusing the pathological characteristics, the transcriptome characteristics, the methylation characteristics and the clinical characteristics, and extracts the lipid metabolism-related marker characteristics of the target patient based on the methylation data to perform a comprehensive liver cancer analysis on the patient. This can more comprehensively reflect the tumor characteristics of the target patient, improve the accuracy of auxiliary diagnosis, provide clinicians with more comprehensive and in-depth pathological information, and provide accurate prognosis predictions for the target patient.

[0054] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0056] Figure 1 Flowchart of a liver cancer analysis method based on multimodal deep learning according to an embodiment of the present application;

[0057] Figure 2 This is a logical principle diagram of a liver cancer analysis system based on multimodal deep learning according to an embodiment of the present application;

[0058] Figure 3 1 is a block diagram of a liver cancer analysis system based on multimodal deep learning according to an embodiment of the present application;

[0059] Figure 4 Schematic diagram of the system structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0061] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0062] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or micro-controller node devices.

[0063] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0064] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0065] The following is a detailed description of the technical background of the embodiments of the present application and the direction of the lipid metabolism-related markers studied in the embodiments of the present application:

[0066] Liver cancer is a global health challenge, and its incidence is increasing worldwide. It is estimated that by 2025, one million people will be affected by liver cancer each year, and hepatocellular carcinoma (HCC) is the most common form of liver cancer, accounting for 90% of cases. Liver cancer is one of the most common malignant tumors in my country. More than 50% of new cases of liver cancer worldwide occur in my country each year, and its incidence and mortality rates remain high. Hepatitis B virus (HBV), hepatitis C virus (HCV), cirrhosis, metabolic syndrome, etc. are generally considered to be predisposing factors for liver cancer. Therefore, accurate diagnosis of liver cancer is particularly important to improve the 5-year survival rate of liver cancer patients. Unfortunately, liver cancer is not sensitive to chemotherapy, so surgical treatment is the first choice for liver cancer patients. However, since most patients with liver cancer are diagnosed in the late stage, they can only receive systemic treatment. In recent years, systemic treatment for liver cancer patients has evolved from single-agent targeted therapy (sorafenib and lenvatinib) to checkpoint inhibitors plus targeted therapy (PD-L1 monoclonal antibody, atezolizumab combined with the VEGF antagonist bevacizumab). Despite significant progress in the clinical treatment of liver cancer, only a small proportion of patients can achieve lasting clinical benefits. Given the current medical situation in my country, the development of a new diagnostic system is of great significance to the growing demand for liver cancer diagnosis.

[0067] The occurrence and development of liver cancer is a multifactorial, multi-stage process. The liver is the central organ of lipid metabolism, and the occurrence and development of its tumors may depend on energy metabolism processes including fat generation and conversion. Lipid metabolism refers to the processes of synthesis, decomposition, transport and storage of lipids in organisms. Abnormal lipid metabolism is usually manifested as abnormal increase or decrease in blood lipid levels, including imbalances in total cholesterol, triglycerides, low-density lipoprotein (LDL) and high-density lipoprotein (HDL). More and more epidemiological and clinical studies have shown that abnormal lipid metabolism is associated with an increased risk of liver cancer. For example, a high-cholesterol diet may be a risk factor for the development of liver cancer, while hypocholesterolemia may increase the risk of liver cancer. Abnormal lipid metabolism is not only related to the occurrence of liver cancer, but may also affect the progression and metastasis of liver cancer.

[0068] Studies have shown that tumor cells can adapt to their rapid proliferation needs by changing lipid metabolism, including processes such as the synthesis, oxidation, and esterification of fatty acids. In addition, changes in lipid metabolism in the tumor microenvironment may also promote tumor angiogenesis, immune escape, and metastasis. Although studies have revealed the connection between abnormal lipid metabolism and liver cancer, the specific mechanism is still not fully understood. In-depth research on the mechanism of action of abnormal lipid metabolism in liver cancer is of great significance for early diagnosis, risk assessment, and the formulation of treatment strategies. In addition, understanding how abnormal lipid metabolism affects the biological behavior and clinical treatment of liver cancer, and exploring the pathological characteristics and molecular markers related to lipid metabolism in various subtypes of liver cancer will help develop new prevention and clinical treatment methods.

[0069] In recent years, deep learning technology has made significant progress in medical image analysis. However, pathology image analysis remains challenging, primarily due to the complex tissue and texture features of pathology images, as well as subtle differences between different pathology types. Although convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have seen initial application in CT image feature extraction and tumor prognosis prediction, these methods primarily focus on single-modality data and fail to comprehensively analyze and analyze multiple types of human tumor data. Therefore, the scalability, reliability, and accuracy of tumor identification and prognosis need to be further improved. Therefore, multimodal deep learning models that can effectively integrate information from different modalities can better simulate human perception and cognitive processes, thereby achieving a more comprehensive and in-depth understanding. Therefore, multimodal deep learning models have become a key cross-disciplinary research area in medicine. In addition to medical imaging and genomic data, pathology and transcriptome data can reveal information about the internal microstructure and gene expression of tumors, further improving diagnosis and prognosis. However, deep learning models based on clinical pathology images combined with multimodal deep learning models have not yet found a solution for practical clinical application. As the costs of various molecular analyses decrease and deep learning techniques advance, this approach may become more cost-effective in the future and facilitate the clinical application of related technologies.

[0070] This application will focus on integrating liver cancer data from different modalities, combining the multifactorial influence of lipid metabolism on liver cancer, and ultimately using multimodal deep learning technology to establish a liver cancer diagnosis and classification, prognosis prediction, and pathology image-based lipid metabolism-related marker expression prediction system, providing a practical diagnostic model for clinical use. This system is expected to be integrated into the clinical process to reduce the workload of pathologists and clinicians, while also promoting the learning and training of less experienced pathologists to support the diagnosis and treatment of liver cancer.

[0071] With the advancement of deep learning technology in medicine, artificial intelligence (AI) technologies, including radiomics and deep learning methods, have emerged as a unique opportunity to improve the scope of clinical care for hepatocellular carcinoma by predicting biological features and prognosis in the field of medical imaging.

[0072] In the field of radiomics, liver cancer diagnosis technology has matured. Currently, diagnostic methods based on medical imaging technologies such as CT and MRI are widely used. Tietz E et al. constructed a multivariate logistic regression model based on CT images to predict new liver cancer in high-risk patients with cirrhosis, demonstrating good efficacy and further confirming the important role of radiomics in liver cancer diagnosis. However, while radiomics can provide objective and quantitative diagnostic information, the limitations of imaging examinations may lead to misdiagnosis or missed diagnosis of some benign solid lesions. To overcome these limitations, researchers are actively exploring the integration of features extracted by deep learning into radiomics models. Gan Fuwen et al. achieved preoperative prediction of liver cancer by combining imaging features extracted from high-throughput T2-weighted imaging with parameter fine-tuning of the EfficientNet-B7 model and deep labels constructed using LightGBM. This study not only enriches the scope of radiomics but also provides new insights for the precise diagnosis of liver cancer.

[0073] Histopathological image examination is the gold standard for oncological diagnosis. Its images reveal the cellular and tissue structure of tissue sections and can be used to determine cell type, tissue structure, the number and morphology of abnormal cells, and assess the specific characteristics of liver cancer. In recent years, with the rapid development of artificial intelligence (AI) technology, histopathological classification using digital whole slide images (WSIs) has become possible, significantly improving the efficiency and accuracy of liver cancer diagnosis. Shao Runhua et al. used convolutional neural networks to conduct in-depth research on liver cancer pathological images, including diagnostic algorithms for liver tumor detection, image segmentation, and preoperative prediction. On May 20, 2024, Nakatsuka T's team proposed a deep learning model that predicts liver cancer progression based on hematoxylin and eosin (HE)-stained HCC pathological images. The model reportedly demonstrated impressive performance, with an accuracy rate of 81.0% and an AUC of 0.80. However, while this achievement is impressive, there is still room for improvement. Therefore, we are considering fusing multiple data modalities to construct a multimodal deep learning model to further improve the accuracy and reliability of liver cancer pathological diagnosis. This is not only the main direction of this study, but also an important development trend in the field of liver cancer diagnosis in the future.

[0074] Currently, the application of artificial intelligence in the analysis of tumor pathological tissue markers has achieved remarkable results. Saha et al. proposed a deep learning-based Her2Net model that can automatically evaluate HER-2 staining in breast cancer. This model not only achieved high accuracy, but also had a false positive rate of only 6.84%. This achievement fully demonstrated the great potential of AI in tumor pathological diagnosis. AI-assisted tumor pathological diagnosis has the advantages of automation, high efficiency, and repeatability. In particular, it has shown great advantages in the quantitative interpretation of tissue pathological markers, which helps to improve the objectivity and accuracy of tumor companion diagnosis. This also provides a new idea for this study, that is, to conduct more in-depth analysis and diagnosis of tumors by integrating multi-model and multi-omics information.

[0075] Serum markers are the most convenient and non-invasive diagnostic methods in clinical applications. Among the many tumor markers, alpha-fetoprotein (AFP), alpha-fetoprotein variants (AFP-L3%), and abnormal prothrombin (DCP) are known as the "three liver cancer tests." Internationally, the liver cancer triple test is a commonly used serum marker combination for early diagnosis of liver cancer. Studies have shown that the combined detection of DCP, AFP, and AFP-L3% can increase the detection rate of liver cancer to 85.9%-94.57%, which can basically meet the needs of liver cancer diagnosis. However, due to methodological limitations, AFP-L3% detection is still a difficult point in the three liver cancer tests. The main problem is that the operation process is cumbersome and time-consuming, and the test results need to be multiplied by the dilution factor, which consumes a lot of manpower. This not only increases the cost of testing, but also limits its widespread application in clinical practice. To this end, Moldogazieva NT et al. proposed that high-throughput proteomics analysis technology can be combined with artificial intelligence algorithms and predictive models to further explore new HCC candidate biomarkers, improve the sensitivity and specificity of HCC detection, and provide strong support for predicting treatment responses.

[0076] With advances in medical imaging and genomics, the diagnosis and prognosis of liver tumors are increasingly reliant on multidimensional data analysis. While medical images such as CT and MRI provide important information about tumor morphology, distinguishing benign from malignant tumors and predicting prognosis based solely on imaging data remains challenging. Liver biopsy remains the gold standard for diagnosing benign and malignant liver tumors, but as an invasive procedure, its clinical application has certain limitations. In recent years, deep learning has achieved significant breakthroughs in medical image analysis. However, research on the classification of benign and malignant liver tumors in pathological images remains insufficient, primarily due to the complex tissue structure and texture features of pathological images, as well as the subtle differences between different pathological types. Therefore, integrating data from multiple modalities for comprehensive analysis has become a key research direction. In addition to medical imaging omics, omics data such as pathomic and transcriptomics can also reveal the microstructure and gene expression information within tumors, improving diagnosis and prognosis prediction.

[0077] Currently, deep learning has seen initial applications in the diagnosis and prognosis of other tumors, such as using convolutional neural networks (CNNs) for CT image feature extraction and classification, and recurrent neural networks (RNNs) for time series medical data modeling and tumor prognosis prediction. We will also be applying deep learning techniques to liver tumors. However, current deep learning prediction approaches for tumors primarily focus on single-modality data and fail to fully exploit the correlation and complementarity between multimodal data. Therefore, there is still room for improvement in accurately distinguishing benign and malignant tumors and predicting prognosis.

[0078] The purpose of this application is to establish a liver cancer diagnosis and classification, prognosis prediction and lipid metabolism-related marker expression prediction system based on pathological images, including: an input unit for obtaining the patient's pathological data, transcriptome data, methylation data and clinical data; a preprocessing unit for preprocessing the pathological data, transcriptome data, methylation data and clinical data respectively; a hierarchical processing unit for inputting the various preprocessed data into a multimodal prediction model (multimodal fusion model) that integrates a pathological model, a transcriptome model, a methylation model and a clinical model to predict the pathological results (pathological characteristics), transcriptome results (transcriptome characteristics), methylation results (methylation characteristics) and clinical results (clinical characteristics); an analysis unit for obtaining the final tumor classification, prognosis prediction and lipid metabolism marker expression prediction results through pathological results (pathological characteristics), transcriptome results (transcriptome characteristics), methylation results (methylation characteristics) and clinical results (clinical characteristics).

[0079] This application will utilize multimodal deep learning technology to effectively integrate various types of tumor data, more comprehensively reflecting tumor characteristics, more effectively assisting clinical diagnosis, and providing doctors with more comprehensive and in-depth pathological information. This approach can be widely applied in clinical practice. In later research, by appropriately adjusting the model structure and data processing flow, it can also be applied to other medical fields, such as disease diagnosis, drug development, gene editing, and other different medical tasks and scenarios.

[0080] The following is a detailed explanation of the terms used in the embodiments of this application:

[0081] 1. Multimodal fusion model: A deep learning model that combines different types of data (such as pathological data, transcriptomic data, methylation data, and clinical data). By fusing data from multiple modalities, it can comprehensively simulate the biological characteristics of liver cancer and provide more accurate information for clinical diagnosis and prognosis.

[0082] 2. Pathology data (whole slide images): High-resolution images obtained from pathology sections in medical diagnosis can reveal the microscopic features of tissue and cellular structures. Whole slide images (WSIs) are a standard format for pathology image analysis and contain complete information about tumor tissue.

[0083] 3. Transcriptome data: Gene expression data obtained through RNA sequencing technology, used to analyze the transcription levels of genes in cells or tissues, revealing the biological characteristics of liver cancer and its response to treatment.

[0084] 4. Methylation Data: DNA methylation is a gene regulation mechanism that affects gene expression by altering the methylation status of DNA. Methylation data are typically obtained through methylation microarray technology and used to analyze genomic modifications in tumors, reflecting epigenetic signatures in liver cancer development.

[0085] 5. Clinical data: Patient information recorded during liver cancer diagnosis and treatment, typically including the patient's age, gender, medical history, cirrhosis status, etc., can provide a personalized basis for tumor risk assessment and treatment plans.

[0086] 6. Lipid metabolism: The synthesis, decomposition and transport process of lipids in organisms is closely related to the occurrence of liver cancer.

[0087] 7. Feature weighting: A technique that calculates the weights of different modal data through the Softmax algorithm and applies the weights to the features to ensure that the importance of each modality is properly considered during multimodal data fusion.

[0088] 8. Layer Normalization: A standardization technique used to adjust the distribution of features in a neural network layer to prevent excessive differences in features between different modalities, thereby stabilizing the model training process.

[0089] 9. Data cleaning: During the data processing process, noise, missing values ​​and outliers in the dataset are processed to improve data quality and model training effect.

[0090] 10. Data augmentation: Generate additional training data through methods such as random cropping, flipping, and color jittering to enhance the generalization ability of the model and reduce overfitting.

[0091] 11. Convolutional Neural Network (CNN): A deep learning model for image processing that extracts spatial features of images through convolution operations and is widely used in medical image analysis.

[0092] 12. Cross-Entropy Loss Function: A commonly used loss function that measures the difference between the predicted probability distribution of the model output and the true label, helping to optimize the model's classification performance.

[0093] 13. Pretraining: The process of training a model on a large dataset to enable it to learn common features, thereby more effectively training for specific tasks in application scenarios with small sample data.

[0094] 14. Hyperparameter optimization: The process of adjusting parameters such as learning rate and regularization coefficient during model training to achieve optimal model performance.

[0095] 15. UMAP (Uniform Manifold Approximation and Projection): A nonlinear dimensionality reduction technique that optimizes data layout while preserving the topological structure of high-dimensional data, making it easier to observe data distribution in a low-dimensional space. It is used to analyze high-dimensional molecular data such as transcriptomes and methylation data.

[0096] 16. Fully Connected Layer: A layer in a neural network that connects every input node to every output node. It is usually used as the last layer in classification tasks to complete the classification output of the model.

[0097] 17. Softmax: A mathematical function used for multi-classification tasks, often used in the output layer of a neural network. The Softmax function converts a set of real numbers into a probability distribution, where the probability of each class is between 0 and 1, and the sum of all class probabilities is 1.

[0098] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0099] According to one aspect of the present application, a liver cancer analysis method based on multimodal deep learning is provided. Figure 1 This is a flow chart of a liver cancer analysis method based on multimodal deep learning according to an embodiment of the present application. The liver cancer analysis method based on multimodal deep learning includes at least steps 110 to 150, which are described in detail as follows:

[0100] In step 110 , pathological data, transcriptome data, methylation data, and clinical data of a target patient are obtained.

[0101] Specifically, the multimodal data mentioned in the embodiments of the present application include at least pathological data, transcriptome data, methylation data and clinical data. The application integrates multiple information sources such as pathological images, transcriptome, methylation and clinical data, significantly improving the accuracy of diagnosis. The present application achieves a more comprehensive understanding of tumors through the feature fusion of multimodal data.

[0102] In step 120, the pathological data, the transcriptome data, the methylation data and the clinical data are preprocessed respectively to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data and target clinical data corresponding to the clinical data.

[0103] In one embodiment of the present application, the pathological data is a full-slice image, and the target pathological data is obtained by the following steps:

[0104] Performing full-slice image segmentation on the pathological data to obtain a plurality of image blocks;

[0105] Screening out target image blocks whose cell tissue area ratio is higher than a preset threshold value from each of the image blocks;

[0106] Adjusting the image parameters of each target image block to adjust the image parameters to preset reference values;

[0107] After the image parameters of each target image block are adjusted to the reference value, generating the target pathology data according to each target image block;

[0108] The image parameters include brightness, contrast, gamma correction and saturation.

[0109] This application provides a pathological model, a transcriptome model, a methylation model, and a clinical model, and then integrates the pathological model, transcriptome model, methylation model, and clinical model to obtain a multimodal prediction model. The following is a detailed explanation of the pathological model, transcriptome model, methylation model, and clinical model:

[0110] Specifically, pathological data, including whole slide images (WSI), plays a vital role in disease diagnosis, research and treatment. Due to its huge amount of data and high resolution, WSI places great demands on the computing resources of deep learning. In order to make more effective use of these images, we will cut the original image and decompose it into image blocks (patches) with lower pixel values, which can reduce the complexity of our data processing. After the initial cutting, the patches are filtered and used for data enhancement. By utilizing various image processing operations, the patches are finally randomly cropped into image blocks with lower pixel values. Then, areas with low tissue area ratio are further eliminated. The elimination step is performed by a skilled pathologist to remove low-quality areas caused by HE staining technical factors. In the embodiment of the present application, by inputting pathological data into the pathological model, the above steps can enhance the generalization of the pathological model, reduce its overfitting to the training data, and at the same time improve the adaptability and robustness of the pathological model to changes in real images.

[0111] Pathology data requires image data preprocessing. By individually adjusting specific parameters for each image block, such as brightness, contrast, gamma correction, and saturation, the image's brightness, contrast, gamma correction, and saturation distribution are adjusted to approach a preset baseline value, or a pre-set average value. The next stage of the processing involves examining the input image preprocessing methods to ensure they match the input requirements of the deep learning model (multimodal fusion model). In particular, each channel of the image needs to be normalized.

[0112] The following is a detailed description of the preprocessing of the transcriptome data, the methylation data, and the clinical data:

[0113] The transcriptome data was thoroughly cleaned and preprocessed. Methylation data was also cleaned, organized, and quality-controlled on the same platform. During model training, it was normalized to eliminate the effects of inter-sample variability and gene length, converting it to a format suitable for model training.

[0114] Clinical data are statistical indicators presented in the form of text descriptions or charts to evaluate the possible outcomes of disease development, as well as the effectiveness of patient treatment and survival prospects. They usually include survival time, disease recurrence, recovery status, etc. They can assist doctors in formulating more personalized treatment plans, help to gain a deeper understanding of the natural course of the disease, influencing factors, etc., and provide a basis for the development of new treatments and drugs. In addition, clinical prognostic data also include patient conditions such as age, gender, and race, which will also affect the patient's tumor prediction to a certain extent. In actual applications, clinical prognostic data can be either a doctor's text description of the patient's prognosis or relevant diagnosis and treatment data recorded in tabular form. Regarding the preprocessing of clinical prognostic data, we will perform missing value processing and outlier processing to ensure the integrity and accuracy of the data.

[0115] In step 130, feature extraction is performed on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data.

[0116] In one embodiment of the present application, the pathological characteristics are obtained by the following steps:

[0117] Performing feature extraction on the target pathological data using a ResNet152 model to obtain the pathological features of the target patient;

[0118] The model weights of the ResNet152 model are obtained by the following steps:

[0119] A cross entropy loss function is used to perform model prediction of the target pathology data using a probability difference distribution, and a back propagation algorithm is used to calculate the gradient of the loss of the probability difference distribution relative to the ResNet152 model parameters;

[0120] The model weights are obtained by using an SGD optimizer according to the gradient calculation.

[0121] Specifically, regarding the construction of the pathological model, deep learning network models such as deep convolutional neural network (CNN), recurrent neural network (RNN) or graph neural network (GNN) can be used as the basic image model to adapt to different data structures and features. This application will select the ResNet152 model as the basic model of the pathological model. As a convolutional neural network model, ResNet152 can effectively extract key features in the image through pre-training on large-scale image datasets. These features have important reference value for tumor diagnosis.

[0122] The construction and training of the image model include: using the pre-trained ResNet152 model as a feature extractor for a specific pathology image classification task. To ensure that the model output matches the classification requirements, we replace the model's last fully connected layer with a new fully connected layer with n output nodes based on actual considerations to adapt to the specific pathology image classification task.

[0123] During training, the model's parameters were fine-tuned for the tumor diagnosis task to improve its adaptability to the task and output tumor labels and their corresponding probabilities. A cross-entropy loss function was used to measure the difference between the model's predicted probability distribution and the true label, and the gradient of the loss with respect to the model parameters was calculated using a backpropagation algorithm. The model weights were updated based on the calculated gradient using the SGD optimizer, with the learning rate initially set to 0.001 and decayed by 0.1 after every a epoch. During b epochs of training, the learning rate was continuously adjusted to optimize model performance. An epoch represents the process by which the model traverses and learns from the entire training dataset. An epoch is the process by which the algorithm completes a complete forward and backward pass on all training samples.

[0124] In one embodiment of the present application, the transcriptome signature and the methylation signature are obtained by the following steps:

[0125] Using a unified manifold approximation and projection method to perform feature mapping on the target transcriptome data and the target methylation data in a high-dimensional space, respectively, to obtain transcriptome mapping data and methylation mapping data in a low-dimensional space;

[0126] determining the transcriptome signature based on the transcriptome mapping data, and determining the methylation signature based on the methylation mapping data;

[0127] The unified manifold approximation and projection method is specifically as follows:

[0128] S1, calculates the neighboring probability of data points in high-dimensional space;

[0129] Wherein, S1 can calculate the proximity probability of any two data points in the target transcriptome data or the target methylation data by the following formula:

[0130]

[0131] Where, dist(x i ,x j ) represents the Euclidean distance between data point i and data point j; ρ iis the nearest neighbor distance of data point i; σ i is the preset scale parameter, p ij is the proximity probability of any two data points.

[0132] S2, calculates the similarity of data points in low-dimensional space;

[0133] Wherein, S2 can calculate the similarity of any two data points in the transcriptome mapping data or the methylation mapping data by the following formula:

[0134]

[0135] Where y i and y j are the positions of data points i and j in the low-dimensional space respectively; a and b are hyperparameters, q ij is the similarity between any two data points;

[0136] S3, determines the loss function of the mapping between high-dimensional space and low-dimensional space by the following formula:

[0137]

[0138] Wherein, L is the loss function.

[0139] In this application, a comprehensive approach is used to analyze transcriptome data and methylation microarray data (methylation data) to monitor tumor progression and evaluate treatment efficacy. By comparing RNA transcriptome data from tumor tissue and normal tissue, biomarkers with diagnostic value can be screened.

[0140] To ensure data accuracy and consistency, the transcriptome data underwent comprehensive data cleaning and preprocessing. Simultaneously, the methylation microarray data were cleaned, organized, and quality-controlled using the same platform. During model training, they were normalized to eliminate the effects of inter-sample variability and gene length, converting them to a format suitable for model training.

[0141] After data preprocessing, we used statistical methods and bioinformatics analysis, combined with the characteristics of lipid metabolism-related genes, to identify key genes for liver tumor classification. Changes in the expression levels of these genes may be closely associated with specific biological processes or signaling pathways in tumor development, which is crucial for improving model prediction accuracy and interpretability. We extracted these genes and their copy numbers and compiled them into a new list for subsequent training.

[0142] Before model building, the goal is to evaluate the independent impact of a single data pattern on classification performance. To this end, we chose UMAP (Uniform Manifold Approximation and Projection), a recently developed nonlinear dimensionality reduction technique that uses local manifold approximation and fuzzy simplicial set representation (local fuzzy simplicial set representation) to construct the topological structure of high-dimensional data. The UMAP algorithm aims to maintain the topological properties of the dataset while reducing the data dimension, minimizing the cross-entropy between high-dimensional and low-dimensional topological representations by optimizing the data layout. We used the UMAP package to reduce the dimensionality of transcriptome data and methylation microarray data, and then mapped the data to a low-dimensional space to observe their distribution, thereby gaining a more comprehensive understanding of the molecular characteristics and treatment response of tumors. Its main steps and formulas are as follows:

[0143] ① Calculation of neighboring probability in high-dimensional space

[0144] For any two data points i and j, UMAP first calculates the probability of their proximity in high-dimensional space. This probability p ij The calculation method is:

[0145]

[0146] Where, dist(x i ,x j ) represents the Euclidean distance between data point i and data point j; ρ i

[0147] is the nearest neighbor distance of data point i; σ i is the preset scale parameter, p ij

[0148] is the proximity probability of any two data points.

[0149] ②Similarity calculation in low-dimensional space

[0150] In low-dimensional space, UMAP uses a formula similar to t-SNE to calculate the similarity q between two data points i and j. ij :

[0151]

[0152] Where y i and y j

[0153] are the positions of data points i and j in the low-dimensional space respectively; a and b are hyperparameters, q ij

[0154] is the similarity between any two data points.

[0155] ③Loss function

[0156] The UMAP loss function optimizes the low-dimensional representation by minimizing the difference in similarity between high-dimensional and low-dimensional spaces. The goal is to minimize the cross-entropy loss:

[0157]

[0158] This loss function ensures that neighboring points in high-dimensional space also remain nearby in low-dimensional space, thereby preserving the local structure and topology of the data.

[0159] Both transcriptomics models and methylation models will be composed of multiple fully connected layers to capture the patterns in gene expression data and the mathematical relationships (non-linear relationships) between variables.

[0160] In one embodiment of the present application, regarding the clinical data model

[0161] Clinical data are statistical indicators presented in the form of text descriptions or charts to evaluate the possible outcomes of disease development, as well as the effectiveness of patient treatment and survival prospects. They usually include survival time, disease recurrence, recovery status, etc. They can assist doctors in formulating more personalized treatment plans, help to gain a deeper understanding of the natural course of the disease, influencing factors, etc., and provide a basis for the development of new treatments and drugs. In addition, clinical prognostic data also include patient conditions such as age, gender, and race, which will also affect the patient's tumor prediction to a certain extent. In actual applications, clinical prognostic data can be either a doctor's text description of the patient's prognosis or relevant diagnosis and treatment data recorded in tabular form.

[0162] Regarding clinical prognostic data preprocessing, missing value processing and outlier processing are performed to ensure data integrity and accuracy. Key information in clinical prognostic data is extracted. Key information has a significant impact on tumor prediction and therefore requires separate processing.

[0163] Clinical input data contains both numerical and categorical features. In the clinical model, an embedding layer is used to concatenate the categories into a single input vector. Feature extraction and dimensionality reduction are then performed on the input vector to introduce nonlinearity and reduce overfitting. The clinical model also exists as a deep learning model that takes the clinical input data as input and outputs a tumor label and its corresponding probability. During the model construction process, the network structure and parameters are adjusted to improve performance and robustness.

[0164] In step 140, feature fusion is performed on the pathological features, the transcriptome features, the methylation features, and the clinical features to obtain liver cancer analysis features of the target patient.

[0165] In one embodiment of the present application, the liver cancer analysis characteristics of the target patient are obtained by the following steps:

[0166] Inputting the pathological features, the transcriptome features, the methylation features, and the clinical features into a pre-set multimodal fusion model to obtain the liver cancer analysis features output by the multimodal fusion model;

[0167] The mathematical expression of the multimodal fusion model is as follows:

[0168]

[0169] Where, is the liver cancer analysis feature output by the multimodal fusion model, W class is a weight matrix corresponding to the target pathological data, the target transcriptomic data, the target methylation data, and the target clinical data. LayerNorm represents layer normalization, f1, f2, f3, and f4 are the pathological features, the transcriptomic features, the methylation features, and the clinical features, respectively.

[0170] Specifically, in order to build a deep learning model (multimodal prediction model) that can accurately diagnose tumors, this application adopts a multi-information fusion strategy to solve the heterogeneity of multimodal data.

[0171] The initial step involves creating a feature fusion layer, which is used to fuse feature vectors (128 dimensions per source) from four different sources (pathology data, transcriptome data, methylation data, and clinical data) into a low-dimensional space. In order to facilitate the effective integration of these features, a feature weighting layer is introduced. The softmax algorithm ensures that the sum of the weights is 1. In the forward propagation phase of the model, weights are first assigned to each feature, and then these weights are multiplied by the corresponding features to achieve feature weighting. Subsequently, the weighted features are combined into a single feature tensor, which provides a comprehensive feature representation for subsequent fusion and classification. In addition, a layer normalization layer is added to normalize the features, thereby further stabilizing the training process. Finally, the fused features are mapped to the final classification output through a classifier layer (linear layer), and the number of classifier output nodes matches the number of task categories.

[0172] The mathematical expression of the multimodal prediction model is shown below.

[0173]

[0174] Represents the output of the multimodal prediction model. W classRepresents the weight matrix output by the pathology model, transcription model, methylation model, and clinical model. LayerNorm represents layer normalization. f1, f2, f3, and f4 represent pathology features, RNA sequencing result features (transcriptome features), methylation microarray data features (methylation features), and clinical features, respectively.

[0175] z i Represents the original output of the i-th category. Through the Softmax function, the model can derive the probability of each category and make classification decisions.

[0176] During the training phase, the fusion model constructed above is trained as a whole, and its performance is continuously improved. After sufficient training and optimization, the deep learning fusion model will have high diagnostic capabilities. In practical applications, this trained model can be used to make predictions about patient samples.

[0177] During the prediction stage, the relevant data corresponding to each patient (sample) is input into the multimodal tumor prediction model. The prediction model will generate corresponding tumor labels based on the knowledge and rules it has learned, thereby providing doctors with a powerful auxiliary tool.

[0178] In step 150, lipid metabolism-related marker features of the target patient are extracted based on the methylation data, and a comprehensive liver cancer analysis is performed on the target patient based on the lipid metabolism-related marker features and the liver cancer analysis features.

[0179] In one embodiment of the present application, performing a comprehensive liver cancer analysis on the target patient based on the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics includes:

[0180] Liver cancer is diagnosed and prognosis is predicted for the target patient based on the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics.

[0181] Specifically, the multimodal prediction model of the present application also includes a lipid metabolism feature prediction module,

[0182] This application introduces a lipid metabolism feature prediction module into a multimodal prediction model to reveal the relationship between abnormal lipid metabolism and the development and progression of liver cancer patients. Based on lipid metabolism-related genes in transcriptome data, this module uses in-depth feature extraction and analysis to predict the expression levels of lipid metabolism-related markers, providing auxiliary decision support for personalized treatment. The technical implementation is as follows:

[0183] Identification and feature extraction of lipid metabolism-related genes:

[0184] During the transcriptome data preprocessing phase, bioinformatics methods were used to identify lipid metabolism genes associated with liver cancer development. These genes include key genes in metabolic pathways such as lipid synthesis, degradation, and transport, such as cholesterol synthesis genes and fatty acid synthases. The selected lipid metabolism gene data were further cleaned and standardized to ensure the consistency and reliability of the input data. Furthermore, dimensionality reduction techniques (such as Unified Mapping) were used to extract the most representative lipid metabolism features to reduce noise interference and improve model stability.

[0185] Feature prediction model design:

[0186] The lipid metabolism signature prediction module utilizes a multi-layer neural network architecture. The input layer receives processed lipid metabolism gene expression data. The hidden layer transforms the features using a nonlinear activation function to extract potential metabolic signature patterns. A softmax layer predicts the expression of various lipid metabolism markers, and the model outputs the expression probability of lipid metabolism-related markers. The resulting probability distribution reflects the degree of lipid metabolism abnormality in patients, providing a visual reference for clinical metabolic status.

[0187] Prediction of abnormal lipid metabolism status and personalized decision support:

[0188] This module, combined with lipid metabolism gene expression profiles, can accurately identify liver cancer patients with abnormal lipid metabolism. These patients may have different prognoses and treatment needs. The model's predictions can help physicians identify high-risk patients and tailor more precise treatment plans for them. For example, in patients with abnormal lipid metabolism, specific lipid metabolism inhibitors or nutritional manipulation may be potential therapeutic strategies. The model's predictions provide a basis for selecting such treatment options and enable more personalized therapeutic interventions in clinical practice.

[0189] 1) Multimodal Deep Learning Fusion Model (Multimodal Prediction Model): This application proposes a multimodal deep learning fusion model that integrates pathology data, transcriptome data, methylation data, and clinical data to achieve accurate diagnosis and prognosis prediction for liver cancer. This model further optimizes the fusion effect through feature weighting and layer normalization, providing higher accuracy and reliability for liver cancer diagnosis.

[0190] (2) UMAP dimensionality reduction mechanism: This application uses the UMAP (Uniform Manifold Approximation and Projection) algorithm to reduce the dimensionality of high-dimensional data, preserving the local topological structure of multimodal data. Through UMAP dimensionality reduction, each modal data can still maintain its dense distribution of features after dimensionality reduction, allowing the model to efficiently capture the potential feature relationships of the data in a low-dimensional space. This mechanism reduces data redundancy and effectively reduces computational complexity, providing a more concise and information-rich feature representation for subsequent feature fusion and classification.

[0191] (3) Feature weighting mechanism: This application introduces a feature weighting mechanism and uses the Softmax algorithm to calculate the weight of each modal feature to ensure that the relative importance of features can be effectively considered when fusing data from different modalities, thereby improving the overall performance of the model.

[0192] (4) Prediction of lipid metabolism-related characteristics: In combination with the multifactorial influence of lipid metabolism on liver cancer, this application uses a model to predict the expression of lipid metabolism-related markers, providing a new research perspective on the role of abnormal lipid metabolism in liver cancer and contributing to clinical precision treatment.

[0193] (5) Flexibility of model structure: The system has a flexible structure that can adjust the modal input of the model according to actual needs. It is suitable for various types of tumor diagnosis and other medical tasks, which improves the versatility and scalability of the model.

[0194] In summary, multimodal prediction models can provide doctors with diagnostic recommendations and prognostic predictions in real time, helping them better formulate treatment plans and make informed decisions. Doctors can combine the predictions from multimodal prediction models with the patient's clinical information to develop more precise treatment plans. Furthermore, multimodal prediction models can be used for early cancer screening and prevention, improving people's quality of life and health. Integrating multimodal prediction models with clinical workflows provides doctors with a convenient, fast, and accurate diagnostic support system. This integrated approach has the potential to improve doctors' work efficiency and better adapt multimodal prediction models to actual clinical needs.

[0195] This application will establish a liver cancer diagnosis classification, prognosis prediction, and lipid metabolism-related marker expression prediction system based on pathological images. In response to the problem of single data in traditional diagnostic methods, deep learning technology is used to effectively integrate pathological genomics, transcriptomics, methylation, and clinical prognosis data, which can more comprehensively reflect the characteristics of the tumor, and is expected to improve the accuracy of auxiliary diagnosis and provide clinicians with more comprehensive and in-depth pathological information. In response to the heterogeneity between multimodal data, technical means are used to eliminate differences and conflicts, ensure the effectiveness and stability of the model, and adaptively adjust different modal data. Technical means are used to accelerate model convergence, expand the training data set, and improve the accuracy and generalization ability of the multimodal prediction model, so that the multimodal prediction model can process data from multiple modalities at the same time and has stronger feature extraction and classification capabilities. The multimodal deep learning model is combined with the clinical workflow to provide doctors with real-time diagnostic advice and prognosis prediction results, which can adapt to the needs of different clinical scenarios.

[0196] The multimodal deep learning model (multimodal prediction model) of this application has important application value in clinical liver cancer management. By combining pathological data, transcriptome data, methylation data and clinical data, the model can provide doctors with more accurate and comprehensive liver cancer diagnostic information. Especially in the early diagnosis of liver cancer, the model uses the complementarity of multimodal data to improve the sensitivity and specificity of detection and reduce the possibility of misdiagnosis and missed diagnosis. For prognosis prediction, the model can provide detailed prognostic information through multimodal data analysis to assist doctors in formulating more personalized treatment plans.

[0197] In addition, this application also includes a predictive function for lipid metabolism characteristics, which provides a new tool for studying the relationship between abnormal lipid metabolism and the development of liver cancer. This function not only helps to identify potential high-risk patients, but also provides a reference for treatment plans targeting lipid metabolism. This multimodal deep learning model has strong clinical promotion potential and can be embedded in the hospital's clinical workflow to help doctors diagnose and manage liver cancer patients more accurately and efficiently in actual work. By providing high-quality auxiliary diagnostic information, this application not only reduces the workload of doctors, but also supports the refined development of clinical liver cancer management and improves patients' survival rate and quality of life.

[0198] Figure 2 This is a logic diagram of the liver cancer analysis system based on multimodal deep learning in an embodiment of the present application. Figure 3 This is a block diagram of a liver cancer analysis system 300 based on multimodal deep learning according to an embodiment of the present application. According to an embodiment of the present application, the liver cancer analysis system 300 based on multimodal deep learning includes: an input unit 301, a preprocessing unit 302, a hierarchical processing unit 303, a fusion processing unit 304, and an analysis unit 305.

[0199] Input unit 301, for obtaining pathological data, transcriptome data, methylation data and clinical data of a target patient;

[0200] a preprocessing unit 302 for preprocessing the pathological data, the transcriptome data, the methylation data, and the clinical data, respectively, to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data;

[0201] a hierarchical processing unit 303 for performing feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data;

[0202] The fusion processing unit 304 is used to perform feature fusion on the pathological features, the transcriptome features, the methylation features and the clinical features to obtain the liver cancer analysis features of the target patient.

[0203] The analysis unit 305 extracts lipid metabolism-related marker features of the target patient based on the methylation data, and performs a comprehensive liver cancer analysis on the target patient according to the lipid metabolism-related marker features and the liver cancer analysis features.

[0204] As another aspect, the present application further provides a computer-readable storage medium having stored thereon a program product capable of implementing the methods provided above in this specification. In some possible implementations, various aspects of the present application may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to perform the steps described in the "Example Method" section above in accordance with various exemplary embodiments of the present application.

[0205] According to an embodiment of the present application, a program product for implementing the above method can be a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0206] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0207] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0208] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0209] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0210] As another aspect, the present application also provides an electronic device capable of implementing the above method.

[0211] Those skilled in the art will appreciate that various aspects of the present application may be implemented as systems, methods, or program products. Therefore, various aspects of the present application may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0212] Refer to the following Figure 4 4 to describe an electronic device 400 according to this embodiment of the present application. Figure 4 The electronic device 400 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0213] like Figure 4As shown, electronic device 400 is implemented as a general-purpose computing device. Components of electronic device 400 may include, but are not limited to, at least one processing unit 410, at least one storage unit 420, and a bus 430 connecting various system components (including storage unit 420 and processing unit 410).

[0214] The storage unit stores program code, which can be executed by the processing unit 410, so that the processing unit 410 performs the steps described in the above "Example Method" section of this specification according to various exemplary embodiments of the present application.

[0215] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0216] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0217] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller node, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0218] The electronic device 400 can also communicate with one or more external devices 1200 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 400, and / or any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 450. Furthermore, the electronic device 400 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 400, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0219] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0220] Furthermore, the above-mentioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present application and are not intended to be limiting. It is readily understood that the processes illustrated in the above-mentioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0221] It should be understood that the present application is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be performed without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A liver cancer analysis method based on multimodal deep learning, characterized in that: The method comprises: Obtain pathological data, transcriptome data, methylation data, and clinical data of target patients; Preprocessing the pathological data, the transcriptome data, the methylation data, and the clinical data respectively to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data; performing feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data; Performing feature fusion on the pathological features, the transcriptome features, the methylation features, and the clinical features to obtain liver cancer analysis features of the target patient; extracting lipid metabolism-related marker features of the target patient based on the methylation data, and performing a comprehensive liver cancer analysis on the target patient according to the lipid metabolism-related marker features and the liver cancer analysis features; The transcriptome characteristics and the methylation characteristics are obtained by the following steps: Using a unified manifold approximation and projection method to perform feature mapping on the target transcriptome data and the target methylation data in a high-dimensional space, respectively, to obtain transcriptome mapping data and methylation mapping data in a low-dimensional space; determining the transcriptome signature based on the transcriptome mapping data, and determining the methylation signature based on the methylation mapping data; The unified manifold approximation and projection method is specifically as follows: S1, calculates the neighboring probability of data points in high-dimensional space; Wherein, S1 calculates the proximity probability of any two data points in the target transcriptome data or the target methylation data using the following formula: Where, Represents data points and data points The Euclidean distance of is a data point The nearest neighbor distance of is the preset scale parameter, is the proximity probability of any two data points; S2, calculates the similarity of data points in low-dimensional space; Wherein, S2 calculates the similarity of any two data points in the transcriptome mapping data or the methylation mapping data using the following formula: Where, and The data points and Position in low-dimensional space; and is a hyperparameter, is the similarity between any two data points; S3, determines the loss function of the mapping between high-dimensional space and low-dimensional space by the following formula: Where, is the loss function.

2. The liver cancer analysis method based on multimodal deep learning according to claim 1, characterized in that: The pathological data is a full-slice image, and the target pathological data is obtained by the following steps: Performing full-slice image segmentation on the pathological data to obtain a plurality of image blocks; Screening out target image blocks whose cell tissue area ratio is higher than a preset threshold value from each of the image blocks; Adjusting the image parameters of each target image block to adjust the image parameters to preset reference values; After the image parameters of each target image block are adjusted to the reference value, generating the target pathology data according to each target image block; The image parameters include brightness, contrast, gamma correction and saturation.

3. The liver cancer analysis method based on multimodal deep learning according to claim 2, characterized in that: The pathological characteristics are obtained by the following steps: Performing feature extraction on the target pathological data using a ResNet152 model to obtain the pathological features of the target patient; The model weights of the ResNet152 model are obtained by the following steps: A cross entropy loss function is used to perform model prediction of the target pathology data using a probability difference distribution, and a back propagation algorithm is used to calculate the gradient of the loss of the probability difference distribution relative to the ResNet152 model parameters; The model weights are obtained by using an SGD optimizer according to the gradient calculation.

4. The liver cancer analysis method based on multimodal deep learning according to claim 3, characterized in that: The liver cancer analysis characteristics of the target patient are obtained by the following steps: Inputting the pathological features, the transcriptome features, the methylation features, and the clinical features into a pre-set multimodal fusion model to obtain the liver cancer analysis features output by the multimodal fusion model; The mathematical expression of the multimodal fusion model is as follows: ； Where, is the liver cancer analysis feature output by the multimodal fusion model, is a weight matrix corresponding to the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, The representation layer normalization, f1, f2, f3 and f4 are the pathological features, the transcriptome features, the methylation features and the clinical features, respectively.

5. The liver cancer analysis method based on multimodal deep learning according to claim 4, characterized in that: The comprehensive liver cancer analysis of the target patient according to the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics includes: Liver cancer is diagnosed and prognosis is predicted for the target patient based on the lipid metabolism-related marker characteristics and the liver cancer analysis characteristics.

6. A liver cancer analysis system based on multimodal deep learning, characterized in that: The system comprises: An input unit, used to obtain pathological data, transcriptome data, methylation data, and clinical data of target patients; a preprocessing unit, configured to preprocess the pathological data, the transcriptome data, the methylation data, and the clinical data, respectively, to obtain target pathological data corresponding to the pathological data, target transcriptome data corresponding to the transcriptome data, target methylation data corresponding to the methylation data, and target clinical data corresponding to the clinical data; a hierarchical processing unit, configured to perform feature extraction on the target pathological data, the target transcriptome data, the target methylation data, and the target clinical data, respectively, to obtain pathological features corresponding to the target pathological data, transcriptome features corresponding to the target transcriptome data, methylation features corresponding to the target methylation data, and clinical features corresponding to the target clinical data; A fusion processing unit is used to fuse the pathological features, the transcriptome features, the methylation features and the clinical features to obtain the liver cancer analysis features of the target patient. an analysis unit, extracting lipid metabolism-related marker features of the target patient based on the methylation data, and performing a comprehensive liver cancer analysis on the target patient based on the lipid metabolism-related marker features and the liver cancer analysis features; The transcriptome features and the methylation features are obtained by the following steps: Using a unified manifold approximation and projection method to perform feature mapping on the target transcriptome data and the target methylation data in a high-dimensional space, respectively, to obtain transcriptome mapping data and methylation mapping data in a low-dimensional space; determining the transcriptome signature based on the transcriptome mapping data, and determining the methylation signature based on the methylation mapping data; The unified manifold approximation and projection method is specifically as follows: S1, calculates the neighboring probability of data points in high-dimensional space; Wherein, S1 can calculate the proximity probability of any two data points in the target transcriptome data or the target methylation data by the following formula: Where, Represents data points and data points The Euclidean distance of is a data point The nearest neighbor distance of is the preset scale parameter, is the proximity probability of any two data points; S2, calculates the similarity of data points in low-dimensional space; Wherein, S2 can calculate the similarity of any two data points in the transcriptome mapping data or the methylation mapping data by the following formula: Where, and The data points and Position in low-dimensional space; and is a hyperparameter, is the similarity between any two data points; S3, determines the loss function of the mapping between high-dimensional space and low-dimensional space by the following formula: Where, is the loss function.

7. A computer-readable storage medium, characterized in that At least one program code is stored in the computer-readable storage medium, and the at least one program code is loaded and executed by the processor to implement the operations performed by the method according to any one of claims 1 to 5.

8. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the method according to any one of claims 1 to 5.

Images