Colorectal cancer transcriptomic feature inference system across modalities based on he-stained sections
By integrating pathological images, transcriptomic data, and clinical data through multi-head attention mechanisms and deep learning, the problems of data fusion and interpretability in multimodal colorectal cancer prognostic assessment have been solved, enabling more accurate predictions and assisting in personalized treatment plans.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHIMEDICAL UNIVERSITY
- Filing Date
- 2026-03-31
- Publication Date
- 2026-07-03
AI Technical Summary
Existing multimodal colorectal cancer prognostic assessment models have shortcomings in data fusion, failing to effectively uncover deep correlations between modalities, and the deep learning models lack interpretability, making it difficult for doctors to trust the prediction results and apply them in clinical practice.
A cross-modal colorectal cancer transcriptome feature inference system based on HE-stained slides was adopted. The system integrates pathological images, transcriptome data and clinical data through multi-head attention mechanism, and combines k-means clustering and deep learning to generate interpretable prognostic indicators and provide standardized reports.
It improves the accuracy and transparency of prognostic assessment, simplifies the operational process, makes the model more credible and convenient in clinical applications, and supports the development of personalized treatment plans.
Smart Images
Figure CN122337549A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of clinical medical technology, specifically to a transmodal inference system for colon cancer transcriptome features based on HE-stained sections. Background Technology
[0002] In the field of colorectal cancer prognostic assessment, current technologies have long relied heavily on single-dimensional clinical parameters such as tumor staging and histological grade. Tumor staging is based on tumor size, depth of invasion, lymph node metastasis, and distant metastasis, while histological grade assesses the degree of malignancy by the differentiation of cancer cells. Although these data are widely used in clinical diagnosis and treatment, they can only provide a preliminary assessment of the disease status at a macroscopic level. Because the development and progression of colorectal cancer involves complex biological processes such as multi-gene regulation, abnormal cell metabolism, and changes in the tumor microenvironment, single-dimensional clinical parameters cannot fully explain the complex biological mechanisms of the disease, resulting in significant biases in predictive results and failing to accurately guide the formulation of clinical treatment plans and the assessment of patient survival prognosis.
[0003] In recent years, with the rapid development of high-throughput sequencing and microscopic imaging technologies, transcriptome data and pathological images have been increasingly integrated into the construction of colorectal cancer prognostic models. Transcriptome data can deeply analyze biological signaling pathways within tumor cells, revealing the molecular mechanisms of tumor occurrence, development, and metastasis at the gene expression level; pathological images, on the other hand, can visually present the morphological characteristics of tumors through the morphology, arrangement, and tissue structure of tumor cells under a microscope. Against this backdrop, multimodal models have emerged, which, by integrating clinical indicators, imaging information, and genomic data, have improved the accuracy of colorectal cancer prognostic prediction to a certain extent, providing more references for clinical decision-making.
[0004] However, existing multimodal modeling technologies still have many problems and shortcomings. Firstly, in terms of data fusion, a simple data splicing strategy is commonly used, mechanically combining data from different modalities without considering the deep connections such as spatiotemporal correlations and causal relationships between clinical, imaging, and genomic data. This makes it difficult to uncover deep correlations between modalities, resulting in poor data fusion effects and failing to fully leverage the synergistic advantages of multimodal data. Secondly, deep learning models, as the core technology of multimodal analysis, generally suffer from the "black box" problem. The features and patterns learned by the model through a large amount of data during training are difficult to interpret in a way that is understandable to humans, failing to meet the stringent clinical requirements for model interpretability. This makes doctors hesitant to trust the model's predictions, limiting its widespread clinical application. Thirdly, most current multimodal colorectal cancer prognostic research results remain at the academic level. In practical applications, they lack user-friendly interfaces and standardized output formats. Doctors face problems such as complex operation and difficulty in interpreting results, making it difficult to truly transform them into practical clinical tools and effectively assist in optimizing clinical diagnosis and treatment processes.
[0005] Therefore, in response to the problems of insufficient multimodal data fusion, poor model interpretability, and difficulties in clinical translation in the existing technology for colorectal cancer prognostic assessment, this invention is committed to developing a colorectal cancer multimodal prognostic assessment system that can efficiently fuse multi-source data, enhance model transparency, and improve the convenience of clinical application. This system will provide colorectal cancer patients with more accurate prognostic predictions, assist clinicians in developing personalized treatment plans, and promote the development of precision medicine for colorectal cancer. Summary of the Invention
[0006] The purpose of this invention is to provide a transmodal inference system for colon cancer transcriptome features based on HE-stained sections, in order to solve the above-mentioned problems.
[0007] This invention provides the following technical solution: a system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections, comprising: The human-computer interaction module is used by users to upload clinical data, pathological images, and transcriptome data, and to select model parameters; The core backend program module is used to process and extract data features from clinical data, pathological images, and transcriptome data uploaded by users. The multimodal large model module is used to receive data features extracted by the backend core program module, integrate and cluster them, and generate prognostic indicators based on the integrated features and clustering results. The output report generation module receives the prediction results and outputs a standardized report, including patient information, prediction results, and clinical recommendations.
[0008] Preferably, the background core program module includes a clinical data processing module, which is used to receive and process the clinical data, check whether the missing values are randomly missing after data cleaning and fill them with the average value, and then extract at least three features such as survival time, survival status and cancer stage through z-score standardization, and output them to the clinical data input terminal of the multimodal large model.
[0009] Preferably, the background core program module includes a transcriptome data processing module, which is used to receive and process transcriptome data, screen out n key pathway features through PCA dimensionality reduction unit, and connect to the transcriptome data input terminal of the multimodal large model.
[0010] Preferably, the background core program module includes a pathological image processing module, which is used to receive and process pathological images, adjust the resolution of the pathological images, extract m image features using ResNet50, and connect to the pathological image data input terminal of the multimodal large model.
[0011] Preferably, the multimodal large model module consists of a feature integration layer, a clustering analysis unit, and a prediction model, wherein: The feature integration layer receives feature inputs from the backend core program module, transcriptome data processing module, and pathological image processing module. It integrates features using a multi-head attention mechanism, dynamically weights and captures complex interactions between modalities. Cluster analysis units are grouped using the k-means algorithm; The predictive model, based on integrated features and clustering results, is trained using deep learning to generate prognostic indicators.
[0012] Preferably, the multi-head attention mechanism dynamically calculates the association weights between pathological features and transcriptomic features as follows: In this context, the Q-query vector represents pathological image features; the K-bond vector represents transcriptomic features; and the V-value vector represents the fused multimodal features, generated by concatenating pathological and transcriptomic features through a fully connected layer. The dimension scaling factor. scaling factor; This is the attention weight matrix; the specific process is as follows: The pathological feature Q and the gene feature K are similar to each other by matrix multiplication to generate an attention weight matrix for feature alignment. Dynamic weighting is performed using normalized weights; Multiply the weighted weight matrix by V to generate the final fused features; Modeling the causal relationship between clinical parameters and molecular characteristics using graph neural networks.
[0013] Preferably, the fused features are input into the risk model, which outputs the patient's survival probability and risk score. A risk score threshold is set, and an early warning is triggered when the threshold is exceeded.
[0014] Preferably, clinical data is in CSV format, pathological images are in JPG / PNG / SVS format, and transcriptome data is in TXT / CSV format.
[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention overcomes the limitations of a single data source, more comprehensively capturing the physiological and pathological characteristics of patients and significantly improving prognostic accuracy. It constructs a multi-head attention mechanism, using multiple parallel attention heads to focus on the interaction relationships between different modalities, enhancing the model's sensitivity to key biological signals. The multimodal colorectal cancer prognostic model is encapsulated into a complete system platform, easily operated by users without professional backgrounds, promoting widespread medical adoption. A bidirectional interpretation mechanism is established, allowing users to infer key pathway issues through the model's pre-training results without expensive high-throughput sequencing, reducing the economic burden on patients. Professional test reports are generated based on patient input information, and tumor heterogeneity is inferred from pathological images, accurately predicting the cancer status of the three modalities and achieving precision medicine. Attached Figure Description
[0016] Figure 1 This is a system model architecture diagram of the present invention; Figure 2 This is a flowchart of the data processing and model training process of this invention; Figure 3 This is a block diagram of the model of the present invention. Detailed Implementation
[0017] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0018] refer to Figure 1 This invention provides a cross-modal transcriptome feature inference system for colorectal cancer based on HE-stained sections. The specific architecture is as follows: Figure 1 As shown, the present invention will now be described in terms of its main modules, including: refer to Figure 3 The system consists of a human-computer interaction module for users to upload clinical data, pathological images, and transcriptome data, and select model parameters; a backend core program module for processing the uploaded clinical data, pathological images, and transcriptome data and extracting data features; a multimodal large model module for receiving the data features extracted by the backend core program module, integrating and clustering them, and generating prognostic indicators based on the integrated features and clustering results; and an output report generation module for receiving prediction results and outputting standardized reports, including patient information, prediction results, and clinical recommendations.
[0019] Front-end human-computer interaction interface: Designed as an intuitive and easy-to-use graphical user interface (GUI), users submit clinical data in CSV format, pathological images in JPG / PNG / SVS format, and transcriptome data in TXT / CSV format through the data upload module, and select model parameters (such as the number of clusters and the number of attention heads) through the parameter setting module. No parameter settings are required for ordinary users; pre-trained optimal parameters are used by default. The interface connects to the backend core program via a network interface, and data and parameters are transmitted to the backend in real time. The results are then returned to the interface for display from the backend.
[0020] refer to Figure 2 The core backend program modules include a clinical data processing module, which receives and processes the clinical data. After data cleaning, it checks for random missing values and fills them with the mean value. Then, it uses z-score standardization to extract at least three features: survival time, survival status, and cancer stage, and outputs the data to the clinical data input terminal of the multimodal large-scale model. The core backend program modules also include a transcriptome data processing module, which receives and processes transcriptome data. It uses PCA dimensionality reduction to filter out 3024 key pathway features and connects them to the transcriptome data input terminal of the multimodal large-scale model. Finally, the core backend program modules include a pathology image processing module, which receives and processes pathology images. It adjusts the resolution of the pathology images to 224×224, extracts 1000 image features using ResNet50, and connects them to the pathology image data input terminal of the multimodal large-scale model. The multimodal large model module consists of a feature integration layer, a clustering analysis unit, and a prediction model. Specifically: the feature integration layer receives feature inputs from the backend core program module, transcriptome data processing module, and pathological image processing module, and integrates features using a multi-head attention mechanism, dynamically weighting and capturing complex interactions between modalities; the clustering analysis unit uses the k-means algorithm for grouping; and the prediction model, based on the integrated features and clustering results, is trained using deep learning to generate prognostic indicators. The output report generation module receives the backend prediction results, learns from the LLAM3 language model, and outputs a standardized report containing patient information, prediction results (such as survival probability and risk score), and clinical recommendations.
[0021] The basic working principle of this invention is as follows: Data processing: Users upload data through the front end, which is then transmitted to the core program in the back end. Clinical data is cleaned, missing values are filled, and z-scores are standardized to generate key features; transcriptome data is subjected to PCA dimensionality reduction to extract pathway features; pathological images are adjusted in resolution and then image features are extracted using ResNet50 to form multimodal feature input.
[0022] Feature integration process: The feature integration layer of the multimodal large model applies a multi-head attention mechanism, which processes the features of the three modalities in parallel through multiple attention heads, dynamically allocates weights, captures the biological dependencies between modalities, and generates fused features.
[0023] Clustering analysis process: Integrate features into clustering analysis units, and use the k-means algorithm for group training.
[0024] Model training and prediction process: The prediction model is trained based on fusion features and clustering results, learns the mapping relationship between features and prognostic results, outputs survival probability and risk score, and generates reports according to user needs, realizing intelligent answers and analysis of disease status.
[0025] The entire system is designed to ensure efficient operation and supports the integration of new modal data or algorithm upgrades.
[0026] To better understand this invention, the following examples further illustrate it: Data input: A pathology slide scanner scans HE-stained slides (color slides viewed under a microscope) of colon cancer in the hospital into electronic images. Genetic testing equipment provides gene expression data from tumor tissue. An electronic medical record system directly retrieves basic information such as the patient's age, tumor stage, and survival time.
[0027] Extract key information from three types of data, and piece together information from different sources like a piecing together... Figure 1 These data are integrated to determine the patient's survival probability and risk level based on the integrated information. This is achieved through three modules: a feature extractor, a cross-modal fusion unit, and a prognostic prediction module.
[0028] Data processing flow: Adjust the scanned pathology images to a uniform resolution (224×224 pixels), remove noise, and highlight the tumor area; Use an improved ResNet50 model (an image recognition algorithm) to extract key features from images, such as the distribution of densely populated areas of cancer cells and tissue texture. Gene expression data were compressed to 3024 key signaling pathways; the top x genes with the largest fluctuations were retained as the core clues for subsequent analysis. Fill in missing values and replace them with the average value; Convert indicators from different units into a unified standard.
[0029] Cross-modal fusion: Employs a multi-head attention mechanism.
[0030] The multi-head attention mechanism dynamically calculates the association weights between pathological features and transcriptomic features as follows: In this context, the Q-query vector represents pathological image features; the K-bond vector represents transcriptomic features; and the V-value vector represents the fused multimodal features, generated by concatenating pathological and transcriptomic features through a fully connected layer. The dimension scaling factor. scaling factor; This is the attention weight matrix; the specific process is as follows: Pathological feature Q and genetic feature K are similarized using matrix multiplication to generate an attention weight matrix for feature alignment. Dynamic weighting is then applied using normalized weights. The weighted matrix is multiplied by V to generate the final fused feature. A graph neural network is used to model the causal relationship between clinical parameters and molecular features. The fused feature is input into a risk model, which outputs the patient's survival probability and risk score. A risk score threshold is set; when the threshold is exceeded, an alert is triggered.
[0031] Prognostic prediction and results output: Predictive Model: Based on the fused data, a deep learning algorithm is used to calculate the patient's survival probability and recurrence risk over the next y years. Specifically, a heatmap can be used to mark the key areas affecting the prediction in the pathology image, key gene pathways can be listed as medical explanations, and a report that conforms to doctors' habits can be generated, including treatment recommendations.
[0032] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it.
Claims
1. A system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections, characterized in that, include: The human-computer interaction module is used by users to upload clinical data, pathological images, and transcriptome data, and to select model parameters; The core backend program module is used to process and extract data features from clinical data, pathological images, and transcriptome data uploaded by users. The multimodal large model module is used to receive data features extracted by the backend core program module, integrate and cluster them, and generate prognostic indicators based on the integrated features and clustering results. The output report generation module receives the prediction results and outputs a standardized report, including patient information, prediction results, and clinical recommendations.
2. The system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections according to claim 1, characterized in that, The backend core program module includes a clinical data processing module, which is used to receive and process the clinical data. After data cleaning, it checks whether the missing values are randomly missing and fills them with the average value. Then, it extracts at least three features, namely survival time, survival status and cancer stage, through z-score standardization and outputs them to the clinical data input terminal of the multimodal large model.
3. The system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections according to claim 1, characterized in that, The background core program module includes a transcriptome data processing module, which is used to receive and process transcriptome data, screen out n key pathway features through PCA dimensionality reduction unit, and connect to the transcriptome data input terminal of the multimodal large model.
4. The system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections according to claim 1, characterized in that, The backend core program module includes a pathological image processing module, which is used to receive and process pathological images, adjust the resolution of the pathological images, extract m image features using ResNet50, and connect to the pathological image data input terminal of the multimodal large model.
5. The system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections according to claim 1, characterized in that, The multimodal large model module consists of a feature integration layer, a clustering analysis unit, and a prediction model, wherein: The feature integration layer receives feature inputs from the backend core program module, transcriptome data processing module, and pathological image processing module. It integrates features using a multi-head attention mechanism, dynamically weights and captures complex interactions between modalities. Cluster analysis units are grouped using the k-means algorithm; The predictive model, based on integrated features and clustering results, is trained using deep learning to generate prognostic indicators.
6. The system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections according to claim 5, characterized in that, The multi-head attention mechanism dynamically calculates the association weights between pathological features and transcriptomic features as follows: In this context, the Q-query vector represents pathological image features; the K-bond vector represents transcriptomic features; and the V-value vector represents the fused multimodal features, generated by concatenating pathological and transcriptomic features through a fully connected layer. The dimension scaling factor. scaling factor; This is the attention weight matrix; the specific process is as follows: The pathological feature Q and the gene feature K are similar to each other by matrix multiplication to generate an attention weight matrix for feature alignment. Dynamic weighting is performed using normalized weights; Multiply the weighted weight matrix by V to generate the final fused features; Modeling the causal relationship between clinical parameters and molecular characteristics using graph neural networks.
7. The system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections according to claim 6, characterized in that, The fused features are input into the risk model, which outputs the patient's survival probability and risk score. A risk score threshold is set, and an alert is triggered when the threshold is exceeded.
8. The system for inferring transmodal transcriptome features of colorectal cancer based on HE-stained sections according to claim 1, characterized in that, Clinical data were presented in CSV format, pathological images in JPG / PNG / SVS format, and transcriptome data in TXT / CSV format.