Pathological image molecular marker prediction method based on multi-task learning

Abstract

The invention discloses a pathological image molecular marker prediction method based on multi-task learning, and the method comprises the steps: firstly extracting patch-level features through a pre-trained CONCH large model, and mapping the patch-level features to a unified dimension through linear projection; modeling is carried out on the global context between patches through a shared multi-layer Transform encoder, and an improved MMoE structure is introduced: a plurality of expert sub-networks generate diversified high-order representation in parallel, a task specific gating network carries out dynamic weighted aggregation on expert output, and the expert output is respectively sent to a classification branch and a regression branch to generate MSI classification and TMB continuous prediction values; the training takes the cross entropy loss of the classification task and the mean square error of the regression task as the reference, and the task loss is weighted and summed through the task weight to realize joint optimization; according to the method, training and external verification are carried out on TCGA and CPTAC data sets, and an interpretive heat map and expert importance analysis can be generated in combination with attention and gating weights, so that clinical molecular diagnosis and treatment decision making are assisted.

Description

Technical Field

[0001] This invention relates to the fields of medical pathological images and bioinformatics technology applications, and in particular to a method for predicting molecular markers in pathological images based on multi-task learning. Background Technology

[0002] Capturing tumor-related changes at different levels of the molecular map helps to gain a deeper understanding of tumorigenesis and promotes the exploration and application of targeted therapies. Precision cancer treatment relies on specific gene alterations, which are usually confirmed by molecular biology detection methods, such as Sanger sequencing and fluorescence in situ hybridization. However, such tests can be a limiting factor in clinical oncology workflows because they often require long turnaround times and depend on comprehensive genomics or transcriptomics analysis capabilities. Furthermore, many medical centers struggle to routinely perform these costly tests.

[0003] Recent advances in computational pathology (CPath) have increasingly demonstrated that histological sections stained with hematoxylin and eosin (H&E) contain information related to known and potential biomarkers, and that molecular profiles can be directly inferred from digitized whole-slide images (WSI). Pioneer studies have confirmed which molecular features and solid tumor types leave a sufficiently strong histological footprint to be inferred from WSI using deep learning alone. However, manual feature engineering adds workload and is susceptible to subjective biases. Furthermore, recent studies predicting molecular markers from pathological images have largely employed classification methods, but many molecular features (such as gene expression, protein abundance, and the degree of immune infiltration) are inherently continuous variables. Binarizing them not only leads to information loss but also limits model performance improvement. In contrast, regression methods can directly utilize continuous labels, more fully capturing changes at the molecular level, thereby improving predictive accuracy and biological interpretability. In molecular biomarker prediction, clinical applications typically require definitive classification results (such as positive / negative MSI or HRD), while many tumor-related biological processes (such as immune infiltration, proliferation, and heterogeneity) naturally exhibit continuous variables, making them more suitable for modeling via regression tasks. Recent studies have shown that joint modeling of classification and regression tasks can play a complementary role in weakly supervised pathology prediction: classification tasks meet clinical diagnostic needs, while regression tasks provide additional continuous signals, helping the model capture finer-grained histological features and improving predictive performance and the biological plausibility of potential representations. Furthermore, this multi-task framework demonstrates better generalization across different cohorts, highlighting its potential application in precision pathology.

[0004] Multiple instance learning (MIL) frameworks typically involve three main steps: 1) tissue segmentation at a single fold and generating low-dimensional embeddings via a block-level feature extractor; 2) pooling the patched embeddings using an aggregator and creating representations for WSI; and 3) mapping the representations to slide-level labels via global pooling or a predictor. Attention-based MIL research is emerging; however, traditional global attention methods rely on the softmax mechanism, which requires calculating the similarity between each pixel and all other pixels. This process generates a context vector for each pixel, leading to significant time and computational resource consumption. Furthermore, exploration of multi-task learning (MTL) in CPath is relatively insufficient. Considering that the expression of different molecular biomarkers in tissues is not isolated, many may be simultaneously highly or poorly expressed, modeling biomarker prediction from WSI as a multi-task learning process is more biologically meaningful. Summary of the Invention

[0005] This invention proposes a multi-task learning-based method for predicting molecular biomarkers in pathological images. It simultaneously performs microsatellite instability (MSI) classification and tumor mutation burden (TMB) regression using whole-section pathological images. The method includes feature extraction, gating and aggregation, and a prediction module. It constructs classification and regression branches that share features and are jointly trained using a multi-expert structure. This method can handle both classification and regression tasks, improving the model's prediction accuracy and generalization ability, and possesses good interpretability, thus aiding in clinical molecular diagnosis and treatment decisions.

[0006] To achieve the above objectives, this invention provides a method for predicting molecular biomarkers in pathological images based on multi-task learning, comprising the following steps:

[0007] S1: Acquire full-view digital pathological images and biomarker expression data of patients with solid tumors;

[0008] S2: The full-view digital pathological images are preprocessed and feature extracted to obtain pathological embedding representations;

[0009] S3: The pathological embedding representation is input into a multi-task neural network combined with an attention module to obtain deep features;

[0010] S4: Input the deep features into the multi-task classification prediction module to obtain classification and regression prediction results.

[0011] Furthermore, the specific process of S1 is as follows:

[0012] Download full-view digital pathology images (WSI) and corresponding clinical information from the TCGA and CPTAC databases.

[0013] Furthermore, the specific process of S2 is as follows:

[0014] S2.1: Use the Otsu thresholding method to segment the tissue regions in each full-view digital pathology image (WSI);

[0015] S2.2: Read the WSI sampling resolution into memory and divide the organization region of each WSI into multiple non-overlapping square patch blocks;

[0016] S2.3: Set a minimum threshold for the median of each 8-bit RGB channel, and filter out valid square blocks based on the threshold to exclude square blocks that lack organization or have insufficient organization.

[0017] S2.4: The selected valid square blocks are extracted using a fine-tuned pre-trained CONCH model to obtain pathological features;

[0018] S2.5: Generate a 512-dimensional feature vector representation for each valid square patch:

[0019] Within the MIL framework, WSI is viewed as a package containing K instances, where a patch is an instance; the patch-level embedding corresponding to patch K is represented as... .

[0020] Furthermore, S3 specifically includes the following steps:

[0021] After obtaining the input features of the pathological slides, a pre-trained image encoder is used to extract patch-level feature representations. And mapped to a unified dimension via a linear projection layer. ;

[0022] The features are input to a shared Transformer encoder, which consists of multiple stacked Transformer encoder layers and is used to model the contextual relationships between different patches within a bag through a self-attention mechanism.

[0023] The formula for calculating the self-attention mechanism is: (1);

[0024] Where Q, K, and V are the query matrix, key matrix, and value matrix obtained from the input features through linear transformation, respectively. The dimension of the key vector;

[0025] Through self-attention, the model can capture global dependencies between patches, obtain context-enhanced feature representations, and generate bag-level representation vectors after average pooling. The bag-level representation is input into multiple expert sub-networks, each of which employs a Transformer-based nonlinear mapping module. The input to each expert sub-network is a single vector, and its self-attention degenerates into residual connections and a feedforward network to achieve higher-order feature mapping.

[0026] The mapping process is formalized as follows: (2);

[0027] in, Represents the mapping function of the i-th expert subnetwork, and outputs... ;

[0028] The outputs of all experts are stacked to form a tensor: (3);

[0029] Where K is the number of experts;

[0030] The tensor serves as the input to a task-specific gating network for subsequent tasks, enabling dynamic feature aggregation for classification and regression tasks.

[0031] By employing an improved Mixture-of-Experts (MMoE) structure, each expert sub-network can learn different feature transformation patterns, thereby enhancing the diversity of feature representations. Simultaneously, the expert networks share underlying structural parameters, reducing model complexity while improving training efficiency. Furthermore, combined with task-specific gating mechanisms, different tasks (such as MSI classification and TMB regression) can adaptively select and weight expert outputs according to their own needs, achieving knowledge sharing and differentiated representation between tasks.

[0032] Furthermore, S4 includes the following steps:

[0033] The multi-task classification prediction module includes a classification task-specific layer, a regression task-specific layer, and a multi-task loss function;

[0034] The classification task-specific layer includes a gating network branch and a classification task network branch. The classification task output is calculated by weighting the gating weights with the expert output using the hybrid features of the classification experts. (4);

[0035] in It is the feature representation after the two branches are combined. The distribution of attention among the experts; The regression task-specific layer includes a gating network branch and a regression task network branch. The regression task output is calculated by weighting the regression expert hybrid features with the gating weights and the expert output. (5);

[0036] The multi-task loss function is composed of a weighted sum of the classification task loss and the regression task loss, with the classification task loss employing the cross-entropy loss function. (6);

[0037] For markers belonging to classification tasks, the model outputs the predicted class probability. ,in This represents the number of categories for task t. The true label of the b-th sample under task t is obtained using one-hot encoding;

[0038] The regression task loss uses mean squared error: (7);

[0039] in These are true, continuous labeled values;

[0040] The total loss for all tasks is constructed using a weighted sum: (8).

[0041] The design concept of this invention is as follows:

[0042] This invention aims to simultaneously classify microsatellite instability (MSI) and regress tumor mutational burden (TMB) from unlabeled H&E whole-slice images, thereby improving the accuracy, interpretability, and clinical accessibility of molecular predictions. The method first uses a pre-trained CONCH large model to extract patch-level features, which are then linearly projected to a unified dimension. Subsequently, a shared multi-layer Transformer encoder models the global context between patches. Based on this representation, an improved MMoE (Mixture-of-Experts) structure is introduced: several expert sub-networks generate diverse high-order representations in parallel, and a task-specific gating network dynamically weights and aggregates the expert outputs, feeding them into the classification and regression branches respectively to produce continuous predictions of MSI classification and TMB. Training is based on the cross-entropy loss of the classification task and the mean squared error of the regression task. Joint optimization is achieved by weighted summation of the losses of each task using task weights, supporting fixed or dynamic weighting strategies to balance the influence between tasks. This method is trained and externally validated on the TCGA and CPTAC datasets. By combining attention and gating weights, it can generate interpretive heatmaps and expert importance analysis, thereby assisting in clinical molecular diagnosis and treatment decisions.

[0043] The beneficial effects of this invention are as follows:

[0044] 1. Implement multi-task joint learning to improve prediction performance and efficiency.

[0045] 2. Supports dynamic feature aggregation for classification and regression, improving model generalization ability. Regression tasks directly learn from continuous values, preserving the information structure of the original distribution of markers, reducing information loss and the impact of label noise, and improving the stability and generalization of the model.

[0046] 3. Introduce the Transformer encoder into the MMoE architecture to model the global relationships of pathological slides and enhance feature representation capabilities. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall framework and process of the present invention;

[0048] Figure 2 This is a visualization example of the results of the method of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] This invention includes the following steps:

[0051] S1: Pathological image data preprocessing and feature extraction

[0052] Download the necessary WSI and clinical information for the experiment from The Cancer Genome Atlas (TCGA) and The Clinical Proteomic Tumor Analysis Consortium (CPTAC) databases.

[0053] In the preprocessing stage, the tissue region in each WSI is automatically segmented using the traditional Otsu thresholding method. The WSI is read into memory at a subsampling resolution (default 64× downscaling), and the tissue region of each WSI is then divided into non-overlapping square blocks with an edge length of 360 μm, resized to 224×224 pixels. Patches lacking tissue or with minimal tissue are excluded by setting a minimum threshold of 50 for the median of each 8-bit RGB channel. After patch segmentation, a finely tuned pre-trained CONCH large model is used to extract pathological features from the preprocessed full-view digital pathology image. A 512-dimensional feature vector representation is generated for each patch. Our analysis is performed within the MIL framework, where the WSI is considered a package and the patch is an instance. Here, for a WSI represented as a package of K instances, we represent the patch-level embedding corresponding to patch K as follows: .

[0054] S2: A Multi-Task Feature Extraction Method Based on Shared Transformer and Hybrid Expert Structure

[0055] After obtaining the input features of the pathological slides, a pre-trained image encoder is used to extract patch-level feature representations. And mapped to a unified dimension via a linear projection layer. Subsequently, the features are input to a shared Transformer encoder, which consists of multiple stacked Transformer encoder layers, used to model the contextual relationships between different patches within a bag through a self-attention mechanism. Specifically, the calculation formula for the self-attention mechanism is as follows: (1);

[0056] Where Q, K, and V are the query matrix, key matrix, and value matrix obtained from the input features through linear transformation, respectively. This represents the dimension of the key vector. Through this mechanism, the model can capture global dependencies between patches, obtaining context-enhanced feature representations. After average pooling, bag-level representation vectors are generated. Next, the bag-level representation is input into multiple expert subnetworks, each of which is a Transformer-based nonlinear mapping module. In this embodiment, since the input is a single vector, the self-attention of the expert subnetwork degenerates into residual connections and a feedforward network, whose core function is to achieve high-order feature mapping. The mapping process can be formalized as follows: (2);

[0057] in, Represents the mapping function of the i-th expert subnetwork, and outputs... The outputs of all experts are stacked to form a tensor: (3);

[0058] Where K is the number of experts. This tensor serves as the input to subsequent task-specific gating networks (class gate and regression gate) to achieve dynamic feature aggregation for classification and regression tasks.

[0059] By employing an improved Mixture-of-Experts (MMoE) structure, each expert sub-network can learn different feature transformation patterns, thereby enhancing the diversity of feature representations. Simultaneously, the expert networks share underlying structural parameters, reducing model complexity while improving training efficiency. Furthermore, combined with task-specific gating mechanisms, different tasks (such as MSI classification and TMB regression) can adaptively select and weight expert outputs according to their own needs, achieving knowledge sharing and differentiated representation between tasks.

[0060] S3: Multi-task classification prediction module

[0061] The classification task-specific layer includes a gating network branch and a classification task network branch. The classification task output is calculated by weighting the gating weights with the expert output using the hybrid features of the classification experts. (4);

[0062] in It is the feature representation after the two branches are combined. The attention distribution of each expert. The regression task-specific layer includes a gating network branch and a regression task network branch. The regression task output is calculated by weighting the regression expert hybrid features with the gating weights and the expert outputs. (5);

[0063] The multi-task loss function consists of a weighted sum of classification loss and regression loss, characterized in that the classification task uses the cross-entropy loss function: (6);

[0064] For markers belonging to classification tasks, the model outputs the predicted class probability. ,in This represents the number of categories for task t. The true label of the b-th sample under task t is obtained using one-hot encoding.

[0065] The regression task loss uses mean squared error: (7);

[0066] in These are true, continuous labeled values.

[0067] The total loss for all tasks is constructed using a weighted sum: (8);

[0068] The objective of this invention is to predict clinically relevant characteristics, including gene expression signatures and biomarkers, directly from histological sections. Clinically actionable molecular alterations were initially planned from systematic study results using TCGA data (https: / / portal.gdc.cancer.gov / ), and an overview of the two TCGA datasets used is included. For certain biomarkers, consensus was used to plot molecular binary labels; for example, considering microsatellite instability (MSI), all patients defined as MSI-h were classified as positive, while patients with microsatellite stable (MSS) and MSI-l were labeled as negative. The values ​​of the continuous biomarker tumor mutation burden (TMB) were represented using actual molecular data. For external validation, colorectal cancer tissue samples were obtained from the CPTAC database, and matching molecular analysis data were collected using the GDC API (https: / / gdc.cancer.gov / developers / gdc-application-programming-interface-api).

[0069] The training and validation datasets were derived from two solid tumor datasets from TCGA, containing 724 pathological sections from 724 patients and corresponding molecular expression information. The external testing dataset consisted of 110 WSIs from 110 colorectal cancer patients in the CPTAC database, along with corresponding MSI and TMB molecular expression information. Since patient diagnostic information is provided at the patient level, rather than on individual sections, all WSIs corresponding to patient cases were treated as a single WSI package during training and validation. The model was trained for a minimum of 50 epochs and a maximum of 200 epochs. After the initial 50 epochs, if the validation set loss (i.e., the sum of losses across all tasks) had not decreased for 20 consecutive epochs, stopping was triggered, and the best model with the lowest validation loss was used to report performance on the validation set. Five-fold cross-validation was used to further evaluate the robustness of model training, where the best model was selected based on performance on the validation set. Model parameters were updated using the Adam optimizer with a learning rate of 1e-2 and... Weight decay was 4e-5. A Dropout layer with α=0.25 was used after each hidden layer to avoid overfitting. The performance of the optimal patient-level classifier in five-fold cross-validation was further examined on an internal test cohort and an external test cohort. Receiver operating characteristic (ROC) analysis was used, and performance was evaluated using the area under the ROC curve (AUC). For each biomarker, performance was reported as the average AUC over 5 folds, and the standard deviation was calculated. Figure 2 ).

[0070] The molecular biomarker prediction method based on multi-task learning proposed in this invention shows significant performance improvements in both MSI classification and TMB regression tasks. The mean area under the receiver operating characteristic (AUROC) curve for MSI state prediction using the model of this invention reaches 0.831 (95% CI: 0.739–0.889), and the five-fold cross-validation results show that the AUC of each fold remains between 0.73 and 0.89 (e.g., ...). Figure 2 This verified the model's generalization performance under different sample distributions.

[0071] Table 1 ;

[0072] Meanwhile, the present invention achieved excellent performance in all four performance indicators (as shown in Table 1), which fully verifies the superiority and stability of the present invention in multi-task joint learning and feature sharing.

Claims

1. A method for predicting molecular biomarkers in pathological images based on multi-task learning, characterized in that, Includes the following steps: S1: Acquire full-view digital pathological images and biomarker expression data of patients with solid tumors; S2: Preprocess and extract features from the full-view digital pathological image to obtain pathological embedding representations; S3: The pathological embedding representation is input into a multi-task neural network combined with an attention module to obtain deep features; S4: Input the deep features into the multi-task classification prediction module to obtain the classification and regression prediction results.

2. The method for obtaining full-view digital pathological images and biomarker expression data of solid tumor patients according to claim 1, characterized in that: The specific process of S1 is as follows: Download full-view digital pathology images (WSI) and corresponding clinical information from the TCGA and CPTAC databases.

3. The method for obtaining full-field digital pathological images and biomarker expression data of solid tumor patients according to claim 1, characterized in that: The specific process of S2 is as follows: S2.1: Use the Otsu thresholding method to segment the tissue regions in each full-view digital pathology image (WSI); S2.2: Read the WSI sub-sampling resolution into memory and divide the organization region of each WSI into multiple non-overlapping square image patches; S2.3: Set a minimum threshold for the median of each 8-bit RGB channel, and filter out valid square image blocks based on the threshold to exclude square image blocks that lack organization or have insufficient organization. S2.4: Using the pre-trained CONCH model, extract the selected valid square image patches to obtain pathological features; S2.5: Generate a 512-dimensional feature vector representation for each valid square image patch: Within the MIL framework, WSI is viewed as a package containing K instances, where a patch is an instance; the patch-level embedding corresponding to patch K is represented as... .

4. The method for predicting molecular biomarkers in pathological images based on multi-task learning according to claim 1, characterized in that, S3 specifically includes the following steps: After obtaining the input features of the pathological slides, a pre-trained image encoder is used to extract patch-level feature representations. And mapped to a unified dimension via a linear projection layer. , This indicates the batch size, which is the number of pathological slices input into the model at one time. This indicates the number of image patches obtained after dividing the entire pathological whole slice into WSI sections; This represents the feature dimension of the model, that is, the dimension of the feature vector of each patch after linear projection; The features are input to a shared Transformer encoder, which consists of multiple stacked Transformer encoder layers, and is used to model the contextual relationships between different patches within a bag through a self-attention mechanism. The formula for calculating the self-attention mechanism is: (1) Where Q, K, and V are the query matrix, key matrix, and value matrix obtained from the input features through linear transformation, respectively. The dimension of the key vector; Through self-attention, the overall model architecture can capture global dependencies between patches, obtain context-enhanced feature representations, and generate bag-level representation vectors after average pooling. The bag-level representation is input into multiple expert sub-networks, each of which employs a Transformer-based nonlinear mapping module. The input to each expert sub-network is a single vector, and its self-attention degenerates into residual connections and a feedforward network to achieve higher-order feature mapping. The mapping process is formalized as follows: (2) in, Represents the mapping function of the i-th expert subnetwork, and outputs... ; The outputs of all experts are stacked to form a tensor: (3) Where K is the number of experts; The tensor serves as the input to a task-specific gating network for subsequent tasks, enabling dynamic feature aggregation for classification and regression tasks.

5. The method for predicting molecular biomarkers in pathological images based on multi-task learning according to claim 1, characterized in that, S4 includes the following steps: The multi-task classification prediction module includes a classification task-specific layer, a regression task-specific layer, and a multi-task loss function; The classification task-specific layer includes a gating network branch and a classification task network branch. The classification task calculates a weighted sum of the gating weights and the expert output through the classification expert hybrid feature branch: (4) in It is the feature representation after the two branches are combined. The distribution of attention among the experts; Indicates the first Feature representations output by each expert subnetwork; The regression task-specific layer includes a gating network branch and a regression task network branch. The regression task calculates a weighted sum of the gating weights and the expert output through the regression expert hybrid feature branch: (5) The multi-task loss function is composed of a weighted sum of the classification task loss and the regression task loss, with the classification task loss employing the cross-entropy loss function. (6) For markers belonging to classification tasks, the model outputs the predicted class probability. in The number of categories representing task t. The true label of the b-th sample under task t. : No. The indexes for each category use one-hot encoding; The regression task loss uses mean squared error: (7) in These are true, continuous labeled values; The total loss for all tasks is constructed using a weighted sum: (8)。

Images