A breast pathological image molecular typing prediction method based on a multi-attribute embedding model

Abstract

The application is a breast pathological image molecular typing prediction method based on a multi-attribute embedding model, and the steps are as follows: step one: a multi-attribute classifier is constructed based on the deep features of the image blocks of digital pathological sections to obtain the instance-level molecular level probability distribution; step two: a feature embedding module is constructed, and the global feature is constructed based on the instance-level deep features and the strongest representation block; step three: a Transformer-MLP classifier is constructed to perform slice-level molecular typing prediction on the global feature; and step four: a two-level classification loss with multi-label constraints is constructed to jointly train the network. The application can construct the correlation between the morphological features of the breast pathological image and the immunohistochemical molecular phenotype, and simultaneously, in view of the weak label problem of the WSI data, a weakly supervised classification method based on multi-attribute feature embedding is designed to realize accurate prediction of the immunohistochemical molecular typing of breast cancer. The application can be combined with various computer pathological auxiliary analysis systems, and has wide market prospects and application value.

Description

Technical Field

[0001] This invention relates to a molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model, belonging to the fields of computational pathology, digital image processing, and computer vision. It primarily involves multi-instance learning, weakly supervised learning, and self-supervised learning. It has broad application prospects in the field of medical pathology analysis. Background Technology

[0002] As the leading cause of death among women, breast cancer can spread to vital organs and tissues such as the brain, lungs, and liver through various means, including direct spread, lymph node metastasis, and hematogenous metastasis. Therefore, timely detection and early treatment can help breast cancer patients achieve good prognoses and high survival rates, effectively preventing the tragedy of incurable cancer due to metastasis. Currently, the most accurate diagnostic method for breast cancer is pathological biopsy, which involves partially or completely removing the tumor through fine-needle aspiration or surgical resection. The specimen is then sent to the pathology department for pathological staining and related immunohistochemical examinations to determine the presence of cancer and, after diagnosis, the pathological type and molecular phenotype of breast cancer—important information closely related to subsequent treatment. With the rapid development of computational pathology, the maturity and widespread application of whole-slide image (WSI) technology have driven the digitization of pathological slides and provided an ideal platform for computer-aided analysis of breast pathology images. This has led to a surge in research on breast cancer pathology analysis using machine learning and deep learning algorithms based on histological and morphological features.

[0003] Computer-aided analysis techniques based on traditional machine learning algorithms typically utilize the abundant spatial, color, and texture information contained in pathological slide images to manually construct histopathological features relevant to breast cancer diagnosis and analysis. Belsare et al. used a spatial color texture map segmentation method to segment the periluminal epithelial cells in breast pathological images, and then designed a linear discriminant classifier based on features such as gray-level co-occurrence matrix, graph run matrix, and Euler number to distinguish between benign and malignant breast tissue (see reference: Belsare, Mushrif, Pangarkar, et al. Classification of breast cancer histopathology images using texture feature analysis. Tencon 2015 IEEE Region 10 Conference). Vu et al. proposed an automatic feature discovery framework and a low-complexity classification method for histopathological image analysis based on a dictionary learning method designed with salient features. The accuracy of this method on a pathological image dataset of breast duct lesions can reach 97.75% (see reference: THVu, HSMousavi, VMonga, et al. Histopathological image classification using discriminative feature-oriented dictionary learning. IEEE Transactions on Medical Imaging, 2015, 35(3):738-751).Anuranjeeta's morphological feature-based auxiliary analysis system achieved an accuracy of 85.7% on a dataset containing 70 breast cancer histopathological images (see reference: Anuranjeeta, K KShukla, A Tiwari, et al. Classification of histopathological images of breast cancerous and non-cancerous cells based on morphological features. Biomedical and Pharmacology Journal, 2017, 10(1):353-366). Spanhol et al. established a large public dataset, BreakHis, for breast cancer histopathological images and conducted experiments based on hand-designed features. They demonstrated that free parameter threshold adjacency statistics features have higher robustness and significance in breast pathology image classification compared to features such as local binary patterns and gray-level co-occurrence matrices (see reference: Spanhol, Oliveira, Petitjean, et al. A dataset for breast cancer histopathological image classification. IEEE Transactions on Biomedical Engineering, 2015, 63(7):1455-1462). Although the above-mentioned traditional machine learning algorithms have good classification results on relevant breast pathology image datasets, limitations such as limited application scenarios, unstable generalization performance, and the need for manual design and feature extraction have become important factors hindering their development.

[0004] Deep learning has powerful nonlinear fitting and generalization capabilities. Due to its advantages such as not requiring manual feature design and being able to directly acquire and process features through multi-layer neural networks, it has gradually become the mainstream of artificial intelligence research in breast cancer pathology analysis. Yao et al. constructed a benign and malignant classification model for breast histology pathology images using a parallel feature extractor based on convolutional neural networks and recurrent neural networks and a perceptual attention mechanism. The classification accuracy exceeded 90% in multiple breast cancer pathology datasets (see reference: Yao, Zhang, Zhou, et al. Parallel structure deep neural network using CNN and RNN with an attention mechanism for breast cancer histology image classification. Cancer, 2019, Vol. 11, 1901.). Hameed et al. proposed using fine-tuned VGG16 and VGG19 neural networks to construct an ensemble model for identifying breast cancer tissue images, and demonstrated that the ensemble network had higher classification accuracy by comparison with other methods (see reference: Hameed, Zahia, Garcia-Zapirain, et al. Breast cancer histopathology image classification using an ensemble of deep learning models. Sensors, 2020, 20(16):4373).Lu et al. constructed a graph neural network for directly predicting HER2 status in breast tissue H&E staining images based on a feature mapping model between hematoxylin-eosin (H&E) stained images and human epidermal growth factor receptor 2 (HER2) immunohistochemical images. Their robustness was demonstrated in the Cancer Genomic Atlas (TCGA) breast cancer WSI database (see: Lu, Toss, Dawood, et al. Slidegraph+: whole slide image level graphs to predict her2 status in breast cancer. Medical Image Analysis, 2022, 80: 102486). Jaber et al. designed a multi-scale image patch-slice two-stage classification algorithm based on Inception-v3 neural network and support vector machine, which achieved a molecular subtype prediction accuracy of 65.92% in breast cancer WSI data. They also discussed the possibility of automatic segmentation of breast pathology images with heterogeneous molecular subtypes (see reference: Jaber, Song, Taylor, et al. A deep learning image-based intrinsic molecular subtype classifier of breast tumors reveals tumor heterogeneity that may affect survival. Breast Cancer Research, 2020, 22(1):1-10).

[0005] In summary, traditional machine learning algorithms, based on manually constructed features, have limitations in extracting information from breast pathology images, making it difficult to extract crucial information relevant at the molecular level. However, with the rapid development of deep learning technology and the widespread availability of WSI data, it has become possible to construct pathology image feature representations strongly correlated with molecular subtyping information using neural network models. However, due to the high resolution, large size, complex information, and weak slice-level labels of WSI data, deep learning often requires image cropping or reduction when processing such data. These preprocessing methods can easily lead to the loss of important global features by the deep learning model, reducing its classification accuracy. Furthermore, effectively utilizing this complex information without over-reliance on the constraints of weak slice-level labels poses a significant challenge to model structure and performance. This invention proposes a molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model. This method can construct the association between morphological features of breast pathology images and immunohistochemical molecular phenotypes. Furthermore, addressing the issues of large WSI data volume and weak labels, a weakly supervised classification method based on multi-attribute feature embedding is designed to effectively achieve accurate prediction of breast cancer immunohistochemical molecular subtyping. Summary of the Invention

[0006] 1. Objective: To address the above-mentioned problems, the objective of this invention is to provide a molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model. This method can construct the association between morphological features of breast pathology images and immunohistochemical molecular phenotypes. Furthermore, to address the issues of large WSI data volume and weak labels, a weakly supervised classification method based on multi-attribute feature embedding is designed to effectively achieve accurate prediction of breast cancer immunohistochemical molecular subtyping.

[0007] 2. Technical Solution: To achieve this objective, the overall approach of this invention is to extract WSI image patch features based on a deep convolutional neural network model, construct a multi-attribute classifier with weakly constrained immunohistochemical labels to obtain image patches in WSI that are strongly correlated with various molecular information, and utilize a feature embedding module to obtain global features. Finally, Transformer-MLP classification is used to predict immunohistochemical molecular typing at the breast cancer slice level. The algorithmic approach of this invention is mainly reflected in the following four aspects:

[0008] 1) Extract WSI image patch features based on convolutional neural network model, and construct a multi-attribute classifier to obtain image patches in WSI that are strongly correlated with various molecular information;

[0009] 2) Construct a feature embedding module to embed image patch features with strong molecular information into instance packages to obtain global features, reduce local differentiation differences within slices and enhance molecular correlation;

[0010] 3) Use a classification framework based on Transformer encoder and multilayer perceptron (MLP) to perform slice-level molecular typing prediction on global features;

[0011] 4) For the multi-attribute embedding model consisting of a multi-attribute classifier, a feature embedding module, and a Transformer-MLP classifier, construct a multi-label constrained secondary classification loss for joint training.

[0012] This invention relates to a molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model. The specific steps of this method are as follows:

[0013] Step 1: Construct a multi-attribute classifier based on the image block depth features of digital pathological slides to obtain instance-level molecular-level probability distributions.

[0014] First, effective image patches from digital pathology slides are obtained through magnification and cropping, removing a large number of blank and contaminated areas from the WSI data. Then, depth features of each image patch are extracted using ResNet18 and SimCLR (a simple framework for contrastive learning of visual representations) as the base network. Four classifiers, with fully connected layers as the backbone, are constructed to classify the deep features of each image patch based on the weak molecular-level labels from the slide-level immunohistochemistry and the final subtyping labels. The probability distribution of each image patch at the molecular level and for each subtyping is calculated, and the strongest representative image patch is obtained. The specific process is as follows:

[0015] S11. Considering that the immunohistochemical classification of breast cancer is mainly determined by three molecular levels, it is presumed that the final classification is more complex in relation to image morphological features than at a single molecular level. Therefore, a multi-attribute classifier is constructed to jointly build the representational association between different molecular levels and the final classification. Based on instance-level deep features, for the expression levels of three molecular classes—estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2)—two binary classifiers with fully connected layers as the backbone are constructed, and the outputs of the corresponding binary classifiers are constrained by weak labels for the three molecular classes. After normalizing the output of each classifier, two probability scores for the corresponding molecular level are obtained: high expression and low expression probability scores. These are then statistically analyzed to obtain the probability scores for each class of all image patches in the entire slice, thus providing the probability distributions of high and low expression for the three molecular classes in the slice. For the final classification, a four-class classifier with a fully connected layer as the backbone is constructed, and the output of the four-class classifier is constrained by weak classification labels. Similarly, after normalizing the output, four probability scores are obtained, corresponding to Luminal A (lumen type A), Luminal... Four subtypes were identified: B (lubricated type B), HER2-enriched (HER2 overexpression type), and Basal-like (triple negative type). After statistical analysis, the probability scores of all image patches in the entire slice were obtained for each of the four subtypes. Based on the global probability distribution map of the slice under different attributes, the image patch with the highest probability in each category under the four attributes (ER, PR, HER2, and final subtype) was selected as the strongest representation patch, indicating that the depth features of the image patch are strongly correlated with the specific category of the corresponding attribute.

[0016] Step 2: Construct a feature embedding module to build global features based on instance-level deep features and the strongest representation image patch.

[0017] The feature embedding module inputs instance packages composed of depth features from all image patches in a single slice, constructing instance package features with the same dimension and size. Based on the depth features of the strongest representation patch under each attribute in the slice, auxiliary features with strong molecular-level correlations are constructed. Global features are constructed by superimposing instance package features and auxiliary features, enhancing molecular-level correlations and weakening local differences within the global features. The specific process is as follows:

[0018] S21. To address the issues of large local differentiation differences in slices, small proportion of effective information, and easy interference to the model, a feature embedding module is designed to construct global features that include local characteristics and highlight dominant information.

[0019] First, considering the variable-length input problem of instance packets (i.e., the number of valid image blocks in each slice is not equal), two fully connected layers are used to extract features from the instance packets, and the calculation formula is as follows: Where D is the instance package composed of the depth features of the image patches in the slice; L1 and L2 are fully connected layers of the same type but with different parameters, and the extracted feature results are K and Q, respectively; N represents the number of image patches in the slice; and C represents the dimension of the depth features of the image patches.

[0020] Secondly, construct instance packet features with uniform dimensions and consistent sizes to standardize subsequent network inputs: A = K T Q, A∈R C ×C Thus, the instance package feature A is obtained. Simultaneously, auxiliary features are constructed based on the deep features of the strongest representation blocks under each attribute: F = L3(H), H ∈ R. (2+2+2+4)×C , F∈R (C-1)×C Here, H represents the channel concatenation result of the 10 strongest representation blocks' deep features in the multi-attribute classifier, which are the sets of the strongest representation blocks' deep features corresponding to ER high expression, ER low expression, PR high expression, PR low expression, HER2 high expression, HER2 low expression, Luminal A, Luminal B, HER2-enriched, and Basal-like, respectively. F is the auxiliary feature. To achieve dynamic correlation between the auxiliary feature and each attribute, a variable parameter matrix P∈R is set and channel-superimposed with the auxiliary feature to obtain the molecularly strongly correlated auxiliary feature F′∈R. C×C Finally, the instance package features and auxiliary features are superimposed: M = A + F′. By preserving the local differences in the instance package features and highlighting molecular correlation information, the global feature M ∈ R is obtained. C×C .

[0021] Step 3: Construct a Transformer-MLP classifier to perform slice-level molecular typing prediction on global features.

[0022] Considering that the global features obtained by the feature embedding module still have noise from weak representation instances after the deep features of the original instance package are embedded into strong representation block features, the global features are encoded using a Transformer encoder to weaken the noise; an MLP classification with two fully connected layers as the backbone is constructed to decode the encoded features, and finally the slice-level molecular subtyping prediction results are calculated.

[0023] Step 4: Construct a multi-label constrained second-level classification loss, using molecular-level labels and genotype labels to constrain features at each level, and enhance molecular associations to balance the global dependency among weak labels.

[0024] Considering the construction of representation associations at different molecular levels and the final subtype using a multi-attribute classifier, an image patch classification loss is constructed using slice-level molecular level labels and subtype labels to enhance molecular associations and reduce the dependence of slice-level subtype labels on the global classification. For the final slice-level molecular subtype prediction results, a slice-level classification loss is constructed. The image patch loss and slice-level loss are combined to construct a multi-label constrained second-level classification loss.

[0025] S41. Construct a second-level classification loss with multi-label constraints.

[0026] First, for the binary classifiers corresponding to the three molecular levels in the multi-attribute classifier, calculate the binary cross-entropy loss Loss for each. ER Loss PR and Loss Her2 The calculation formula is: Where x t The probability x′ is obtained by normalizing the output of the binary classifier network for the t-th slice after passing through the sigmoid activation function. t y t The molecular-level label represents the t-th slice, where T is the total number of samples.

[0027] Furthermore, for the four-class classifier corresponding to the four subtypes of breast cancer in the multi-attribute classifier, the four-class cross-entropy loss is calculated. IHC The calculation formula is: Where x ti The output of the four-class classifier network for the t-th slice and the i-th class is normalized to have a probability of x′. ti y ti w represents the label of the i-th category in the t-th slice. i Let l be the hyperparameter corresponding to the i-th category to handle the imbalance of samples between labels. t Let be the cross-entropy loss corresponding to the t-th slice.

[0028] Secondly, based on the slice-level classification results of the Transformer-MLP classifier, the four-class cross-entropy loss (Loss) is calculated. Slide The calculation formula is: Where v ti The normalized probability of the MLP network output for the t-th slice and the i-th class is v′. ti .

[0029] Finally, construct the second-level classification loss: Loss = α(Loss ER +Loss PR +Loss Her2 +Loss IHC )+βLoss Slide , where α and β are the weight parameters corresponding to the image patch and slice level losses, respectively.

[0030] The flowchart of the molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model is as follows: Figure 1 As shown, depth features are extracted from image patches obtained by magnifying and cropping WSI based on image patch feature extraction network. Multi-attribute classifier is used to obtain the probability distribution of image patches at each molecular level and under each type of subtype. After statistics, the strongest representation block is obtained and global features of digital pathology slides are constructed through feature embedding module. Finally, Transformer-MLP classifier is constructed to predict the molecular subtype of breast pathology images of cases.

[0031] The advantages and effects of this invention are as follows: This invention proposes a molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model. It designs a multi-attribute classifier to jointly construct the correlation between morphological features and immunohistochemical molecular phenotypes of breast pathology images; it constructs a feature embedding module to enhance the correlation between global features and molecular levels, weakening the dependence of global features on slice-level subtyping labels; it effectively achieves molecular subtyping prediction for breast cancer cases using a Transformer-MLP classifier; and it uses a multi-label-constrained secondary classification loss in conjunction with the multi-attribute classifier and the Transformer-MLP classifier for joint training, simultaneously constraining the probability distribution of image patches and slices for the four subtyping categories. Compared with traditional breast pathology image analysis techniques, this invention's algorithm delves deeper into the correlation between images and molecules, possesses high accuracy in immunohistochemical molecular subtyping prediction, and can be combined with various application systems based on computer-aided pathology analysis technology, demonstrating broad market prospects and application value. Attached Figure Description

[0032] Figure 1 This is a flowchart of the overall process for predicting molecular subtyping of breast pathological images based on a multi-attribute embedding model.

[0033] Figure 2 The diagram shows the framework of a multi-attribute embedding model consisting of a multi-attribute classifier, a feature embedding module, and a Transformer-MLP classifier.

[0034] Figure 3 This is a schematic diagram of the second-level classification loss under multi-label constraints.

[0035] Figure 4 This is a probability distribution diagram of image patches statistically analyzed for each attribute by a multi-attribute classifier.

[0036] Figure 5 This is a schematic diagram illustrating the test results of the method of the present invention on the breast cancer WSI dataset. Detailed Implementation

[0037] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0038] This invention is a molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model. Its overall process is as follows: Figure 1 As shown, the specific implementation details for each part are as follows:

[0039] Step 1: Construct a multi-attribute classifier based on the image patch depth features of digital pathology slides to obtain instance-level molecular-level probability distributions;

[0040] S11. Considering that breast cancer immunohistochemical typing is mainly determined by three molecular levels, it is assumed that the association between the final typing and image morphological features is more complex than that at a single molecular level. Therefore, a multi-attribute classifier is constructed to jointly build the representational association between different molecular levels and the final typing. First, using a contrastive learning framework based on ResNet18 and SimCLR as the backbone, unsupervised training is performed on image patches across all slices. After model convergence, 512-dimensional deep features are extracted from the image patches. Based on this, for the three molecular levels of ER, PR, and Her2, binary classifiers containing a fully connected layer and a ReLU activation layer are constructed respectively to calculate the probability distribution of high and low expression at the molecular level for image patches. For the final typing, a four-class classifier with a fully connected layer and a ReLU activation layer as the backbone is constructed to calculate the probability distribution of image patches for four typing types: Luminal A, Luminal B, HER2-enriched, and Basal-like. Finally, the probability distribution map of the entire slice under different attributes is statistically analyzed, and the image patch with the highest probability is the strongest representation patch. Its framework structure is as follows: Figure 2 As shown, the multi-attribute classifier takes the image patch depth features as input, uses multiple classifiers and different weak labels to calculate the probability of the image patch in different categories under each attribute, calculates the probability scores of all image patches in the whole slice, selects the image patch with the highest probability as the strongest representation patch, and inputs the depth features of the representation patch into the feature embedding module to construct global features.

[0041] Step 2: Construct a feature embedding module to build global features based on instance-level deep features and the strongest representation block.

[0042] S21. First, considering the variable-length input problem of instance packets (i.e., the number of valid image blocks in each slice is not equal), two fully connected layers are used to extract features from the instance packets. The calculation formula is as follows: Where D represents the instance packet composed of the depth features of image patches in the slice; L1 and L2 are fully connected layers of the same type but with different parameters, whose input and output feature dimensions remain unchanged, and whose outputs are K and Q, respectively; N represents the number of image patches in the slice; C represents the dimension of the image patch depth features, which is taken as C = 512 in this invention. Next, instance packet features with uniform dimensions and consistent sizes are constructed to standardize the subsequent network input: A = K T Q, A∈RC×C Thus, the instance package feature A is obtained. Simultaneously, auxiliary features are constructed based on the deep features of the strongest representation blocks under each attribute: F = L3(H), H ∈ R. (2+2+2+4)×C , F∈R (C-1)×C Here, H corresponds to the concatenation result of the deep feature channels of the strongest representation block in the multi-attribute classifier, and F is the auxiliary feature. To achieve dynamic association between the auxiliary feature and each attribute, a variable parameter matrix P∈R is set. C Channel superposition with auxiliary features yields molecularly strongly correlated auxiliary features F′∈R C×C During training, the parameter matrix P is dynamically updated to fine-tune the distribution of auxiliary features. Finally, the instance package features and auxiliary features are superimposed: M = A + F′, to obtain the global features M ∈ R. C×C Its framework structure is as follows: Figure 2 As shown.

[0043] Step 3: Construct a Transformer-MLP classifier to perform slice-level molecular typing prediction on global features.

[0044] Step 4: Construct a multi-label constrained second-level classification loss, using molecular-level labels and genotype labels to constrain features at each level, and enhance molecular associations to balance the global dependency among weak labels.

[0045] S41. Construct a multi-label constrained second-level classification loss, specifically composed of the following: For the two classifiers corresponding to the three molecular levels in the multi-attribute classifier, calculate the binary cross-entropy loss Loss for each. ER Loss PR and Loss Her2 The calculation formula is: Where x t The probability x′ is obtained by normalizing the output of the binary classifier network for the t-th slice after passing through the sigmoid activation function. t y t Let represent the molecular-level label of the t-th slice, and T be the total number of samples. For the four-class classifier corresponding to the four subtypes of breast cancer in a multi-attribute classifier, calculate the four-class cross-entropy loss (Loss). IHC The calculation formula is: Where x ti The output of the four-class classifier network for the t-th slice and the i-th class is normalized to have a probability of x′. ti y ti w represents the label of the i-th category in the t-th slice. i Let l be the hyperparameter corresponding to the i-th category to handle the imbalance of samples between labels. t Let be the cross-entropy loss corresponding to the t-th slice. Calculate the four-class cross-entropy loss Loss based on the slice-level classification results of the Transformer-MLP classifier.Slide The calculation formula is: Where v ti The normalized probability of the MLP network output for the t-th slice and the i-th class is v′. ti Finally, construct the second-order classification loss: Loss = α(Loss ER +Loss PR +Loss Her2 +Loss IHC )+βLoss Slide α and β are the loss weight parameters for image patch and slice levels, respectively, and can take values ​​of α = β = 0.5.

[0046] Its specific composition is as follows Figure 3 As shown, the image patch classification loss, combined with a multi-attribute classifier, constructs a global probability distribution map of slices under different attributes and selects the strongest representation patch. The slice-level classification loss, combined with an MLP classifier, is used for supervised training to constrain the distance between the genotyping prediction result and the slice label to continuously narrow. Specifically, the labels used in calculating each level of loss are all weak slice-level labels, and the three molecular-level labels (ER, PR, and HER2) comprehensively determine the final IHC (immunohistochemistry) molecular genotyping. The secondary loss utilizes the joint label to fully constrain the network, balancing the dependence of the weak genotyping labels on the global distribution.

[0047] Furthermore, during training, the multi-attribute embedding model, consisting of a multi-attribute classifier, a feature embedding module, and a Transformer-MLP classifier, uses the ADAM optimizer to update parameters. Gradient backpropagation is used to adjust model parameters to reduce loss, with the initial learning rate set to lr0 = 2 × 10⁻⁶. -4 The learning rate is then dynamically adjusted using the Cosine Annealing function as the number of iterations increases, with the minimum learning rate set to lr. min =5×10 -6 The maximum number of iterations is ep_num = 200.

[0048] To visually demonstrate the effects of the present invention, Figure 4 This is a probability distribution map of image patches statistically analyzed by a multi-attribute classifier across various attributes. The WSI slices are classified as Luminal A by immunohistochemistry, showing high expression levels for ER and PR, and low expression levels for Her2. Figure 4 The heatmap of ER, PR, and Her2 expression in the slices shows that darker colors indicate a higher probability of high molecular expression at the molecular level, while lighter colors indicate a higher probability of low molecular expression at the molecular level. High ER and PR expression slices constitute a large proportion of the slices, while low Her2 expression slices are dominant, consistent with the molecular levels observed in the slices. Figure 4The probability heatmap of the IHC four-class classification shows that there are more image patches classified as Luminal A in the slice than the other three classifications, which is consistent with the slice-level classification, proving the effectiveness of the multi-attribute classifier in realizing the correspondence between image representation and molecular level.

[0049] Figure 5 This presents the model test results based on breast cancer WSI data for this invention. Using TCGA and breast cancer WSI data from the Cancer Hospital of the Chinese Academy of Medical Sciences, with 1421 digital pathological slides randomly selected as the training set, 203 as the validation set, and 407 as the test set, this method achieved an accuracy of 73% in predicting molecular subtyping of breast pathological images. Furthermore, the AUC values ​​(areas of return on curve) for the four subtyping categories—Luminal A, Luminal B, HER2-enriched, and Basal-like—were 0.87, 0.70, 0.88, and 0.90, respectively. Therefore, this invention can effectively interpret molecular subtyping of breast pathological images. This invention can be combined with various computer-aided pathology analysis systems and widely applied in related fields of pathology analysis, possessing broad market prospects and application value.

Claims

1. A molecular subtyping prediction method for breast pathology images based on a multi-attribute embedding model, characterized in that, The steps are as follows: Step 1: Construct a multi-attribute classifier based on the image block depth features of digital pathology slides to obtain instance-level molecular probability distributions; firstly, obtain effective image blocks from digital pathology slides through magnification and block cropping, and remove a large number of blank and contaminated areas from the WSI data; Then, the deep features of each image patch are extracted based on the ResNet18 and SimCLR contrastive learning framework. Four classifiers with fully connected layers as the backbone are constructed for the weak labels at the molecular level of slice-level immunohistochemistry and the final classification labels. The deep features of the image patches are classified respectively, and the probability distribution of each image patch at the molecular level and each classification is calculated to obtain the image patch with the strongest representation. Step 2: Construct a feature embedding module to build global features based on instance-level deep features and the strongest representation block; Using instance packages composed of depth features of all image blocks in a single slice as module input, we construct instance package features with the same dimension and size; based on the depth features of the strongest representation block under each attribute in the slice, we construct auxiliary features with strong correlation at the molecular level; by superimposing instance package features and auxiliary features, we construct global features, enhance the molecular level correlation in global features and weaken local differences. Step 3: Construct a Transformer-MLP classifier to perform slice-level molecular typing prediction on global features; Considering that the global features obtained by the feature embedding module still have noise from weak representation instances after the deep features of the original instance package are embedded into strong representation block features, the global features are encoded using a Transformer encoder to weaken the noise; an MLP classification with two fully connected layers as the backbone is constructed to decode the encoded features, and finally the slice-level molecular subtyping prediction results are calculated. Step 4: Construct a multi-label constrained second-level classification loss, using molecular-level labels and genotype labels to constrain features at each level, and enhance molecular associations to balance the dependence of each weak label on the global classification. The specific process is as follows: Construct a multi-label constrained second-level classification loss; first, for the two classifiers corresponding to the three molecular levels in the image patch multi-attribute classifier, calculate the binary classification cross-entropy loss respectively. , and The calculation formula is: ,in For the first The output of the image patch binary classifier network is normalized by the sigmoid activation function to obtain the probability. , Representing the Molecular-level labels on individual slices The total number of samples; For the four-class classifier corresponding to breast cancer quintet in the multi-attribute classifier of image patches, calculate the four-class cross-entropy loss. The calculation formula is: ,in For the first The slice The output of the image patch four-classifier network, after normalization, has the following probability: , Indicates the first The slice The tag, For the first The corresponding hyperparameters are used to handle the imbalance of samples between labels. For the first The cross-entropy loss corresponding to each slice.

2. The method for molecular subtyping prediction of breast pathology images based on a multi-attribute embedding model according to claim 1, characterized in that: In step one, the specific process is as follows: Considering that the immunohistochemical typing of breast cancer is determined by a combination of three molecular levels, it is hypothesized that the final typing is more complex in association with image morphological features than at a single molecular level. Therefore, a multi-attribute classifier is constructed to jointly build the representational association between different molecular levels and the final typing. Based on instance-level deep features, binary classifiers with fully connected layers as the backbone are constructed for the expression levels of three molecular classes: estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2). The output results of the corresponding binary classifiers are constrained using weak labels for the three molecular classes. Simultaneously, the output results of each classifier are normalized to obtain two probability scores at the corresponding molecular level: high expression and low expression probability scores. After statistical analysis, the probability scores of each class for all image patches in the entire slice are obtained. Therefore, the high and low expression probability distributions of the three molecular classes in the slice can be obtained.

3. The method for molecular subtyping prediction of breast pathology images based on a multi-attribute embedding model according to claim 2, characterized in that: For the final classification, a four-class classifier with a fully connected layer as the backbone is constructed, and the output of the four-class classifier is constrained by the weak classification label. Similarly, the output results are normalized to obtain four probability scores, corresponding to four classifications: Luminal A (Luminal A), Luminal B (Luminal B), HER2-enriched (HER2-overexpressing), and Basal-like (Triple-negative). After statistical analysis, the four probability scores of all image patches in the entire slice are obtained. Based on the global probability distribution map of the slice under different attributes, the image patch with the highest probability in different categories under the four attributes ER, PR, HER2, and the final classification is selected as the strongest representation patch, indicating that the depth features of the image patch are strongly correlated with the specific category of the corresponding attribute.

4. The method for molecular subtyping prediction of breast pathological images based on a multi-attribute embedding model according to claim 1, characterized in that: In step two, the specific process is as follows: To address the issues of large local differentiation differences in slices, small proportion of effective information, and easy interference with the model, a feature embedding module is designed to construct global features that include local characteristics and highlight dominant information. First, considering the variable-length input problem of instance packets (i.e., the number of effective image blocks in each slice is unequal), two fully connected layers are used to extract features from the instance packets, with the calculation formula as follows: ;in, An instance package consisting of depth features of image patches within a slice; and For fully connected layers of the same type but with different parameters, the extracted feature results are as follows: and ; Indicates the number of image patches in a slice; First, the depth feature dimension of the image patch is represented; second, instance packet features with uniform dimension and size are constructed to standardize the subsequent network input. Thus, the instance package characteristics are obtained. Simultaneously, auxiliary features are constructed based on the deep features of the strongest representation blocks under each attribute: ,in The deepest representation block corresponding to the highest and lowest expression levels of three types of molecules and the four subtypes in the multi-attribute classifier. Auxiliary features are used; to achieve dynamic correlation between auxiliary features and various attributes, a variable parameter matrix is ​​set. Channel superposition with auxiliary features yields auxiliary features with strong molecular-level correlation. Finally, the instance package features and auxiliary features are superimposed: Global features are obtained by preserving local differences in instance package features and highlighting molecular correlation information. .

5. The method for molecular subtyping prediction of breast pathology images based on a multi-attribute embedding model according to claim 1, characterized in that: Calculate the four-class cross-entropy loss based on the slice-level classification results of the slice-level Transformer-MLP classifier. The calculation formula is: ,in For the first The slice The output of the slice-level MLP network, after normalization, has the following probability: Finally, construct the second-order classification loss: ,in and These are the weight parameters corresponding to the image patch and slice level losses, respectively, with values... .

6. The method for molecular subtyping prediction of breast pathology images based on a multi-attribute embedding model according to claim 5, characterized in that: During training, the multi-attribute embedding model, consisting of a multi-attribute classifier, a feature embedding module, and a Transformer-MLP classifier, uses the ADAM optimizer to update parameters. Gradient backpropagation is used to adjust model parameters to reduce loss. The initial learning rate is set to... The learning rate is then dynamically adjusted using the Cosine Annealing function as the number of iterations increases, with the minimum learning rate set to [value missing]. The maximum number of iterations is 200.

Images