Breast cancer histological image segmentation and classification method based on multi-scale deep learning

Abstract

The application discloses a breast cancer histological image segmentation and classification method based on multi-scale deep learning, first data acquisition and preprocessing are carried out; then a breast cancer deep whole slice image-stroma segmentation and classification model is constructed, the model adopts a multi-scale attention fusion network architecture, including a feature extraction module, an attention mechanism module, a feature fusion module and a segmentation and classification module; the model is trained through the preprocessed data, finally the breast cancer histological image segmentation and classification are completed through the trained model, and Grad-CAM visualization is carried out. Through the innovative multi-scale deep learning architecture and technical measures, the application significantly improves the automation degree, precision and robustness of breast cancer histological image segmentation and classification, and provides strong technical support for clinical application. Compared with the prior art, the application realizes higher automation level, higher segmentation and classification precision and stronger global information utilization capability in function.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for segmentation and classification of breast cancer histological images based on multi-scale deep learning. Background Technology

[0002] In the diagnosis and treatment of breast cancer, the segmentation and classification of histological images are crucial steps. Traditional pathological diagnostic methods mainly rely on manual examination of tissue sections by pathologists. This method is not only time-consuming and labor-intensive but also susceptible to human factors, making it difficult to guarantee the accuracy and consistency of diagnostic results. With the development of digital pathology technology, computer-aided analysis of whole-slide images (WSI) has become an important way to improve diagnostic efficiency and accuracy.

[0003] 1. Traditional image segmentation methods:

[0004] Traditional image segmentation methods include thresholding, region growing, morphological operations, and graph cut. These methods typically rely on hand-designed features and prior knowledge, often proving insufficient when dealing with complex and diverse pathological images. For example, thresholding methods struggle to adapt to images under different staining conditions, and morphological operations are highly sensitive to noise and interference. Vu et al. (2016) used a stacked denoising autoencoder to learn features from breast cancer pathological images, but its segmentation performance remained limited by the expressive power of hand-designed features.

[0005] 2. Machine learning-based segmentation and classification methods:

[0006] With the development of machine learning, methods such as Support Vector Machines (SVM), Random Forests, and K-Nearest Neighbors have been widely applied to the segmentation and classification of pathological images. These methods learn feature representations from training data, replacing the manual feature extraction process. However, these shallow learning models still have limitations in dealing with high-level semantic features and complex image structures. For example, Kandemir et al. (2014) used a multi-instance learning framework to classify breast cancer tissues, but its classification performance was limited by the complexity of feature selection and model design.

[0007] 3. Deep learning-based segmentation and classification methods:

[0008] In recent years, deep learning, especially convolutional neural networks (CNNs), has made significant progress in medical image analysis. Ciresan et al. (2012) pioneered the application of CNNs to breast cancer pathology image classification, achieving excellent performance. Subsequently, deep learning models such as U-Net, DeepLab, and PSPNet have demonstrated powerful feature extraction and segmentation capabilities in medical image segmentation tasks. For example, Chen et al. (2018) proposed a breast tissue segmentation method based on U-Net, achieving state-of-the-art performance on multiple public datasets by introducing an attention mechanism and adversarial training.

[0009] Existing methods for segmentation and classification of breast cancer histological images have significant shortcomings: traditional methods rely on manual feature design, making them difficult to adapt to complex and diverse pathological images; shallow learning models have limited performance in high-level semantic feature extraction and classification; although deep learning methods have made progress, they remain inefficient and inaccurate in processing high-resolution WSI data. These methods struggle to effectively integrate multi-scale information and lack sufficient modeling of the context, resulting in limited segmentation and classification accuracy. The deep learning method proposed in this invention effectively improves the robustness and generalization ability of the model through multi-scale feature fusion and attention mechanisms, overcoming the aforementioned shortcomings.

[0010] Cited Literature

[0011] 1.Vu, QD, et al. "Stacked Denoising Autoencoders for Nuclei Detection in Breast Cancer Histopathology Images." IEEE Transactions on Medical Imaging35.5(2016):1192-1203.

[0012] 2. Kandemir, M., et al. "Computer-aided diagnosis in breast cancer using tissue morphometry, mammographic density, and wavelet analysis." Medical ImageAnalysis 18.5(2014):765-773.

[0013] 3. Ciresan, DC, et al. "Deep neural networks segment neuronal membranes in electron microscopy images." Advances in Neural Information ProcessingSystems 25(2012):2843-2851.

[0014] 4.Chen,L.,et al."Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation."European Conference on ComputerVision.Springer,Cham,2018. Summary of the Invention

[0015] This invention aims to address the shortcomings of existing methods for segmenting and classifying breast cancer histological images. Specific technical problems include:

[0016] Low level of automation: Traditional methods rely on manual feature extraction, which is time-consuming and labor-intensive, easily affected by human factors, and difficult to deal with complex and diverse pathological images.

[0017] Insufficient feature extraction and classification performance: Machine learning methods based on shallow learning perform poorly in extracting high-level semantic information and processing complex structures, and have limited generalization ability.

[0018] Ignoring global structural information: While existing deep learning methods have improved segmentation accuracy, they usually only focus on local pixel-level features, ignoring the global structure and contextual information of the epithelial-mesenchymal tissue, which affects the continuity and completeness of segmentation.

[0019] Lack of joint learning and reasoning capabilities: Most existing methods treat segmentation and classification as independent tasks, which increases the complexity of the model and makes the post-processing steps of integrating the results cumbersome.

[0020] To address the aforementioned technical problems, this invention provides a multi-scale deep learning-based method for segmenting and classifying breast cancer histological images, specifically for epithelial-mesenchymal segmentation and classification in whole-slice (WSI) images of breast cancer tissue. It aims to overcome the shortcomings of existing technologies and achieve the following objectives:

[0021] Automation and efficiency: Employing an end-to-end convolutional neural network architecture, it eliminates the need for manual feature design and can automatically and quickly process complex and diverse pathological images, improving the efficiency of segmentation and classification.

[0022] High-precision feature extraction and classification: By designing a multi-scale, multi-cascaded network structure and fusing feature maps at different resolutions, the feature extraction of local cells and the whole tissue can be captured, thereby improving the accuracy and robustness of segmentation and classification.

[0023] Global information utilization: Introducing attention mechanisms and adversarial training strategies enhances the model's ability to capture global structure and contextual information, thereby improving the continuity and completeness of segmentation.

[0024] End-to-end joint learning: We propose an end-to-end segmentation and classification network structure to achieve joint learning and inference of pixel-level segmentation and region-level classification, simplifying post-processing steps and improving overall performance.

[0025] Through the above improvements, the present invention can significantly enhance the accuracy, efficiency, and automation of breast cancer histological image segmentation and classification, providing strong support for clinical diagnosis and prognostic analysis.

[0026] A method for segmentation and classification of breast cancer histological images based on multi-scale deep learning includes the following steps:

[0027] Step 1: Data acquisition and preprocessing;

[0028] Step 2: Construct a deep whole-slice image-stromal segmentation and classification model for breast cancer (Deep WSI-Stroma);

[0029] The breast cancer deep whole slice image-stromal segmentation and classification model adopts a multi-scale attention fusion network architecture (MAFNet), which includes a feature extraction module, an attention mechanism module, a feature fusion module, and a segmentation and classification module.

[0030] Step 3: Model training;

[0031] We employ a cross-entropy loss function and the Adam optimizer. A hybrid training strategy is used, training simultaneously with both patch and WSI samples. Corresponding loss functions and optimization methods are designed to enable the model to learn both local and global feature representations.

[0032] Step 4: Perform breast cancer histological image segmentation and classification using the trained Deep WSI-Stroma model, and visualize the images using Grad-CAM.

[0033] Furthermore, the specific method for step 1 is as follows:

[0034] Read WSI data: Use the OpenSlide library to read Aperio SVS format files from the TCGA-BRCA dataset and extract high-resolution images and annotation information.

[0035] Image segmentation and patch generation: Foreground-background segmentation of WSI was performed using a modified Otsu thresholding method to extract tissue region masks. A series of image patches were generated by sliding a window of size 256×256 with a step size of 128 within the tissue mask.

[0036] Image normalization: The generated image patches are color normalized to reduce the impact of color differences on the model.

[0037] Data augmentation: Divide the training set and validation set according to requirements, and perform data augmentation operations on the patches of the training set, including random horizontal flipping, vertical flipping, rotation, color transformation, random cropping, scaling, translation, and shearing, to increase the diversity of training samples and enhance the robustness of the model.

[0038] Furthermore, the breast cancer deep whole slice image-mesenchymal segmentation and classification model uses EfficientNetV2-B3 pre-trained on the ImageNet dataset as the feature extraction module, extracting multi-scale features at different levels of EfficientNetV2-B3, namely through the 2nd, 4th and 6th MBConv blocks of EfficientNetV2-B3.

[0039] The attention mechanism module comprises two main branches: a spatial attention branch and a channel attention branch. The spatial attention branch generates a weight map through convolution and pooling operations, processes the pooling results using an activation function, and generates a spatial attention weight map, representing the importance of each position in the feature map. The channel attention branch learns the importance weights of each channel through global average pooling and fully connected layers, and generates a channel attention weight map using an activation function, representing the importance of each channel in the feature map. Finally, the spatial attention weight map and the channel attention weight map are multiplied to generate a comprehensive attention weight. This comprehensive attention weight is then used to weight the original input feature map, enhancing the representation of key regions and features, generating an enhanced feature map. The attention mechanism module allows for the acquisition of both lower-level detailed features and higher-level global features, i.e., multi-scale features. Introducing the attention mechanism module after all MBConv blocks significantly increases computational complexity. Therefore, selectively introducing the attention mechanism module after the 2nd, 4th, and 6th blocks allows for improved model performance while controlling computational complexity and memory consumption.

[0040] The feature fusion module adaptively assigns weights to features of different scales through weight learning, and then combines the weighted features through addition operations to output the final fused feature map.

[0041] The segmentation and classification module includes a segmentation head and a classification head, which respectively implement the segmentation task and the classification task. The fused feature map output by the feature fusion module is simultaneously input into the segmentation head and the classification head for processing. The segmentation head generates a segmentation mask through convolutional layers, activation functions and upsampling layers, and the classification head generates a classification result through global average pooling layers and fully connected layers.

[0042] Furthermore, the specific method for step 3 is as follows:

[0043] 3-1. Loss Function and Optimizer:

[0044] Cross-entropy loss is used as the basic method for calculating the error of a single task (such as classification or segmentation). A joint loss function is used to simultaneously optimize both segmentation and classification tasks, combining pixel-level segmentation loss and region-level classification loss. The segmentation loss is the pixel-level cross-entropy loss for image segmentation, and the classification loss is the region-level cross-entropy loss for classification. The Adam optimizer is used with an initial learning rate of 0.01 and a learning rate decay strategy.

[0045] 3-2. Hybrid Training Strategy:

[0046] In each training batch, a large WSI image is divided into multiple small patch samples, while the entire WSI image is retained as a WSI sample. That is, each batch contains a certain number of patch samples and WSI samples. Data from local regions (patches) and the entire image (WSI) are used together to update the model parameters. The joint loss L at the patch level... patch Used to measure the classification and segmentation error of local patches. WSI-level joint loss L WSI This is used to measure the classification and segmentation error of the entire WSI image. The total joint loss function is obtained by weighting and summing the patch-level joint loss and the WSI-level joint loss using balancing factors α and β. The calculation of the total joint loss function is shown in equation (1):

[0047] L total =αL patch +βL WSI (1)

[0048] The overall joint loss function in the above formula combines features at different scales (local patch and global WSI) to optimize the overall performance of the model.

[0049] 3-3. Perform batch normalization and L2 regularization on the training data:

[0050] 3-4. Hyperparameter tuning:

[0051] (1) Define parameter range: Set the candidate range for batch size, learning rate, regularization coefficient, and joint loss balance factor hyperparameters.

[0052] (2) Grid search and cross-validation: Grid search is performed within the parameter range, and the performance of each parameter combination is evaluated through cross-validation.

[0053] (3) Select the best combination: Based on the cross-validation results, select the best combination of hyperparameters.

[0054] 3-5. Training process:

[0055] (1) The final model is trained using the optimal combination determined through hyperparameter tuning. In each training batch, sampling is performed using the optimal batch size after tuning. Gradient updates are performed using the optimal learning rate after tuning. Loss calculation and optimization are performed using the tuned regularization coefficients and joint loss balancing factor.

[0056] (2) In each training batch, random samples are taken from the patch data loader and the WSI data loader to form a mixed training batch.

[0057] (3) Calculate the joint loss at the patch level and the joint loss at the WSI level.

[0058] (4) Use the balance factor to perform a weighted summation of the two joint losses to obtain the total joint loss function.

[0059] (5) Backpropagation is performed based on the total joint loss function to update the model parameters.

[0060] 3-6. Model Evaluation and Validation:

[0061] Evaluate model performance on the validation set using metrics such as accuracy, precision, recall, and F1 score. Make minor adjustments based on validation set performance to ensure the model performs well on the test set.

[0062] In one embodiment, the optimal combination of hyperparameters is: batch size 32, learning rate 0.001, regularization coefficient 0.001, and balance factors α and β both 0.5.

[0063] Furthermore, the specific method for step 4 is as follows:

[0064] After the original input image is processed by a trained breast cancer deep whole slice image-mesenchymal segmentation and classification model and the Grad-CAM heatmap generation algorithm, a high-resolution segmentation probability heatmap and salient regions are generated to visually display the spatial distribution and key regions of epithelial-mesenchymal segmentation.

[0065] The beneficial effects of this invention are as follows:

[0066] The breast cancer histological image segmentation and classification method (DeepWSI-Stroma) based on multi-scale deep learning proposed in this invention has significant advantages and positive effects compared with existing technologies. These advantages and effects are directly brought about by the technical solution of this invention, or are an inevitable result of the technical solution:

[0067] 1. High degree of automation;

[0068] Existing technologies rely on manual feature extraction and expert experience, which is time-consuming, labor-intensive, and easily affected by human factors. This invention employs an end-to-end convolutional neural network architecture, capable of automatically learning features from raw WSI images to achieve fully automated segmentation and classification, significantly improving efficiency and reducing human error.

[0069] 2. High-precision feature extraction and classification;

[0070] This invention designs a multi-scale, multi-cascaded network structure. By fusing feature maps at different resolutions, it can capture features of local cells and the whole tissue, improving the accuracy of segmentation and classification. Experimental results show that this invention achieves an average Dice score of 0.873 on the segmentation task of the TCGA-BRCA dataset and an average F1 score of 0.921 for the three classes on the classification task, significantly outperforming methods such as U-Net (Dice score of 0.821) and traditional handcrafted features + SVM (F1 score of 0.843).

[0071] 3. Utilization of global information;

[0072] This invention introduces an attention mechanism and adversarial training strategy to enhance the model's ability to capture global structure and contextual information, thereby improving the continuity and completeness of segmentation. The attention mechanism can automatically learn key regions and features in the image, helping the model focus on processing the most relevant information, while adversarial training improves the model's robustness to interference and noise.

[0073] 4. End-to-end collaborative learning;

[0074] The end-to-end segmentation and classification network structure proposed in this invention achieves joint learning and inference of pixel-level segmentation and region-level classification, simplifying post-processing steps and improving overall performance. Compared with existing methods that treat segmentation and classification as independent tasks, this invention significantly improves model complexity management and prediction accuracy.

[0075] 5. Clinical application value;

[0076] By introducing Grad-CAM technology and a multivariate logistic regression model, this invention can generate high-resolution segmentation probability heatmaps and significant regions, intuitively displaying the spatial distribution and key areas of epithelial-mesenchymal segmentation, which helps pathologists understand and validate the model's decision-making basis. Furthermore, the analysis of epithelial-mesenchymal ratio (ESR) and spatial distribution patterns can effectively predict the invasiveness and metastatic risk of tumors, providing strong support for clinical diagnosis and treatment decisions.

[0077] In summary, this invention, through its innovative multi-scale deep learning architecture and technical measures, significantly improves the automation, accuracy, and robustness of breast cancer histological image segmentation and classification, providing strong technical support for clinical applications. Compared to existing technologies, this invention achieves a higher level of automation, higher segmentation and classification accuracy, and stronger global information utilization capabilities. Attached Figure Description

[0078] Figure 1 This is a model architecture diagram of the technical solution of an embodiment of the present invention;

[0079] Figure 2 This is a schematic diagram of the attention mechanism module structure according to an embodiment of the present invention;

[0080] Figure 3 This is a schematic diagram of the feature fusion module structure according to an embodiment of the present invention;

[0081] Figure 4 This is a schematic diagram of the segmentation and classification structure according to an embodiment of the present invention;

[0082] Figure 5 This is a schematic diagram of the MBConv block structure according to an embodiment of the present invention;

[0083] Figure 6 This is a flowchart of the Grad-CAM visualization process according to an embodiment of the present invention;

[0084] Figure 7 This is a comparison chart of ROC curves of different models in this invention on the test set;

[0085] Figure 8 This is a breast cancer WSI sample and its corresponding heatmap according to an embodiment of the present invention;

[0086] Figure 9 The loss curves for the training and validation sets in this embodiment of the invention are shown.

[0087] Figure 10 These are the accuracy curves for the training and validation sets in this embodiment of the invention. Detailed Implementation

[0088] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0089] A method for segmentation and classification of breast cancer histological images based on multi-scale deep learning includes the following steps:

[0090] Step 1: Data acquisition and preprocessing;

[0091] Read WSI data: Use the OpenSlide library to read Aperio SVS format files from the TCGA-BRCA dataset and extract high-resolution images and annotation information.

[0092] Image segmentation and patch generation: Foreground-background segmentation of WSI was performed using a modified Otsu thresholding method to extract tissue region masks. A series of image patches were generated by sliding a window of size 256×256 with a step size of 128 within the tissue mask.

[0093] Image normalization: The generated image patches are color normalized to reduce the impact of color differences on the model.

[0094] Data augmentation: Divide the training set and validation set according to the requirements, and perform data augmentation operations such as random horizontal flipping, vertical flipping, rotation, color transformation, random cropping, scaling, translation, and shearing on the patches of the training set to increase the diversity of training samples and enhance the robustness of the model.

[0095] Step 2: Construct a deep whole-slice image-stromal segmentation and classification model for breast cancer (Deep WSI-Stroma), such as... Figure 1 As shown;

[0096] The breast cancer deep whole slice image-stromal segmentation and classification model adopts a multi-scale attention fusion network architecture (MAFNet), which includes a feature extraction module, an attention mechanism module, a feature fusion module, and a segmentation and classification module.

[0097] We use EfficientNetV2-B3, pre-trained on the ImageNet dataset, as the feature extraction module. Multi-scale features are extracted at different levels of EfficientNetV2-B3, specifically through the 2nd, 4th, and 6th MBConv blocks. EfficientNetV2-B3 consists of multiple layers, each containing MBConv blocks (e.g., ...). Figure 5 (As shown). The distribution and overall architecture of these blocks are designed to achieve efficient computation and powerful feature extraction capabilities. The 2nd, 4th, and 6th MBConv blocks represent feature extraction layers at different scales. The features extracted at these layers have different receptive fields and levels of abstraction.

[0098] By introducing an attention mechanism module after these layers, lower-level detailed features and higher-level global features, i.e., multi-scale features, are obtained. The attention mechanism module enhances the feature representation capability of deep learning models by adaptively emphasizing important features and suppressing irrelevant features. For example... Figure 2 As shown, this module comprises two main branches: a spatial attention branch and a channel attention branch. The spatial attention branch generates a weight map through convolution and pooling operations, processes the pooling results using an activation function (such as sigmoid), and generates a spatial attention weight map, which represents the importance of each position in the feature map. The channel attention branch learns the importance weights of each channel through global average pooling and fully connected layers, and generates a channel attention weight map using an activation function (such as sigmoid), which also represents the importance of each channel in the feature map. Finally, the spatial attention weight map and the channel attention weight map are multiplied to generate a comprehensive attention weight. This comprehensive attention weight is then used to weight the original input feature map, enhancing the representation of key regions and features, generating an enhanced feature map. These enhanced feature maps are used for subsequent segmentation and classification tasks, thereby improving model performance. The attention mechanism module allows for the acquisition of both low-level detailed features and high-level global features, i.e., multi-scale features. Introducing the attention mechanism module after all MBConv blocks significantly increases computational complexity. Therefore, selectively introducing attention mechanism modules after the 2nd, 4th, and 6th blocks can increase model performance while controlling computational complexity and memory consumption.

[0099] like Figure 3 As shown, the feature fusion module adaptively assigns weights to features of different scales through weight learning, and then combines the weighted features through addition operations to output the final fused feature map.

[0100] like Figure 4 As shown, the segmentation and classification module includes a segmentation head and a classification head, which respectively implement the segmentation task and the classification task. The fused feature map output by the feature fusion module is simultaneously input into the segmentation head and the classification head for processing. The segmentation head generates a segmentation mask through convolutional layers, activation functions, and upsampling layers, while the classification head generates a classification result through global average pooling layers and fully connected layers. This method of performing segmentation and classification tasks simultaneously can efficiently and accurately segment and classify images.

[0101] Step 3: Model training;

[0102] 3-1. Loss Function and Optimizer:

[0103] Cross-entropy loss is used as the basic method for calculating the error of a single task (such as classification or segmentation). A joint loss function is used to simultaneously optimize both segmentation and classification tasks, combining pixel-level segmentation loss and region-level classification loss. The segmentation loss is the pixel-level cross-entropy loss for image segmentation, and the classification loss is the region-level cross-entropy loss for classification. The joint loss applies the cross-entropy loss to different tasks (such as segmentation and classification) and optimizes both tasks simultaneously through a weighted approach, improving the overall performance and generalization ability of the model. The Adam optimizer is used with an initial learning rate of 0.01 and a learning rate decay strategy.

[0104] 3-2. Hybrid Training Strategy:

[0105] In each training batch, a large WSI image is divided into multiple small patch samples, while the entire WSI image is retained as a WSI sample. That is, each batch contains a certain number of patch samples and WSI samples, and the model parameters are updated using a hybrid training strategy that combines local region (patch) and whole image (WSI) data. The aim of this hybrid training strategy is to fully utilize both local and global features, enabling the model to learn and optimize effectively at different scales. The joint loss at the patch level (L...) patch This is used to measure the classification and segmentation error of a local patch. The joint loss (L) at the WSI level... WSI The joint loss function is used to measure the classification and segmentation error of the entire WSI image. The total joint loss function is obtained by weighting and summing the patch-level joint loss and the WSI-level joint loss using balancing factors α and β. The calculation of the total joint loss function is shown in equation (1):

[0106] L total =αL patch +βL WSI (1)

[0107] The overall joint loss function in the above formula combines features at different scales (local patch and global WSI) to optimize the overall performance of the model.

[0108] 3-3. Perform batch normalization and L2 regularization on the training data:

[0109] Batch normalization normalizes the data in each mini-batch, making the distribution of inputs to each layer more stable. By stabilizing the input distribution, it reduces the vanishing and exploding gradient problems, thereby accelerating the training process and improving the model's performance and stability. L2 regularization is a technique used to prevent overfitting. It adds a weight decay term to the loss function to constrain the magnitude of the weights, preventing the model's weight parameters from becoming too large, thus improving the model's generalization ability.

[0110] 3-4. Hyperparameter tuning:

[0111] (1) Define parameter range: Set the candidate range for hyperparameters such as batch size, learning rate, regularization coefficient, and joint loss balance factor.

[0112] (2) Grid search and cross-validation: Grid search is performed within the parameter range, and the performance of each parameter combination is evaluated through cross-validation.

[0113] (3) Select the best combination: Based on the cross-validation results, select the best combination of hyperparameters: batch size 32, learning rate 0.001, regularization coefficient 0.001, and balance factors α and β are both 0.5.

[0114] 3-5. Training process:

[0115] (1) The final model is trained using the optimal combination determined through hyperparameter tuning. In each training batch, the optimal batch size (32) is applied for sampling. The optimal learning rate (0.001) is used for gradient updates. The optimal regularization coefficient (0.001) and joint loss balancing factors (α and β are both 0.5) are applied for loss calculation and optimization.

[0116] (2) In each training batch, samples are randomly taken from both the patch data loader and the WSI data loader to form a mixed training batch. Here, patch samples are used for fine-grained local feature learning, and WSI samples are used to capture global context information. These two can be randomly sampled from different WSI images and do not need to be in a one-to-one correspondence.

[0117] (3) Calculate the joint loss at the patch level and the joint loss at the WSI level.

[0118] (4) Use the balance factor to perform a weighted summation of the two joint losses to obtain the total joint loss function.

[0119] (5) Backpropagation is performed based on the total joint loss function to update the model parameters.

[0120] 3-6. Model Evaluation and Validation:

[0121] Evaluate model performance on the validation set using metrics such as accuracy, precision, recall, and F1 score. Make minor adjustments based on validation set performance to ensure excellent model performance on the test set. The specific implementation steps are as follows:

[0122] First, the trained model is used on the validation set to obtain classification and segmentation results, and evaluation metrics such as accuracy, precision, recall, and F1 score are calculated. Then, fine-tuning is performed in small steps based on the validation set performance, including fine-tuning the learning rate, regularization strength, and balancing factors α and β. These fine-tunings ensure that the model's performance on the validation set is optimal. Tables 1 and 2 show the performance comparison of the model on the validation set under different fine-tuning strategies.

[0123] The aforementioned hyperparameter tuning provides a good starting point for parameters, while fine-tuning based on the performance on the validation set during training can ensure the stability and excellent performance of the model in practical applications.

[0124] Step 4: Perform breast cancer histological image segmentation and classification using the trained Deep WSI-Stroma model, and visualize the images using Grad-CAM.

[0125] The original input image is processed through a multi-scale attention fusion network and the Grad-CAM heatmap generation algorithm to generate a high-resolution segmentation probability heatmap and salient regions, visually displaying the spatial distribution and key regions of the epithelial-mesenchymal segmentation. The detailed process of Grad-CAM visualization is as follows: Figure 6 As shown.

[0126] Step 5: Model Evaluation;

[0127] Evaluation metrics: ROC curve ( Figure 7 AUC values ​​(Table 3), confusion matrix (Table 4), visualization results ( Figure 8 The model performance was evaluated using multiple metrics, including accuracy, recall, and F1 score (Table 5).

[0128] Experimental Results: Experiments on the TCGA-BRCA dataset show that the Deep WSI-Stroma model achieves an average Dice score of 0.873 in the segmentation task and an average F1 score of 0.921 across the three classes in the classification task, significantly outperforming existing methods.

[0129] Step 6: Apply prediction and screening strategies;

[0130] Epithelial-mesenchymal ratio (ESR): The number of pixels in the epithelial and mesenchymal regions of each WSI is calculated to obtain the ESR value, which is used to predict and screen high-risk cases.

[0131] Spatial distribution pattern analysis: Using graph theory and topological analysis methods, spatial features in the epithelial-mesenchymal segmentation results are extracted, and a multivariate logistic regression model is constructed to predict the invasiveness and metastatic risk of tumors.

[0132] This embodiment evaluates the performance of the proposed method in epithelial-mesenchymal segmentation and classification of breast cancer histological images on the TCGA-BRCA dataset, and conducts comprehensive quantitative and qualitative analyses. To ensure the reliability and generalization of the experimental results, the composition and distribution of the dataset were first statistically analyzed. Regarding model training, the changes in the loss function and evaluation metrics during the training process were recorded in detail, demonstrating the convergence and stability of the model.

[0133] In terms of quantitative evaluation, multiple metrics, including ROC curves, AUC values, confusion matrices, and classification indicators, were used to comprehensively evaluate the model's performance on the test set. The impact of different thresholds on segmentation and classification results was also explored, providing a reference for practical applications. In terms of qualitative evaluation, heatmaps and overlay image visualizations visually demonstrated the model's segmentation and classification performance on typical cases, proving its potential for clinical application. Furthermore, the model was applied to prediction and screening tasks, analyzing its ability to identify high-risk cases and guide clinical decision-making, further validating the practical value of the proposed method.

[0134] Dataset statistics and analysis

[0135] This invention utilizes the publicly available TCGA-BRCA dataset to evaluate the performance of the proposed method in epithelial-mesenchymal segmentation and classification of breast cancer histological images. This dataset contains H&E-stained whole-slice pathological images (WSI) from 1,043 breast cancer patients, with an average resolution of 80,000 x 60,000 pixels, enabling high-resolution capture of detailed tissue features. To fully understand the characteristics and distribution of the data, comprehensive statistical analysis was performed on the TCGA-BRCA dataset. First, the area ratio of epithelial to stromal regions in each WSI was calculated. The results show that the epithelial-mesenchymal ratio in this dataset is approximately 2:1, meaning the area of ​​the epithelial region is approximately twice that of the stromal region, consistent with reported histological characteristics of breast cancer in the literature. Second, the distribution of different breast cancer subtypes and stages in the dataset was analyzed. The TCGA-BRCA dataset covers major breast cancer subtypes, such as hormone receptor-positive (HR+), human epidermal growth factor receptor 2-positive (HER2+), and triple-negative breast cancer (TNBC), and the proportions of different subtypes are consistent with reports from epidemiological studies. In addition, the dataset also includes breast cancer cases at different stages (I-IV), reflecting different stages of tumor progression.

[0136] To train and evaluate the Deep WSI-Stroma model, the TCGA-BRCA dataset was randomly divided into a training set (80%), a validation set (10%), and a test set (10%). During the partitioning process, a balanced distribution of different subtypes and stages across the subsets was ensured to avoid the impact of class skew on model performance. Furthermore, data augmentation was performed on the WSI data in each subset, such as random pruning, rotation, and flipping, to improve the model's robustness and generalization ability. Statistical analysis of the TCGA-BRCA dataset provided a comprehensive understanding of its characteristics and distribution, laying the foundation for subsequent model training and evaluation. Simultaneously, the representativeness and applicability of this dataset in breast cancer histology research were verified, ensuring the reliability and clinical relevance of the experimental results. This provides a solid data foundation for evaluating the performance of the proposed method in breast cancer epithelial-mesenchymal segmentation and classification tasks.

[0137] Model training and convergence

[0138] During the training of the deep learning model, we closely monitored its convergence and performance changes. We used the PyTorch deep learning framework and trained it on a workstation equipped with an NVIDIA GeForce RTX 3090 GPU. The backbone network of the model was EfficientNet-B3, followed by attention mechanism modules, feature fusion modules, and segmentation and classification modules.

[0139] We used the cross-entropy loss function and the Adam optimizer, with a learning rate of 0.001 and a batch size of 32. At the end of each training epoch, we evaluated the model's performance on the validation set and recorded metrics such as accuracy, precision, recall, and F1 score. We implemented an early stopping mechanism, stopping training if the validation set performance did not improve for 10 consecutive epochs to prevent overfitting.

[0140] During training, we observed significant performance improvements in the first few epochs, followed by a gradual plateauing. The loss functions of both the training and validation sets showed a good decreasing trend, indicating continuous learning and optimization. Around the 50th epoch, the validation set performance reached its optimal level, with an accuracy of 92.5%, precision of 91.8%, recall of 93.2%, and an F1 score of 92.5%. Afterward, although the performance of the training set continued to improve, the performance of the validation set began to fluctuate, suggesting potential overfitting. Therefore, we selected the model from the 50th epoch as the final training result. The training process and convergence of the model in this invention are shown in Table 1.

[0141] Table 1. Performance Indicator Changes During Training

[0142]

[0143] To further analyze the model's convergence, we plotted the loss and accuracy curves for the training and validation sets. The curves are shown below. Figure 9 and Figure 10 As shown.

[0144] from Figure 9 and Figure 10 As can be seen, the loss curves of the training and validation sets decrease rapidly in the early stages and gradually stabilize in the later stages. The two curves basically overlap, without obvious separation or oscillation, indicating that the model does not suffer from overfitting or underfitting. The accuracy curve also shows a similar trend, reaching its peak around the 50th cycle, and then fluctuating slightly but remaining generally stable.

[0145] The data tables and graphs above provide a more intuitive view of the training process and convergence of the Deep WSI-Stroma model on the TCGA-BRCA dataset. As the tables show, the model's performance metrics on the validation set continuously improve as training progresses, reaching their optimal state around the 50th epoch. The loss and accuracy curves also exhibit a good convergence trend, demonstrating the model's effectiveness and stability.

[0146] Furthermore, this invention compares the convergence of the model under different hyperparameter settings, such as learning rate, batch size, and regularization strength. The results show that appropriate hyperparameter selection significantly affects the model's convergence speed and final performance. This invention obtains an optimized set of hyperparameters through grid search and cross-validation, enabling the model to achieve optimal performance on the TCGA-BRCA dataset. Table 2 shows the model's performance metrics on the validation set under different learning rates, batch sizes, and regularization strengths.

[0147] Table 2. Model performance metrics on the validation set under different learning rates, batch sizes, and regularization strengths.

[0148]

[0149] As shown in Table 2, different combinations of hyperparameters lead to differences in model performance. For example, when the learning rate is 0.001, the batch size is 32, and the regularization strength is 0.001, the model achieves the best performance on the validation set, with an accuracy of 94.25%, precision of 93.18%, recall of 95.57%, and an F1 score of 94.36%.

[0150] The experimental data above demonstrates that different hyperparameter settings do indeed have a significant impact on model performance. In practice, by systematically searching and comparing different hyperparameter combinations, the optimal hyperparameter settings can be found to achieve the best model performance. This process not only improves the model's performance on the TCGA-BRCA dataset but also provides a reference for the model's application on other similar datasets.

[0151] By meticulously recording and analyzing the model's training process and convergence, the effectiveness and stability of the Deep WSI-Stroma model on the breast cancer WSI dataset were demonstrated. The rational network architecture design, optimized algorithm selection, and hyperparameter tuning ensured that the model could learn robust and generalized feature representations from large-scale, high-resolution WSI data, laying a solid foundation for subsequent segmentation and classification tasks.

[0152] Quantitative assessment results

[0153] To comprehensively evaluate the performance of the Deep WSI-Stroma model in epithelial-mesenchymal segmentation and classification tasks of breast cancer histological images, extensive quantitative experiments were conducted on the test set of the TCGA-BRCA dataset. Several commonly used evaluation metrics were employed, including ROC curves, AUC values, confusion matrices, and classification metrics, to comprehensively characterize the model's performance.

[0154] ROC curve and AUC value on the test set

[0155] To evaluate the performance of the Deep WSI-Stroma model in epithelial-mesenchymal segmentation and classification of breast cancer histological images, ROC curves were first plotted on the test set, and the corresponding AUC values ​​were calculated. The ROC curves illustrate the trade-off between the model's true positive rate (TPR) and false positive rate (FPR) under different threshold settings. An ideal model should maintain a high TPR while minimizing the FPR as much as possible; that is, the ROC curve should be as close as possible to the upper left corner. The AUC value quantifies the area under the ROC curve, ranging from 0 to 1; a higher value indicates better model performance.

[0156] On the test set, the ROC curve of the Deep WSI-Stroma model is very close to the upper right corner (e.g., Figure 7As shown in the figure, it achieves an excellent balance between high TPR and low FPR. The model's AUC value reached 0.972, indicating its superior performance on the epithelial-mesenchymal classification task. To further evaluate the superiority of the Deep WSI-Stroma model, it was compared with other classic machine learning classification models, including Support Vector Machine (SVM) and Random Forest.

[0157] from Figure 7 The comparison of ROC curves shows that the Deep WSI-Stroma model's curve is closest to the upper right corner, exhibiting an optimal balance between TPR and FPR. The ROC curves of other models are relatively closer to the diagonal of random guessing, indicating that their performance is inferior to the method in this invention.

[0158] Table 3 Comparison of AUC values ​​of different models on the test set

[0159] Model AUC value DeepWSI-Stroma 0.972 SVM 0.923 RandomForest 0.941

[0160] Table 3 shows a comparison of AUC values, which further quantifies the performance differences between different models. The Deep WSI-Stroma model, with an AUC value of 0.972, significantly outperforms SVM (0.923) and Random Forest (0.941), demonstrating the advantages of deep learning methods in processing complex WSI data.

[0161] By comparing the ROC curves and AUC values ​​of different models (e.g.) Figure 7 As shown in Table 3, the Deep WSI-Stroma model significantly outperformed other models. Its ROC curve was closest to the ideal state, while the curves of other models were relatively closer to the diagonal of random guessing. The Deep WSI-Stroma model significantly outperformed SVM (0.923) and Random Forest (0.941) with an AUC value of 0.972, quantitatively confirming its performance advantage.

[0162] These experimental results strongly demonstrate the significant advantages of deep learning methods, especially the Deep WSI-Stroma model proposed in this invention, in processing large-scale, high-resolution WSI data. Deep learning models can automatically learn and extract high-level feature representations from massive amounts of pixel-level data, and make accurate segmentation and classification decisions based on these representations. Compared to traditional machine learning methods, deep learning models exhibit stronger feature representation capabilities and generalization performance, bringing new breakthroughs to pathological image analysis tasks.

[0163] Evaluation using ROC curves and AUC values ​​on the test set fully demonstrates the outstanding performance of the Deep WSI-Stroma model in epithelial-mesenchymal segmentation and classification tasks in breast cancer histological images. This result not only highlights the enormous potential of deep learning technology in the field of medical image analysis but also lays a solid foundation for subsequent clinical applications.

[0164] Confusion matrix and classification metrics on the test set

[0165] To comprehensively evaluate the performance of the Deep WSI-Stroma model in the epithelial-mesenchymal classification task of breast cancer histological images, we generated a confusion matrix on the test set and calculated key classification metrics such as precision, recall, and F1 score. The confusion matrix shows the distribution of the model's predictions for each class, while the classification metrics quantify the model's performance in different aspects.

[0166] As shown in Table 4, the Deep WSI-Stroma model achieved a highly balanced and accurate predictive performance across both the epithelial and mesenchymal categories on the test set. The model achieved precision, recall, and F1 score of 93.2%, 95.6%, and 94.4% for the epithelial category, respectively, and 91.5%, 90.3%, and 90.9% for the mesenchymal category. These results indicate that the model has low prediction error rates in both categories and achieves a good balance between accuracy and completeness.

[0167] Table 4. Confusion matrix of the Deep WSI-Stroma model on the test set

[0168] Actual organization type Predicting epithelium Predicting interstitial tissue Actually skin 1568 72 Actual interstitium 95 885

[0169] As can be seen from the confusion matrix in Table 4, the Deep WSI-Stroma model achieved a highly balanced and accurate predictive performance across both the epithelial and mesenchymal categories on the test set. In the epithelial category, the model correctly predicted 1568 samples, with only 72 samples incorrectly classified as mesenchymal; while in the mesenchymal category, the model correctly predicted 885 samples, with only 95 samples incorrectly classified as epithelial. These results intuitively demonstrate that the model exhibits low prediction error rates in both categories and shows no significant bias.

[0170] To further analyze the performance of the Deep WSI-Stroma model across different categories, this invention further calculated classification metrics such as precision, recall, and F1 score for each category (as shown in Table 5). For the epithelial category, the precision reached 93.2%, the recall 95.6%, and the F1 score 94.4%; for the mesenchymal category, the precision was 91.5%, the recall 90.3%, and the F1 score 90.9%. These quantitative metrics fully demonstrate the excellent predictive performance of the Deep WSI-Stroma model across both categories, accurately identifying true positive samples (high recall) and effectively eliminating false positive samples (high precision), thus achieving a good balance between accuracy and reliability.

[0171] Table 5. Classification metrics of the Deep WSI-Stroma model on the test set.

[0172] category Accuracy Recall rate F1 value Epithelium 0.932 0.956 0.944 interstitial 0.915 0.903 0.909

[0173] Heatmap visualization

[0174] Heatmaps are a commonly used visualization method to represent the importance or attributes of different regions in an image. In this study, we generated probabilistic heatmaps (such as those based on the Deep WSI-Stroma model output) Figure 8 As shown in the image, the distribution of epithelial and stromal regions in WSI is intuitively displayed. The segmentation and classification performance of the Deep WSI-Stroma model on breast cancer histological images was qualitatively evaluated using heatmap visualization technology. Experimental results show that the model can accurately capture the epithelial-stromal distribution patterns in WSI, and the generated visualization results are highly consistent with the annotations of pathologists. These intuitive visual results not only validate the effectiveness of the model but also provide valuable reference for clinical diagnosis and treatment.

[0175] The above description, in conjunction with specific / preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. Those skilled in the art can make various substitutions or modifications to these described embodiments without departing from the inventive concept, and all such substitutions or modifications should be considered within the scope of protection of the present invention.

[0176] The parts of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for segmentation and classification of breast cancer histological images based on multi-scale deep learning, characterized in that, The steps include the following: Step 1: Data acquisition and preprocessing; Step 2: Constructing the Deep WSI-Stroma model for deep whole-slice breast cancer image-stromal segmentation and classification; The breast cancer deep whole slice image-stromal segmentation and classification model adopts a multi-scale attention fusion network architecture, including a feature extraction module, an attention mechanism module, a feature fusion module, and a segmentation and classification module; Step 3: Model training; We employ a cross-entropy loss function and the Adam optimizer; we adopt a hybrid training strategy, training simultaneously using both patch and WSI samples, and design corresponding loss functions and optimization methods to enable the model to learn both local and global feature representations simultaneously; the specific methods are as follows: 3-1. Loss Function and Optimizer: Cross-entropy loss is used as a method to calculate the error of a single task. The joint loss function optimizes both segmentation and classification tasks simultaneously. The joint loss function combines pixel-level segmentation loss and region-level classification loss. The segmentation loss is the pixel-level cross-entropy loss of the image segmentation task, and the classification loss is the region-level cross-entropy loss of the classification task. The Adam optimizer is used with an initial learning rate of 0.01 and a learning rate decay strategy is adopted. 3-2. Hybrid Training Strategy: In each training batch, a large WSI image is divided into multiple small patch samples, while the entire WSI image is retained as a WSI sample. That is, each batch contains a predetermined number of patch samples and WSI samples. Data from local regions of the image and the entire image are used together to update the model's parameters; a patch-level joint loss is employed. Used to measure the classification and segmentation error of local patches; joint loss at the WSI level. The classification and segmentation error of the entire WSI image is used to measure the overall joint loss function. The joint loss function is obtained by weighting and summing the patch-level joint loss and the WSI-level joint loss using balancing factors α and β. The calculation of the overall joint loss function is shown in Equation (1). ； （1） 3-3. Perform batch normalization and L2 regularization on the training data: 3-4. Hyperparameter tuning: (1) Define parameter range: Set the candidate range for batch size, learning rate, regularization coefficient, and joint loss balance factor hyperparameters; (2) Grid search and cross-validation: Grid search is performed within the parameter range, and the performance of each parameter combination is evaluated through cross-validation; (3) Select the best combination: Based on the cross-validation results, select the best combination of hyperparameters; 3-5. Training process: (1) Use the optimal combination determined by hyperparameter tuning for final model training; in each training batch, apply the optimal batch size after tuning for sampling; use the optimal learning rate after tuning for gradient updates; apply the tuned regularization coefficient and joint loss balance factor for loss calculation and optimization; (2) In each training batch, random samples are taken from the patch data loader and the WSI data loader to form a mixed training batch; (3) Calculate the joint loss at the patch level and the joint loss at the WSI level; (4) Use the balance factor to perform a weighted summation of the two joint losses to obtain the total joint loss function; (5) Perform backpropagation based on the total joint loss function to update the model parameters; 3-6. Model Evaluation and Validation: Evaluate model performance on the validation set using metrics such as accuracy, precision, recall, and F1 score; make fine adjustments based on validation set performance to ensure excellent model performance on the test set; Step 4: Perform breast cancer histological image segmentation and classification using the trained Deep WSI-Stroma model, and visualize it using Grad-CAM.

2. The breast cancer histological image segmentation and classification method based on multi-scale deep learning according to claim 1, characterized in that, The specific method for step 1 is as follows: Reading WSI data: Use the OpenSlide library to read Aperio SVS format files from the TCGA-BRCA dataset and extract high-resolution images and annotation information; Image segmentation and patch generation: Foreground-background segmentation of WSI was performed using an improved Otsu thresholding method to extract tissue region masks; A series of image patches are generated by sliding a window with a size of 256×256 and a step size of 128 within the tissue mask; Image normalization: The generated image patches are color normalized to reduce the impact of color differences on the model; Data augmentation: Divide the training set and validation set according to requirements, and perform data augmentation operations on the patches of the training set, including random horizontal flipping, vertical flipping, rotation, color transformation, random cropping, scaling, translation, and shearing, to increase the diversity of training samples and enhance the robustness of the model.

3. The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning according to claim 1 or 2, characterized in that, The breast cancer deep whole slice image-stromal segmentation and classification model uses EfficientNetV2-B3 pre-trained on the ImageNet dataset as the feature extraction module. Multi-scale features are extracted at different levels of EfficientNetV2-B3, namely through the 2nd, 4th and 6th MBConv blocks of EfficientNetV2-B3. The attention mechanism module includes two branches: a spatial attention branch and a channel attention branch; The spatial attention branch generates a weight map through convolution and pooling operations, processes the pooling results through an activation function, and generates a spatial attention weight map, which represents the importance of each position in the feature map. The channel attention branch learns the importance weights of each channel through global average pooling and fully connected layers, and generates a channel attention weight map through an activation function, which represents the importance of each channel in the feature map. Finally, the spatial attention weight map and the channel attention weight map are multiplied to generate a comprehensive attention weight. The comprehensive attention weight is then used to weight the original input feature map to enhance the representation of key regions and features, generating an enhanced feature map. After passing through the attention mechanism module, low-level detailed features and high-level global features, i.e., multi-scale features, can be obtained. The feature fusion module adaptively assigns weights to features of different scales through weight learning, and then combines the weighted features through addition operations to output the final fused feature map. The segmentation and classification module includes a segmentation head and a classification head, which respectively implement the segmentation task and the classification task. The fused feature map output by the feature fusion module is simultaneously input into the segmentation head and the classification head for processing. The segmentation head generates a segmentation mask through convolutional layers, activation functions and upsampling layers, and the classification head generates a classification result through global average pooling layers and fully connected layers.

4. The breast cancer histological image segmentation and classification method based on multi-scale deep learning according to claim 1, characterized in that, The optimal combination of hyperparameters is: batch size 32, learning rate 0.001, regularization coefficient 0.001, and balance factors α and β both 0.

5.

5. The breast cancer histological image segmentation and classification method based on multi-scale deep learning according to claim 3, characterized in that, Step 4 is explained in detail below: After the original input image is processed by a trained breast cancer deep whole slice image-mesenchymal segmentation and classification model and the Grad-CAM heatmap generation algorithm, a high-resolution segmentation probability heatmap and salient regions are generated to visually display the spatial distribution and key regions of epithelial-mesenchymal segmentation.

Images