Breast cancer histological image segmentation and classification method based on multi-scale deep learning

Abstract

The invention discloses a breast cancer histological image segmentation and classification method based on multi-scale deep learning. The method comprises the following steps: firstly, carrying out data acquisition and preprocessing; then, a breast cancer deep full-slice image-interstitial segmentation and classification model is constructed, and the model adopts a multi-scale attention fusion network architecture and comprises a feature extraction module, an attention mechanism module, a feature fusion module and a segmentation and classification module; and carrying out model training through the preprocessed data, finally completing breast cancer histological image segmentation and classification through the trained model, and carrying out Grad-CAM visualization. Through an innovative multi-scale deep learning architecture and technical measures, the automation degree, precision and robustness of breast cancer histological image segmentation and classification are remarkably improved, and powerful technical support is provided for clinical application. Compared with the prior art, the method has the advantages that higher automation level, higher segmentation and classification precision and higher global information utilization capability are realized functionally.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a breast cancer histological image segmentation and classification method based on multi-scale deep learning. Background Art

[0002] In the process of breast cancer diagnosis and treatment, the segmentation and classification of histological images are crucial steps. Traditional pathological diagnosis methods mainly rely on pathologists' manual inspection of tissue sections, which is not only time-consuming and labor-intensive, but also easily affected by human factors, making it difficult to ensure the accuracy and consistency of the diagnostic results. With the development of digital pathology technology, the use of computer-assisted analysis of whole slide pathology images (WSI) has become an important way to improve diagnostic efficiency and accuracy.

[0003] 1. Traditional image segmentation method:

[0004] Traditional image segmentation methods include threshold segmentation, region growing, morphological operations, and graph cuts. These methods usually rely on manually designed features and prior knowledge, and often show inadequacies when processing complex and diverse pathological images. For example, the threshold segmentation method is difficult to adapt to images under different staining conditions, and morphological operations are sensitive to noise and interference. Vu et al. (2016) used stacked denoising autoencoders to learn features of breast cancer pathology images, but their segmentation performance was still limited by the expressive power of manual features.

[0005] 2. Segmentation and classification methods based on machine learning:

[0006] With the development of machine learning, methods such as support vector machine (SVM), random forest and K nearest neighbor have been widely used in the segmentation and classification of pathological images. These methods replace the process of manual feature extraction by learning feature representation from training data. However, these shallow learning models still have limitations in dealing with high-level semantic features and complex image structures. For example, Kandemir et al. (2014) used a multi-instance learning framework to classify breast cancer tissues, but their classification performance was limited by the complexity of feature selection and model design.

[0007] 3. Segmentation and classification methods based on deep learning:

[0008] In recent years, deep learning, especially convolutional neural networks (CNNs), has made significant progress in medical image analysis. Ciresan et al. (2012) first applied CNN to the classification of breast cancer pathology images and achieved excellent performance. Subsequently, deep learning models such as U-Net, DeepLab, and PSPNet have demonstrated powerful feature extraction and segmentation capabilities in medical image segmentation tasks. For example, Chen et al. (2018) proposed a breast tissue segmentation method based on U-Net, which achieved optimal performance on multiple public datasets by introducing attention mechanisms and adversarial training.

[0009] Existing methods for segmentation and classification of breast cancer histology images have significant deficiencies: traditional methods rely on manual feature design and are difficult to adapt to complex and diverse pathological images; shallow learning models are limited in high-level semantic feature extraction and classification performance; despite advances in deep learning methods, they are still inefficient and inaccurate in processing high-resolution WSI data. These methods are difficult to effectively fuse multi-scale information and lack sufficient modeling of context, resulting in limited segmentation and classification accuracy. The deep learning method proposed in the present invention effectively improves the model's robustness and generalization ability through multi-scale feature fusion and attention mechanism, overcoming the above shortcomings.

[0010] Cited Literature

[0011] 1.Vu, QD, et al. "Stacked Denoising Autoencoders for Nuclei Detection in Breast Cancer Histopathology Images." IEEE Transactions on Medical Imaging35.5(2016):1192-1203.

[0012] 2. Kandemir, M., et al. "Computer-aided diagnosis in breast cancer using tissue morphometry, mammographic density, and wavelet analysis." Medical ImageAnalysis 18.5(2014):765-773.

[0013] 3. Ciresan, DC, et al. "Deep neural networks segment neuronal membranes in electron microscopy images." Advances in Neural Information ProcessingSystems 25(2012):2843-2851.

[0014] 4.Chen,L.,et al."Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation."European Conference on ComputerVision.Springer,Cham,2018. Summary of the invention

[0015] The present invention aims to solve the shortcomings of existing breast cancer histology image segmentation and classification methods. Specific technical problems include:

[0016] Low degree of automation: Traditional methods rely on manual feature extraction, which is time-consuming and labor-intensive, easily affected by human factors, and difficult to cope with complex and diverse pathological images.

[0017] Inadequate feature extraction and classification performance: Machine learning methods based on shallow learning perform poorly in extracting high-level semantic information and processing complex structures, and have limited generalization capabilities.

[0018] Ignoring global structural information: Although existing deep learning methods have improved segmentation accuracy, they usually only focus on local pixel-level features and ignore the global structure and contextual information of epithelial-mesenchymal tissue, affecting the continuity and integrity of segmentation.

[0019] Lack of joint learning and reasoning capabilities: Most existing methods treat segmentation and classification as independent tasks, which increases the complexity of the model and makes the post-processing steps of integrating the results cumbersome.

[0020] In view of the above technical problems, the present invention provides a breast cancer histological image segmentation and classification method based on multi-scale deep learning, which is used for epithelial-stromal segmentation and classification in breast cancer histological whole slice images (WSI). It aims to overcome the shortcomings of the prior art and achieve the following goals:

[0021] Automation and efficiency: It adopts an end-to-end convolutional neural network architecture, which does not require manual feature design. It can automatically and quickly process complex and diverse pathological images and improve the efficiency of segmentation and classification.

[0022] High-precision feature extraction and classification: Designing a multi-scale, multi-cascade network structure and fusing feature maps at different resolutions can capture the characteristics of local cells and overall tissues, improving the accuracy and robustness of segmentation and classification.

[0023] Utilization of global information: Introducing attention mechanism and adversarial training strategy to enhance the model's ability to capture global structure and contextual information and improve the continuity and completeness of segmentation.

[0024] End-to-end joint learning: An end-to-end segmentation and classification network structure is proposed to achieve joint learning and reasoning of pixel-level segmentation and region-level classification, simplify post-processing steps, and improve overall performance.

[0025] Through the above improvements, the present invention can significantly improve the accuracy, efficiency and automation level of breast cancer histological image segmentation and classification, and provide strong support for clinical diagnosis and prognosis analysis.

[0026] The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning includes the following steps:

[0027] Step 1: Data acquisition and preprocessing;

[0028] Step 2: Construct a deep whole-slice image-stroma segmentation and classification model for breast cancer (Deep WSI-Stroma);

[0029] The breast cancer deep full-slice image-stroma segmentation and classification model adopts a multi-scale attention fusion network architecture (MAFNet), including a feature extraction module, an attention mechanism module, a feature fusion module, and a segmentation and classification module.

[0030] Step 3: Model training;

[0031] The cross entropy loss function and Adam optimizer are used. A hybrid training strategy is adopted, using both patch and WSI samples for training, and the corresponding loss function and optimization method are designed to enable the model to learn local and global feature representations at the same time.

[0032] Step 4: Complete breast cancer histological image segmentation and classification through the trained Deep WSI-Stroma model, and perform Grad-CAM visualization;

[0033] Furthermore, the specific method of step 1 is as follows:

[0034] Read WSI data: Use the OpenSlide library to read the Aperio SVS format files in the TCGA-BRCA dataset and extract high-resolution images and annotation information.

[0035] Image segmentation and patch generation: The improved Otsu thresholding method is used to perform foreground-background segmentation on WSI and extract the tissue region mask. A sliding window with a size of 256×256 and a step size of 128 is generated within the tissue mask to generate a series of image patches.

[0036] Image normalization: Perform color normalization on the generated image patches to reduce the impact of coloring differences on the model.

[0037] Data enhancement: Divide the training set and validation set according to the needs, and perform data enhancement operations on the patches of the training set, including random horizontal flipping, vertical flipping, rotation, color change, random cropping, scaling, translation, and shearing, to increase the diversity of training samples and enhance the robustness of the model.

[0038] Furthermore, the breast cancer deep whole-slice image-stroma segmentation and classification model uses EfficientNetV2-B3 pre-trained on the ImageNet dataset as a feature extraction module, and extracts multi-scale features at different levels of EfficientNetV2-B3, namely through the 2nd, 4th and 6th MBConv blocks of EfficientNetV2-B3.

[0039] The attention mechanism module includes two main branches: spatial attention branch and channel attention branch. The spatial attention branch generates a weight map through convolution and pooling operations, and processes the pooling result through activation function to generate a spatial attention weight map, which represents the importance of each position in the feature map. The channel attention branch learns the importance weight of each channel through global average pooling and fully connected layer, and generates a channel attention weight map through activation function, which represents the importance of each channel in the feature map. Finally, the spatial attention weight map and the channel attention weight map are multiplied to generate a comprehensive attention weight, and then the comprehensive attention weight is used to weight the original input feature map to enhance the representation of key areas and features and generate an enhanced feature map. After the attention mechanism module, lower-level detail features and higher-level global features, i.e., multi-scale features, can be obtained. Introducing the attention mechanism module after all MBConv blocks will greatly increase the computational complexity. Therefore, selectively introducing the attention mechanism module after the 2nd, 4th, and 6th blocks can control the computational complexity and memory consumption while increasing the model performance.

[0040] The feature fusion module adaptively assigns weights to features of different scales through weight learning, and then combines the weighted features through an addition operation to output a final fused feature map.

[0041] The segmentation and classification module includes a segmentation head and a classification head, which respectively implement segmentation tasks and classification tasks; the fusion feature map output by the feature fusion module is simultaneously input into the segmentation head and the classification head for processing respectively, the segmentation head generates a segmentation mask through a convolution layer, an activation function and an upsampling layer, and the classification head generates a classification result through a global average pooling layer and a fully connected layer.

[0042] Furthermore, the specific method of step 3 is as follows:

[0043] 3-1. Loss function and optimizer:

[0044] The cross entropy loss is used as the basic method to calculate the error of a single task (such as classification or segmentation). The segmentation and classification tasks are optimized simultaneously through a joint loss function, which combines the pixel-level segmentation loss and the region-level classification loss. The segmentation loss is the cross entropy loss of the pixel-level image segmentation task, and the classification loss is the cross entropy loss of the region-level classification task. The Adam optimizer is used, the initial learning rate is set to 0.01, and the learning rate decay strategy is adopted.

[0045] 3-2. Mixed training strategy:

[0046] In each training batch, the large-size WSI image is divided into multiple small patch samples, while the entire WSI image is retained as the WSI sample, that is, each batch contains a certain number of patch samples and WSI samples, and the data of the local area (patch) and the whole image (WSI) are mixed to update the parameters of the model. Patch-level joint loss L patch Used to measure the classification and segmentation error of local patches. The joint loss L at the WSI level WSI It is used to measure the classification and segmentation errors of the entire WSI image. The total joint loss function is obtained by weighted summing the joint loss at the patch level and the joint loss at the WSI level using the balance factors α and β. The calculation of the total joint loss function is shown in formula (1):

[0047] L total =αL patch +βL WSI ; (1)

[0048] The total joint loss function in the above formula combines features of different scales (local patch and global WSI) to optimize the overall performance of the model.

[0049] 3-3. Perform batch normalization and L2 regularization on the training data:

[0050] 3-4. Hyperparameter tuning:

[0051] (1) Define parameter ranges: Set candidate ranges for batch size, learning rate, regularization coefficient, and joint loss balance factor hyperparameters.

[0052] (2) Grid search and cross-validation: Grid search is performed within the parameter range, and the performance of each parameter combination is evaluated through cross-validation.

[0053] (3) Select the best combination: Based on the cross-validation results, select the best hyperparameter combination.

[0054] 3-5. Training process:

[0055] (1) Use the best combination determined by hyperparameter tuning for final model training. In each training batch, apply the tuned best batch size for sampling. Use the tuned best learning rate for gradient updates. Apply the tuned regularization coefficient and joint loss balance factor for loss calculation and optimization.

[0056] (2) In each training batch, random samples are taken from the patch data loader and the WSI data loader to form a mixed training batch.

[0057] (3) Calculate the joint loss at the patch level and the joint loss at the WSI level.

[0058] (4) Use the balance factor to perform weighted summation of the two joint losses to obtain the total joint loss function.

[0059] (5) Perform back propagation based on the total joint loss function to update the model parameters.

[0060] 3-6. Model evaluation and validation:

[0061] Evaluate the model performance on the validation set using accuracy, precision, recall, and F1 score metrics. Make minor adjustments based on the validation set performance to ensure the model performs well on the test set.

[0062] In one embodiment, the optimal hyperparameter combination is: batch size 32, learning rate 0.001, regularization coefficient 0.001, and balance factors α and β are both 0.5.

[0063] Furthermore, the specific method of step 4 is as follows:

[0064] After the original input image is passed through the trained breast cancer deep full-slice image-stroma segmentation and classification model and the Grad-CAM heat map generation algorithm, a high-resolution segmentation probability heat map and significant areas are generated to intuitively display the spatial distribution and key areas of epithelial-stroma segmentation.

[0065] The beneficial effects of the present invention are as follows:

[0066] The breast cancer histological image segmentation and classification method (DeepWSI-Stroma) based on multi-scale deep learning proposed in the present invention has significant advantages and positive effects compared with the existing technology. These advantages and effects are directly brought about by the technical solution of the present invention, or are the inevitable results of the technical solution:

[0067] 1. High degree of automation;

[0068] The existing technology relies on manual feature extraction and expert experience, which is time-consuming and labor-intensive, and is easily affected by human factors. The present invention adopts an end-to-end convolutional neural network architecture, which can automatically learn features from the original WSI image, realize fully automatic segmentation and classification, greatly improve efficiency, and reduce human errors.

[0069] 2. High-precision feature extraction and classification;

[0070] The present invention designs a multi-scale, multi-cascade network structure, which can capture the characteristics of local cells and overall tissues by fusing feature maps at different resolutions, thereby improving the accuracy of segmentation and classification. Experimental results show that the present invention achieves an average Dice score of 0.873 in the segmentation task on the TCGA-BRCA dataset, and an average F1 value of 0.921 for the three categories in the classification task, which is significantly better than methods such as U-Net (Dice score of 0.821) and traditional manual features + SVM (F1 value of 0.843).

[0071] 3. Global information utilization;

[0072] The present invention introduces the attention mechanism and adversarial training strategy to enhance the model's ability to capture global structure and contextual information, and improve the continuity and integrity of segmentation. The attention mechanism can automatically learn the key areas and features in the image, helping the model to focus on the most relevant information, while adversarial training improves the model's robustness to interference and noise.

[0073] 4. End-to-end joint learning;

[0074] The end-to-end segmentation and classification network structure proposed in this invention realizes the joint learning and reasoning of pixel-level segmentation and region-level classification, simplifies the post-processing steps, and improves the overall performance. Compared with the existing methods that treat segmentation and classification as independent tasks, this invention significantly improves the complexity management and prediction accuracy of the model.

[0075] 5. Clinical application value;

[0076] By introducing Grad-CAM technology and multivariate logistic regression models, the present invention can generate high-resolution segmentation probability heat maps and significant regions, intuitively displaying the spatial distribution and key areas of epithelial-stromal segmentation, and helping pathologists understand and verify the basis for model decision-making. In addition, epithelial-stromal ratio (ESR) and spatial distribution pattern analysis can effectively predict the invasiveness and metastasis risk of tumors, providing strong support for clinical diagnosis and treatment decisions.

[0077] In summary, the present invention significantly improves the automation, accuracy and robustness of breast cancer histological image segmentation and classification through innovative multi-scale deep learning architecture and technical measures, providing strong technical support for clinical applications. Compared with the existing technology, the present invention functionally achieves a higher level of automation, higher segmentation and classification accuracy, and stronger global information utilization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a model architecture diagram of the technical solution of an embodiment of the present invention;

[0079] Figure 2 This is a schematic diagram of the structure of the attention mechanism module according to an embodiment of the present invention;

[0080] Figure 3 This is a schematic diagram of the structure of a feature fusion module according to an embodiment of the present invention;

[0081] Figure 4 A schematic diagram of the segmentation and classification structure of an embodiment of the present invention;

[0082] Figure 5 Schematic diagram of the MBConv block structure of an embodiment of the present invention;

[0083] Figure 6 A flowchart of the Grad-CAM visualization process according to an embodiment of the present invention;

[0084] Figure 7 It is a comparison diagram of ROC curves of different models on the test set according to the embodiment of the present invention;

[0085] Figure 8 A breast cancer WSI sample and its corresponding heat map according to an embodiment of the present invention;

[0086] Fig. 9 : are the loss curves of the training set and the validation set of the embodiment of the present invention;

[0087] Fig.10 It is the accuracy curve of the training set and the validation set of the embodiment of the present invention. DETAILED DESCRIPTION

[0088] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0089] The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning includes the following steps:

[0090] Step 1: Data acquisition and preprocessing;

[0091] Read WSI data: Use the OpenSlide library to read the Aperio SVS format files in the TCGA-BRCA dataset and extract high-resolution images and annotation information.

[0092] Image segmentation and patch generation: The improved Otsu thresholding method is used to perform foreground-background segmentation on WSI and extract the tissue region mask. A sliding window with a size of 256×256 and a step size of 128 is generated within the tissue mask to generate a series of image patches.

[0093] Image normalization: Perform color normalization on the generated image patches to reduce the impact of coloring differences on the model.

[0094] Data enhancement: Divide the training set and validation set according to the needs, and perform data enhancement operations such as random horizontal flipping, vertical flipping, rotation, color change, random cropping, scaling, translation, and shearing on the patches of the training set to increase the diversity of training samples and enhance the robustness of the model.

[0095] Step 2: Construct a deep whole-slice image-stroma segmentation and classification model for breast cancer (Deep WSI-Stroma), such as Figure 1 As shown;

[0096] The breast cancer deep full-slice image-stroma segmentation and classification model adopts a multi-scale attention fusion network architecture (MAFNet), including a feature extraction module, an attention mechanism module, a feature fusion module, and a segmentation and classification module.

[0097] EfficientNetV2-B3 pre-trained on the ImageNet dataset is used as the feature extraction module. Multi-scale features are extracted at different levels of EfficientNetV2-B3, namely, through the 2nd, 4th and 6th MBConv blocks of EfficientNetV2-B3. EfficientNetV2-B3 consists of multiple layers, each of which contains MBConv blocks (such as Figure 5 ). The distribution and overall architecture of these blocks are designed to achieve efficient computation and powerful feature extraction capabilities. The 2nd, 4th, and 6th MBConv blocks represent feature extraction layers at different scales. The features extracted by these layers have different receptive fields and levels of abstraction.

[0098] By introducing the attention mechanism module after these layers, we can obtain lower-level detail features and higher-level global features, i.e., multi-scale features. The attention mechanism module is used to enhance the feature representation capability of the deep learning model by adaptively emphasizing important features and suppressing irrelevant features. Figure 2 As shown in Figure 1, the module contains two main branches: spatial attention branch and channel attention branch. The spatial attention branch generates a weight map through convolution and pooling operations, and processes the pooling result through an activation function (such as Sigmoid) to generate a spatial attention weight map. This weight map represents the importance of each position in the feature map. The channel attention branch learns the importance weights of each channel through global average pooling and a fully connected layer, and generates a channel attention weight map through an activation function (such as Sigmoid). This weight map represents the importance of each channel in the feature map. Finally, the spatial attention weight map and the channel attention weight map are multiplied to generate a comprehensive attention weight, and then the comprehensive attention weight is used to weight the original input feature map to enhance the representation of key areas and features and generate enhanced feature maps. These enhanced feature maps are used for subsequent segmentation and classification tasks, thereby improving the performance of the model. After the attention mechanism module, lower-level detail features and higher-level global features, namely multi-scale features, can be obtained. Introducing the attention mechanism module after all MBConv blocks will greatly increase the computational complexity. Therefore, selectively introducing the attention mechanism module after the 2nd, 4th, and 6th blocks can increase the model performance while controlling the computational complexity and memory consumption.

[0099] like Figure 3 As shown, the feature fusion module adaptively assigns weights to features of different scales through weight learning, and then combines the weighted features through addition operations to output the final fusion feature map.

[0100] like Figure 4 As shown, the segmentation and classification module includes a segmentation head and a classification head, which respectively implement segmentation tasks and classification tasks; the fusion feature map output by the feature fusion module is simultaneously input into the segmentation head and the classification head for processing respectively, the segmentation head generates a segmentation mask through a convolution layer, an activation function and an upsampling layer, and the classification head generates a classification result through a global average pooling layer and a fully connected layer. This method of performing segmentation and classification tasks simultaneously can efficiently and accurately perform image segmentation and classification.

[0101] Step 3: Model training;

[0102] 3-1. Loss function and optimizer:

[0103] The cross entropy loss is used as the basic method to calculate the error of a single task (such as classification or segmentation). The segmentation and classification tasks are optimized simultaneously through a joint loss function. The joint loss function combines the pixel-level segmentation loss and the region-level classification loss. The segmentation loss is the cross entropy loss of the pixel-level image segmentation task, and the classification loss is the cross entropy loss of the region-level classification task. The joint loss applies the cross entropy loss to different tasks (such as segmentation and classification), and optimizes the two tasks simultaneously in a weighted manner to improve the overall performance and generalization ability of the model. The Adam optimizer is used, the initial learning rate is set to 0.01, and the learning rate decay strategy is adopted.

[0104] 3-2. Mixed training strategy:

[0105] In each training batch, the large-size WSI image is divided into multiple small patch samples, while the entire WSI image is retained as the WSI sample. That is, each batch contains a certain number of patch samples and WSI samples, and the data of the local area (patch) and the whole image (WSI) of the image are mixed to update the parameters of the model. The purpose of this hybrid training strategy is to make full use of local features and global features so that the model can effectively learn and optimize at different scales. The joint loss at the patch level (L patch ) is used to measure the classification and segmentation errors of local patches. The joint loss at the WSI level (L WSI ) is used to measure the classification and segmentation errors of the entire WSI image. The total joint loss function is obtained by weighted summing the joint loss at the patch level and the joint loss at the WSI level using the balance factors α and β. The calculation of the total joint loss function is shown in formula (1):

[0106] L total =αL patch +βL WSI ; (1)

[0107] The total joint loss function in the above formula combines features of different scales (local patch and global WSI) to optimize the overall performance of the model.

[0108] 3-3. Perform batch normalization and L2 regularization on the training data:

[0109] Batch normalization normalizes the data of each small batch to make the distribution of input to each layer more stable. By stabilizing the input distribution, it reduces the gradient vanishing and gradient exploding problems, thereby accelerating the training process and improving the performance and stability of the model. L2 regularization is a technique used to prevent model overfitting. It constrains the size of the weight by adding a weight decay term to the loss function, so that the weight parameters of the model will not become too large, thereby improving the generalization ability of the model.

[0110] 3-4. Hyperparameter tuning:

[0111] (1) Define parameter ranges: Set candidate ranges for hyperparameters such as batch size, learning rate, regularization coefficient, and joint loss balance factor.

[0112] (2) Grid search and cross-validation: Grid search is performed within the parameter range, and the performance of each parameter combination is evaluated through cross-validation.

[0113] (3) Select the best combination: Based on the cross-validation results, select the best hyperparameter combination: batch size 32, learning rate 0.001, regularization coefficient 0.001, and balance factors α and β are both 0.5.

[0114] 3-5. Training process:

[0115] (1) The final model training is performed using the best combination determined by hyperparameter tuning. In each training batch, the optimal batch size (32) after tuning is applied for sampling. The gradient update is performed using the optimal learning rate (0.001) after tuning. The loss calculation and optimization are performed using the tuned regularization coefficient (0.001) and the joint loss balance factor (α and β are both 0.5).

[0116] (2) In each training batch, random samples are taken from the patch data loader and the WSI data loader to form a mixed training batch. Here, patch samples are used for fine-grained local feature learning, and WSI samples are used to capture global context information. The two can be randomly sampled from different WSI images and do not need to correspond one to one.

[0117] (3) Calculate the joint loss at the patch level and the joint loss at the WSI level.

[0118] (4) Use the balance factor to perform weighted summation of the two joint losses to obtain the total joint loss function.

[0119] (5) Perform back propagation based on the total joint loss function to update the model parameters.

[0120] 3-6. Model evaluation and validation:

[0121] Evaluate the model performance on the validation set, using accuracy, precision, recall, and F1 value metrics. Make minor adjustments based on the validation set performance to ensure that the model performs well on the test set. The specific implementation steps are as follows:

[0122] First, the trained model is used to obtain classification and segmentation results on the validation set, and the evaluation indicators of accuracy, precision, recall, and F1 value are calculated. Then, fine-tuning is performed in small steps according to the performance of the validation set, including fine-tuning the learning rate, regularization strength, and balance factors α and β. Through these fine-tuning, the performance of the model on the validation set is ensured to be optimal. Tables 1 and 2 show the performance comparison of the models on the validation set under different fine-tuning strategies.

[0123] The aforementioned hyperparameter tuning provides a good starting point for parameters, and fine-tuning during training based on the performance of the validation set can ensure the stability and excellent performance of the model in practical applications.

[0124] Step 4: Complete breast cancer histological image segmentation and classification through the trained Deep WSI-Stroma model, and perform Grad-CAM visualization;

[0125] After the original input image is processed through the multi-scale attention fusion network and the Grad-CAM heat map generation algorithm, a high-resolution segmentation probability heat map and salient regions are generated to intuitively display the spatial distribution and key areas of epithelial-mesenchymal segmentation. The detailed process of Grad-CAM visualization is as follows: Figure 6 shown.

[0126] Step 5: Model evaluation;

[0127] Evaluation indicators: Use ROC curve ( Figure 7 ), AUC value (Table 3), confusion matrix (Table 4), visualization results ( Figure 8 ), precision, recall and F1 were used to evaluate the model performance (Table 5).

[0128] Experimental results: Experiments were conducted on the TCGA-BRCA dataset. The Deep WSI-Stroma model achieved an average Dice score of 0.873 in the segmentation task and an average F1 value of 0.921 for the three categories in the classification task, which is significantly better than existing methods.

[0129] Step 6: Apply prediction and screening strategies;

[0130] Epithelial-stromal ratio (ESR): The number of pixels in the epithelial and stromal regions in each WSI was calculated to obtain the ESR value, which was used to predict and screen high-risk cases.

[0131] Spatial distribution pattern analysis: Graph theory and topological analysis methods are used to extract spatial features from the epithelial-stromal segmentation results, and a multivariate logistic regression model is constructed to predict the invasiveness and metastasis risk of tumors.

[0132] This example evaluates the performance of the proposed method in epithelial-stromal segmentation and classification of breast cancer histological images on the TCGA-BRCA dataset, and conducts comprehensive quantitative and qualitative analysis. In order to ensure the reliability and generalization of the experimental results, the composition and distribution of the dataset are firstly statistically analyzed. In terms of model training, the changes in the loss function and evaluation indicators during the training process are recorded in detail, proving the convergence and stability of the model.

[0133] In terms of quantitative evaluation, multiple indicators such as ROC curve, AUC value, confusion matrix and classification index were used to comprehensively evaluate the performance of the model on the test set. At the same time, the influence of different thresholds on the segmentation and classification results was explored, providing a reference for practical applications. In terms of qualitative evaluation, the segmentation and classification effects of the model on typical cases were intuitively demonstrated through heat map and overlay image visualization, proving its potential for clinical application. In addition, the model was applied to prediction and screening tasks, and its ability to identify high-risk cases and guide clinical decision-making was analyzed, further verifying the practical value of the proposed method.

[0134] Dataset statistics and analysis

[0135] The present invention uses the public TCGA-BRCA dataset to evaluate the performance of the proposed method in epithelial-stromal segmentation and classification of breast cancer histological images. The dataset contains H&E-stained whole-slice pathological images (WSI) from 1,043 breast cancer patients, with an average resolution of 80,000x 60,000 pixels, which can capture the detailed characteristics of tissues at high resolution. In order to fully understand the characteristics and distribution of the data, the TCGA-BRCA dataset was comprehensively statistically analyzed. First, the area ratio of epithelial and stromal regions in each WSI was calculated. The results show that in this dataset, the epithelial-stromal ratio is about 2:1, that is, the area of ​​the epithelial area is about twice that of the stromal area, which is consistent with the histological characteristics of breast cancer reported in the literature. Secondly, the distribution of different breast cancer subtypes and stages in the dataset is analyzed. The TCGA-BRCA dataset covers the main subtypes of breast cancer, such as hormone receptor-positive (HR+), human epidermal growth factor receptor 2-positive (HER2+), and triple-negative breast cancer (TNBC), and the proportions of different subtypes are consistent with those reported in epidemiological studies. In addition, the dataset also includes breast cancer cases of different stages (I-IV), reflecting different stages of tumor progression.

[0136] In order to train and evaluate the Deep WSI-Stroma model, the TCGA-BRCA dataset was randomly divided into a training set (80%), a validation set (10%), and a test set (10%). During the division process, the distribution of different subtypes and stages in each subset was balanced to avoid the impact of category skew on model performance. In addition, data augmentation was performed on the WSI in each subset, such as random cropping, rotation, and flipping, to improve the robustness and generalization ability of the model. Through statistics and analysis of the TCGA-BRCA dataset, the characteristics and distribution of the data were fully understood, laying the foundation for subsequent model training and evaluation. At the same time, the representativeness and applicability of the dataset in breast cancer histology research were verified, ensuring the reliability and clinical relevance of the experimental results. This provides a solid data foundation for evaluating the performance of the proposed method in breast cancer epithelial-stromal segmentation and classification tasks.

[0137] Model training and convergence

[0138] During the training of the deep learning model, we paid close attention to the convergence and performance changes of the model. We used the PyTorch deep learning framework and trained it on a workstation equipped with an NVIDIA GeForce RTX 3090 GPU. The backbone network of the model is EfficientNet-B3, followed by an attention mechanism module, a feature fusion module, and a segmentation and classification module.

[0139] We use the cross entropy loss function and the Adam optimizer, with the learning rate set to 0.001 and the batch size set to 32. At the end of each training cycle, we evaluate the performance of the model on the validation set and record the accuracy, precision, recall, and F1 value. We set an early stopping mechanism to stop training when the validation set performance does not improve for 10 consecutive cycles to prevent overfitting.

[0140] During the training process, we observed that the performance of the model improved significantly in the first few cycles, and then gradually leveled off. The loss functions of both the training set and the validation set showed a good downward trend, indicating that the model is constantly learning and optimizing. Around the 50th cycle, the performance of the validation set reached the best, with an accuracy of 92.5%, a precision of 91.8%, a recall rate of 93.2%, and an F1 value of 92.5%. After that, although the performance of the training set was still improving, the performance of the validation set began to fluctuate, indicating that the model may have begun to overfit. Therefore, we selected the model of the 50th cycle as the final training result. The training process and convergence of the model in the present invention are shown in Table 1:

[0141] Table 1 Performance index changes during training

[0142]

[0143] In order to further analyze the convergence of the model, we plotted the loss curve and accuracy curve of the training set and the validation set. Fig. 9 and Fig.10 shown.

[0144] from Fig. 9 and Fig.10 It can be seen that the loss curves of the training set and the validation set drop rapidly in the early stage, gradually stabilize in the later stage, and the two curves basically overlap without obvious separation or oscillation, indicating that the model is not overfitting or underfitting. The accuracy curve also shows a similar trend, reaching the highest point around the 50th cycle, and then fluctuating slightly but generally stable.

[0145] The above data tables and graphs can more intuitively show the training process and convergence of the Deep WSI-Stroma model on the TCGA-BRCA dataset. As can be seen from the table, as the training progresses, the performance indicators of the model on the validation set are constantly improving, and reach the optimal level around the 50th cycle. The loss curve and accuracy curve also show a good convergence trend, proving the effectiveness and stability of the model.

[0146] In addition, the present invention also compares the convergence of the model under different hyperparameter settings, such as learning rate, batch size, and regularization strength. The results show that appropriate hyperparameter selection has a significant impact on the convergence speed and final performance of the model. In the present invention, a set of optimized hyperparameter combinations are obtained through grid search and cross-validation methods, so that the model achieves the best performance on the TCGA-BRCA dataset. Table 2 shows the performance indicators of the model on the validation set under different learning rates, batch sizes, and regularization strengths.

[0147] Table 2 Performance indicators of the model on the validation set under different learning rates, batch sizes and regularization strengths

[0148]

[0149] It can be seen from Table 2 that different hyperparameter combinations will lead to differences in model performance. For example, when the learning rate is 0.001, the batch size is 32, and the regularization strength is 0.001, the model achieves the best performance on the validation set, with an accuracy of 94.25%, a precision of 93.18%, a recall of 95.57%, and an F1 value of 94.36%.

[0150] The above experimental data prove that different hyperparameter settings do have a significant impact on the performance of the model. In practice, by systematically searching and comparing different hyperparameter combinations, we can find the optimal hyperparameter settings and achieve the best performance of the model. This process not only improves the performance of the model on the TCGA-BRCA dataset, but also provides a reference for the application of the model on other similar datasets.

[0151] By recording and analyzing the training process and convergence of the model in detail, the effectiveness and stability of the Deep WSI-Stroma model on the breast cancer WSI dataset are demonstrated. Reasonable network architecture design, optimization algorithm selection and hyperparameter tuning ensure that the model can learn robust and generalized feature representations from large-scale, high-resolution WSI data, laying a solid foundation for subsequent segmentation and classification tasks.

[0152] Quantitative evaluation results

[0153] In order to comprehensively evaluate the performance of the Deep WSI-Stroma model in the epithelial-stromal segmentation and classification tasks of breast cancer histological images, extensive quantitative experiments were conducted on the test set of the TCGA-BRCA dataset. Several commonly used evaluation indicators, including ROC curves, AUC values, confusion matrices, and classification indicators, were used to comprehensively characterize the performance of the model.

[0154] ROC curve and AUC value on the test set

[0155] In order to evaluate the performance of the Deep WSI-Stroma model in the task of epithelial-stromal segmentation and classification of breast cancer histological images, the ROC curve was first drawn on the test set and the corresponding AUC value was calculated. The ROC curve shows the trade-off between the true positive rate (TPR) and the false positive rate (FPR) of the model under different threshold settings. An ideal model should reduce the FPR as much as possible while maintaining a high TPR, that is, the ROC curve should be as close to the upper left corner as possible. The AUC value quantifies the area under the ROC curve, ranging from 0 to 1, and the larger the value, the better the performance of the model.

[0156] On the test set, the ROC curve of the Deep WSI-Stroma model is very close to the upper right corner (such as Figure 7The model achieved an excellent balance between high TPR and low FPR. The AUC value of the model reached 0.972, indicating that it achieved excellent performance in the epithelial-mesenchymal classification task. To further evaluate the superiority of the Deep WSI-Stroma model, it was compared with other classic machine learning classification models, including support vector machine (SVM) and random forest.

[0157] from Figure 7 From the ROC curve comparison chart in , we can see that the curve of the Deep WSI-Stroma model is closest to the upper right corner, showing the best balance between TPR and FPR. The ROC curves of other models are relatively closer to the diagonal line of random guessing, indicating that their performance is worse than that of the method in the present invention.

[0158] Table 3 Comparison of AUC values ​​of different models on the test set

[0159] Model AUC value DeepWSI-Stroma 0.972 SVM 0.923 RandomForest 0.941

[0160] The AUC value comparison table is shown in Table 3. The data in the table further quantifies the performance differences of different models. The Deep WSI-Stroma model significantly outperforms SVM (0.923) and random forest (0.941) with an AUC value of 0.972, proving the advantages of deep learning methods in processing complex WSI data.

[0161] By comparing the ROC curves and AUC values ​​of different models (such as Figure 7 and Table 3), it was found that the Deep WSI-Stroma model performed significantly better than the other models. Its ROC curve was closest to the ideal state, while the curves of the other models were relatively closer to the diagonal of random guessing. The Deep WSI-Stroma model significantly exceeded SVM (0.923) and random forest (0.941) with an AUC value of 0.972, quantitatively confirming its performance advantage.

[0162] These experimental results strongly demonstrate the significant advantages of deep learning methods, especially the Deep WSI-Stroma model proposed in this paper, in processing large-scale, high-resolution WSI data. Deep learning models can automatically learn and extract high-level feature representations from massive pixel-level data, and make accurate segmentation and classification decisions based on this. Compared with traditional machine learning methods, deep learning models show stronger feature representation capabilities and generalization performance, bringing new breakthroughs to pathological image analysis tasks.

[0163] The ROC curve and AUC value evaluation on the test set fully demonstrated the excellent performance of the Deep WSI-Stroma model in the epithelial-stromal segmentation and classification tasks of breast cancer histological images. This result not only highlights the great potential of deep learning technology in the field of medical image analysis, but also lays a solid foundation for subsequent clinical applications.

[0164] Confusion matrix and classification metrics on the test set

[0165] In order to comprehensively evaluate the performance of the Deep WSI-Stroma model in the task of epithelial-stromal classification of breast cancer histological images, we generated a confusion matrix on the test set and calculated key classification indicators such as precision, recall, and F1 value. The confusion matrix shows the distribution of the model's prediction results for each category, while the classification indicators quantify the performance of the model in different aspects.

[0166] As can be seen from Table 4, the Deep WSI-Stroma model achieved highly balanced and accurate prediction performance in both the epithelial and mesenchymal categories of the test set. The model's precision, recall, and F1 value for the epithelial category reached 93.2%, 95.6%, and 94.4%, respectively, while the corresponding indicators for the mesenchymal category also reached 91.5%, 90.3%, and 90.9%, respectively. These results show that the model has a low prediction error rate for both categories and achieves a good balance between accuracy and completeness.

[0167] Table 4 Confusion matrix of Deep WSI-Stroma model on the test set

[0168] Actual tissue type Predict epithelium Predicting interstitial In fact, the leather 1568 72 Actual interstitial 95 885

[0169] From the confusion matrix in Table 4, we can see that the Deep WSI-Stroma model achieved highly balanced and accurate prediction performance in both epithelial and mesenchymal categories of the test set. In the epithelial category, the model correctly predicted 1568 samples, and only 72 samples were incorrectly classified as mesenchymal; while in the mesenchymal category, the model correctly predicted 885 samples, and only 95 samples were incorrectly classified as epithelial. These results intuitively show that the model has a low prediction error rate in both categories and does not show obvious bias.

[0170] In order to deeply analyze the performance of the Deep WSI-Stroma model in different categories, the present invention further calculated the classification indicators such as precision, recall rate and F1 value of each category (as shown in Table 5). The precision of the epithelial category reached 93.2%, the recall rate was 95.6%, and the F1 value was 94.4%; the precision of the interstitial category was 91.5%, the recall rate was 90.3%, and the F1 value was 90.9%. These quantitative indicators fully demonstrate the excellent predictive performance of the Deep WSI-Stroma model in both categories, which can accurately identify true positive samples (high recall rate) and effectively exclude false positive samples (high precision), thus achieving a good balance between accuracy and reliability.

[0171] Table 5 Classification indicators of Deep WSI-Stroma model on the test set

[0172] category Accuracy Recall F1 value Epithelial 0.932 0.956 0.944 Interstitial 0.915 0.903 0.909

[0173] Heatmap Visualization

[0174] Heatmap is a commonly used visualization method to represent the importance or attributes of different regions in an image. In this study, we generated a probability heatmap based on the output of the Deep WSI-Stroma model (e.g. Figure 8 The Deep WSI-Stroma model was used to segment and classify breast cancer histological images. The results show that the model can accurately capture the epithelial-stromal distribution pattern in WSI, and the generated visualization results are highly consistent with the annotation results of pathologists. These intuitive visual results not only verify the effectiveness of the model, but also provide valuable reference for clinical diagnosis and treatment.

[0175] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0176] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A method for segmentation and classification of breast cancer histological images based on multi-scale deep learning, characterized in that: The steps include: Step 1: Data acquisition and preprocessing; Step 2: Construct a deep whole-slice image-stroma segmentation and classification model for breast cancer (Deep WSI-Stroma); The breast cancer deep full-slice image-stroma segmentation and classification model adopts a multi-scale attention fusion network architecture, including a feature extraction module, an attention mechanism module, a feature fusion module, and a segmentation and classification module; Step 3: Model training; Use the cross entropy loss function and Adam optimizer; adopt a hybrid training strategy, use both patch and WSI samples for training, design corresponding loss functions and optimization methods, so that the model can learn local and global feature representations at the same time; Step 4: Complete breast cancer histological image segmentation and classification through the trained Deep WSI-Stroma model, and perform Grad-CAM visualization.

2. The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning according to claim 1, characterized in that: Step 1: Read WSI data: Use the OpenSlide library to read the Aperio SVS format files in the TCGA-BRCA dataset and extract high-resolution images and annotation information; Image segmentation and patch generation: Use the improved Otsu threshold method to perform foreground-background segmentation on WSI and extract tissue region masks; A series of image patches were generated by sliding the window with a size of 256 × 256 and a step size of 128 within the tissue mask; Image normalization: Perform color normalization on the generated image patches to reduce the impact of coloring differences on the model; Data enhancement: Divide the training set and validation set according to the needs, and perform data enhancement operations on the patches of the training set, including random horizontal flipping, vertical flipping, rotation, color change, random cropping, scaling, translation, and shearing, to increase the diversity of training samples and enhance the robustness of the model.

3. The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning according to claim 1 or 2, characterized in that: The breast cancer deep whole-slice image-stroma segmentation and classification model uses EfficientNetV2-B3 pre-trained on the ImageNet dataset as a feature extraction module, and extracts multi-scale features at different levels of EfficientNetV2-B3, namely, through the 2nd, 4th and 6th MBConv blocks of EfficientNetV2-B3; The attention mechanism module includes two main branches: spatial attention branch and channel attention branch; The spatial attention branch generates a weight map through convolution and pooling operations, and processes the pooling results through activation functions to generate a spatial attention weight map. This weight map represents the importance of each position in the feature map. The channel attention branch learns the importance weights of each channel through global average pooling and fully connected layers, and generates a channel attention weight map through activation functions. This weight map represents the importance of each channel in the feature map. Finally, the spatial attention weight map and the channel attention weight map are multiplied to generate a comprehensive attention weight, which is then used to weight the original input feature map to enhance the representation of key areas and features and generate an enhanced feature map. After the attention mechanism module, lower-level detail features and higher-level global features, i.e., multi-scale features, can be obtained. The feature fusion module adaptively assigns weights to features of different scales through weight learning, and then combines the weighted features through addition operations to output a final fusion feature map; The segmentation and classification module includes a segmentation head and a classification head, which respectively implement segmentation tasks and classification tasks; the fusion feature map output by the feature fusion module is simultaneously input into the segmentation head and the classification head for processing respectively, the segmentation head generates a segmentation mask through a convolution layer, an activation function and an upsampling layer, and the classification head generates a classification result through a global average pooling layer and a fully connected layer.

4. The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning according to claim 3, characterized in that: Step 3: 3-1. Loss function and optimizer: The cross entropy loss is used as the basic method for calculating the error of a single task. The segmentation and classification tasks are optimized simultaneously through a joint loss function. The joint loss function combines the pixel-level segmentation loss and the region-level classification loss. The segmentation loss is the cross entropy loss of the pixel-level image segmentation task, and the classification loss is the cross entropy loss of the region-level classification task. The Adam optimizer is used, the initial learning rate is set to 0.01, and the learning rate decay strategy is adopted. 3-2. Mixed training strategy: In each training batch, the large-size WSI image is divided into multiple small patch samples, while the entire WSI image is retained as the WSI sample. That is, each batch contains a certain number of patch samples and WSI samples, and the local area of ​​the image and the data of the entire image are mixed to update the parameters of the model; the joint loss at the patch level Used to measure the classification and segmentation errors of local patches; joint loss at the WSI level It is used to measure the classification and segmentation error of the entire WSI image. The total joint loss function is obtained by weighted summing the joint loss at the patch level and the joint loss at the WSI level using the balance factors α and β. The calculation of the total joint loss function is shown in formula (1): ； （1） The total joint loss function in the above formula combines features of different scales (local patch and global WSI) to optimize the overall performance of the model; 3-3. Perform batch normalization and L2 regularization on the training data: 3-4. Hyperparameter tuning: (1) Define parameter ranges: set candidate ranges for batch size, learning rate, regularization coefficient, and joint loss balance factor hyperparameters; (2) Grid search and cross-validation: Grid search is performed within the parameter range, and the performance of each parameter combination is evaluated through cross-validation; (3) Select the best combination: Based on the cross-validation results, select the best hyperparameter combination; 3-5. Training process: (1) Use the best combination determined by hyperparameter tuning for final model training; in each training batch, apply the tuned best batch size for sampling; use the tuned best learning rate for gradient update; apply the tuned regularization coefficient and joint loss balance factor for loss calculation and optimization; (2) In each training batch, randomly sample from the patch data loader and the WSI data loader to form a mixed training batch; (3) Calculate the joint loss at the patch level and the joint loss at the WSI level; (4) Use the balance factor to perform weighted summation of the two joint losses to obtain the total joint loss function; (5) Perform back propagation based on the total joint loss function to update the model parameters; 3-6. Model evaluation and validation: Evaluate model performance on the validation set using accuracy, precision, recall, and F1 score metrics; make minor adjustments based on validation set performance to ensure the model performs well on the test set.

5. The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning according to claim 4, characterized in that: The best hyperparameter combination is: batch size 32, learning rate 0.001, regularization coefficient 0.001, and balance factors α and β are both 0.

5.

6. The method for segmentation and classification of breast cancer histological images based on multi-scale deep learning according to claim 4, characterized in that: Step 4: After the original input image is passed through the trained breast cancer deep full-slice image-stroma segmentation and classification model and the Grad-CAM heat map generation algorithm, a high-resolution segmentation probability heat map and significant areas are generated to intuitively display the spatial distribution and key areas of epithelial-stroma segmentation.

Images