Prediction method and device for precancerous lesions of breast cancer

By employing a student-teacher model architecture and knowledge distillation techniques, and utilizing EfficientNet-B4 and DeepLabV3++ structures, the problems of high computational cost and poor interpretability in the diagnosis of precancerous lesions of breast cancer are solved, achieving efficient and accurate prediction of precancerous lesions of breast cancer.

CN118942707APending Publication Date: 2024-11-12THE CENTRAL HOSPITAL OF WUHAN (WUHAN NO 2 HOSPITAL WUHAN CANCER RESEARCH INSTITUTE)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411165043.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Current technologies for diagnosing precancerous lesions of breast cancer suffer from high computational costs and poor interpretability, especially in deep learning models based on whole-slice scan images, where it is difficult to effectively utilize cell nuclear features for accurate classification.

Method used

A student-teacher model architecture is adopted, in which the teacher model is used to segment and classify cell nuclei and generate pseudo-labels for cell nucleus classification. These pseudo-labels are then passed to the student model through knowledge distillation, which improves the classification accuracy and interpretability of the model at the ROI level. EfficientNet-B4 and DeepLabV3++ architectures are used for feature extraction and segmentation, and data augmentation techniques are combined to optimize model training.

Benefits of technology

It improves the diagnostic accuracy and interpretability of precancerous lesions of breast cancer, reduces the consumption of computational resources, achieves efficient prediction of precancerous lesions of breast cancer, conforms to the diagnostic thinking of pathologists, and improves the prediction accuracy of the model at the ROI level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942707A_ABST
    Figure CN118942707A_ABST
Patent Text Reader

Abstract

The invention provides a breast cancer precancerous lesion prediction method and device, and the method comprises the steps: obtaining the WSI of a breast tissue after HE dyeing, and selecting an ROI based on the WSI; the ROIs are input into the trained student model one by one to execute a classification task, the type of each ROI is obtained, the student model and the teacher model adopt the same encoder and share the weight of the encoder in the trained teacher model, the student model is trained based on a first ROI data set, the first ROI data set is a third ROI data set with ROI classification labels, and the student model and the teacher model are trained based on the third ROI data set. The teacher model carries out training based on a second ROI data set to realize segmentation and classification of the cell nucleus, wherein the second ROI data set is a third ROI data set with a cell nucleus classification false label; and generating a prediction result of the WSI according to the risk levels of all the ROI types. The invention provides a method and a device for predicting precancerous lesions of breast cancer, which are used for solving the problems of large calculation amount and poor interpretability in the prior art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and device for predicting precancerous lesions of breast cancer. Background Art

[0002] Breast Intraductal Proliferative Lesions (BIDPLs) are a group of heterogeneous epithelial proliferative lesions that originate from the glandular epithelium, progenitor cells or stem cells of the terminal lobular unit of the mammary duct. BIDPLs subtypes include Usual Ductal Hyperplasia (UDH), Flat Epithelial Atypia (FEA), Atypical Ductal Hyperplasia (ADH) and Ductal Carcinoma In Situ (DCIS), among which: UDH is a benign proliferation of glandular epithelium differentiated from progenitor cells or stem cells, composed of progenitor cells, intermediate epithelium and mature glandular epithelium, ADH and DCIS are malignant monoclonal hyperplasias of glandular epithelium, and FEA is a columnar cell lesion. The biological characteristics of different subtypes are highly heterogeneous. Among them, FEA, ADH and DCIS are precancerous lesions, and the risk of developing into invasive carcinoma (IC) is significantly higher than that of UDH. Therefore, differential diagnosis of BIDPLs subtypes is of great significance in the clinical management of breast cancer precancerous lesions.

[0003] Currently, pathologists usually identify BIDPLs subtypes by epithelial atypia and immunohistochemical phenotypes in hematoxylin-eosin (HE) histopathological images, where epithelial atypia refers to the deviation of the nucleus, cells and tissue structure morphology between proliferative epithelium and normal epithelium. However, the evaluation of atypia is a subjective and time-consuming process that relies on the pathologist's expertise and experience in image diagnosis. At the same time, since the morphological differences between UDH, ADH and low-grade DCIS subtypes are extremely subtle, the consistency and repeatability of BIDPLs diagnosis based on HE histopathological images are not ideal.

[0004] In recent years, with the application of whole slide image (WSI) and distributed data storage, the development of digital pathology and artificial intelligence has promoted the formation and rapid development of computational pathology. Computational pathology uses machine learning and image processing technology to realize the auxiliary detection, classification, quantification, and prognostic evaluation of pathological images, and has broad application prospects in breast pathology image analysis. The exploration of machine learning in BIDPLs diagnosis includes non-neural models based on manual features, such as logistic regression, decision trees, and support vector machines, as well as deep learning models, such as convolutional neural networks (CNNs) and graph neural networks (GNNs). However, WSI has a very large resolution (about 100,000×100,000 pixels) and usually contains a large amount of background information, blank areas without information, tissue folds, staining impurities or cracks, etc. Therefore, WSI cannot be directly input into deep learning models. Usually, WSI is cropped into image patches (patches) or regions of interest (ROIs) and then input into deep learning models. Such algorithms have a large amount of computation, and the accuracy of WSI-level prediction obtained by ROI-level prediction needs to be improved. In addition, compared with models based on manual features, deep learning models have poor interpretability in their autonomous learning, feature extraction, model fitting and other processes due to the black box characteristics of their internal structure. Summary of the invention

[0005] The present invention provides a method and device for predicting precancerous lesions of breast cancer, which are used to solve the problems of large amount of calculation and poor interpretability in the prior art.

[0006] In a first aspect, the present invention provides a method for predicting precancerous lesions of breast cancer, comprising:

[0007] Obtaining WSI of breast tissue after HE staining, and selecting ROI based on the WSI;

[0008] Inputting the ROIs into the trained student model one by one to perform the classification task, and obtaining the type of each ROI, the student model and the teacher model use the same encoder and share the weight of the encoder in the trained teacher model, the student model is trained based on the first ROI data set, the first ROI data set is a third ROI data set with ROI classification labels, the teacher model is trained based on the second ROI data set to achieve segmentation and classification of cell nuclei, and the second ROI data set is a third ROI data set with cell nucleus classification pseudo labels;

[0009] The prediction results of the WSI are generated according to the risk levels of all ROI types.

[0010] In the above scheme, the present invention provides pseudo labels for nucleus classification when training the teacher model, so that the KDCNN-NSAC model can focus on the characteristics of nuclei. Unlike the existing algorithms based on patch-level or ROI-level diagnosis, the present invention conforms to the diagnostic thinking of pathologists. It not only pays full attention to the key information of BIDPLs diagnosis, but also uses nucleus segmentation to guide ROI classification, and improves the interpretability of the KDCNN-NSAC model to a certain extent. Compared with other WSI-level processing algorithms, such as neural networks, packet classification, and multi-instance classification, the present invention does not need to perform post-processing such as image stitching to predict WSI after ROI classification, saving a lot of computing resources.

[0011] Optionally, obtaining the type of each ROI includes:

[0012] The type with the highest probability among all types output by the student model is used as the type of each ROI.

[0013] Optionally, the classification task is determined by the ROI classification label and the number of output channels of the decoder in the student model.

[0014] Optionally, the teacher model includes the encoder and the decoder, the encoder adopts the EfficientNet-B4 structure, the decoder adopts three branch structures, each branch structure is based on the upsampling strategy of DeepLabV3++, and the encoder is respectively provided with a jump connection with the three branch structures, and the three branch structures are divided into a mask branch, a horizontal-vertical branch and a classification branch, wherein:

[0015] The mask branch performs pixel-level binary segmentation on the cell nucleus;

[0016] The horizontal-vertical branch uses the position information of the cell nuclei to separate the overlapping or stacked cell nuclei;

[0017] The classification branch performs pixel-level classification on the cell nuclei after the binary segmentation.

[0018] In the above scheme, the three-branch structure not only ensures the advantage of the teacher model in position feature extraction, but also retains the fine-grained features at the cell nucleus level, thereby greatly improving the accuracy of cell segmentation. The cell nucleus type information provided by the classification branch is also more conducive to disease diagnosis.

[0019] Optionally, the student model includes the encoder, decoder and fully connected layer, the encoder adopts the EfficientNet-B4 structure, and the decoder is based on the upsampling strategy of DeepLabV3++.

[0020] Optionally, the cell nucleus classification pseudo label is generated on the third ROI data set based on a trained Hover-Net, and the Hover-Net is trained based on a public data set PanNuke.

[0021] Optionally, generating the prediction result of the WSI according to the risk levels of all ROI types includes taking a type with the highest risk level among all ROI types as the prediction result of the WSI.

[0022] In a second aspect, the present invention provides a device for predicting precancerous lesions of breast cancer, comprising:

[0023] A selection module, used for obtaining WSI of breast tissue after HE staining, and selecting ROI based on the WSI;

[0024] A classification module, used for inputting ROIs into the trained student model one by one to perform classification tasks, and obtaining the type of each ROI, wherein the student model and the teacher model use the same encoder and share the weights of the encoder in the trained teacher model, wherein the student model is trained based on a first ROI data set, wherein the first ROI data set is a third ROI data set with ROI classification labels, and the teacher model is trained based on a second ROI data set to segment and classify cell nuclei, wherein the second ROI data set is a third ROI data set with cell nucleus classification pseudo labels;

[0025] The prediction module is used to generate the prediction result of the WSI according to the risk levels of all ROI types.

[0026] In a third aspect, the present invention provides a computing device, comprising:

[0027] A memory for storing program instructions;

[0028] The processor is used to call the program instructions stored in the memory and execute any of the above methods according to the obtained program.

[0029] In a fourth aspect, the present invention provides a computer-readable non-volatile storage medium, comprising computer-readable instructions, which enable a computer to execute any of the above methods when the computer reads and executes the computer-readable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A schematic diagram of a flow chart of a method for predicting precancerous lesions of breast cancer provided by an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of the structure of KDCNN-NSAC provided in an embodiment of the present invention, wherein:

[0033] A is the structure of the teacher model, and B is the structure of the student model;

[0034] Figure 3 A schematic diagram of the effect of cell nucleus segmentation and classification based on the output result of the teacher model provided in an embodiment of the present invention, wherein:

[0035] A is the original input ROI, B is the cell nucleus segmentation and classification map, and C is the gradient-weighted class activation map heat map (Grad-CAM) of cell nucleus segmentation;

[0036] Figure 4 A schematic diagram of the effect of ROI classification obtained based on the output result of the student model provided in an embodiment of the present invention, wherein:

[0037] A is the original input ROI, B is the cell nucleus segmentation and classification map, and C is the gradient-weighted class activation map heat map (Grad-CAM) of cell nucleus segmentation;

[0038] Figure 5 A schematic diagram of the effect of WSI prediction provided by an embodiment of the present invention, wherein:

[0039] A is the original WSI, B is the WSI cell nucleus segmentation and classification map, C is the gradient-weighted class activation map heat map (Grad-CAM) of cell nucleus segmentation, and D is the prediction result of WSI;

[0040] Figure 6 A schematic diagram of the structure of a device for predicting precancerous lesions of breast cancer provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Figure 1 The process of a method for predicting precancerous lesions of breast cancer provided by an embodiment of the present invention is shown in detail, including:

[0043] Step 101, obtaining the WSI of breast tissue after HE staining, and selecting ROI based on the WSI.

[0044] Among tissue pathology images, HE staining images are more complex than immunohistochemistry images and immunofluorescence images, and are more challenging to use for image analysis.

[0045] In order to save computing power, when selecting ROI, try to avoid blank background, tissue folding, staining impurities and cracks, and only select ROIs that are potentially significant for diagnosis. Based on the acquired WSI, breast tissue can be differentially diagnosed. The types of breast tissue usually diagnosed include normal breast tissue (N), pathological benign lesions (PB), UDH, FEA, ADH, DCIS and IC. However, considering that computing power is limited, in one example, only the WSI of HE tissue sections that are considered by pathologists to belong to the BIDPLs disease spectrum after initial screening can be obtained. The BIDPLs disease spectrum is the development process spectrum of breast tissue from benign to borderline to malignant epithelial hyperplasia, that is, from UDH, FEA, ADH to DCIS.

[0046] Step 102, input the ROIs one by one into the trained student model to perform the classification task and obtain the type of each ROI.

[0047] Specifically, the types of ROI include N, PB, UDH, FEA, ADH, DCIS, and IC, among which UDH, FEA, ADH, and DCIS belong to BIDPLs subtypes. In the differential diagnosis of BIDPLs subtypes, the atypia of the nucleus, cells, and tissue structure is the key factor, and the differential diagnosis of UDH, ADH, and low-grade DCIS is relatively difficult due to the similar tissue morphological characteristics. In the HE histopathological images, UDH has obvious cell polymorphism, and various types of cells with different morphologies are crowded or overlapping, while ADH and DCIS have a single cell morphology but nuclear atypia, which is significantly different from normal cell nuclei in size, shape, and polarity. The polymorphism of UDH may lead to its misdiagnosis as intermediate-grade DCIS, because the polymorphism can easily be misinterpreted as moderate atypia. In addition, ADH also has similar cell characteristics with low-grade DCIS, and there is no essential difference between ADH and low-grade DCIS, so the immunohistochemical phenotype is also consistent. There is currently no consensus on the differential diagnosis of ADH and low-grade DCIS. The most commonly used criteria are: low-grade DCIS lesions involving ≥2 ducts or a maximum diameter ≥2 mm.

[0048] Therefore, for the classification task of ROI, it is necessary to focus on the cell nuclei with important diagnostic significance. The embodiment of the present invention proposes a knowledge distillation framework using Convolutional Neural Network-based Nuclei Segmentation-assisted Classification (KDCNN-NSAC) based on CNN cell nuclei segmentation: the teacher model is used to segment and classify the cell nuclei, and the student model is used to classify the ROI.

[0049] It should be noted that in the structure of KDCNN-NSAC, the student model and the teacher model use the same encoder and share the weights of the encoder in the trained teacher model, such as Figure 2 shown.

[0050] Figure 2 A is the structure of the teacher model, including an encoder and a decoder. The encoder uses the EfficientNet-B4 structure for downsampling, aiming to extract multi-level and multi-scale features in the image. The decoder uses a three-branch structure, each of which is based on the upsampling strategy of DeepLabV3++. The three branch structures are divided into a mask branch, a horizontal-vertical branch, and a classification branch, where:

[0051] The mask branch performs binary segmentation on the cell nucleus;

[0052] The horizontal-vertical branch uses the positional information of nuclei to separate overlapping or stacked nuclei;

[0053] The classification branch performs pixel-level classification on the cell nuclei after binary segmentation.

[0054] The above three branch structures not only ensure the advantages of the teacher model in position feature extraction, but also retain the fine-grained features at the cell nucleus level, thereby greatly improving the accuracy of cell segmentation. The cell nucleus type information provided by the classification branch is also more conducive to disease diagnosis.

[0055] In the teacher model, the encoder first downsamples to extract feature maps of different resolutions, and then the low-resolution feature maps are gradually upsampled through the three branches of the decoder using transposed convolution to match the size of the high-resolution feature maps. Since multiple downsampling will cause the feature map to be compressed, resulting in a decrease in prediction accuracy and loss of boundary information, in order to fully integrate feature information at different levels, the teacher model uses skip connection technology, that is, skip connections are set between the encoder and the three branch structures. This connection method helps to capture rich contextual information, especially when dealing with complex scenes, it can significantly improve the performance of the model.

[0056] The output of the teacher model is the pixel binary mask, the de-stacked binary mask, and the probability of the cell nucleus type to which the pixel belongs. The above outputs capture fine-grained nuclear level information, position features, and nuclear classification information. The fine-grained nuclear level information includes the edge, shape, and size of the cell nucleus. The output of the teacher model is only used to verify the accuracy of the teacher model, and its output will not be input into the student model.

[0057] Figure 2 B is the structure of the student model. Compared with the teacher model, the student model is more streamlined in structure, including an encoder, a decoder, and a fully connected layer. The encoder adopts the EfficientNet-B4 structure to fully capture the multi-scale features in the image. The decoder adopts the upsampling strategy of DeepLabV3++ to accurately restore the spatial information of the image. A fully connected layer is added after the decoder to further perform the ROI classification task (i.e. Figure 2 ROI classification prediction in ).

[0058] The output of the student model is the probability of each type in the classification task.

[0059] In one example, obtaining the type of each ROI includes:

[0060] The type with the highest probability among all types output by the student model is taken as the type of each ROI.

[0061] In one example, the classification task consists of ROI classification labels (i.e. Figure 2 The number of output channels of the student model can be adjusted according to different downstream tasks. The number of output channels represents the number of all types in the classification task, and the specific type is determined by the ROI classification label. For example, if the ROI classification labels include N, BIDPLs, and IC, and the number of output channels is set to 3, the output result of the student model is the probabilities of the three types of N, BIDPLs, and IC. Based on the 7 types of ROI, the student model can be trained to perform tasks such as 7-classification, 6-classification (except PB), 3-classification (N, BIDPLs, and IC), 2-classification (BIDPLs vs N or IC), and 2-classification within BIDPLs.

[0062] The KDCNN-NSAC model uses a simple and easy offline distillation, that is, a one-way knowledge transfer and a two-stage training procedure. First, the teacher model is trained, and then the teacher model is used to guide the training of the student model, realizing the migration of the multi-task teacher model to the single-task student model. In the one-way knowledge transfer, the knowledge of the teacher model is passed to the student model through the output of the teacher model (such as the output of the softmax layer or the features of the intermediate layer). The student model tries to imitate the behavior of the teacher model to improve its performance.

[0063] Specifically, the student model is trained based on the first ROI dataset, which is the third ROI dataset with ROI classification labels, and the teacher model is trained based on the second ROI dataset, which is the third ROI dataset with cell nucleus classification pseudo labels.

[0064] Before training the model, the first ROI dataset and the second ROI dataset can be divided into a training set, a validation set, and a test set in a ratio of 6:2:2. Furthermore, in order to increase the diversity and quantity of the training set and improve the generalization and robustness of the model, data augmentation can be performed. At the same time, since the data augmentation operation is performed before the model is trained, it will not have a direct impact on the training process of the model, nor will it increase the computational complexity of the model. Data augmentation is performed randomly, and a 50% probability can be set for data augmentation. When performing data augmentation, the images of the training set will be cropped into patches of the same size, and the augmented data will be used together with the original training set to train the model.

[0065] Before training the teacher model, the third ROI dataset needs to be annotated with nuclei to generate pseudo labels for nuclei classification, thus obtaining the second ROI dataset. When training the teacher model, pseudo labels for nuclei classification are provided, allowing the KDCNN-NSAC model to focus on nuclei features, which not only fully utilizes the pathological characteristics of BIDPLs, but also improves the interpretability of the KDCNN-NSAC model to a certain extent.

[0066] In one example, a pseudo label for nucleus classification is generated on a third ROI dataset based on a trained Hover-Net, which is trained on the public dataset PanNuke. Hover-Net is a model with a relatively high accuracy in the multi-classification task of nuclei in breast cancer tissue pathology images among existing algorithms. It can utilize the rich instance information encoded in the vertical and horizontal distances from the nucleus pixel to the centroid, and simultaneously obtain accurate nucleus instance segmentation and classification. Therefore, the pseudo label for nucleus classification generated by Hover-Net is regarded as a hard label during the training of the teacher model. Hover-Net includes an encoder and a decoder, and the decoder also adopts a three-branch structure. However, unlike the teacher model, the encoder of Hover-Net adopts ResNet50, and each branch of the decoder is based on the upsampling strategy of DenseNet. The structures of the encoder and decoder in Hover-Net and the teacher model are completely different, indicating that the two models use different architectures to perform the same task. PanNuke is a pan-cancer HE image set, in which 51,077 cell nuclei come from five types of cell nuclei in breast tissue, namely normal glandular epithelial cells, tumor epithelial cells, stromal connective tissue, stromal inflammatory cells and necrotic cells. Accordingly, the trained teacher model can classify the cell nuclei into five categories: normal glandular epithelial cells, tumor epithelial cells, stromal connective tissue, stromal inflammatory cells and necrotic cells.

[0067] It should be noted that training the teacher model based on the cell nucleus classification pseudo labels generated by Hover-Net, that is, transferring the cell nucleus classification pseudo labels from Hover-Net to the teacher model, is also a process of knowledge distillation.

[0068] Before training the student model, the weights of the encoder in the trained teacher model need to be shared with the student model, so as to transfer the teacher model's ability in cell information extraction to the student model, and then train it based on the first ROI data set. The shared weights can be frozen at the beginning of the student model training, that is, the encoder weights are not updated during training, and then unfrozen after a certain number of training rounds, and the encoder weights are updated with the training. This strategy helps the student model quickly obtain effective feature representations at the beginning of training, thereby improving the overall performance. In addition, due to the different task settings of the student model and the teacher model, the introduction of knowledge distillation speeds up the convergence of the classification network, enabling the student model to handle classification tasks in a more lightweight manner, while making the actual training process simple and controllable during deployment.

[0069] The general training steps for the model are as follows:

[0070] Data preparation: collect and organize relevant training data, and perform preprocessing;

[0071] Initialize the model: set the initial values ​​of the model parameters;

[0072] Generation training: In each round of iteration, data is input into the model, the loss function is calculated, and the model parameters are updated through the back-propagation algorithm;

[0073] Adjust hyperparameters: Adjust hyperparameters such as learning rate and regularization parameters according to the training results;

[0074] Monitoring and evaluation: Continuously monitor indicators during training, such as accuracy, loss, etc., to evaluate model performance;

[0075] Repeat the steps of training -> adjusting hyperparameters -> monitoring and evaluation: until the model achieves satisfactory performance or meets the stopping criteria.

[0076] In one example, the first ROI dataset and the third ROI dataset are derived from a public dataset (BRACS). The KDCNN-NSAC model is constructed and tested in the BRACS dataset, which contains HE histopathological images of N, PB, UDH, FEA, ADH, DCIS, and IC. The ROIs provided by BRACS can be annotated with pathologists (i.e., ROI classification labels) or without pathologists' annotations. An ROI set without ROI classification labels can be selected in BRACS as the third ROI dataset for download, and the ROI classification labels can be added based on the ROI set as the first ROI dataset.

[0077] Both the teacher model and the student model are based on Ubuntu 20.04 and PyTorch 1.15.1, and are trained on four 2080Ti GPUs. The student model is trained using the AdamW optimizer, with the initial learning rate set to 0.001 and the weight decay set to 10 -3 The batch size of the data is set to 4, and the data is enhanced using Gaussian noise, random motion blur, median filtering, and contrast adjustment (with probabilities of occurrence of 0.2, 0.5, 0.2, and 0.5, respectively). The gradient descent scheme uses cosine annealing warm restarts (warm-up cosine annealing) as the cosine adjustment cycle every two calendar hours, and the training is carried out for a total of 200 calendar hours.

[0078] Figure 3 A schematic diagram of the effect of cell nucleus segmentation and classification obtained based on the output result of the teacher model provided in an embodiment of the present invention is shown, wherein: A is the original input ROI, B is the cell nucleus segmentation and classification map, and C is the gradient-weighted class activation mapping heat map (Grad-CAM) of cell nucleus segmentation.

[0079] Figure 4A schematic diagram of the effect of ROI classification obtained based on the output result of the student model provided in an embodiment of the present invention is shown, wherein: A is the original input ROI, B is the cell nucleus segmentation and classification map, and C is the gradient-weighted class activation mapping heat map (Grad-CAM) of cell nucleus segmentation.

[0080] The models were evaluated by comparing the predictions of the KDCNN-NSAC model with the annotations assigned by pathologists using the F1 score, area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and accuracy as evaluation indicators. The KDCNN-NSAC model outperformed the state-of-the-art algorithms proposed on the BRACS dataset in 7-class classification with an average F1 score of 0.63. When the influence of PB was removed and all BIDPLs subtypes were merged for 3-class classification, the misprediction rate decreased. The KDCNN-NSAC model performed well in differential diagnosis of BIDPLs from IC (F1 score = 0.98) and N (F1 score = 0.94), indicating that the KDCNN-NSAC model has a considerable ability to distinguish BIDPLs from IC or N at the ROI level. Further binary classification was performed within the BIDPLs disease spectrum. The F1 scores of the KDCNN-NSAC model in distinguishing FEA from DCIS, FEA from UDH, FEA from ADH, UDH from ADH, ADH from DCIS, and UDH from DCIS were 0.91, 0.90, 0.89, 0.81, 0.72, and 0.69, respectively. It can be seen that the KDCNN-NSAC model also performed quite well in distinguishing FEA from DCIS, and UDH from ADH.

[0081] Step 103: Generate WSI prediction results according to the risk levels of all ROI types.

[0082] In one example, generating a prediction result of a WSI according to the risk levels of all ROI types includes taking the type with the highest risk level among all ROI types as the prediction result of the WSI. In the BIDPLs subtype, the risk level of IC gradually increases from UDH, FEA, ADH to DCIS. Among the ROI types, the risk levels are ranked from low to high as N, PB, UDH, FEA, ADH, ADH, DCIS, and IC. When the types of all ROIs in a WSI include N, PB, UDH, FEA, ADH, DCIS, and IC, the WSI is predicted to be IC.

[0083] In one example, the highest risk category can be found by following these steps:

[0084] Initialize the prediction result of WSI to N with the lowest risk level;

[0085] Compare the prediction results of WSI with the types of all ROIs one by one. When the risk level of a certain ROI type is higher than that of the prediction results of WSI, assign the type of this ROI to the prediction results of WSI.

[0086] After all ROI types are compared, the final prediction result of WSI is obtained.

[0087] Figure 5 A schematic diagram of the effect of WSI prediction provided by an embodiment of the present invention is shown, wherein: A is the original WSI, B is the WSI cell nucleus segmentation and classification map, C is the gradient-weighted class activation map heat map (Grad-CAM) of cell nucleus segmentation, D is the prediction result of WSI, and score is the probability that ROI belongs to a certain type.

[0088] Different from the existing algorithms based on patch-level or ROI-level diagnosis, the design concept of the embodiment of the present invention is consistent with the diagnosis concept of pathologists, fully focusing on the key information of BIDPLs diagnosis (i.e., nucleus information), and using nucleus segmentation to guide ROI classification.

[0089] Compared with other WSI-level processing algorithms, such as neural network, packet classification, and multi-instance classification, the embodiments of the present invention do not require post-processing such as image stitching to predict WSI after ROI classification, thus saving a lot of computing resources.

[0090] The kappa consistency test was used to calculate the consistency between the pathologist's annotations and the prediction results of the KDCNN-NSAC model at the WSI level. Different from the existing algorithms based on patch-level and ROI-level diagnosis, the WSI in the embodiment of the present invention is diagnosed based on the risk level of all ROI prediction results within it. The total accuracy of the WSI-level prediction is 90.12% (310 / 344), which is highly consistent with the pathologist's annotations (kappa value is 0.88).

[0091] Based on the same inventive concept, Figure 6 The schematic diagram of the structure of a device for predicting precancerous lesions of breast cancer provided by an embodiment of the present invention is shown, comprising:

[0092] A selection module is used to obtain the WSI of breast tissue after HE staining and select ROI based on the WSI;

[0093] A classification module is used to input ROIs into the trained student model one by one to perform classification tasks and obtain the type of each ROI. The student model and the teacher model use the same encoder and share the weights of the encoder in the trained teacher model. The student model is trained based on the first ROI dataset, which is a third ROI dataset with ROI classification labels. The teacher model is trained based on the second ROI dataset to achieve segmentation and classification of cell nuclei. The second ROI dataset is a third ROI dataset with pseudo labels for cell nuclei classification.

[0094] The prediction module is used to generate prediction results of WSI according to the risk levels of all ROI types.

[0095] Based on the same inventive concept, an embodiment of the present invention provides a computing device, including:

[0096] A memory for storing program instructions;

[0097] The processor is used to call the program instructions stored in the memory and execute the above method according to the obtained program.

[0098] Based on the same inventive concept, an embodiment of the present invention provides a computer-readable non-volatile storage medium, including computer-readable instructions. When a computer reads and executes the computer-readable instructions, the computer executes the above method.

[0099] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0101] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0103] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0104] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for predicting precancerous lesions of breast cancer, characterized in that: include: Obtaining WSI of breast tissue after HE staining, and selecting ROI based on the WSI; Inputting the ROIs into the trained student model one by one to perform the classification task, and obtaining the type of each ROI, the student model and the teacher model use the same encoder and share the weight of the encoder in the trained teacher model, the student model is trained based on the first ROI data set, the first ROI data set is a third ROI data set with ROI classification labels, the teacher model is trained based on the second ROI data set to achieve segmentation and classification of cell nuclei, and the second ROI data set is a third ROI data set with cell nucleus classification pseudo labels; The prediction results of the WSI are generated according to the risk levels of all ROI types.

2. The method according to claim 1, characterized in that: The types of each ROI include: The type with the highest probability among all types output by the student model is used as the type of each ROI.

3. The method according to claim 1, characterized in that: The classification task is determined by the ROI classification label and the number of output channels of the decoder in the student model.

4. The method according to claim 1, characterized in that: The teacher model includes the encoder and the decoder. The encoder adopts the EfficientNet-B4 structure. The decoder adopts three branch structures. Each branch structure is based on the upsampling strategy of DeepLabV3++. Jump connections are set between the encoder and the three branch structures respectively. The three branch structures are divided into mask branches, horizontal-vertical branches and classification branches, wherein: The mask branch performs pixel-level binary segmentation on the cell nucleus; The horizontal-vertical branch uses the position information of the cell nuclei to separate the overlapping or stacked cell nuclei; The classification branch performs pixel-level classification on the cell nuclei after the binary segmentation.

5. The method according to claim 4, characterized in that: The student model includes the encoder, decoder and fully connected layer. The encoder adopts the EfficientNet-B4 structure, and the decoder is based on the upsampling strategy of DeepLabV3++.

6. The method according to claim 1, characterized in that: The cell nucleus classification pseudo label is generated on the third ROI data set based on the trained Hover-Net, and the Hover-Net is trained based on the public data set PanNuke.

7. The method according to claim 1, characterized in that: Generating the prediction result of the WSI according to the risk levels of all ROI types includes taking the type with the highest risk level among all ROI types as the prediction result of the WSI.

8. A device for predicting precancerous lesions of breast cancer, characterized in that: include: A selection module, used for obtaining WSI of breast tissue after HE staining, and selecting ROI based on the WSI; A classification module, used for inputting ROIs into the trained student model one by one to perform classification tasks, and obtaining the type of each ROI, wherein the student model and the teacher model use the same encoder and share the weights of the encoder in the trained teacher model, wherein the student model is trained based on a first ROI data set, wherein the first ROI data set is a third ROI data set with ROI classification labels, and the teacher model is trained based on a second ROI data set to segment and classify cell nuclei, wherein the second ROI data set is a third ROI data set with cell nucleus classification pseudo labels; The prediction module is used to generate the prediction result of the WSI according to the risk levels of all ROI types.

9. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 7 according to the obtained program.

10. A computer-readable non-volatile storage medium, characterized in that: The method comprises computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the computer executes the method according to any one of claims 1 to 7.