An automatic scoring method for HER2 immunohistochemical images of breast cancer

Abstract

The present invention discloses an automatic scoring method for immunohistochemical images of breast cancer HER2. First, the image block set is inferred using a cell nucleus instance segmentation model and a tumor tissue semantic segmentation model to output a cell nucleus segmentation map and a tumor region segmentation map; then the tumor region contour in the tumor region segmentation map is extracted and the convex hull ratio and curvature features are calculated; then the invasive cancer region and the carcinoma in situ region are classified by a Gaussian mixture model and a naive Bayes classifier; then the cell membrane segmentation of the cell nucleus segmentation map is performed to obtain the cell membrane region and the cell staining degree is classified to obtain tumor cells with different staining levels in the invasive cancer region; finally, counting and proportion statistics are performed and the HER2 scoring of the breast cancer immunohistochemical image is completed according to the clinical guidelines. The present invention reduces human errors, ensures the consistency and reliability of the results, greatly shortens the diagnosis time, and provides immediate support for clinical decision-making.

Description

Technical Field

[0001] The invention belongs to the technical field of pathological image processing, and in particular relates to an automatic scoring method for immunohistochemical images of HER2 in breast cancer. Background Art

[0002] HER2 (Human Epidermal Growth Factor Receptor 2) is an important biomarker in the diagnosis and treatment of breast cancer. The expression level of HER2 directly affects the patient's treatment plan and prognosis assessment, so accurate HER2 scoring is crucial. Immunohistochemistry (IHC) is one of the common methods for detecting HER2 expression. It displays HER2 protein on the cell membrane through staining, thus providing a basis for subsequent treatment.

[0003] Traditional IHC image scoring mainly relies on manual evaluation by pathologists. Pathologists need to use a microscope to observe the entire area of ​​the pathological slide, determine the degree of staining of tumor cells in the invasive cancer area and estimate their proportion. Traditional immunohistochemistry scoring has two defects: first, poor accuracy and long time consumption: usually there are millions of cells in the pathological slide, and pathologists need to observe different positions of the pathological slide and make an overall estimate of the proportion of tumor cells with four degrees of staining in the pathological slide, and cannot get an accurate score; second, poor consistency and reproducibility: different pathologists have different professional backgrounds, experience and technical levels, and the scores of the same IHC image may be different. At the same time, due to the heterogeneity of pathological images and the differences in staining reagents, the complexity of subjective judgment is increased. With the advancement of digitalization and the development of artificial intelligence technology, the application of these technologies has not only improved the speed and accuracy of diagnosis, but also opened up new diagnosis and treatment methods and improved the treatment effect of patients. Existing automatic scoring methods can be divided into segmentation-based and classification-based methods. Among them, classification-based automatic scoring methods usually use convolutional neural networks to extract embedded representations of image blocks, and then use statistical decision fusion (such as "Tewary S, Mukhopadhyay S. AutoIHCNet: CNN architecture and decision fusion for automated HER2 scoring[J]. Applied Soft Computing, 2022,119: 108572."), support vector machine / multilayer perceptron (such as "Han Z, Lan J, Wang T, et al. A deep learning quantification algorithm for HER2 scoring of gastric cancer[J]. Frontiers in Neuroscience, 2022, 16: 877229.") and multiple instance learning (Multiple Instance Learning, MIL) (such as "Pisula JI, Datta RR, Valdez LB, et al. Predicting the HER2 status in oesophageal cancerfrom tissue microarrays using convolutional neural networks[J]. British Journal of Cancer, 2023, 128(7): 1369-1376.》) method combines the classification results of multiple sub-image patches.The automatic scoring method based on segmentation follows the clinical guidelines (Rakha EA, Pinder SE, Bartlett JMS, et al. Updated UK Recommendations for HER2 assessment in breast cancer [J]. Journal of clinical pathology, 2015, 68(2):93-99.). It generally segments the cell nucleus first, then uses the color deconvolution algorithm to evaluate the staining level of the cell membrane area in the DAB color space, and finally calculates the proportion of positive tumor cells of different degrees to the total number of tumor cells in the invasive cancer (such as Ruifrok AC, Johnston DA. Quantification of histochemical staining by color deconvolution [J]. Analytical and quantitative cytology and histology, 2001, 23(4): 291-299.).

[0004] Although existing HER2 scoring methods have made some progress in accuracy and efficiency, they still have some significant shortcomings. For classification-based automatic scoring methods: 1) They rely heavily on training data and require a large amount of high-quality annotated data for training. If the data lacks diversity, the classification model may perform poorly when faced with complex tissue morphology that has never been seen. 2) They have poor interpretability. The deep learning model is a black box and it is difficult to explain its decision-making process. In a clinical setting, this may reduce doctors' trust in the model results. 3) They are not explicitly modeled according to the scoring criteria, but are trained in a data-driven way. The immunohistochemistry score depends on the experience of the pathologist and is somewhat subjective, so the cost of obtaining high-quality annotations is extremely expensive. For automatic scoring methods based on segmentation: 1) The accuracy of cell nucleus segmentation is limited. The paper "Masmoudi H, Hewitt SM, Petrick N, et al. Automated quantitative assessment of HER-2 / neu immunohistochemical expression in breast cancer [J]. IEEE transactions on medical imaging, 2009, 28 (6): 916-925." uses a marker-based watershed algorithm to perform cell nucleus segmentation. However, pathological images are highly heterogeneous. Especially in highly dense tumor areas, traditional segmentation methods are prone to mis-segmentation and over-segmentation. 2) Although the current segmentation-based methods are modeled according to the scoring guidelines, the algorithm process is simplified and fails to effectively distinguish between invasive cancer and carcinoma in situ areas. 3) When classifying cell staining levels, the threshold setting or supervised learning paradigm is used. On the one hand, the threshold setting method has low generalization performance. When using different stains / scanners, different thresholds need to be used. On the other hand, the supervised learning paradigm requires a large number of labeled samples, which is costly. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, provide an automatic scoring method for immunohistochemical images of breast cancer HER2, apply deep learning technology to automatic HER2 scoring of immunohistochemical images of breast cancer, improve the consistency and reproducibility of HER2 scoring, significantly shorten the diagnosis time, and provide immediate support for clinical decision-making.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] On the one hand, a method for automatic scoring of immunohistochemical images of breast cancer HER2 is provided, characterized in that it comprises the following steps:

[0008] Acquire breast cancer immunohistochemistry images and perform preprocessing to obtain image block sets;

[0009] The cell nucleus instance segmentation model and tumor tissue semantic segmentation model are used to infer the image block set and output the cell nucleus segmentation map and tumor area segmentation map of breast cancer immunohistochemistry image respectively;

[0010] Extract the tumor region contour in the tumor region segmentation map, and calculate the convex hull ratio and curvature characteristics of the tumor region contour;

[0011] The convex hull ratio and curvature features of the tumor region contour were clustered using a Gaussian mixture model to obtain the feature prior distribution of the tumor region segmentation map, and the naive Bayes classifier was used to classify the tumor region segmentation map to obtain the invasive cancer region and the carcinoma in situ region.

[0012] Performing cell membrane segmentation on the cell nucleus segmentation map to obtain the cell membrane region, extracting the DAB channel features of the cell membrane region, and classifying the cell staining degree based on the staining level using a naive Bayes classifier to obtain tumor cells with different staining levels in the invasive cancer region; the staining level includes strong staining, medium staining, weak staining and no staining;

[0013] Tumor cells with different staining grades in the invasive cancer area were counted and their proportions were statistically analyzed, and the HER2 score of breast cancer immunohistochemistry images was completed according to clinical guidelines.

[0014] As a preferred technical solution, the preprocessing of breast cancer immunohistochemistry images is specifically as follows:

[0015] Downsampling of breast cancer immunohistochemistry images and conversion to grayscale images I gray , expressed as: I gray = DS ( I original ),in, DS () is the downsampling function, I original Immunohistochemistry images for breast cancer;

[0016] Apply histogram equalization technology to obtain the global contrast map of grayscale images I eq ;

[0017] Adaptive threshold algorithm is used to segment the global contrast map to obtain the tissue region segmentation map. M ;

[0018] Set the sliding window size w × h With sliding step s, traverse the entire tissue region segmentation map through the sliding window technology M , record the coordinates of the image blocks captured by each sliding window;

[0019] For each image block coordinate ( x m , y m ), calculate its segmentation map in the tissue area M The average value of the pixels at the corresponding position in Am m ,like Am m Greater than the preset threshold θ , then add it to the pending queue Q Get an image block set;

[0020] The pending queue Q Used as input for subsequent cell nucleus instance segmentation model and tumor tissue semantic segmentation model.

[0021] As a preferred technical solution, the cell nucleus instance segmentation model is a trained Hover-Net model;

[0022] When the Hover-Net model is trained, the H color channel of the HE image is extracted using color deconvolution as training data; then the cell nucleus annotation data of the prepared breast cancer immunohistochemistry image is used to fine-tune the parameters.

[0023] As a preferred technical solution, the tumor tissue semantic segmentation model is a trained U-Net model;

[0024] The U-Net model uses the residual network ResNet as the feature encoder, performs full-supervised training, and implements semantic segmentation by overlapping sliding windows. The specific process is as follows:

[0025] Read the image block set, save the segmentation map output by each image block according to the coordinates of the image block, and perform the nearest neighbor downsampling operation on each segmentation map. z times to obtain the downsampled segmentation map, and finally concatenate all the downsampled segmentation map sets to obtain the segmentation map of the entire tumor area of ​​the breast cancer immunohistochemistry image.

[0026] As a preferred technical solution, the extraction of all tumor area contours from the tumor area segmentation map is specifically as follows:

[0027] Performing connected component labeling on the tumor region segmentation map to obtain a connected component map, traversing each connected component map to extract the coordinate range of the connected component map, obtaining the tumor region segmentation map of the corresponding image block according to the coordinate range index and splicing them to obtain the tumor region contour in the tumor region segmentation map;

[0028] The convex hull ratio β The calculation formula is: β = A contour / A convexhull ,in, A contour is the area of ​​the tumor region contour, A convexhull is the area of ​​the convex hull contour of the tumor region;

[0029] The curvature feature uses the average curvature of the tumor area contour Ak Said, the formula is:

[0030] ,

[0031] in, n is the number of contour points contained in the contour of a tumor area; k u For the u The curvature value of a contour point is calculated based on two adjacent contour points.

[0032] As a preferred technical solution, the cell membrane segmentation refers to the segmentation of the cell membrane outside the cell nucleus outline in the cell nucleus segmentation diagram. p The range of pixels is taken as the cell membrane area.

[0033] As a preferred technical solution, the DAB channel features of the cell membrane region are extracted using a color deconvolution algorithm.

[0034] As a preferred technical solution, the naive Bayes classifier is used to classify the cell staining degree based on the staining level, specifically:

[0035] The average value of the DAB channel feature of the cell membrane region is used to preliminarily screen the cells in the cell nucleus segmentation map based on the staining level to obtain a collection of cells at different staining levels;

[0036] Calculating statistics of each cell set in cell sets at different staining levels; the statistics include mean, median, maximum, variance, kurtosis and range of DAB channel pixels in the cell membrane region of each cell set;

[0037] A multivariate naive Bayes classifier was established by fitting the statistics of cell collections at different staining levels using Gaussian distribution.

[0038] The statistics of tumor cells in the invasive cancer area at different staining levels are used as input to determine the staining level of tumor cells. The determination formula is:

[0039] ,

[0040] in, y i For the i Dyeing level, P ( y i ∣ X ) is the dyeing grade characteristic X Judged as the i The probability of a coloring level, P ( X ) is the dyeing grade characteristic X The marginal probability of P ( y i ) is the i The prior probability of the coloring level, P ( X ∣ y i ) is in i Dyeing grade characteristics under each dyeing grade X The probability distribution of is calculated by Gaussian distribution:

[0041] ,

[0042] in, f () is the probability density function of the Gaussian distribution, that is:

[0043] ,

[0044] in, N is the number of statistics, x j Characteristic of dyeing grade X Middle j Statistics, ( μ ij , σ ij )for x j exist y i The parameters satisfy the Gaussian distribution.

[0045] As a preferred technical solution, the HER2 score is completed according to the clinical guidelines, specifically:

[0046] When the number of tumor cells with strong staining in the invasive cancer area is greater than 10% and the cell membrane staining is complete and uniform, the HER2 score of the whole section image of breast cancer is 3+, and the breast cancer is positive;

[0047] When the number of tumor cells with medium or weak staining in the invasive cancer area is greater than 10% and the cell membrane staining is complete, or when the number of tumor cells with strong staining in the invasive cancer area is less than or equal to 10% and the cell membrane staining is complete, the HER2 score of the whole section image of breast cancer is 2+, and further detection and observation are required;

[0048] When the number of weakly stained tumor cells in the invasive cancer area is greater than 10% and the cell membrane staining is incomplete, the HER2 score of the whole section image of breast cancer is 1+, and the breast cancer is negative;

[0049] When the staining level in the invasive cancer area is no staining, or when the number of tumor cells with weak staining level in the invasive cancer area is less than or equal to 10% and the cell membrane staining is incomplete, the HER2 score of the whole section image of breast cancer is 0, and the breast cancer is negative.

[0050] On the other hand, an automatic scoring system for immunohistochemical images of breast cancer HER2 is provided, comprising an image processing module, an image segmentation module, a contour acquisition module, a region division module, a staining classification module and a scoring determination module;

[0051] The image processing module is used to obtain breast cancer immunohistochemistry images and perform preprocessing to obtain an image block set;

[0052] The image segmentation module is used to use a cell nucleus instance segmentation model and a tumor tissue semantic segmentation model to infer the image block set and output a cell nucleus segmentation map and a tumor region segmentation map of a breast cancer immunohistochemistry image respectively;

[0053] The contour acquisition module is used to extract the tumor region contour in the tumor region segmentation map, and calculate the convex hull ratio and curvature characteristics of the tumor region contour;

[0054] The region segmentation module is used to cluster the convex hull ratio and curvature features of the tumor region contour through a Gaussian mixture model to obtain a feature prior distribution of the tumor region segmentation map, and to classify the tumor region segmentation map using a naive Bayes classifier to obtain an invasive cancer region and a carcinoma in situ region;

[0055] The staining classification module is used to perform cell membrane segmentation on the cell nucleus segmentation map to obtain the cell membrane region, extract the DAB channel features of the cell membrane region, and classify the cell staining degree based on the staining level through a naive Bayes classifier to obtain tumor cells with different staining levels in the invasive cancer region; the staining level includes strong staining, medium staining, weak staining and no staining;

[0056] The scoring module is used to count and proportion tumor cells of different staining grades in the invasive cancer area, and complete the HER2 score of breast cancer immunohistochemistry images according to clinical guidelines.

[0057] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0058] 1. Improve accuracy and efficiency: In view of the inconsistent scoring caused by the time-consuming and subjective traditional manual scoring method, this invention aims to significantly improve the accuracy of HER2 scoring, reduce human errors, and ensure the consistency and reliability of the results through the combination of deep learning and Bayesian theorem.

[0059] 2. Enhanced interpretability: Since deep learning-based classification methods usually have poor interpretability, this invention not only improves automation but also incorporates multi-level models and optimization strategies that are more in line with clinical logic, enhancing the interpretability of the model, enabling doctors to understand the reasons behind the scoring and increase their trust in the results.

[0060] 3. Solve the problem of lack of labeled data: Due to the scarcity of high-quality cell nucleus labeled data, the present invention adopts a transfer learning strategy to train the cell nucleus segmentation model, using the features learned by the pre-trained model on a large-scale external dataset and fine-tuning it on breast cancer IHC images to reduce the dependence on a large amount of labeled data while maintaining high segmentation accuracy.

[0061] 4. Accurately distinguish between carcinoma in situ and invasive cancer areas: It is very difficult to judge carcinoma in situ and invasive cancer in HER2 immunohistochemistry images. It requires observing the morphological features of the tumor area at both high resolution and low resolution. The semantic segmentation method cannot solve this problem. However, the present invention combines the contour features of the tumor area (such as convex hull area ratio and curvature features) with the naive Bayes classifier, and further distinguishes the tumor area into invasive cancer area and carcinoma in situ area based on morphological features, thereby improving the accuracy of the distinction.

[0062] 5. Optimize staining grade judgment: The subjectivity of cell membrane staining is very strong. In addition, the difference in staining of different antibodies is very large. Doctors need to mark each antibody, which is very expensive. Therefore, in view of the shortcomings in cell membrane staining grade classification, the present invention proposes a cell membrane staining grade classification method based on a naive Bayes classifier, which does not require instance-level annotation, only clinical scoring results are required, which greatly reduces the annotation cost and can be applied to pathological sections of different antibodies on a large scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 The figure is a flow chart of an automatic scoring method for immunohistochemical images of HER2 in breast cancer according to an embodiment of the present invention.

[0065] Figure 2 Schematic diagram of the process of tumor region classification in an embodiment of the present invention.

[0066] Figure 3 Schematic diagram of the process of tumor cell staining grade classification in an embodiment of the present invention.

[0067] Figure 4 Schematic diagram of training data of the Hover-Net model in an embodiment of the present invention.

[0068] Figure 5 The figure is a schematic diagram of the structure of an automatic scoring system for immunohistochemical images of HER2 in breast cancer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0070] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0071] In the diagnosis and treatment of breast cancer, HER2 immunohistochemistry (IHC) image assessment faces several challenges: the first problem is that the traditional manual scoring method is time-consuming and easily affected by subjective judgment, resulting in fluctuating scoring results; secondly, although the model based on multiple instance learning (MIL) has been tried for automated evaluation, its performance is insufficient and its interpretability is poor, making it difficult to clearly reveal the logical basis behind the scoring; thirdly, due to the dense distribution of cell nuclei in the image, the fine pixel-level annotation work becomes extremely difficult, and there is a lack of public IHC image cell nucleus annotation data; finally, the current technology has obvious shortcomings in distinguishing between carcinoma in situ and invasive cancer areas. Even if a large amount of labeled data is invested in training the semantic segmentation model, the effect is still unsatisfactory, which is crucial for the accurate evaluation of HER2.

[0072] In view of this, the present invention provides a solution to the HER2 scoring problem of breast cancer IHC images through deep learning technology, aiming to significantly enhance the accuracy of scoring, the reliability and consistency of clinical diagnosis. The present invention uses cutting-edge image processing technology and artificial intelligence algorithms, namely semantic segmentation and instance segmentation, to accurately segment tumor areas and cell nuclei; on this basis, key features are extracted from the DAB staining channels of tumor areas and cell membranes, and these features are modeled using a Gaussian mixture model to achieve accurate classification of carcinoma in situ and invasive cancer areas and their cell membrane staining levels; finally, combined with clinical practice and clear diagnostic guidelines, the immunohistochemistry score can be accurately calculated to provide strong support for clinical decision-making. The technical solution of the present invention has undoubtedly brought new breakthroughs in the intersection of medical image processing, computational pathology and artificial intelligence.

[0073] like Figure 1 , Figure 2 , Figure 3 As shown, this embodiment provides a method for automatically scoring immunohistochemical images of breast cancer HER2, comprising:

[0074] Step 1: Image segmentation processing:

[0075] Acquire breast cancer immunohistochemistry images and perform preprocessing to obtain image block sets;

[0076] The cell nucleus instance segmentation model and tumor tissue semantic segmentation model are used to infer the image block set and output the cell nucleus segmentation map and tumor area segmentation map of breast cancer immunohistochemistry image respectively;

[0077] Step 2: Tumor area classification:

[0078] Extract the tumor area contour in the tumor area segmentation map, and calculate the convex hull ratio and curvature characteristics of the tumor area contour ( Figure 2Statistical features in );

[0079] The convex hull ratio and curvature features of the tumor region contour were clustered using a Gaussian mixture model to obtain the feature prior distribution of the tumor region segmentation map, and the naive Bayes classifier was used to classify the tumor region segmentation map to obtain the invasive cancer region and the carcinoma in situ region.

[0080] Step 3: Cell staining and grading:

[0081] The cell membrane is segmented on the cell nucleus segmentation map to obtain the cell membrane region, the DAB channel features of the cell membrane region are extracted, and the cell staining level is classified based on the staining level by the naive Bayes classifier to obtain tumor cells with different staining levels in the invasive cancer region; wherein the staining level includes strong staining, medium staining, weak staining and no staining;

[0082] Step 4: Image HER2 scoring:

[0083] Tumor cells with different staining grades in the invasive cancer area were counted and their proportions were statistically analyzed, and the HER2 score of breast cancer immunohistochemistry images was completed according to clinical guidelines.

[0084] Furthermore, in the field of digital pathology, whole slide images (WSI) are known for their ultra-high resolution and massive data volume, which requires sophisticated strategies to optimize model inference time and efficiently use computing resources. Therefore, the preprocessing link is particularly critical. It is not only the cornerstone for improving the efficiency of subsequent analysis, but also a key step to ensure that data processing focuses on core information; breast cancer immunohistochemistry images (IHC images) are obtained by whole slide scanning and preprocessed, specifically:

[0085] First, breast cancer immunohistochemistry images (IHC images) were downsampled to reduce image resolution without excessive information loss, and then converted into grayscale images. I gray , expressed as: I gray = DS ( I original ),in, DS () is the downsampling function, I original Immunohistochemistry images for breast cancer;

[0086] Next, the histogram equalization technique is applied to obtain the global contrast map of the grayscale image. I eq ;

[0087] Afterwards, the global contrast map was segmented using an adaptive threshold algorithm to obtain the tissue region segmentation map. M , expressed as:

[0088] ,

[0089] in, m x,y is the global contrast map I eq Middle pixel ( x,y ), 1 is the indicator function, I eq ( x , y ) is the global contrast map I eq Middle pixel ( x , y ), T otsu It is the global threshold calculated by Otsu's method;

[0090] Finally, set the sliding window size w × h With sliding step s , traverse the entire tissue region segmentation map through the sliding window technology M , record the coordinates of the image blocks captured by each sliding window;

[0091] For each image block coordinate ( x m , y m ), calculate its segmentation map in the tissue area M The average value of the pixels at the corresponding position in Am m ,like Am m Greater than the preset threshold θ , then add it to the pending queue Q = {( x m , y m ) | Am m > θ} (i.e., a set of image blocks composed of multiple image blocks) is used as the input of the subsequent cell nucleus instance segmentation model and tumor tissue semantic segmentation model.

[0092] Furthermore, the present invention uses the trained Hover-Net model as the cell nucleus segmentation model; due to the lack of publicly available IHC (immunohistochemistry) image cell nucleus segmentation annotated datasets, while annotated HE (hematoxylin-eosin) staining image datasets are relatively abundant, the present application is trained on a large number of HE image annotated datasets, and then fine-tuned on a small amount of prepared IHC image data. In order to improve the initial performance of the training model during transfer learning, reducing the difference between HE images and IHC images is an effective method, as shown in the following specific methods: Figure 4 As shown:

[0093] When training the Hover-Net model, color deconvolution is used to extract the H color channel of the HE image as training data. After color deconvolution, the difference between the H channel images of the HE image and the IHC image is smaller. In the fine-tuning stage, only a small amount of nucleus annotation data of IHC images is needed to complete the training of the nucleus segmentation model.

[0094] The color deconvolution in this embodiment is set with reference to the document "Ruifrok AC, Johnston DA. Quantification of histochemical staining by color deconvolution [J]. Analytical and quantitative cytology and histology, 2001, 23 (4): 291-299.".

[0095] Further, according to the clinical guidelines for HER2 scoring of breast cancer (Rakha EA, Pinder SE, Bartlett JMS, et al. Updated UK Recommendations for HER2assessment inbreast cancer [J]. Journal of clinical pathology, 2015, 68 (2): 93-99.), HER2 immunohistochemistry image scoring only focuses on the staining of cancer cells in the invasive cancer tissue area in the tissue section, and needs to exclude the carcinoma in situ and the matrix area (including normal tissue, fat, cartilage, etc.). Therefore, a semantic segmentation model is first required to achieve the segmentation of the tumor area. Since the invasive cancer area and the carcinoma in situ area are particularly close in image features, p63-stained images are usually used for auxiliary judgment in clinical practice; at the same time, the distinction between invasive cancer and carcinoma in situ needs to be judged in combination with large context information, which is difficult to achieve in the semantic segmentation algorithm using a sliding window method and a fixed receptive field. Therefore, the present invention adopts the method of first segmenting the tumor area and then performing the classification of invasive cancer and carcinoma in situ according to the manual features of the tumor area contour. In the present invention, a trained U-Net model is used as a semantic segmentation model for tumor tissues. The feature encoder thereof adopts a residual network ResNet. The model uses a 512×512 image block for full supervision training and implements semantic segmentation of IHC images by overlapping sliding windows. The specific process is as follows:

[0096] Reading image block sets Q , save the segmentation map output by each image block according to the coordinates of the image block, and perform the nearest downsampling operation on each segmentation map to scale it by z times to obtain the downsampled segmentation map. Finally, all the downsampled segmentation map sets are spliced ​​to obtain the segmentation map of the entire tumor area of ​​the breast cancer immunohistochemistry image.

[0097] In this embodiment, the residual network ResNet is constructed with reference to the document "He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.", and the nearest neighbor downsampling operation is scaled 16 times.

[0098] Furthermore, the classification of invasive cancer and carcinoma in situ requires the use of the convex hull ratio and curvature features of the tumor area contour. Therefore, it is necessary to extract the contours of all tumor areas from the tumor area map. However, due to the high resolution of WSI, saving the segmentation map of the original resolution size has the problem of memory overflow / slow processing speed and high computational complexity and space complexity. Saving the downsampled result will result in insufficient resolution of contour features and loss of information. Therefore, this application uses the tumor area segmentation map obtained by downsampling the tumor tissue semantic segmentation model to perform connected domain analysis and extract contour features from the original resolution result, specifically:

[0099] Connected component labeling is performed on the tumor region segmentation map to obtain a connected component map, each connected component map is traversed to extract the coordinate range of the connected component map, the tumor region segmentation map of the corresponding image block is obtained according to the coordinate range index and spliced ​​to obtain the tumor region contour in the tumor region segmentation map.

[0100] Among them, the convex hull ratio of the tumor area contour is β The calculation formula is: β = A contour / A convexhull ,in, A contour is the area of ​​the tumor region contour, A convexhull is the area of ​​the convex hull contour of the tumor region;

[0101] The curvature feature uses the average curvature of the tumor region contour Ak Said, the formula is:

[0102] ,

[0103] in, n is the number of contour points contained in the contour of a tumor area; k u For the u The curvature value of a contour point is calculated based on two adjacent contour points.

[0104] For u The curvature value of a contour point is calculated by using the adjacent points before and after ( x u-1 ,y u-1 ),( x u+1 ,y u+1 ) to estimate the curvature; specifically, the curve equation is first expressed as a parametric equation, and for the curve parameter t, there exists:

[0105] ,

[0106] make , , so that the parameterized curve meets the boundary conditions at a specific point, that is:

[0107] ,

[0108] Solving the above equations we can calculate a 1. a 2. a 3. b 1. b 2 and b 3; Finally, according to the following curvature formula, the first u The curvature value of the contour point is as follows:

[0109] ,

[0110] in, , , , .

[0111] Furthermore, the convex hull ratio and curvature features of the tumor region contour were clustered using the Gaussian Mixture Model (GMM), and the feature prior distribution of the tumor region segmentation map was fitted. Then, the naive Bayes classifier was used to classify the tumor region segmentation map to obtain the invasive cancer region and the carcinoma in situ region.

[0112] In the immunohistochemistry scoring task, since we only care about the proportion of tumor cells in the invasive cancer area, the tumor cells in the invasive cancer area mentioned below refer to the tumor cells located inside the invasive area. The coordinate points calculated by the cell nucleus segmentation can be used to confirm whether the cells are located in the invasive cancer area. This is a conventional technical means and will not be repeated in this application. After completing the classification of carcinoma in situ and invasive cancer areas, it is necessary to interpret the cell membrane staining level of all tumor cells in the invasive cancer area and complete the number statistics of cells with different staining levels. However, the marking of the staining level is also difficult to obtain, and manual evaluation is highly subjective. Therefore, the present application provides a cell staining grading method.

[0113] Since HER2 is a stain for cell membranes, the staining level of cells is expressed in the image as the pixel intensity value in the DAB color channel. Therefore, it is necessary to segment the cell membrane area first, and then segment the cell nucleus outside the cell nucleus outline in the cell nucleus segmentation map. p The pixel range is taken as the cell membrane region. In this embodiment, the range of 10 pixels outside the cell nucleus outline is taken as the cell membrane region of the cell.

[0114] Furthermore, the color deconvolution algorithm is used to extract the DAB channel information of the cell membrane area, which is expressed as: X DAB ={ x 1, x 2, x 3,..., x n},in, n is the number of pixels in the cell membrane area.

[0115] Furthermore, the naive Bayes classifier is used to classify the degree of cell staining based on the staining level, specifically:

[0116] In order to preliminarily distinguish cell samples with different staining levels, the average value of the DAB channel feature in the cell membrane region is first used to preliminarily screen the cells in the cell nucleus segmentation map based on the staining level to obtain cell sets with different staining levels. The average value of the DAB channel feature in the cell membrane region is calculated as:

[0117] ,

[0118] in, is the average value of DAB channel characteristics of all cell membrane regions, x i For the i DAB channel characteristics in each cell membrane region, n is the number of cell membrane regions.

[0119] Then, the statistics of each cell set at different staining levels are calculated to describe the staining level characteristics of each cell set; wherein the statistics include the mean, median, maximum value, variance, kurtosis and range of the DAB channel pixels in the cell membrane region of each cell set.

[0120] Then, Gaussian distribution was used to fit the statistics of cell collections at different staining levels to establish a multivariate naive Bayes classifier;

[0121] Finally, the statistics of tumor cells in the invasive cancer area at different staining levels are used as input to determine the staining level of tumor cells. The determination formula is:

[0122] ,

[0123] in, y i For the i Dyeing level, P ( y i ∣ X ) is the dyeing grade characteristicX Judged as the i The probability of a coloring level, P ( X ) is the dyeing grade characteristic X The marginal probability of , as the denominator, will not participate in the final calculation; P ( y i ) is the i The prior probability of each staining level can be obtained statistically from a small amount of prepared IHC image data; P ( X ∣ y i ) is in i Dyeing grade characteristics under each dyeing grade X The probability distribution of is calculated by Gaussian distribution:

[0124] ,

[0125] in, f () is the probability density function of the Gaussian distribution, that is:

[0126] ,

[0127] in, N is the number of statistics, x j Characteristic of dyeing grade X Middle j Statistics, ( μ ij , σ ij )for x j exist y i The parameters satisfy the Gaussian distribution.

[0128] In this embodiment, if the tumor cell staining level is determined to be strong staining, it is a strongly positive tumor cell; if the tumor cell staining level is determined to be moderate staining, it is a moderately positive tumor cell; if the tumor cell staining level is determined to be weak staining, it is a weakly positive tumor cell; if the tumor cell staining level is determined to be no staining, it is a negative tumor cell.

[0129] Furthermore, once the staining levels of all tumor cells in the invasive cancer area are determined, the HER2 score can be calculated based on the ratio of tumor cells with different staining levels according to the clinical guidelines, specifically:

[0130] When the number of tumor cells with strong staining in the invasive cancer area is greater than 10% and the cell membrane staining is complete and uniform, the HER2 score of the whole section image of breast cancer is 3+, and the breast cancer is positive;

[0131] When the number of tumor cells with medium or weak staining in the invasive cancer area is greater than 10% and the cell membrane staining is complete, or when the number of tumor cells with strong staining in the invasive cancer area is less than or equal to 10% and the cell membrane staining is complete, the HER2 score of the whole section image of breast cancer is 2+, and further detection and observation are required;

[0132] When the number of weakly stained tumor cells in the invasive cancer area is greater than 10% and the cell membrane staining is incomplete, the HER2 score of the whole section image of breast cancer is 1+, and the breast cancer is negative;

[0133] When the staining level in the invasive cancer area is no staining, or when the number of tumor cells with weak staining level in the invasive cancer area is less than or equal to 10% and the cell membrane staining is incomplete, the HER2 score of the whole section image of breast cancer is 0, and the breast cancer is negative.

[0134] The clinical guidelines in this application refer to the document "Rakha EA, Pinder SE, Bartlett JMS, et al. Updated UK Recommendations for HER2 assessment in breast cancer[J]. Journal of clinical pathology, 2015, 68(2): 93-99.".

[0135] In summary, the present invention applies deep learning technology to the automatic scoring of breast cancer HER2 immunohistochemistry images, realizes multi-level processing from macro to micro, improves the accuracy and efficiency of image analysis, and compared with the prior art, the present invention directly simulates the diagnostic process of pathologists through a deep learning model, and enhances the diagnostic logic and interpretability of the model. The present invention not only improves the consistency and reproducibility of the scoring, but also greatly shortens the diagnosis time, provides immediate support for clinical decision-making, and takes an important step in improving the automation, standardization and clinical practicality of the automatic scoring of breast cancer HER2 immunohistochemistry images.

[0136] Furthermore, this embodiment performs performance tests on this method (Ours) and other existing methods (ABMIL, CLAM, TransMIL) on the HER2ZJY test dataset. The HER2ZJY dataset is a private dataset collected in a hospital. Three pathologists were invited to perform HER2 scoring, so four groups of annotation results can be obtained, namely pathologist one, pathologist two, pathologist three, and expert voting. Among them, the result of the expert vote is to observe whether the scoring results of the three pathologists are consistent. If they are inconsistent, voting is adopted. If the results of the three pathologists are all different, the judgment results of the doctor with higher seniority are adopted. Then, the HER2 scores predicted by different methods are compared respectively, and the accuracy ACC and F1 score are selected as evaluation indicators. Among them, the classification-based algorithm ABMIL was constructed with reference to the literature (Ilse M, Tomczak J, Welling M. Attention-based deep multiple instance learning [C] / / International conference on machine learning. PMLR, 2018: 2127-2136.); CLAM was constructed with reference to the literature (Lu MY, Williamson DFK, Chen TY, et al. Data-efficient and weakly supervised computational pathology on whole-slide images [J]. Nature biomedical engineering, 2021, 5 (6): 555-570.); TransMIL was constructed with reference to the literature (Shao Z, Bian H, Chen Y, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification [J]. Advances in neural information processing systems, 2021, 34: 2136-2147.). It is worth noting that these MIL-based methods all performed five-fold cross validation on the HER2ZJY test data, while the method of the present invention did not use the annotation of the above HER2ZJY test data for training. The performance test results are shown in Table 1 below:

[0137] Table 1 Performance test results

[0138]

[0139] It can be seen from the table that the accuracy ACC and F1 Score indicators of the present invention are better than those of the three existing MIL-based methods, and the MIL-based algorithm cannot explain its prediction results, so it lacks practicality in clinical practice.

[0140] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0141] Based on the same idea as the automatic scoring method for immunohistochemical images of breast cancer HER2 in the above embodiment, the present invention also provides an automatic scoring system for immunohistochemical images of breast cancer HER2, which can be used to execute the automatic scoring method for immunohistochemical images of breast cancer HER2. For ease of explanation, the structural schematic diagram of an embodiment of an automatic scoring system for immunohistochemical images of breast cancer HER2 only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0142] like Figure 5 As shown, another embodiment of the present invention provides an automatic scoring system for immunohistochemical images of breast cancer HER2, including an image processing module, an image segmentation module, a contour acquisition module, a region division module, a staining classification module and a scoring determination module;

[0143] Among them, the image processing module is used to obtain breast cancer immunohistochemistry images and perform preprocessing to obtain image block sets;

[0144] The image segmentation module is used to use the cell nucleus instance segmentation model and the tumor tissue semantic segmentation model to infer the image block set and output the cell nucleus segmentation map and tumor area segmentation map of the breast cancer immunohistochemistry image respectively;

[0145] The contour acquisition module is used to extract the tumor region contour in the tumor region segmentation map and calculate the convex hull ratio and curvature characteristics of the tumor region contour;

[0146] The region segmentation module is used to cluster the convex hull ratio and curvature features of the tumor region contour through a Gaussian mixture model to obtain the feature prior distribution of the tumor region segmentation map, and to classify the tumor region segmentation map using a naive Bayes classifier to obtain the invasive cancer region and the carcinoma in situ region;

[0147] The staining classification module is used to perform cell membrane segmentation on the cell nucleus segmentation map to obtain the cell membrane region, extract the DAB channel features of the cell membrane region, and classify the cell staining degree based on the staining level through the naive Bayes classifier to obtain tumor cells with different staining levels in the invasive cancer region; the staining levels include strong staining, medium staining, weak staining and no staining;

[0148] The scoring module is used to count and proportion tumor cells of different staining grades in the invasive cancer area, and complete the HER2 score of breast cancer immunohistochemistry images according to clinical guidelines.

[0149] It should be noted that the automatic scoring system for immunohistochemical images of breast cancer HER2 of the present invention corresponds one to one with the automatic scoring method for immunohistochemical images of breast cancer HER2 of the present invention. The technical features and beneficial effects described in the embodiment of the automatic scoring method for immunohistochemical images of breast cancer HER2 are applicable to the embodiment of the automatic scoring system for immunohistochemical images of breast cancer HER2. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0150] In addition, in the implementation of the automatic scoring system for immunohistochemical images of breast cancer HER2 in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the automatic scoring system for immunohistochemical images of breast cancer HER2 is divided into different program modules to complete all or part of the functions described above.

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for automatically scoring immunohistochemical images of breast cancer HER2, characterized in that: The steps include: Obtain breast cancer immunohistochemistry images and perform preprocessing to obtain image block sets, specifically: Downsampling of breast cancer immunohistochemistry images and conversion to grayscale images I gray , expressed as: I gray = DS ( I original ),in, DS () is the downsampling function, I original Immunohistochemistry images for breast cancer; Apply histogram equalization technology to obtain the global contrast map of grayscale images I eq ; Adaptive threshold algorithm is used to segment the global contrast map to obtain the tissue region segmentation map. M ; Set the sliding window size w × h With sliding step s , traverse the entire tissue region segmentation map through the sliding window technology M , record the coordinates of the image blocks captured by each sliding window; For each image block coordinate ( x m , y m ), calculate its segmentation map in the tissue area M The average value of the pixels at the corresponding position in Am m ,like Am m Greater than the preset threshold θ , then add it to the pending queue Q Get an image block set; The pending queue Q Used as input for subsequent cell nucleus instance segmentation model and tumor tissue semantic segmentation model; The cell nucleus instance segmentation model and tumor tissue semantic segmentation model are used to infer the image block set and output the cell nucleus segmentation map and tumor area segmentation map of breast cancer immunohistochemistry image respectively; Extract the tumor region contour in the tumor region segmentation map, and calculate the convex hull ratio and curvature characteristics of the tumor region contour; The convex hull ratio and curvature features of the tumor region contour were clustered using a Gaussian mixture model to obtain the feature prior distribution of the tumor region segmentation map, and the naive Bayes classifier was used to classify the tumor region segmentation map to obtain the invasive cancer region and the carcinoma in situ region. Performing cell membrane segmentation on the cell nucleus segmentation map to obtain the cell membrane region, extracting the DAB channel features of the cell membrane region, and classifying the cell staining degree based on the staining level using a naive Bayes classifier to obtain tumor cells with different staining levels in the invasive cancer region; the staining level includes strong staining, medium staining, weak staining and no staining; Tumor cells with different staining grades in the invasive cancer area were counted and their proportions were statistically analyzed, and the HER2 score of breast cancer immunohistochemistry images was completed according to clinical guidelines.

2. The method for automatic scoring of immunohistochemical images of breast cancer HER2 according to claim 1, characterized in that: The cell nucleus instance segmentation model is a trained Hover-Net model; When the Hover-Net model is trained, the H color channel of the HE image is extracted using color deconvolution as training data; then the cell nucleus annotation data of the prepared breast cancer immunohistochemistry image is used to fine-tune the parameters.

3. The method for automatic scoring of immunohistochemical images of breast cancer HER2 according to claim 1, characterized in that: The tumor tissue semantic segmentation model is a trained U-Net model; The U-Net model uses the residual network ResNet as the feature encoder, performs full-supervised training, and implements semantic segmentation by overlapping sliding windows. The specific process is as follows: Read the image block set, save the segmentation map output by each image block according to the coordinates of the image block, and perform the nearest neighbor downsampling operation on each segmentation map. z times to obtain the downsampled segmentation map, and finally concatenate all the downsampled segmentation map sets to obtain the segmentation map of the entire tumor area of ​​the breast cancer immunohistochemistry image.

4. The method for automatic scoring of immunohistochemical images of breast cancer HER2 according to claim 1, characterized in that: The extraction of all tumor area contours from the tumor area segmentation map is specifically as follows: Performing connected component labeling on the tumor region segmentation map to obtain a connected component map, traversing each connected component map to extract the coordinate range of the connected component map, obtaining the tumor region segmentation map of the corresponding image block according to the coordinate range index and splicing them to obtain the tumor region contour in the tumor region segmentation map; The convex hull ratio β The calculation formula is: β = A contour / A convexhull ,in, A contour is the area of ​​the tumor region contour, A convexhull is the area of ​​the convex hull contour of the tumor region; The curvature feature uses the average curvature of the tumor area contour Ak Said, the formula is: ， in, n is the number of contour points contained in the contour of a tumor area; k u For the u The curvature value of a contour point is calculated based on two adjacent contour points.

5. The method for automatic scoring of immunohistochemical images of breast cancer HER2 according to claim 1, characterized in that: The cell membrane segmentation refers to the segmentation of the cell nucleus outside the cell nucleus outline in the cell nucleus segmentation diagram. p The range of pixels is taken as the cell membrane area.

6. The method for automatic scoring of immunohistochemical images of breast cancer HER2 according to claim 1, characterized in that: The DAB channel features of the cell membrane region are extracted using a color deconvolution algorithm.

7. The method for automatic scoring of immunohistochemical images of breast cancer HER2 according to claim 1, characterized in that: The naive Bayes classifier is used to classify the degree of cell staining based on the staining level, specifically: The average value of the DAB channel feature of the cell membrane region is used to preliminarily screen the cells in the cell nucleus segmentation map based on the staining level to obtain a collection of cells at different staining levels; Calculating statistics of each cell set in cell sets at different staining levels; the statistics include mean, median, maximum, variance, kurtosis and range of DAB channel pixels in the cell membrane region of each cell set; A multivariate naive Bayes classifier was established by fitting the statistics of cell collections at different staining levels using Gaussian distribution. The statistics of tumor cells in the invasive cancer area at different staining levels are used as input to determine the staining level of tumor cells. The determination formula is: ， in, y i For the i Dyeing level, P ( y i ∣ X ) is the dyeing grade characteristic X Judged as the i The probability of a coloring level, P ( X ) is the dyeing grade characteristic X The marginal probability of P ( y i ) is the i The prior probability of the coloring level, P ( X ∣ y i ) is in i Dyeing grade characteristics under each dyeing grade X The probability distribution of is calculated by Gaussian distribution: ， in, f () is the probability density function of the Gaussian distribution, that is: ， in, N is the number of statistics, x j Characteristic of dyeing grade X Middle j Statistics, ( μ ij , σ ij )for x j exist y i The parameters satisfy the Gaussian distribution.

8. An automatic scoring system for immunohistochemical images of breast cancer HER2, characterized in that: The system includes an image processing module, an image segmentation module, a contour acquisition module, a region division module, a staining classification module and a scoring and determination module; The image processing module is used to obtain breast cancer immunohistochemistry images and perform preprocessing to obtain an image block set, specifically: Downsampling of breast cancer immunohistochemistry images and conversion to grayscale images I gray , expressed as: I gray = DS ( I original ),in, DS () is the downsampling function, I original Immunohistochemistry images for breast cancer; Apply histogram equalization technology to obtain the global contrast map of grayscale images I eq ; Adaptive threshold algorithm is used to segment the global contrast map to obtain the tissue region segmentation map. M ; Set the sliding window size w × h With sliding step s , traverse the entire tissue region segmentation map through the sliding window technology M , record the coordinates of the image blocks captured by each sliding window; For each image block coordinate ( x m , y m ), calculate its segmentation map in the tissue area M The average value of the pixels at the corresponding position in Am m ,like Am m Greater than the preset threshold θ , then add it to the pending queue Q Get an image block set; The pending queue Q Used as input for subsequent cell nucleus instance segmentation model and tumor tissue semantic segmentation model; The image segmentation module is used to use a cell nucleus instance segmentation model and a tumor tissue semantic segmentation model to infer the image block set and output a cell nucleus segmentation map and a tumor region segmentation map of a breast cancer immunohistochemistry image respectively; The contour acquisition module is used to extract the tumor region contour in the tumor region segmentation map, and calculate the convex hull ratio and curvature characteristics of the tumor region contour; The region segmentation module is used to cluster the convex hull ratio and curvature features of the tumor region contour through a Gaussian mixture model to obtain a feature prior distribution of the tumor region segmentation map, and to classify the tumor region segmentation map using a naive Bayes classifier to obtain an invasive cancer region and a carcinoma in situ region; The staining classification module is used to perform cell membrane segmentation on the cell nucleus segmentation map to obtain the cell membrane region, extract the DAB channel features of the cell membrane region, and classify the cell staining degree based on the staining level through a naive Bayes classifier to obtain tumor cells with different staining levels in the invasive cancer region; the staining level includes strong staining, medium staining, weak staining and no staining; The scoring module is used to count and proportion tumor cells of different staining grades in the invasive cancer area, and complete the HER2 score of breast cancer immunohistochemistry images according to clinical guidelines.

Images