Methods and systems for automated HER2 scoring

Abstract

A method and system for generating a predicted HER2 score using a machine learning model are disclosed. The exemplary method generally includes using a first machine learning model to identify a plurality of nuclei and membrane segments in a region of interest within an input image. For the plurality of nuclei and membrane segments identified within the input image, a plurality of features are extracted and classified into one of a plurality of feature categories. A predicted HER2 score indicating the likelihood that the stained tissue sample captured in the input image is HER2 positive or HER2 negative is generated based on the classifications assigned to the plurality of extracted features associated with the plurality of segments using a second machine learning model.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to cell tissue sample scoring using image data, and more particularly, to methods and systems for automatically generating IHC HER2 score predictions for images of IHC stained tissue samples.

Background Art

[0002] Human epidermal growth factor receptor 2 (HER2) is a protein found in cells, and it has been found to be an important component in the regulation of cell growth. However, when the HER2 protein is altered in mutant cells such as cancer cells, there is a possibility that extra HER2 protein receptors are created. Overexpression of the HER2 protein receptor causes an increase in cell growth and proliferation. HER2 protein overexpression in cancer cells has been found to be a predictive marker for treatment based on HER2 targeted therapy. Based on this finding, a multiplex staining-based HER2 test has been developed for invasive cancer and assists physicians in making treatment decisions. Therefore, the HER2 test has become a routine practice for screening cancer cells diagnosed in pathology.

[0003] In the screening of breast cancer cells, HER2 test results are represented by a HER2 "score" in the range of IHC0 to IHC3. Then, the HER2 score of the analyzed tissue sample is utilized to evaluate the patient's HER2 status diagnosis (HER2 positive or HER2 negative). Conventionally, the process of evaluating the HER2 "score" of a cell tissue sample has been based simply on visual analysis of an IHC stained tissue sample by a pathologist or physician performing the examination. This process is time-consuming and the results are often inconsistent.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Therefore, in the art, there is a need for an improved system and method for automatically IHC HER2 scoring of cell tissue samples to assist in the treatment of breast cancer, for example, to provide a HER2 score prediction of an analyzed cancer tissue sample to assist in HER2 status diagnosis.

[0005] This "Summary of the Invention" is provided to introduce in a simplified form a selection of concepts that are further described below in the "Detailed Description of the Invention". This "Summary of the Invention" is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Other features, details, utilities, and advantages of the claimed subject matter will become apparent from the following written "Detailed Description of the Invention", which includes aspects defined in the appended claims and shown in the accompanying drawings.

Means for Solving the Problems

[0006] In one embodiment, the present disclosure provides a computer-implemented system and method for generating a predicted HER2 score for an image of an IHC-stained cancer tissue sample. According to a particular embodiment of the present disclosure, the HER2 score prediction system presented herein is designed to provide a prediction of a regional HER2 score for each contiguous region of IHC-stained cancer tissue cells in the image, as well as a prediction of an overall HER2 score for the entire sample captured in the image. The overall HER2 score prediction is based on one or more generated regional HER2 score predictions.

[0007] In one embodiment, the method generally includes specifying one or more regions of interest in a patient image that includes an IHC-stained tissue sample, and identifying a plurality of segments in the patient image using a first machine learning model, where each of the plurality of segments includes nuclei or membranes within one or more regions of interest. A plurality of features are extracted from the plurality of segments identified in the input image by the first machine learning model. The plurality of features can then be classified into one of a plurality of feature categories. A predicted HER2 score is generated using a second machine learning model based on the classifications assigned to the plurality of extracted features associated with the plurality of segments. The predicted HER2 score can indicate the likelihood that the stained tissue sample captured in the input image is HER2 positive or HER2 negative.

[0008] Yet further embodiments provide a computer-implemented method for training a predicted HER2 tissue scoring model to generate a predicted HER2 score for an image of a stained tissue sample using a machine learning model. The exemplary method generally includes receiving a training data set that includes a plurality of images of stained tissue samples. In one aspect, the plurality of images of stained tissue samples within the training data can be labeled to identify segments of membranes and nuclei captured in the plurality of images. In another aspect, the labeled membrane and nuclear features within the plurality of images can be classified into one of a plurality of feature categories. A first machine learning model is trained based on the training data set to classify segments of an input image as nuclei or membranes. A second machine learning model is trained based on the training data set to generate a predicted HER2 score for the input image. A predicted HER2 score that indicates the likelihood that the stained tissue sample captured in the input image is HER2 positive or HER2 negative.

[0009] Other embodiments provide a processing system configured to perform the above-described method and the methods described herein, a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the above-described method and the methods described herein, a computer program product embodied in a computer-readable storage medium including code for performing the above-described method and the methods further described herein, and a processing system including means for performing the above-described method and the methods further described herein.

[0010] The following description and the related drawings detail specific exemplary features of one or more embodiments.

[0011] To enable a more detailed understanding of the features described above of the present disclosure, a more detailed description of the present disclosure, briefly summarized above, may be made with reference to the embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings show only exemplary embodiments of the present disclosure and should not be considered as limiting its scope, as other equally effective embodiments may be recognized.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 6C

Figure 7

Figure 8A

Figure 8B

Figure 8C

[0013] For ease of understanding, where possible, the same reference numbers are used to designate the same elements common to the figures. It is intended that the elements and features of one embodiment may be beneficially incorporated into other embodiments without further elaboration.

[0014] Among the major laboratory techniques conventionally used to determine the HER2 status of breast cancer cells, one is immunohistochemical (IHC) assay staining analysis to examine the level of HER2 protein expression in tissue samples. IHC is generally utilized in histology to detect the presence of specific protein markers that can correspond to a particular tumor. For the accuracy and reproducibility uniformity of the HER2 test in breast cancer cells, the American Society of Clinical Oncology (ASCO) / College of American Pathologists (CAP) jointly issued instructions regarding the HER2 test for diagnosed breast cancer patients and HER2 scoring guidelines. The ASCO / CAP guidelines for IHC HER2 scoring include the recommendation that the HER2 status be initially evaluated by IHC using a semi - quantitative scoring system. Therefore, the common practice of the HER2 test in the industry was to use IHC as a screening test and FISH as a confirmatory test when the IHC of HER2 was uncertain.

[0015] To evaluate the IHC HER2 score of a tissue sample using IHC analysis, a stained tissue sample from the IHC assay is placed on a slide glass to form a static sample for observation by a microscope or other microscopic observation device. An image of the static sample can be acquired to generate a whole slide image (WSI) file of the static sample for storage and later review. The WSI of the stained tissue sample can then be reviewed and evaluated by a pathologist, who then makes objective and subjective decisions using the ASCO / CAP guidelines to generate the IHC HER2 score of the analyzed and observed stained tissue sample within the WSI. According to the ASCO / CAP guidelines, the IHC HER2 score of an invasive breast cancer specimen ranges from IHC0 to IHC3+. In the determination of HER2 status using the IHC HER2 scoring results, an evaluated stained specimen with an IHC3+ score is considered HER2 positive, a specimen with an IHC0 or IHC1+ score is considered HER2 negative, and when having an IHC2+, it is considered uncertain about HER2 protein expression and requires further testing and confirmation using a second IHC or FISH test.

[0016] The IHC assay detects HER2 protein overexpression using a monoclonal or polyclonal antibody that binds to the HER2 protein. The HER2 scoring method disclosed herein assumes that the IHC assay used is standardized by strict quality control. Examples of FDA-approved methods for HER2 evaluation using an IHC assay kit include HerceptTest™ (DAKO, Glostrup, Denmark) and the HER2 / neu (4B5) rabbit monoclonal primary antibody (Ventana, Tucson, Arizona). Such kits are known to contain high-quality reagents that are well standardized with respect to known specificity and sensitivity. Standardization applies to all parameters of the test, including all aspects of pre-analytical tissue sample handling, type and duration of fixation, tissue processing, assay performance, interpretation, and reporting.

[0017] However, there is no universal gold standard that can accurately measure the expression level of the HER2 protein. IHC assays cannot quantitatively measure the expression level of the HER2 protein. Instead, the HER2 expression scoring method based on the ASCO / CAP guidelines relies on the semi - quantitative visual assessment of the cancer cell staining pattern by an examining pathologist who determines the HER2 score based on the observed membrane staining in the stained tissue sample. Specifically, static tissue samples are evaluated based on (1) the completeness of the membrane staining observed in the stained cancer cells, if any, (2) the intensity of the membrane staining, if present, and (3) the percentage of cancer cells in which staining is observed, if any. Therefore, pathologist subjectivity can be involved, which leads to false - positive or false - negative determinations and low reproducibility of the determinations.

[0018] To overcome these and other drawbacks, according to certain embodiments of the HER2 scoring system and method disclosed herein, the HER2 score prediction system is designed to provide a HER2 score prediction for an image of an IHC - stained breast cancer tissue sample to assist a physician in evaluating a HER2 status diagnosis.

[0019] In certain embodiments, the HER2 score prediction system described herein can provide an automated real-time HER2 score prediction for an image of an IHC-stained breast cancer tissue sample using various algorithms or artificial intelligence (AI) models, such as deep learning models or machine learning models, trained based on images of IHC-stained and confirmed breast cancer cells. According to the ASCO / CAP guidelines for IHC-HER2 scoring, certain aspects are directed to algorithms and / or machine learning models designed to detect nuclei and membranes within an image of an IHC-stained tissue sample, analyze and use the IHC staining of each nucleus and membrane within the cancer tissue sample, thereby generating a HER2 score prediction for each analyzed sample. The algorithms and / or machine learning models may be used in combination with one or more other image processing software or tools configured to extract and classify features from the analyzed images relevant to the ASCO / CAP HER2 scoring guidelines. In particular, the algorithms and / or machine learning models can take into account parameters such as the intensity and completeness of the staining of the nuclei and membranes within the stained tissue sample being analyzed. Based on these parameters, the algorithms and / or machine learning models can provide a prediction of the HER2 score of the stained tissue sample within the analyzed image.

[0020] According to certain embodiments, prior to deployment, the machine learning model is trained with training data, such as labeled nucleus and membrane image segments. As described in more detail herein, a human observer can prepare the training data by reviewing and labeling each image segment of a WSI showing an IHC-stained breast cancer tissue sample to indicate and identify the stained nucleus and membrane segments shown in the WSI. After all image segments of the WSI have been reviewed and labeled as showing a nucleus, a membrane, or neither, the labeled image segments are used to train a deep learning machine model, such as a convolutional neural network (CNN), thereby enabling the detection of nuclei and membranes within the whole slide image.

[0021] In some embodiments, to train the machine learning model to be scanner-independent, the image segments used as training data can be enhanced using various contrasts, brightnesses, blurs, sharpnesses, trimmings, cuttings and mixings, elastic transformations, and colors to account for variations in the WSI due to differences in scanners, microscopes, and cameras used to generate the WSI of the stained tissue sample. The image segments for training can also be rotated and input in various orientations to account for various positions and orientations corresponding to the placement of the tissue sample slide when the WSI is captured and generated.

[0022] Next, when nuclei and membranes within a WSI are detected by a deep learning machine learning model, features associated with the detected nuclei and membranes are extracted and classified based on HER2 scoring related factors in the ASCO / CAP guidelines for use in generating a HER2 score prediction. In certain embodiments, data associated with the extracted and classified features corresponding to the analyzed nuclei and membrane segments may further be used to train another machine learning model, such as a random forest or support vector machine model, to classify the HER2 score of the analyzed stained tissue sample shown in the WSI. As described in more detail herein, the training feature data may include extracted and classified feature data corresponding to labeled nuclei and membrane image segments and may be provided in the form of a dataset including the data records. Data labeling is the process of adding one or more meaningful and beneficial labels to the data to provide context for learning by a machine learning model. Each data record is characterized (e.g., refined into a set of one or more features or predictor variables) and labeled based on the extracted and classified nuclear and / or membrane features associated with HER2 scoring. In certain embodiments, each data record is labeled using one or more nuclear and / or membrane features and the classification of each respective feature based on the nuclei and / or membranes shown in the image segment. The features associated with each data record can be used as an input to a second machine learning model, and the generated output can be compared to the label assigned to each of the data records (e.g., the HER2 score output for each respective training nucleus / membrane image). The model can calculate a loss based on the difference between the generated output and the provided label. This loss can then be used to modify the internal parameters or weights of the model. By repeatedly processing the features associated with each data record corresponding to each past patient, the model can be repeatedly refined to generate an accurate prediction of the HER2 score of the image of the stained tissue sample based on the features associated with the staining of the nuclei and membranes within the image.

[0023] Once the machine learning model is trained, the model can be used as part of the HER2 score prediction system described herein to perform local and global HER2 score predictions on IHC stained tissue samples captured in new WSIs. In certain embodiments, the HER2 scoring system can include supporting the visualization of images based on the generated HER2 score predictions to assist physicians and pathologists in their own determination of the final HER2 score and HER2 status diagnosis.

[0024] In one embodiment, in addition to generating HER2 score predictions for stained tissue samples in WSIs, a second machine learning model can further generate one or more regional HER2 scores for one or more regions (regions of interest) of a homogeneous and contiguous population of infiltrating cells shown in the WSI in the WSI input image. In certain embodiments, each region of interest (ROI) in the WSI is classified using the regional HER2 score prediction, along with the overall HER2 score prediction output based on the various regional HER2 score predictions generated for the WSI.

[0025] By using a machine learning model and / or algorithm to predict the IHC HER2 score of an image of a stained tissue sample, a HER2 score prediction is provided to assist physicians and pathologists during their own HER2 score assessment and diagnosis of stained tissue samples within a WSI, enabling consistent real-time assistance. In addition, human subjectivity is minimized during the scoring process, thereby enabling better patient outcomes.

[0026] Exemplary embodiments are presented with respect to systems and methods for automated IHC HER2 scoring of breast cancer cells, but the methods and systems presented herein can generally be applied to the scoring of other IHC stained tissue cells.

[0027] FIG. 1 is a schematic diagram showing an exemplary automatic HER2 scoring system 100 for generating a predicted HER2 score of a stained tissue sample image. In one embodiment, the system 100 includes an image data source 110, a HER2 scoring system 120, a result visualization module 130, and a user interface and visualizer 140. The image data source 110 can include a microscope, an optical microscope, an electron microscope, etc. for examining the microscopic structure of the stained tissue sample. The image data source 110 can further include a camera for capturing an image of the stained tissue sample as WSI input data 102 for use in the system 100.

[0028] In the illustrated embodiment, the HER2 scoring system 120 in the system 100 processes the WSI input data 102 received from the image data source 110 and outputs a predicted HER2 score of the input WSI input data 102. The HER2 scoring system 120 includes a detection system 122, a processing and feature extraction module 124, and a HER2 prediction system 126. The detection system 122 can be implemented in a deep learning framework having a convolutional neural network (CNN) 128. Deep learning models using CNNs can be used for image analysis. Thus, the deep learning framework in the detection system 122 can be trained and utilized to analyze the WSI input data 102 of the stained tissue sample to detect each of the nuclei and membranes shown in the WSI.

[0029] The detection system 122 processes the WSI input data 102 and outputs the processed data as WSI detection data 104 to the processing and feature extraction module 124. The WSI detection data 104 includes a portion of an image from the image data source 110 classified as showing nuclei and / or membranes by the detection system 122. Next, the processing and feature extraction module 124 prepares and analyzes the WSI detection data 104 for scoring, including reassembling together portions of the images within the WSI detection data 104 and extracting and classifying one or more of a plurality of features from the nuclei and membranes detected by the detection system 122 within the WSI detection data 104. Next, the processing and feature extraction module 124 outputs the WSI extraction data 106 to the HER2 prediction system 126 to be classified with one or more regional HER2 score predictions 108. In one aspect, the regional HER2 score predictions 108 can be generated for each group of homogeneous and contiguous infiltrating cell populations captured in the WSI.

[0030] In the illustrated example, the HER2 prediction system 126 includes a machine learning (ML) model 132. According to certain embodiments, the detection system 122 can alternatively implement one or more other types of machine learning models including, but not limited to, a graph neural network, a recurrent neural network, a capsule neural network, or other deep learning models capable of learning to recognize and detect images or portions of images. According to certain embodiments, the ML model 132 can include one or more of a random forest walk, a support vector machine, a decision tree, a convolutional neural network, or other ML models capable of classifying images or portions of images.

[0031] When one or more regional HER2 score predictions 108 are obtained for the WSI input data 102, the HER2 scoring system 120 outputs the one or more regional HER2 score predictions 108 to the result visualization module 130, and additional processing of the WSI input data 102 is performed using one or more of the one or more regional HER2 score predictions 108, the WSI detection data 104, or the WSI extraction data 106 before generating a visual output to the user interface and visualizer 140. When a visual output for the user interface and visualizer 140 is generated, pathologists and physicians can use the user interface and visualizer 140 to review the one or more regional HER2 score predictions 108 and, if there are two or more regional HER2 score predictions, the overall HER2 score prediction 109 of the WSI based on the one or more regional HER2 score predictions 108. In one aspect, the user interface and visualizer 140 can generate and provide a region mask 114 that is overlaid on the WSI showing each of the designated ROIs analyzed by the HER2 scoring to generate the HER2 score prediction. In another aspect, the user interface and visualizer 140 can generate a heat mask 116 showing each of the membranes and nuclei detected in each of the ROIs within the WSI for reference to the one or more regional HER2 score predictions 108 output by the HER2 prediction system 126. The results and summaries from the user interface and visualizer 140 can be further exported to a portable format or shared with a third-party healthcare management system for further review.

[0032] The HER2 scoring system 120 is shown using a detection system 122, a processing and feature extraction module 124, and a HER2 prediction system 126, but according to certain embodiments, one or more of these may be located physically remote from the system 100 and accessed via a network.

[0033] Figure 2 shows an exemplary process flow 200 for the operation of IHC HER2 scoring using the system 100 according to a particular embodiment. In block 201, a WSI (whole slide image) of an IHC stained breast cancer tissue sample is obtained for analysis by the system 100.

[0034] In one aspect, the WSI image may include both cancer cell populations and normal cell populations. In step 202, each of one or more regions of the cancer cell population shown in the WSI is designated as a region of interest (ROI) within the WSI for analysis by the HER2 scoring system 120. The WSI may include one or more ROIs depending on the number of regions of the continuous cancer cell population. In one embodiment, one or more ROIs within the WSI may be manually designated by a user prior to the deployment of the HER2 scoring system 120. Alternatively, one or more ROIs within the WSI may be analyzed and designated using an artificial intelligence (AI) model such as software, an algorithm, or a deep learning model or a machine learning model. The WSI with ROI markings is then input as WSI input data 102 into the detection system 122 of the HER2 scoring system 120 to detect nuclei and membranes in each of the ROIs within the WSI.

[0035] In block 204, the detection system 122 determines whether the first trained machine learning is available to detect nuclei and membranes in the WSI input data 102. In the illustrated embodiment, the first machine learning model can be the CNN model 128 from the system 100. If the trained version of the CNN model 128 is not available, the process proceeds to block 206 where the CNN model 128 is trained with labeled training data. Otherwise, the process proceeds to block 208.

[0036] In block 208, the WSI input data 102 is analyzed using the trained CNN model 128, and in block 210, a WSI detection data 134 output is provided that identifies nuclei and membrane segments captured in the WSI image. In one example, since the image of the WSI input data 102 may be too large to be analyzed by the CNN model 128 at once, the CNN model 128 may instead analyze the image of the WSI input data 102 in segments. Specifically, to preserve the spatial relationships between pixels within the image of the WSI input data 102, the CNN model 128 can create a matrix called a filter (or kernel) that will slide across the matrix of pixels within the image of the WSI input data 102. Therefore, in certain embodiments, when analyzing the WSI input data 102, the CNN model 128 analyzes the WSI input data 102 in 512×512 pixel image segments and can classify each image segment as indicating either a nucleus or a membrane. Image segments in which neither a nucleus nor a membrane is detected can be ignored as background by the CNN model 128. Therefore, the WSI detection data 134 output by the CNN model 128 can include various classified 512×512 pixel image segments of nuclei and membranes captured within the WSI input data 102. In one embodiment, the CNN model 128 can instead analyze and output the WSI detection data 134 in 256×256 pixel image segments, 96×96 pixel image segments, or any other size of image segment that can be processed by the neural network.

[0037] Next, in block 212, the WSI detection data 134 is provided to the processing and feature extraction module 124 for further processing and analysis in preparation for the HER2 prediction system 126. In block 214, the WSI detection data 134 is processed by the processing and feature extraction module 124 and output to the HER2 prediction system 126 as the WSI extraction data 106.

[0038] In one aspect, the processing and feature extraction module 124 can stitch together or reassemble the plurality of classified 512×512 pixel nuclear and membrane image segments output from the CNN model 128 back together (the "reassembled WSI"), such that the stitched together image segments are similar to the original stained tissue sample image of the WSI input data 102. By reassembling the classified nuclear and membrane image segments from the CNN model 128, each of the detected nuclei and membranes within the WSI can be viewed as a whole in relation to each of the ROIs within the captured original tissue sample image. After the plurality of classified image segments are stitched together, the extraction module 124 analyzes the reassembled WSI and extracts a plurality of features from each of the identified nuclear and membrane image segments detected in each of one or more ROIs within the reassembled WSI. Depending on the tissue sample captured in the WSI, the plurality of features can be extracted by module 124 from the detected nuclear and membrane segments in each of one or more ROIs within the reassembled WSI.

[0039] The plurality of features extracted by module 124 from each of the ROIs within the reassembled WSI are related to the HER2 scoring of the tissue sample (in accordance with the ASCO / CAP guidelines) and will be used to generate one or more regional HER2 score predictions 108 for each of the one or more respective ROIs within the reassembled WSI, including features related to the membranes and nuclei detected by the CNN model 128 in each of the respective ROIs.

[0040] As an example, the extraction module 124 can extract features including, but not limited to, membrane staining intensity, base color of the membrane - intensity of the base color of the membrane, membrane staining deviation - standard deviation of the intensity detected across the entire captured image of the membrane, membrane staining completeness ratio - ratio of the complete membrane staining to the total detected membrane area, nuclear staining intensity, base color of the nucleus - intensity of the base color of the nucleus, nuclear staining deviation - standard deviation of the intensity detected across the entire captured image of the nucleus, membrane skewness - skewness coefficient in the detected membrane's histogram curve, nuclear skewness - skewness coefficient in the detected nucleus's histogram curve, ratio of nucleus to membrane - ratio of the number of detected membranes / number of detected nuclei, membrane filling area - area of the complete membrane staining based on the area of the detected nucleus, tumor area - tumor volume / area captured by the image, cell percentage - percentage of the completely stained membranes, and DAB average color - average RGB color value of the DAB component in the HED (hematoxylin - eosin - DAB) color space in the image from the nuclei and membranes in each of one or more ROIs in the reconstructed WSI.

[0041] To analyze and extract one or more features from the nuclei and membranes in each of one or more ROIs in the reconstructed WSI, the processing and feature extraction module 124 can include and use one or more of the following image processing libraries or tools for processing the reconstructed WSI, including, but not limited to, OpenCV (CV2), Scikit - Image (skimage), SciPy, Pillow / PIL, NumPy, Mahotas, SimplelTK, Pgmagick, etc.

[0042] When one or more of the features of multiple membranes and nuclei are extracted from the ROI within the reassembled WSI, module 124 further classifies each of the multiple extracted features into one of a plurality of feature categories related to the ASCO / CAP guidelines for IHC HER2 scoring. As an example, the feature of "membrane staining intensity" extracted from the membrane shown in the ROI of the reassembled WSI can be further classified into one of a plurality of categories including "strongly stained", "moderately stained", or "not stained". Other examples of classified extracted features include whether the image segment shows a fully stained membrane, a partially stained membrane, a stained membrane biased more towards brown, and / or a stained membrane biased more towards white.

[0043] Next, one or more of the multiple extracted and classified features of the nuclei and membranes in each ROI can be output to the HER2 prediction system 126 as WSI extraction data 106 to generate one or more regional HER2 score predictions 108 according to the ASCO / CAP guidelines.

[0044] In block 216, the HER2 prediction system 126 determines whether a trained second machine learning for scoring each ROI is available. In the described embodiment, the second machine learning model can be the ML model 132 described in FIG. 1. If a trained version of the ML model 132 is not available, the process proceeds to block 218 where the ML model 132 is trained with labeled training data. Otherwise, the process proceeds to block 220.

[0045] In block 220, the HER2 prediction system 126 uses the trained ML model 132 to classify each of the one or more ROIs using one or more regional HER2 score predictions 108 based on the WSI extraction data 106 from each respective ROI. In one aspect, the one or more regional HER2 score predictions 108 output by the trained ML model 132 are based on the ASCO / CAP guidelines for IHC HER2 scoring. The ML model 132 can be retrained in HER2 scoring to accommodate changes and updates to the ASCO / CAP guidelines for IHC HER2 scoring of IHC-stained cancer tissue samples.

[0046] Finally, in block 222, an overall HER2 score prediction 109 of the WSI based on the one or more regional HER2 score predictions 108 is output to the user interface 140. If the entire tissue sample within the WSI is designated as a single ROI, the overall HER2 score prediction 109 can be the same as the regional HER2 score prediction output by the ML model 132 trained for the single designated ROI.

[0047] As an example, according to the ASCO / CAP guidelines for IHC HER2 scoring, a region of IHC-stained breast cancer cells is scored as IHC3+ if the stained cell membranes of the IHC-stained tissue cells show complete, very strong, homogeneous and continuous circumferential membrane staining in more than 10% of the tumor cells observed within an infiltrating cell population. Thus, in block 220, if the WSI extraction data 106 analyzed for a region of stained tissue cells by the trained ML model 132 includes feature data classified as "membrane strongly stained" and "membrane completely stained", and the percentage or ratio of the number of strongly and completely stained membranes detected to the total number of nuclei detected in each region exceeds 10%, the trained ML model 132 is likely to output a regional HER2 score of IHC3+ for each analyzed region of the stained tissue cells.

[0048] If the total number of nuclei detected in each evaluated region is adopted as the total number of tumor cells observed in the scored region, then the fact that the percentage representing the ratio of the number of stained membranes to all nuclei detected in the evaluated region exceeds 10% indicates that the number of tumor cells with strongly and completely stained membranes by IHC exceeds 10% of the tumor cells in the region.

[0049] As described above, the image of the infiltrative stained tissue sample captured by the WSI may include a number of groups / regions of stained tissue cells or ROIs. In one embodiment, one or more region HER2 score predictions 108 corresponding to each of one or more designated ROIs within the WSI will be output by the HER2 prediction system 126. In an example where two or more region HER2 score predictions 108 are generated for a number of ROIs within the WSI, the HER2 prediction system 126 also outputs an overall HER2 score prediction 109 for the entire WSI based on one or more region HER2 score predictions 108 and the ASCO / CAP guidelines.

[0050] Figure 3 shows an example of a deep learning image analysis computing system 300 utilized by the detection system 122 to detect nuclei and membranes within the WSI input data 102, according to a particular embodiment.

[0051] Deep learning is a subset of machine learning methods based on learned representations in data. Deep learning uses a set of algorithms to model high-level abstractions in data using deep graphs with many processing layers, including linear and non-linear transformations. The term "deep" refers to hierarchical and / or layered learning. While many machine learning systems are seeded with initial features and / or network weights that are modified through learning of the machine learning network, deep learning networks train themselves to identify "good" features for analysis. The basic building block of a deep learning neural network is the perceptron. A perceptron is an algorithm for supervised learning of binary classifiers consisting of a linear component (weighted sum of inputs) and a non-linear component (activation). By combining perceptrons in multiple layers, it becomes possible to represent complex features to address many real-world problems and per-substrate differences.

[0052] As discussed previously, the ASCO / CAP guidelines require that when evaluating the IHC3+ or IHC2+ HER2 score of each tissue sample, at least 10% of the cancer cells in the tissue sample be evaluated. The 10% requirement is similarly applied when evaluating the regional HER2 score for each ROI of the scored tissue sample, by evaluating each of the one or more ROIs indicated in the WSI at that time. Therefore, since each cancer cell indicated in one or more ROIs of the WSI can be presumed to have a single nucleus, the total number of nuclei detected in one or more ROIs of the WSI may correspond to the total number of cancer cells in the WSI. Therefore, information regarding the number of nuclei in one or more ROIs of the WSI can be utilized to determine, for each of the one or more ROIs, the percentage of cancer cells in the entire WSI based on which the predicted regional HER2 score is based.

[0053] As an example, in order for the ROI within the WSI to receive a HER2 score prediction of IHC1+, IHC2+, or IHC3+, the number of cancer cells in each ROI for which a regional HER2 score prediction is generated must not exceed 10% of the cancer cells in the WSI. That is, the number of detected nuclei in the ROI to be scored must be more than 10% of the total number of detected nuclei in all of the one or more ROIs within the combined WSI. Therefore, due to the 10% ASCO / CAP guideline requirement, an ROI that receives a HER2 regional score prediction that is not IHC0 (e.g., based on the classification of other extraction features such as membrane staining, etc.) but contains less than 10% of the nuclei will nevertheless be given an IHC0 score. Therefore, if the number of cancer cells can be quantified across the entire WSI, it can be ensured that the regional HER2 score prediction and the overall HER2 score prediction of the tissue sample shown in the WSI meet the 10% ASCO / CAP guideline.

[0054] As shown in FIG. 3, in one embodiment, the computing system 300 can include a memory 302, one or more processors 304, and a processing circuit for executing a machine learning system 306 having a deep neural network (DNN) 308 composed of a plurality of layers 310A - 310G (collectively "layer 310"). The DNN 308 can be one of various types of deep neural networks (DNNs), such as a convolutional neural network (CNN), such as CNN128, a feedforward neural network, a recurrent neural network (RNN), etc.

[0055] Memory 302 can store information for processing during the operation of computing system 300. Memory 302 can store program instructions and / or data related to one or more of the systems and modules described according to one or more aspects of the present disclosure. One or more processors 304 can execute instructions, and one or more storage devices for memory 302 can store instructions and / or data for one or more modules. The combination of processor 304 and memory 302 can fetch, store, and / or execute instructions and / or data for one or more applications, modules, or software.

[0056] One or more processors 304 and memory 302 can provide an operating environment or platform for one or more modules or units, which can be implemented as software, but in some examples can include any combination of hardware, firmware, and software. One or more processors 304 and / or memory 302 can also be operatively coupled to one or more other software and / or hardware components, including, but not limited to, one or more of the components and / or systems shown in FIG. 1 and other figures of the present disclosure.

[0057] In the example of FIG. 3, the DNN 308 having a convolutional neural network such as CNN 128 receives input data from the input data set 312 and generates output data 314. The input data set 312 and the output data 314 can include various types of information. For example, the input data set 312 can include a plurality of image data related to one or more IHC-stained tissue samples. The output data 314 can include classification data, translated text data, and / or image classification data related to the input data set 312. As disclosed in the above-described process 200, the input data set 312 can correspond to a captured WSI image of a tissue sample such as the WSI input data 102. As disclosed in process 200, the output data 314 can correspond to classification data of the image data segments from the input data set 312, similar to the provision of the WSI detection data 134 by the CNN model 128 in the detection system 122.

[0058] In one embodiment, the DNN 308 can include a plurality of layers 310A-310G (collectively "layer 310"). Each of the layers 310 can include a respective set of artificial neurons. The layer 310 includes an input layer 310A, an output layer 310G, and one or more hidden layers (e.g., layers 310B-310F). The output from the first layer 310A is fed to the intermediate hidden convolutional layers 310B-310F that analyze each of the available image segments. The intermediate hidden layers 310B-310F are fed to the final output layer 310G that identifies and classifies whether the analyzed image segments from the WSI input data 1020 show an image of a nucleus, an image of a membrane, or an image that is neither (background).

[0059] The layer 310 can include a fully connected layer, a convolutional layer, a pooling layer, and / or other types of layers. In a fully connected layer, the output of each neuron in the previous layer forms the input of each neuron in the fully connected layer.

[0060] In the convolutional layer, each neuron in the convolutional layer processes the input from neurons related to the receptive field of the neuron. Specifically, each of the convolutional layers within layer 310 can include one or more filters. The filters can detect patterns within the image and form layers within each convolutional layer. All neurons within a filter share the same weights, and each neuron within one filter is connected to its counterpart within the next filter, such that the output of one filter is passed to a corresponding set of neurons within the next filter.

[0061] The purpose of each filter is to detect different patterns within the image, and when the DNN 308 is trained, the DNN 308 converges or "learns" about which patterns (e.g., features related to nuclear / membrane morphology) within each filter enable the DNN 308 to recognize each image. The output of each filter is called a feature map, and it is the feature maps that are used by the DNN 308 to break down the image into component pieces. Neurons within the filter develop a specific pattern, fire when they see that pattern, and output the result to the next filter. Then, all of these feature maps are constructed into the final image and can be used to process it.

[0062] The pooling layer combines the output of a cluster of neurons in one layer into a single neuron in the next layer. Unlike the convolutional layer, the pooling layer cannot be trained. Its purpose is to subsample the image to emphasize the most important regions that the network processes. The pooling layer is used to reduce computational complexity and decrease the dimensions of the image.

[0063] An example of a pooling layer is the "max pooling layer". The max pooling layer looks at all of the pixels within the receptive field, selects the pixel with the highest value (the maximum value), and passes it to the next layer of the network. Alternatively, other well-known forms of pooling, including but not limited to average pooling, may be used by the DNN 308.

[0064] Memory 302 stores a plurality of training weights 316 for DNN 308. Each input of each artificial neuron in each layer 310 of DNN 308 is associated with a corresponding training weight 316. By using DNN 308 with a plurality of training weights 316, output data 314 is produced.

[0065] As will be described in more detail below, as part of executing a training process, machine learning system 306 can perform a feed-forward stage in which machine learning system 306 uses the plurality of training weights 316 within DNN 308 to determine output data 314 based on the input data of input data set 312. Further, machine learning system 306 can perform a backpropagation method that calculates the gradient of a loss function. The loss function creates a cost value based on the output data. Machine learning system 306 can then update the plurality of training weights 316 based on the gradient of the loss function. Machine learning system 306 can perform the feed-forward method and the backpropagation method many times using different input data. During or after the training process, machine learning system 306 can generate output data 314 based on new untrained input data using the weights trained in the evaluation inference process.

[0066] In another aspect of the embodiment, as described above, all neurons within the filter of the convolutional layer share the same weighting. Thus, the machine learning system 306 can utilize the shared training weights 316 to classify both the nuclei and membranes detected in the analyzed image segment inputs. By using the shared training weights 316, it becomes possible for the machine learning system 306 to detect both nuclei and membranes without additional computation. For example, compared to the use of parallel binary classifiers to analyze each image segment input to separately detect membranes and nuclei, the utilization of the shared training weights 316 by the machine learning system 306 represents a technical improvement in computational efficiency in that it enables the detection of multiple structures without additional computation. Further, since the weights of the filter are tied, this means that the filter does not change as it analyzes each part of the entire slide image. This shared weighting means that after the filter has been trained to recognize a particular shape or feature within an image, the same filter is used, so the network will recognize a particular shape or feature regardless of its position within the image. The use of shared weights reduces the amount of weights (or parameters) that need to be optimized when training the machine learning system 306. The reduced computation also indicates that the machine learning system 306 can be trained faster and often with less data.

[0067] As discussed herein, the deep learning neural network 308 trained and deployed can, in some embodiments, include a machine learning model trained to recognize spatial relationships between pixel data within an input data set where the machine learning model can make a determination or classification. For example, the deep learning model can include other multi-layer neural network-based models that can learn pixel information and relationships between different items within a spatial sequence, including, but not limited to, single shot detection (SSD), region-based convolutional neural network (R-CNN), faster region-based convolutional neural network (Faster R-CNN), and you only look once (YOLO).

[0068] FIG. 4 shows a schematic diagram illustrating an example of the training and use of deep learning machine learning, such as CNN 128, for detecting nuclei and membranes in an input image of stained tissue cells according to a particular embodiment. As discussed in this disclosure, deep learning is a subset of machine learning based on learning representations within data. Machine learning, including deep learning, explores the research and construction of algorithms, which are also referred to herein as tools, that can learn from existing data and make predictions about new data. In one embodiment, such a tool operates to construct and train a trained deep learning CNN model 410 from exemplary training data 402 to make a data-driven prediction or determination represented as an output or evaluation 404. Exemplary embodiments are presented with respect to several machine learning tools, but the principles presented herein may be applied to other machine learning tools.

[0069] In the illustrated embodiment, the training of the deep learning CNN model involves utilizing one or more features 406 to analyze the training data 402 and generate one or more evaluations 404. The features 406 are individual measurable characteristics of the observed phenomenon. The concept of features is related to the concept of explanatory variables used in statistical techniques such as linear regression. In one exemplary embodiment, the features 406 utilized to detect membranes and nuclei within segments of an image can include one or more features related to nuclear morphology and membrane morphology. In another aspect, the training data 402 regarding the features 406 can be of different types and can include one or more of image data, color, size, shape, position, luminance, etc. One of the many advantages of deep learning is the automatic extraction of features for current classification problems, as opposed to engineering handcrafted features. Thus, the CNN model utilizes the training data 402 to distinguish and learn meaningful features 406 useful in classifying one or more images or portions of images from the WSI input data 102, thereby producing a result or evaluation 404.

[0070] In some exemplary embodiments, to train the CNN model to detect nuclei and membranes within the WSI input data 102, the training data 402 can be reexamined and labeled prior to training to provide structured and supervised learning when training the deep learning CNN model. Thus, in one embodiment, the training data 402 can include labeled histological cell image segments of nuclei and membranes. In addition to labeling the training data 402, the training data 402 may also be further prepared and enhanced to more robustly train the CNN model to be scanner-independent.

[0071] In one embodiment, the images from the image data source 110 may vary due to being used by different imaging systems to capture the WSI training data 402. The images within the WSI training data 402 may also vary due to the various protocols, procedures, and labs used to prepare and stain tissue samples prior to investigation. To train the deep learning CNN model to be less scanner-dependent, the image segments used as the training data 402 are further modified to train the CNN model to detect images of nuclei and membranes of various sizes, colors, intensities, positions, and orientations. For example, prior to training the CNN model, the training data 402 may be modified and enhanced using, but not limited to, various contrast, brightness, blur, sharpness, trimming, cropping and mixing, elastic transformation, and color to adapt to different types of scanners. The tissue sample slides used to generate the image segments for the training data 402 may also be captured at various rotations and orientations to adapt to the various positions and orientations of the slide during the capture of the images of the stained tissue samples to be scored. Such training of the CNN model may enable the system 100 to be more robust, scanner-agnostic, and capable of analyzing and generating HER2 score predictions for images of stained tissue samples of various qualities, settings, and characteristics.

[0072] Using the training data 402 and the learned features 406, the CNN model is trained in operation 408 to yield a trained CNN model 410. When the trained CNN model 410 is used to perform an evaluation, new data 412 is provided as input to the trained CNN model 410, and the CNN model 410 generates an evaluation 404 as output based on the features 406 learned by the CNN model 410. For example, when a future WSI of a stained tissue sample is analyzed by the trained CNN model 410, the CNN model 410 utilizes the features 406 learned regarding the morphology of the membrane and nucleus to detect each of the nuclei and membranes in the image from the new data 412. Another advantage of deep learning is the ability to perform transfer learning, i.e., once a model is trained on a large dataset, it is not necessary to retrain the model from scratch for a new dataset.

[0073] Therefore, referring to the system 100 of FIG. 1, when the CNN model 128 is trained in the same manner as the trained CNN model 410 discussed above, the CNN model 128 is used by the detection system 122 to detect and identify the segments of nuclei and membranes in the input image obtained from the image data source 110.

[0074] FIG. 5 shows a schematic diagram illustrating an example of the training and use of a machine learning model for classifying the HER2 score prediction of the region of one or more ROIs in a reconstructed WSI according to a particular embodiment.

[0075] In the illustrated embodiment, the machine learning model can generate an evaluation 504 by utilizing one or more features 506 for analyzing the training data 502. In one exemplary embodiment, the features 506 utilized for classifying one or more regional HER2 score predictions 108 can include a plurality of membrane and nuclear feature classification data of image segments previously detected by the detection system 122 and classified by the processing and feature extraction module 124. For example, the data related to the features 506 within the ROI image segment used by the machine learning model to generate the evaluation 504 can include, without limitation, classification data corresponding to the following plurality of membrane and nuclear features. Membrane staining intensity, base color of the membrane - intensity of the base color of the membrane, membrane staining deviation - standard deviation of the intensity detected across the captured image of the membrane, membrane staining completeness ratio - ratio of the complete membrane staining to the total detected membrane area, nuclear staining intensity, base color of the nucleus - intensity of the base color of the nucleus, nuclear staining deviation - standard deviation of the intensity detected across the captured image of the nucleus, membrane skewness - skewness coefficient in the histogram curve of the detected membrane, nuclear skewness - skewness coefficient in the histogram curve of the detected nucleus, ratio of nucleus to membrane - ratio of the number of detected membranes / number of detected nuclei, membrane filling area - area of the complete membrane staining based on the area of the detected nucleus, tumor area - tumor volume / area captured by the image, cell percentage - percentage of cells that completed the membrane, and DAB average color - average RGB color value of the DAB component in the HED (hematoxylin - eosin - DAB) color space in the image.

[0076] In one embodiment, classification data corresponding to one or more of the above - mentioned plurality of nuclear and membrane features in a plurality of image segments of the nucleus and membrane is used as the training data 502 to provide structured and supervised learning, thereby training the machine learning model to classify each of one or more ROIs within the WSI input data 102 using the regional HER2 score predictions 108 in the range of IHC0 to IHC3 +.

[0077] The machine learning model uses the training data 502 to find correlations between the identified features 506 that affect the result or evaluation 504. In some exemplary embodiments, the training data 502 includes known labeled data regarding one or more identified features 506 for each nucleus and membrane detected in a plurality of image segments, and one or more results / classifications including corresponding HER2 score prediction evaluations based on the ASCO / CAP guidelines for the captured and classified images of the stained nuclei and membrane cells used in the training data 502.

[0078] Using the training data 502 and the identified features 506, the machine learning model is trained in operation 510. The result of the training is the trained ML model 512. Filters for the trained ML model 512 extract the identified features 506 as feature maps from future images in order to obtain optimal classification performance.

[0079] According to certain embodiments, the ML model 512 can include one or more of a random forest walk, a support vector machine, a decision tree, a convolutional neural network, or other machine learning models that can classify an image or a portion of an image based on being trained by the training data 502. These machine learning models can be models that can segment an image into multiple categories, for example, according to classifications (e.g., strong complete membrane staining, weak complete membrane staining, weak incomplete membrane staining, and no staining) assigned to one or more of a plurality of features 506 within the ROI image segment. When segmenting an image into multiple categories, the machine learning model can generate a segmentation map that divides the image into multiple segments. Each segment of the multiple segments can be associated with one of a plurality of features and categories for which the machine learning model was trained. Thus, the segmentation map can identify segments of the image as segments classified as strong complete membrane staining, segments indicating weak complete membrane staining, unstained membrane patterns, and other types of membrane staining patterns, and based on the classification of the plurality of features and categories for each image segment, can generate a corresponding regional HER2 score prediction.

[0080] When the ML model 512 is used to perform an evaluation, new data 508 is provided as input to the trained ML model 512, and the ML model 512 generates an evaluation 504 (regional HER2 score prediction) as output. For example, when future captured and classified tissue cell images of stained nuclei and membranes are analyzed by the trained ML model 512, the ML model 512 utilizes classification data for one or more of the plurality of features 506 of the nuclei and membranes in the image of the new data 508 to output one or more HER2 score predictions in the evaluation 504 for each analyzed ROI in the image of the new data 508.

[0081] In another aspect of the present disclosure, the algorithm parameters corresponding to the training of the ML model 512 may be further adjusted and calibrated after initial use by the user to further train the ML model 512 to be more robust in accurately generating HER2 score predictions for new staining and image variations that were not previously considered during the training of the ML model 512.

[0082] FIG. 6A shows a flowchart of an exemplary method 600A for training and using the system of FIG. 1 using the processes of FIGS. 2, 4, and 5 according to certain embodiments.

[0083] In block 602, a WSI of an IHC stained tissue sample is obtained by examining the IHC stained tissue sample under a microscope and capturing an image of the microscopic structure of the stained tissue sample.

[0084] In block 604, a first machine learning model is trained to detect and identify nuclei and membranes within the image of the stained tissue sample captured in the WSI. According to certain embodiments, the first machine learning model is a deep learning model having a convolutional neural network such as the CNN model 128 of the detection system 122. In one aspect, the first machine learning model analyzes the WSI with 512×512 pixel image segments and outputs each classified 512×512 image segment. The image segments output from the first machine learning model may be classified as (1) "nucleus" if a nucleus is detected in each image segment, (2) "membrane" if a membrane is detected in each image segment, or (3) skipped if neither a nucleus nor a membrane is detected.

[0085] In block 606, the classified nuclear and membrane image segments from the first machine learning model are processed by being reassembled and stitched together so that the detected nuclei and membranes in the WSI can be investigated and analyzed as a whole in relation to the entire stained tissue sample captured in the original WSI. In block 608, features associated with the detected membranes and nuclei in the reassembled WSI image, which are related to IHC HER2 scoring, are extracted and classified based on features related to IHC HER2 scoring guidelines.

[0086] In block 610, a second machine learning model is trained to analyze and classify regions of cells in the WSI using regional IHC HER2 score prediction based on the classified extracted features of the stained nuclei and membranes detected in the cells within each respective region of the WSI. According to certain embodiments, the second machine learning model can be one of a random forest machine learning, a support vector machine, a decision tree model, or any other machine learning model capable of learning and solving classification problems, such as the ML model 132 of the HER2 prediction system 126.

[0087] In block 612, a WSI of a patient's IHC stained tissue sample is received and one or more regions of interest (e.g., stained cancer cells) are designated for scoring. In block 614, a plurality of segments are output by the first machine learning, and each of the plurality of segments corresponds to nuclei and / or membranes detected in each of the one or more regions of interest (ROIs) designated in the WSI. In block 616, the plurality of nuclear and membrane image segment data output by the trained first machine learning model are processed to correspond to the original WSI input and reassembled together. Then, in one or more ROIs of the WSI, data related to the plurality of features of the nuclei and membranes detected by the trained first machine learning model are extracted and classified into one of a plurality of categories related to the plurality of features according to the ASCO / CAP guidelines for IHC HER2 scoring.

[0088] In block 618, each of one or more ROIs corresponding to each homogeneous and continuous infiltrating stained cell population shown in the reconstructed WSI is classified using region HER2 score prediction with a trained second machine learning model based on the classification assigned to a plurality of extracted features related to nuclei and membranes detected in each of the respective ROIs within the WSI by a trained first machine learning model. During the scoring of each of the one or more ROIs by the trained second machine learning model, one or more intermediate outputs are output to the user interface visualization output 140, and details and data regarding how one or more region HER2 score predictions are generated can be provided.

[0089] In one aspect of the embodiment, in block 618 where each ROI shown in the reconstructed WSI is classified by region HER2 score prediction, the region HER2 score prediction is generated based on the classification assigned to a plurality of extracted features related to nuclei and membranes detected in each of the respective ROIs within the WSI by a trained first machine learning model. Therefore, since the classified extracted features are related to both nuclei and membranes detected by the trained first machine learning model and are based thereon (regions where only membranes are detected are discarded and not used for generating the HER2 score prediction), method 600 generates a more accurate HER2 score prediction than conventional methods that rely solely on the detection of stained membranes.

[0090] Finally, at block 620, the overall HER2 score prediction 109 is output and provided for user reference when evaluating the final HER2 score of the WSI and status diagnosis for the IHC-stained tissue sample of the patient obtained at block 612. When two or more regional HER2 score predictions 108 are generated for a single WSI based on two or more ROIs specified in the WSI, the overall HER2 score prediction 109 is output by the HER2 scoring system 120 based on one or more regional HER2 score predictions 108 generated for each of the one or more specified ROIs. In one embodiment, the overall HER2 score prediction output for the WSI by the HER2 scoring system 120 corresponds to the highest regional HER2 score prediction 108 generated by the HER2 scoring system 120 based on the one or more ROIs analyzed and scored.

[0091] FIG. 6B shows a flowchart of an exemplary method 600B for using the system of FIG. 1 to generate a predicted HER2 score for a WSI of an IHC-stained breast cancer tissue sample of a patient, according to a particular embodiment.

[0092] At block 630, a WSI of the IHC-stained tissue sample of the patient is obtained from the image data source 110 for analysis and scoring by the HER2 scoring system 120. Obtaining the patient's WSI from the image data source 110 can include the user examining and capturing a prepared stained slide specimen via a digital camera and / or digital microscope. Alternatively, obtaining the patient's WSI from the image data source 110 can include uploading the patient's WSI from a third-party source and / or external database.

[0093] In block 632, one or more regions of the stained cancer cells in the WSI are designated as the region of interest (ROI) in the WSI for scoring by the HER2 scoring system 120. In block 634, each of the ROIs in the WSI is analyzed by a first machine learning, a plurality of segments are output by the first machine learning, and each of the plurality of segments corresponds to the nuclei and / or membranes detected in each of the one or more ROIs designated in the WSI. In block 636, feature data is extracted from the plurality of nuclear and membrane image segment data output by the trained first machine learning model and classified for IHC HER2 scoring. To extract the feature data, the plurality of nuclear and membrane image segment data are processed and reassembled together into a reassembled WSI corresponding to the original WSI input for evaluating the extracted feature data by the corresponding designated ROI. The extracted feature data is classified into one of a plurality of categories according to the ASCO / CAP guidelines for IHC HER2 scoring based on a plurality of features corresponding to the nuclei and membranes detected by the first machine learning model trained in each of the one or more ROIs.

[0094] Next, in block 638, each of the one or more ROIs shown in the reassembled WSI is classified using the region HER2 score prediction 108 by a second machine learning model based on the classification assigned to the plurality of extracted features associated with block 636. During the scoring of each of the one or more ROIs by the second machine learning model, one or more intermediate outputs are output to the user interface visualization output, and details and data regarding how the one or more region HER2 score predictions are generated can be provided.

[0095] In one embodiment, when the second machine learning model determines between IHC0 and IHC1+ as the region HER2 score prediction 108 of any one of one or more ROIs in the reassembled WSI, the second machine learning model can finally determine between IHC0 and IHC1+ using the DAB average color of each ROI. The DAB average color of one or more ROIs refers to the DAB component in the HED (hematoxylin - eosin DAB) color space of the detected membrane for evaluating the "brownness" of the ROI in the image. In some embodiments, the DAB average color includes an RGB value range between 0 and 255.

[0096] The DAB average color of each ROI can be calculated by first taking in the original RGB image of the membrane detected within the ROI by the first machine learning model and applying a mask that transforms / shows the color / brownness of the space within the image under the membrane. Then, the HED color space components of the ROI are calculated for the masked image, and the DAB channel or component of HED can be extracted by setting the other two components (hematoxylin and eosin) to zero. Immediately after the extraction of the DAB HED component, the image is converted back to the RGB color space, and the average of the DAB components corresponding to the ROI is obtained. A threshold of the RGB value corresponding to the DAB average color of the ROI may be used by the second machine learning model to finally determine whether to classify the ROI as IHC0 or as IHC1+, and as a result, it can be understood that it may affect the overall HER2 score prediction 109 of the WSI for determining the state diagnosis of the corresponding sampled tissue. For example, the threshold utilized by the second machine learning model can be 189 for the DAB average color of the ROI. For an ROI where the second machine learning model is to determine between IHC0 and IHC1+, the second machine learning model classifies it as IHC1+ if the DAB average color of each ROI is less than 189 and classifies it as IHC0 if the DAB average color is greater than 189.

[0097] In block 640, the overall HER2 score prediction 109 is output to the user interface 140 for WSI based on one or more regional HER2 score predictions 108 and provided for reference by the user when evaluating the final HER2 score for WSI and status diagnosis for the IHC-stained tissue sample of the patient obtained in block 612. When two or more regional HER2 score predictions 108 are generated for a single WSI based on two or more ROIs specified in the WSI, the overall HER2 score prediction 109 is output by the HER2 scoring system 120 based on one or more regional HER2 score predictions generated for each of the one or more specified ROIs. In one embodiment, the overall HER2 score prediction 109 output for the WSI by the HER2 scoring system 120 corresponds to the highest regional HER2 score prediction 108 generated based on one or more ROIs analyzed and scored by the HER2 scoring system 120.

[0098] Finally, in block 642, to assist the user in evaluating the final HER2 score of the WSI and to provide the user with the support data for the overall HER2 score prediction 109 generated by the HER2 scoring system 120 to enable the user to re - consider and confirm the prediction 109, the user interface 140 can provide a reference visual display on the WSI to support and show how the prediction 109 was generated. The visual display on the user interface 140 can include overlaying on the original acquired WSI a mask indicating the designated ROI within the WSI and each of the corresponding regional HER2 score predictions 108 generated for each of the ROIs. To provide the user with further support regarding how each of the regional HER2 score predictions 108 was generated by the HER2 scoring system 120 for each of the designated ROIs, the user interface 140 can further overlay on the original WSI another mask identifying each of the nuclei and membranes detected by the first machine learning model for each of the designated ROIs within the WSI. The identification of the nuclei and membranes for each ROI in the user interface 140 also provides the corresponding feature data that was extracted, classified, and used to generate the regional HER2 score predictions 108 for re - consideration by the user.

[0099] FIG. 6C shows a flowchart of an exemplary method 600C for using the system of FIG. 1 to generate a predicted HER2 score, according to a particular embodiment.

[0100] In block 650, one or more regions of the stained cells are designated as regions of interest (ROIs) within a patient image 700 that includes an IHC - stained tissue sample for scoring by the HER2 scoring system 120. In block 652, each of the ROIs within the WSI is analyzed by a first machine learning to identify a plurality of segments, and each of the plurality of segments includes a nucleus or a membrane within one or more of the ROIs.

[0101] In block 654, a plurality of features are extracted from a plurality of segments output by a first machine learning model. In block 656, the plurality of features extracted from the plurality of segments are classified into one of a plurality of feature categories.

[0102] In block 658, a predicted HER2 score 702 is generated using a second machine learning model based on the classification assigned to the plurality of features extracted in block 654.

[0103] In block 660, a display of the predicted HER2 score 702 is provided for reference by a user in determining a final HER2 score 704 and a HER2 status diagnosis 706 for an IHC stained tissue sample.

[0104] FIG. 7 shows an exemplary processing system 700 according to a particular embodiment that can execute the processes and methods described herein, such as the process for IHC HER2 scoring with respect to FIG. 2 and the method for HER2 score classification with respect to FIGS. 6A and 6B.

[0105] The processing system 700 includes a central processing unit (CPU) 706 connected to a data bus 736. The CPU 706 is configured to process computer-executable instructions stored, for example, in the memory 712, and to cause a server to execute, for example, the methods described herein with respect to FIG. 2. The CPU 706 is included to represent a single CPU, multiple CPUs, a single CPU having multiple processing cores, and other forms of processing architectures capable of executing computer-executable instructions. The memory 712 includes one or more memory devices, such as volatile memory, such as RAM, cache, or other short-term memory that may be implemented in hardware or emulated in software, hard drives, solid state drives, or other long-term memory that may be implemented in hardware or emulated in software, or a combination of volatile and non-volatile memory. Moreover, the one or more memory devices that make up the memory 712 may be located remotely from the processing system 700 and accessed via a network.

[0106] The processing system 700 further includes input / output (I / O) devices 708 and an interface 704, which enable the processing system 700 to interface with the input / output devices 708, such as a keyboard, display, mouse device, pen input, and other devices that enable interaction with the processing system 700. Note that the processing system 700 can be connected to external I / O devices (e.g., external display devices) via physical and wireless connections.

[0107] The processing system 700 further includes a network interface 702, which enables the processing system 700 to access an external network 710 and thereby external computing devices.

[0108] The processing system 700 further includes a memory 712, which, in this example, includes a processing component 714, an extraction component 716, a classification component 718, a receiving component 720, a providing component 722, a CNN model training component 724, an ML model training component 726, and a detection component 728, which can be used when performing the operations described in FIGS. 2 and 6. The memory 712, in this example, further includes ML model data 730, CNN model data 732, WSI image data 734, and one or more software applications 738, which can be used when performing the operations described in FIGS. 2 and 6.

[0109] The processing system 700 can include one or more software applications 736 and media data stored by a memory 712 used by the CPU 706 to execute the processes 200 and methods 600 described herein. In some configurations, the CPU 706 includes a digital signal processor (DSP), an application specific integrated circuit (ASIC), and / or a combination of such units. The CPU 706 is configured to execute one or more software applications 736 and process stored media data that can each be included within the memory 712. The processing system 700 controls data and file transfers between the various systems and devices described in FIG. 1. The memory 712 is also configured to store instructions corresponding to any operation of the method 600 according to the embodiments described herein.

[0110] [[ID=]] Although shown as a single memory 712 in FIG. 7 for simplicity, it should be noted that the various aspects stored in the memory 712 may be stored in various physical memories, including memories remote from the processing system 700, all of which are accessible by the CPU 706 via an internal data connection such as a bus 736.

[0111] Embodiments of the present disclosure may be provided to an end user via a cloud computing infrastructure. Cloud computing refers to the provision of scalable computing resources as a service over a network. More formally, cloud computing can be defined as a computing capability that provides an abstraction between computing resources and the underlying technical architecture (e.g., servers, storage, network), enabling convenient on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal administrative effort or service provider interaction. Thus, cloud computing enables a user to access virtual computing resources (e.g., storage, data, applications, even complete virtualized computing systems, etc.) within the "cloud" regardless of the physical systems (or their locations) underlying the computing resources being provided.

[0112] Generally, cloud computing resources are provided to users on a pay-per-use basis, and users are billed only for the computing resources actually used (e.g., the amount of storage space consumed by the user or the number of virtualized systems instantiated by the user). The user can access any of the resources present in the cloud from anywhere on the Internet at any time. In the context of the present disclosure, the user can access software routines available within the cloud (e.g., one or more software applications 738 corresponding to the HER2 scoring system 120 for executing processes 200 and methods 600A and 600B) or related data. For example, the software routines can be executed on a computing system within the cloud. In such a case, the software routines can maintain spatial and non-spatial data in a storage location within the cloud. By doing so, the user can access this information from any computing system attached to a network (e.g., the Internet) connected to the cloud.

[0113] Figures 8A - 8C illustrate an exemplary user interface visualization output 140 according to a particular embodiment. When one or more regional HER2 score predictions 108 are obtained for each of one or more ROIs within a whole slide image of an IHC stained tissue sample, the HER2 scoring system 120 outputs the one or more regional HER2 score predictions 108 and the corresponding analyzed whole slide image to a result visualization module 130. The result visualization module 130 then further processes the corresponding algorithm results associated with the analyzed whole slide image as well as the WSI detection data 104 and the WSI extraction data 106 for output to the user interface visualization output 140. Specifically, in one embodiment, due to the size of the whole slide image, the result visualization module 130 can process and prepare a portion of the whole slide image at a time for viewing in the user interface visualization output 140. The user interface visualization output 140 enables a user to navigate and change the viewing field of the user interface 140 to examine a particular portion or ROI of the WSI analyzed by the HER2 scoring system 120 in order to confirm and reference the corresponding regional HER2 score predictions 108 generated by the HER2 scoring system 120.

[0114] In the user interface visualization output 140, the prediction summary 112 is also prepared using, for each of the respective ROIs, the one or more regional HER2 score predictions 108 output for each, the details for each of the one or more ROIs specified in the WSI and analyzed by the HER2 scoring system 120, the support - related data on which each of the one or more regional HER2 score predictions 108 for each of the analyzed ROIs is based, and the overall HER2 score prediction for the tissue sample within the WSI when two or more regions of interest within the WSI are scored. FIG. 8A shows an example of a user interface visualization output 140 having a prediction summary 112 with an image of the original WSI analyzed by the HER2 scoring system.

[0115] In one embodiment, the prediction summary 112 includes information regarding each of one or more ROIs specified in the WSI corresponding to the percentage of nuclei positive. The percentage of nuclei positive refers to the percentage of nuclei in the ROI of the tissue sample captured in the WSI determined to be HER2 positive. The percentage of nuclei positive for each of the one or more ROIs within the WSI can be determined by dividing the number of nuclei detected in each respective ROI by the total number of nuclei detected within the WSI.

[0116] In some embodiments, the information related to the percentage of nuclei positive for one or more ROIs can be separately based on the membrane integrity detected in the ROI of the WSI. For example, the percentage of nuclei positive in the ROI can be based on the membrane integrity that can be divided into four categories including when a complete membrane is detected, when no membrane is detected, when both a complete membrane and an incomplete membrane are detected, and when only an incomplete membrane is detected. Calculating the percentage of nuclei positive for each of these categories can include magnifying the nuclei in the image to determine the total number of nuclei detected within the WSI and using the magnified image as a mask to quantify the membrane instances within the WSI. Next, the quantified membrane instances are used to calculate the pixels within each membrane instance regarding each instance of the nuclei, thereby determining the ratio of the membrane to the nuclei for each instance and binning the pixel / ratio data for each membrane into the four above-mentioned membrane integrity categories to determine the percentage of nuclei positive based on each.

[0117] The user interface visualization output 140 further includes an option to overlay the region mask 114 and the heat mask 116 on the WSI image shown in FIG. 8A. The region mask 114 indicates each of the ROIs specified and analyzed by the HER2 scoring system 120 to generate one or more regional HER2 score predictions 108. The heat mask 116 indicates each of the nuclei and membranes detected by the CNN model 128 for each respective ROI and utilized by the HER2 scoring system 120 to generate HER2 score predictions. FIG. 8B shows an example of a region mask 114 with each of the ROIs specified by the user overlaid on the WSI for reference. In one aspect, the user can alternatively change the ROI symbol display within the WSI by the user interface visualization output 140 and generate a new HER2 score prediction based on the new ROI symbol display.

[0118] FIG. 8C shows an example of a heat mask 116 generated and overlaid on each of the respective ROIs within the WSI specified in FIG. 8B. The heat mask 116 indicates each of the detected nuclei and membranes with different colors or markings for comparison at each of the specified ROIs. An image of the original WSI of the stained tissue sample may also be output to the user interface 140 for visual cross-reference with the region mask 114 and / or the heat mask 116. Considering that ROIs that do not meet the 10% ASCO / CAP guidelines are ignored by the HER2 scoring system 120 for the purpose of generating HER2 scoring predictions, the heat mask 116 enables the user to further reexamine any ROI that may have been ignored to confirm that the treatment by the HER2 scoring system 120 was appropriate. In one aspect, the heat mask 116 that identifies and indicates the nuclei and membranes detected by the HER2 scoring system 120 at each respective ROI can assist the user in confirming whether the ignored ROI indeed did not meet the 10% ASCO / CAP guidelines.

[0119] In another aspect of the embodiment, the overlay view of the heat mask 116 on the WSI still enables the user to examine the original colors and stains of the various underlying microscopic components of the tissue sample captured in the WSI, and also enables tracking of the nuclei and membranes in each of the ROIs within the WSI. This further enables the user to confirm whether the nuclei and membranes detected within the WSI by the HER2 scoring system 120 are accurate.

[0120] The prediction summary 112 can include information regarding the number of nuclei and membranes detected in each of the ROIs within the WSI, as well as the respective regional HER2 score predictions 108 for each of the ROIs. Additional information regarding the extracted features of the membranes and nuclei detected in each of the ROIs used to classify one or more of the regional HER2 score predictions 108 can be provided. An overall HER2 score prediction for the entire WSI can further be provided based on one or more of the regional HER2 score predictions 108 output by the HER2 scoring system 120. The number of nuclei and membranes detected can be used to ensure that each of the regional HER2 score predictions 108 meets the 10% requirement in the ASCO / CAP guidelines for IHC HER2 scoring of breast cancer tissue cells.

[0121] In another aspect, after the prediction summary 112 of the WSI is output by the HER2 prediction system 126, the user can optionally adjust the ROIs analyzed by the HER2 prediction system 126 (perhaps in response to an error found by the user when specifying one or more ROIs within the WSI, or in another way) and generate a new prediction summary 112 for the adjusted ROIs. In another embodiment, the user can also alternatively select a specific ROI within the WSI and generate a specific corresponding HER2 score prediction.

[0122] Once the prediction summary 112 is provided to the user interface and visualizer 140, the prediction summary 112 can be used by a user, such as a physician or pathologist, in determining the final HER2 score and status diagnosis for the stained tissue sample captured within the WSI. The prediction summary 112 can also be exported in a portable format or shared by a hospital management system to communicate the prediction summary 112 to a third party for consideration in determining the final HER2 score and status diagnosis for the stained tissue sample captured in the WSI.

[0123] The foregoing is directed to embodiments of the present disclosure, but other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, and the scope of the present disclosure is determined by the following claims.

Claims

1. A method for generating a predicted HER2 score, comprising: designating one or more regions of interest in a patient image comprising an IHC-stained tissue sample; identifying a plurality of segments in the patient image using a first machine learning model, each of the plurality of segments comprising a nucleus or membrane within the one or more regions of interest; extracting a plurality of features from the plurality of segments in the patient image based on the nuclei or membranes within the plurality of segments; classifying each of the plurality of features into one of a plurality of feature categories; generating a predicted HER2 score using a second machine learning model based on the classification of the plurality of features associated with the plurality of segments; providing a display of the predicted HER2 score on the patient image for use in determining a final HER2 score and HER2 status diagnosis for the IHC-stained tissue sample A method comprising.

2. The method of claim 1, wherein designating one or more regions of interest in the patient image comprises designating one or more regions of interest using a third machine learning model.

3. The method of claim 1, wherein each of the one or more regions of interest comprises a homogeneous and contiguous population of cancer cells captured in the patient image.

4. The method of claim 1, wherein the first machine learning model comprises a deep learning neural network machine learning model.

5. The method of claim 1, wherein the second machine learning model comprises one or more of a random forest machine learning model, a support vector machine, a decision tree, a convolutional neural network, or any other machine learning model capable of learning and solving classification problems.

6. The method of claim 1, further comprising partitioning each of the one or more regions of interest in the patient image into a plurality of partitions, each of the plurality of segments identified by the first machine learning model comprising a respective partition of the patient image.

7. The method of claim 1, wherein the plurality of features comprises features corresponding to one or more of the following: intensity of membrane staining, completeness of membrane staining, basal color of the nucleus, basal color of the membrane, ratio of membrane to nucleus, membrane staining deviation, nuclear staining deviation, area of completely stained membrane, percentage of membrane-stained cells, and DAB average color. Claim 8 The method according to claim 1, wherein the first machine learning model uses shared training weights to identify nuclei and membranes in each of the plurality of segments in the patient image. Claim 9 The method according to claim 1, further comprising generating the predicted HER2 score for each of the one or more regions of interest using the second machine learning model, wherein generating the predicted HER2 score is based on the ASCO / CAP guidelines for HER2 scoring of IHC-stained breast cancer tissue cells. Claim 10 The method according to claim 7, further comprising generating a heatmap that identifies the one or more regions of interest and / or nuclei and membranes in each of the one or more regions of interest in the patient image based on the plurality of segments identified by the first machine learning model. Claim 11 A method for training a predicted HER2 tissue scoring model, comprising: receiving a first training dataset including a first plurality of images including stained tissue samples; training a first machine learning model to classify segments of an input image as nuclei or membranes based on the first training dataset, wherein the first plurality of images in the first training dataset are labeled to identify membranes and nuclei in the plurality of images; training a second machine learning model to generate a predicted HER2 score for the input image based on a second training dataset, wherein the second training dataset includes a second plurality of images labeled to identify membranes and / or nuclei in the plurality of images, and the second plurality of images are classified into a plurality of categories corresponding to a plurality of features related to membranes and nuclei in the second plurality of images; and the method includes. Claim 12 The method according to claim 11, wherein the first machine learning model includes a deep learning neural network machine learning model. Claim 13 The method according to claim 11, wherein the plurality of features related to membranes and nuclei in the second plurality of images include one or more of the following features: intensity of membrane staining, integrity of membrane staining, base color of nucleus, base color of membrane, ratio of membrane to nucleus, membrane staining deviation, nucleus staining deviation, area of completely stained membrane, and percentage of stained membrane cells. Claim 14 The method according to claim 11, wherein the second machine learning model includes one or more of a random forest machine learning model, a support vector machine, a decision tree, a convolutional neural network, or any other machine learning model capable of learning and solving classification problems.

15. Training the first machine learning model to classify segments of the input image as nuclei or membranes includes training the first machine learning model to classify both nuclear segments and membrane segments using shared training weights, the method according to claim 11.

16. Training the second machine learning model to generate a predicted HER2 score for the input image based on the second training dataset further includes training the second machine learning model based on the plurality of features related to the ASCO / CAP guidelines for HER2 scoring of IHC stained breast cancer tissue cells, the method according to claim 11.

17. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause a computer system to identifying, using a first machine learning model, a plurality of segments within a patient image including an IHC stained cancer tissue sample, the patient image further including one or more regions of interest, each of the plurality of segments including a nucleus or a membrane within the one or more regions of interest; extracting, from the plurality of segments within the patient image, a plurality of features based on nuclei or membranes within the plurality of segments; classifying each of the plurality of features into one of a plurality of feature categories; generating a predicted HER2 score using a second machine learning model based on the classification of the plurality of features associated with the plurality of segments; and providing a display of the predicted HER2 score on the patient image for use in determining a final HER2 score and a HER2 status diagnosis for the stained tissue sample captured in the patient image A non-transitory computer-readable medium that causes the computer system to perform the steps.

18. The non-transitory computer-readable medium according to claim 17, wherein the first machine learning model includes a deep learning neural network machine learning model.

19. The non-transitory computer-readable medium according to claim 17, wherein the second machine learning model includes one or more of a random forest machine learning model, a support vector machine, a decision tree, a convolutional neural network, or any other machine learning model capable of learning and solving classification problems.

20. The non-transitory computer-readable medium according to claim 17, wherein the plurality of features includes features corresponding to one or more of the intensity of membrane staining, the completeness of membrane staining, the base color of the nucleus, the base color of the membrane, the ratio of the membrane to the nucleus, membrane staining deviation, nuclear staining deviation, the area of the completely stained membrane, the percentage of stained membrane cells, and the average DAB color.

Images