Automatic membrane staining quantification

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a staining score. In one aspect, a method comprises: receiving a microscopic image of a tissue sample that has been stained using a cell staining technique; processing the microscopic image of the tissue sample using a machine learning model, in accordance with values of a set of machine learning model parameters, to generate data identifying a plurality of cells shown in the microscopic image of the tissue sample; processing the microscopic image of the tissue sample to generate, for each of the plurality of cells, a classification of a staining intensity of the cell; and generating a staining score characterizing staining intensity in the microscopic image of the tissue sample based on the classifications of the staining intensities of the plurality of cells shown in the microscopic image of the tissue sample.

Description

AUTOMATIC MEMBRANE STAINING QUANTIFICATIONTECHNICAL FIELD

[0001] This specification relates to generating staining scores characterizing staining intensity in microscopic images of tissue samples using machine learning models.BACKGROUND

[0002] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0003] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to a received input to generate an output.

[0004] Staining of microscopic tissue samples is a method for enhancing the visualization of cellular and molecular structures. This technique involves applying specific chemical dyes to tissue sections, which selectively color different cellular components, thereby improving contrast and detail under microscopic examination. Staining methods include Hematoxylin and Eosin (H&E), which differentiates cell nuclei and cytoplasm, and the Gram stain, used for bacterial classification in microbiology. Specialized stains, such as silver staining for proteins and immunohistochemistry for specific proteins, are also employed for detailed molecular analysis.SUMMARY

[0005] This specification describes a system implemented as computer programs on one or more computers in one or more locations that can generate a staining score characterizing staining intensity in a microscopic image of a tissue sample.

[0006] According to a first aspect there is provided a method performed by one or more computers, comprising: receiving a microscopic image of a tissue sample that has been stained using a cell staining technique; processing the microscopic image of the tissue sample using a machine learning model, in accordance with values of a set of machine learning model parameters, to generate data identifying a plurality of cells shown in the microscopic image of the tissue sample; processing the microscopic image of the tissue sample to generate, for each of the plurality of cells, a classification of a staining intensity of the cell; and generating a staining score characterizingstaining intensity in the microscopic image of the tissue sample based on the classifications of the staining intensities of the plurality of cells shown in the microscopic image of the tissue sample.

[0007] In some implementations, the data identifying the plurality of cells shown in the microscopic image of the tissue sample comprises: a respective segmentation of each of the plurality of cells.

[0008] In some implementations, the data identifying the plurality of cells shown in the microscopic image of the tissue sample comprises: a respective segmentation of a nucleus of each of the plurality of cells.

[0009] In some implementations, for each of the plurality of cells, generating the classification of the staining intensity of the cell comprises: identifying a region of interest in the cell; generating a respective classification of a staining intensity of each of a plurality of pixels included in the region of interest in the cell; and generating the classification of the staining intensity of the cell based on the classifications of the staining intensities of the plurality of pixels included in the region of interest of the cell.

[0010] In some implementations, for each of the plurality of cells, the region of interest of the cell comprises a cell membrane of the cell.

[0011] In some implementations, for each of the plurality of cells, identifying the region of interest in the cell comprises: obtaining a segmentation of the cell in the microscopic image of the tissue sample; and identifying the cell membrane of the cell by applying a morphological operation to the segmentation of the cell.

[0012] In some implementations, identifying the cell membrane of the cell by applying the morphological operation to the segmentation of the cell comprises: identifying each pixel that is: (i) included in the cell, and (ii) located within a threshold distance from a boundary of the cell, as being included in the cell membrane of the cell.

[0013] In some implementations, for each of the plurality of cells, identifying the region of interest in the cell comprises: obtaining a segmentation of a nucleus of the cell in the microscopic image of the tissue sample; and identifying the cell membrane of the cell by applying a morphological operation to the segmentation of the nucleus of the cell.

[0014] In some implementations, identifying the cell membrane of the cell by applying the morphological operation to the segmentation of the nucleus of the cell comprises: identifying each pixel that is: (i) outside the nucleus of the cell, and (ii) located within a threshold distance from the nucleus of the cell, as being included in the membrane of the cell.

[0015] In some implementations, for each of the plurality of cells, generating the respective classification of the staining intensity of each of the plurality of pixels included in the region ofinterest of the cell comprises, for each of the plurality of pixels: classifying the staining intensity of the pixel into a predefined set of possible staining intensities based on a comparison of a statistic derived from data defining a color of the pixel to one or more classification thresholds.

[0016] In some implementations, the set of possible staining intensities comprises: a weak staining intensity, a medium staining intensity, and a strong staining intensity.

[0017] In some implementations, for one or more of the plurality of pixels, classifying the staining intensity of the pixel based on the comparison of the statistic derived from data defining the color of the pixel to one or more classification thresholds comprises: classifying the staining intensity of the pixel as a weak staining intensity based on the statistic derived from data defining the color of the pixel being below a first classification threshold.

[0018] In some implementations, for one or more of the plurality of pixels, classifying the staining intensity of the pixel based on the comparison of the statistic derived from data defining the color of the pixel to one or more classification thresholds comprises: classifying the staining intensity of the pixel as a medium staining intensity based on the statistic derived from data defining the color of the pixel being between a first classification threshold and a second classification threshold.

[0019] In some implementations, for one or more of the plurality of pixels, classifying the staining intensity of the pixel based on the comparison of the statistic derived from data defining the color of the pixel to one or more classification thresholds comprises: classifying the staining intensity of the pixel as being a strong staining intensity based on the statistic derived from data defining the color of the pixel being above a second classification threshold.

[0020] In some implementations, the one or more classification thresholds are determined by performing operations comprising: obtaining a set of calibration images, wherein each calibration image is a microscopic image of a calibration tissue sample that has been stained using a cell staining technique; obtaining, for each of the calibration images, a respective target staining score for the calibration image; and iteratively adjusting the one or more classification thresholds to minimize, for each calibration image, a discrepancy between: (i) a predicted staining score generated for the calibration image based on the one or more classification thresholds, and (ii) the target staining score for the calibration image.

[0021] In some implementations, generating the classification of the staining intensity of the cell based on the classifications of the staining intensities of the plurality of pixels included in the region of interest of the cell comprises: determining, for each staining intensity in a predefined set of possible staining intensities, a number of pixels included in the region of interest of the cell that are classified as having the staining intensity; and classifying the staining intensity of the cellas a staining intensity associated with a highest number of pixels included in the region of interest of the cell.

[0022] In some implementations, generating the staining score characterizing staining intensity in the microscopic image of the tissue sample based on the classifications of the staining intensities of the plurality of cells shown in the microscopic image of the tissue sample comprises: determining, for each staining intensity in a predefined set of possible staining intensities, a fraction of the plurality of cells that are classified as having the staining intensity; and determining the staining score characterizing the staining intensity in the microscopic image of the tissue sample as a linear combination of, for each staining intensity in the set of possible staining intensities, the fraction of the plurality of cells that are classified as having the staining intensity.

[0023] In some implementations, the values of the set of machine learning model parameters are trained values that have been determined by a machine learning training technique.

[0024] In some implementations, determining the trained values of the set of machine learning model parameters by the machine learning training technique comprises: training the machine learning model on a set of training examples, wherein each training example comprises: (i) a training microscopic image of a training tissue sample that has been stained using a cell staining technique, and (ii) target data identifying a plurality of cells shown in the training microscopic image.

[0025] In some implementations, training the machine learning model on the set of training examples comprises, for each of the plurality of training examples: training the machine learning model to reduce a discrepancy between: (i) predicted data generated by the machine learning model by processing the training microscopic image of the training example, and (ii) the target data specified by the training example.

[0026] In some implementations, the machine learning model comprises a neural network.

[0027] In some implementations, the method further comprises generating a visualization of the classifications of the staining intensities of the plurality of cells.

[0028] In some implementations, the method further comprises providing data defining the visualization for display on a user device.

[0029] In some implementations, the microscopic image of the tissue sample comprises: an immunohistochemistry (IHC) image, or a hematoxylin and eosin stained image, or a fluorescent stained image, Giemsa stained image, or a Wright-Giemsa stained image, or a Nissl stained image.

[0030] In some implementations, the microscopic image of the tissue sample comprises an IHC image and the staining score comprises a predicted H-score.

[0031] In some implementations, the method further comprises generating, based on the staining score characterizing staining intensity in the microscopic image of the tissue sample, a recommendation for a treatment to be applied to a subject from whom the tissue sample was obtained.

[0032] In some implementations, the method further comprises generating, based on the staining score characterizing staining intensity in the microscopic image of the tissue sample, a predicted diagnosis of a medical condition in a subject from whom the tissue sample was obtained.

[0033] In some implementations, the method further comprises generating, based on the staining sore characterizing the staining intensity in the microscopic image of the tissue sample, a predicted prognosis for a medical condition in a subject from whom the tissue sample was obtained.

[0034] In some implementations, the method further comprises applying a treatment to a subject from whom the tissue sample was obtained based at least in part on the staining score characterizing staining intensity in the microscopic image of the tissue sample.

[0035] In some implementations, wherein the microscopic image of the tissue sample has been generated using a virtual staining technique.

[0036] In some implementations, the microscopic image of the tissue sample comprises at least one million pixels.

[0037] In some implementations, generating data identifying the plurality of cells shown in the microscopic image of the tissue sample comprises: generating data identifying at least one thousand cells shown in the microscopic image of the tissue sample.

[0038] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0039] The score generation system described in this specification leverages machine learning models to measure a staining score for microscopic images efficiently and accurately. The system uses machine learning models to segment cells in an image and approximates cell membranes with morphological operations. The score generation system uses machine learning models to solve the task of cell segmentation. The score generation system then leverages morphological operations that leverage the understanding of cell structure to perform a more fine-grained task of cell membrane segmentation.

[0040] When compared with conventional pathological approaches that involve manual observation, using machine learning models to segment cells exhibits increased reproducibility and is less time consuming. This can improve the analysis of staining scores for microscopic images and provide researchers with an objective assessment of membrane staining in various pathological contexts. The previous manual approaches could take hours to generate an accuratestaining score for a single image. The score generation system described in this specification can process large, high-resolution images with millions of pixels in seconds or minutes in a fully- automated manner.

[0041] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG. l is a block diagram of an example score generation system.

[0043] FIG. 2 is a flow diagram of an example process for generating a staining score.

[0044] FIG. 3 is a flow diagram of an example process for generating a classification of a staining intensity for a cell.

[0045] FIG. 4 is a flow diagram of an example process for generating a classification of a staining intensity for a cell.

[0046] FIG. 5 is a flow diagram of an example process for determining one or more classification thresholds.

[0047] FIG. 6 is a flow diagram of an example process for training a machine learning model.

[0048] FIG. 7 shows an example of identifying a region of interest in a cell.

[0049] FIG. 8 shows another example of identifying a region of interest in a cell.

[0050] FIG. 9 shows an example of a visualization of staining classifications for cells in a microscopic image of a tissue sample.

[0051] FIG. 10 shows another example 900 of a visualization of staining classifications for cells in a microscopic image of a tissue sample.

[0052] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0053] FIG. 1 shows an example score generation system 100. The score generation system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0054] The score generation system includes a machine learning model 112, a classification system 142, and a generator engine 104.

[0055] The machine learning model 112 is configured to process a microscopic image 110 to generate identification data 114. The microscopic image 110 can be an image of a tissue sample that has been stained using a cell staining technique. The tissue sample can be obtained, for example, from a biopsy of a patient. The identification data 114 can identify a plurality of cells shown in the microscopic image of the tissue sample.

[0056] The microscopic image 110 can be, for example, an imunohistochemistry (IHC) image that is obtained by using an antibody to bind a specific antigen in a tissue section, a hematoxylin and eosin stained image, a fluorescent stained image, Giemsa stained image, a Wright-Giesma stained image, Nissl stained image, or any other type of stained microscopic image. The cell staining technique can include using a fluorescent or colored antibodies in either a direct or indirect staining method. The cell staining technique can any technique that can be used to visualize cells and cell components under a microscope.

[0057] In some implementations, the identification data 114 can be a respective segmentation of each of the cells in the microscopic image 110. The segmentation can determine boundaries that separate one cell from another cell. The segmentation can index the cells in the image. In some examples, the segmentation can, for each cell, specify a set of pixels in the image that are included in the cell. In other examples, the segmentation can, for each cell, specify a set of pixels in the image that are included in nucleus of the cell.

[0058] In some implementations, the identification data 114 can be a respective segmentation of a nucleus of each of the cells in the microscopic image 110.

[0059] The machine learning model 112 can have any appropriate machine learning model architecture that enables the machine learning model to perform its described functions. For instance, the machine learning model 112 can be implemented, e.g., as a neural network model, or a random forest model, or a support vector machine model, or a decision tree model, or a linear regression model, etc. In implementations, where the machine learning model 112 is implemented as a neural network model, the machine learning model 112 can include any appropriate types of neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) in any appropriate number (e.g., 5 layers, 10 layers, or 50 layers) and connected in any appropriate configuration (e.g., as a directed graph of layers). In implementations where the machine learning model 112 is implemented as a decision tree model, the machine learning model 112 can include any appropriate number of vertices and can implement any appropriate splitting function at each vertex.

[0060] The machine learning model 112 can include a set of machine learning model parameters. For instance, for a machine learning model 112 implemented as a neural network model, the setof machine learning model parameters can define the weights and biases of the neural network layers of the machine learning model. As another example, for a machine learning model 112 implemented as a decision tree, the set of machine learning model parameters can define parameters of a respective splitting function used at each vertex of the decision tree. To generate a model output, the machine learning model 112 can process a model input in accordance with values of the set of machine learning model parameters.

[0061] The score generation system 100 can use a training system to train the machine learning model 112 on a set of training examples. More specifically, the training system can determine trained values of the set of machine learning model parameters of the machine learning model 112 by a machine learning training technique. Training the machine learning model 112 is described in more detail with reference to FIG. 6.

[0062] The classification system 142 processes the microscopic image 110 of the tissue sample and the identification data 114 to generate a classification 108 of a staining intensity of the cell for each cell in the microscopic image. The classification system 142 can, for each cell identified in the microscopic image, identify a region of interest in the cell. The classification system 142 can generate a respective classification of a staining intensity of each of a plurality of pixels included in the region of interest in the cell. The classification system 142 can generate the classification of the staining intensity of the cell based on the classifications of the staining intensities of the plurality of pixels included in the region of interest of the cell.

[0063] The generator engine 104 processes the classifications 108 of the staining intensities of each cell in the microscopic image 110 to generate a staining score 102 for the microscopic image 110. The staining score 102 characterizes staining intensity in the microscopic image 110 of the tissue sample based on the classifications of the staining intensities of the cells shown in the microscopic image 110 of the tissue sample. When the microscopic image 110 is an H4C image, the staining score 102 can be a predicted H-score.

[0064] The generator engine 104 can determine, for each staining intensity in a predefined set of possible staining intensities, a fraction of cells in the microscopic image 110 that are classified as having the staining intensity. The generator engine 104 can determine the staining score 102 as a linear combination of, for each staining intensity in the set of possible staining intensities, the fraction of the cells in the microscopic image 110 that are classified as having the staining intensity.

[0065] When the staining score 102 is a predicted H-score, the set of possible staining intensities can include a weak staining intensity, a medium staining intensity, and a strong staining intensity. The H-score can be calculated as follows:H-score = 3*(% of cells with strong staining intensity) + 2*(% of cells with medium staining intensity) + 1*(% of cells with weak staining intensity).

[0066] FIG. 2 is a flow diagram of an example process for generating a staining score. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a score generation system, e.g., the score generation system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0067] The system can receive a microscopic image of a tissue sample that has been stained using a cell staining technique (step 202). The system can receive the image in any appropriate way. For instance, the system can receive the image by way of a user interface (e.g., a graphical user interface (GUI)) or an application programming interface (API) made available by the system.

[0068] The microscopic image can be, for example, an imunohistochemistry (IHC) image that is obtained by using an antibody to bind a specific antigen in a tissue section, a hematoxylin and eosin stained image, a fluorescent stained image, Giemsa stained image, a Wright-Giesma stained image, Nissl stained image, or any other type of stained microscopic image. The microscopic image can include a number of pixels e.g., 10,000, 1 million, 3 million etc.

[0069] The cell staining technique can include using a fluorescent or colored antibodies in either a direct or indirect staining method. The cell staining technique can any technique that can be used to visualize cells and cell components under a microscope. In some examples, the microscopic image can be generated using a virtual staining technique e.g., a method that digitally generates histological stains using computational techniques, e.g., machine learning techniques (e.g., deep learning).

[0070] The system can process the microscopic image of the tissue sample using a machine learning model to generate to generate data identifying a plurality of cells shown in the microscopic image of the tissue sample (step 204). The machine learning model can be any appropriate type of machine learning model that is configured to process an image and generate data that identifies a plurality of cells in the image. The machine learning model can be, for example, the machine learning model of StarDist: Cell Detection with Star-Convex Polygons, Uwe Schmidt, Martin Weigert, Coleman Broaddus & Gene Myers, MICCAI 2018, the machine learning model of Cellpose: a generalist algorithm for cellular segmentation, Carsen Stringer, Tim Wang, Michalis Michaelos & Marius Pachitariu, Nature Methods 2021, or any other appropriate machine learning model.

[0071] In some implementations, the data identifying a plurality of cells can be a respective segmentation of each of the cells in the microscopic image. The segmentation can determineboundaries that separate one cell from another cell. In other implementations, the data identifying a plurality of cells can be a respective segmentation of a nucleus of each of the cells in the microscopic image. The identification data can identify a plurality of cells in the microscopic image, e.g., 1000 cells, 2000 cells, 10,000 cells, and so forth.

[0072] The system can process the microscopic image of the tissue sample to generate, for each of the plurality of cells, a classification of a staining intensity of the cell (step 206). The system can classify each cell as being included in a staining category from a predefined set of staining categories. The predefined set of staining categories can include a strong staining intensity, a medium staining intensity, and a weak staining intensity. Generating a classification of a staining intensity of a cell is described in further detail below with reference to FIG. 3.

[0073] The system can generate a staining score characterizing staining intensity in the microscopic image of the tissue sample (step 208). The system can generate the staining score based on the classifications of the staining intensities of the plurality of cells shown in the microscopic image. When the microscopic image is an IHC image, the staining score 102 can be a predicted H-score.

[0074] The system can determine, for each staining intensity in a predefined set of possible staining intensities, a fraction of cells in the microscopic image that are classified as having the staining intensity. The system can determine the staining score as a linear combination of, for each staining intensity in the set of possible staining intensities, the fraction of the cells in the microscopic image that are classified as having the staining intensity.

[0075] In some implementations, the system can also generate a treatment recommendation based on the staining score. The treatment can be applied to a subject from whom the tissue sample was obtained. For example, the system can generate a recommendation for surgery, medications, or both based on the staining score. In some implementations, a healthcare provider can apply the identified treatment to the subject.

[0076] In some implementations, the system can also generate a predicted diagnosis based on the staining score. The treatment can be applied to a subject from whom the tissue sample was obtained. For example, the system can generate a diagnosis of a medical condition (e.g., immunoreactive, cancerous, healthy, etc.) based on the staining score. In some examples, the system can transmit the diagnosis over a data communication network for presentation on a display of a user device.

[0077] In some implementations, the system can also generate a predicted prognosis based on the staining score. The treatment can be applied to a subject from whom the tissue sample wasobtained. For example, the system can generate a prognosis of a medical condition (e.g., will worsen, will improve, etc.) based on the staining score.

[0078] In some implementations the system can generate a visualization of the classifications of the staining intensities of the plurality of cells. The system can provide data defining the visualization for display on a user device.

[0079] FIG. 3 is a flow diagram of an example process for generating a classification of a staining intensity for a cell. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a classification system, e.g., the classification system 142 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0080] The system can generate a classification of a staining intensity of each of a plurality of pixels in a region of a cell (step 302). The region of the cell can be, for example, the cell membrane. For each pixel, the system can classify the staining intensity of the pixel into a predefined set of possible staining intensities. The classification of a pixel as having a particular staining intensity in the predefined set can be based on a comparison of a statistic derived from data defining a color of the pixel to one or more classification thresholds. The statistic derived from data defining a color of the pixel can be any appropriate statistic, e.g., an average of multiple components of a vector (e.g., RGB vector) defining the color of the pixel, or a particular component (e.g., the "G" component) from a vector (e.g., RGB vector) defining the color of the pixel.

[0081] In some implementations, the predefined set can include a weak staining intensity, a medium staining intensity, and a strong staining intensity. The system can classify a pixel as having a weak staining intensity if the statistic derived from data defining a color of the pixel is below a first classification threshold. The system can classify a pixel as having a medium staining intensity if statistic is between the first classification threshold and a second classification threshold. The system can classify a pixel as having a strong staining intensity if the statistic is above the second classification threshold. Determining the classification thresholds is described in further detail below with reference to FIG. 5.

[0082] The system can use the classification of the staining intensity of each of the plurality of pixels in a region of a cell to generate a classification of a staining intensity of the cell (step 304).

[0083] The system can determine, for each staining intensity in the predefined set of possible staining intensities, a number of pixels included in the region of interest of the cell that are classified as having the staining intensity. The system can classify the staining intensity of the cell as a staining intensity associated with the highest number of pixels included in the region of interest of the cell. For example, if a cell includes 100 pixels that are in a region of interest, the system canidentify that 60 have a strong staining intensity, 30 have a medium staining intensity, and 10 have a weak staining intensity. The system can classify the staining intensity of the cell as strong.

[0084] FIG. 4 is a flow diagram of an example process for generating a classification of a staining intensity for a cell. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a classification system, e.g., the classification system 142 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.

[0085] The system can identify a region of interest in the cell (step 402). The region of interest in the cell can be a cell membrane of the cell.

[0086] In some implementations, the system can identify the cell membrane by first obtaining a segmentation of the cell in a microscopic image of a tissue sample. The system can then identify the cell membrane of the cell by applying a morphological operation to the segmentation of the cell. The morphological operation can include identifying each pixel that is included in the cell and located within a threshold distance from a boundary of the cell e.g., within 10 pixels inside an outer boundary of the cell, within 5 pixels outside an inner boundary of a cell, etc. The system can identify pixels that are both within the cell and within the threshold distance of the boundary of the cell as being included in the cell membrane of the cell. An example illustration of the described morphological operation is provided in FIG. 7.

[0087] In some implementations, the system can identify the cell membrane by first obtaining a segmentation of a nucleus of the cell in the microscopic image of the tissue sample. The system can then identify the cell membrane of the cell by applying a morphological operation to the segmentation of the nucleus of the cell. The morphological operation can include identifying each pixel that is included in the cell and located within a threshold distance from the nucleus of the cell e.g., within 20 pixels from the nucleus, within 15 pixels from the nucleus, etc. The system can identify pixels that are within the threshold distance from the nucleus of the cell and outside an inner boundary of the cell as being included in the cell membrane of the cell. An example illustration of the described morphological operation is provided in FIG. 8.

[0088] The system can generate a respective classification of a staining intensity of each of a plurality of pixels included in the region of interest in the cell (step 404).

[0089] In some implementations, the predefined set can include a weak staining intensity, a medium staining intensity, and a strong staining intensity. The system can classify a pixel as having a weak staining intensity if the color of the pixel is below a first classification threshold. The system can classify a pixel as having a medium staining intensity if the color of the pixel is between the first classification threshold and a second classification threshold. The system can classify apixel as having a strong staining intensity if the color of the pixel is above the second classification threshold.

[0090] The system can generate a classification of the staining intensity of the cell based on the classifications of the staining intensities of the plurality of pixels included in the region of interest of the cell (step 406).

[0091] The system can determine, for each staining intensity in the predefined set of possible staining intensities, a number of pixels included in the region of interest of the cell that are classified as having the staining intensity. The system can classify the staining intensity of the cell as a staining intensity associated with the highest number of pixels included in the region of interest of the cell. For example, if a cell boundary includes 100 pixels that are in a region of interest, the system can identify that 60 have a strong staining intensity, 30 have a medium staining intensity, and 10 have a weak staining intensity. The system can classify the staining intensity of the cell as strong.

[0092] FIG. 5 is a flow diagram of an example process for determining one or more classification thresholds. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a classification system, e.g., the classification system 142 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.

[0093] The system can obtain a set of calibration images (step 502). Each calibration image can be a microscopic image of a calibration tissue sample that has been stained using a cell staining technique.

[0094] The system can obtain, for each of the calibration images, a respective target staining score for the calibration image (step 504). The target staining score can be a staining score from a predetermined set of training scores e.g., weak, medium, and strong. The target staining scores can be, for example, manually generated.

[0095] The system can iteratively adjust one or more classification thresholds to minimize a discrepancy for each calibration image (step 506). The discrepancy can measure a difference between a predicted staining score generated for the calibration image based on the one or more classification thresholds and the target staining score for the calibration image. The classification thresholds can be scalar thresholds.

[0096] FIG. 6 is a flow diagram of an example process for training a machine learning model. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a score generation system, e.g., the scoregeneration system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 600.

[0097] The system can receive a set of training examples (step 602). Each training example can include a training microscopic image of a training tissue sample that has been stained using a cell staining technique and target data identifying a plurality of cells shown in the training microscopic image. The target data can be, for example, manually generated.

[0098] The microscopic image can be, for example, an immunohistochemistry (IHC) image that is obtained by using an antibody to bind a specific antigen in a tissue section, a hematoxylin and eosin stained image, a fluorescent stained image, Giemsa stained image, a Wright-Giesma stained image, Nissl stained image, or any other type of stained microscopic image.

[0099] In some implementations, the data identifying a plurality of cells can be a respective segmentation of each of the cells in the microscopic image. The segmentation can determine boundaries that separate one cell from another cell. In other implementations, the data identifying a plurality of cells can be a respective segmentation of a nucleus of each of the cells in the microscopic image.

[0100] The system can train a machine learning model using the training examples to reduce a discrepancy (step 604). The discrepancy can measure a difference between predicted data generated by the machine learning model by processing the training microscopic image of the training example and the target data specified by the training example.

[0101] The machine learning model can have any appropriate machine learning model architecture that enables the machine learning model to perform its described functions. For instance, the machine learning model 112 can be implemented, e.g., as a neural network model, or a random forest model, or a support vector machine model, or a decision tree model, or a linear regression model, etc.

[0102] The machine learning model can include a set of machine learning model parameters. For instance, for a machine learning model implemented as a neural network model, the set of machine learning model parameters can define the weights and biases of the neural network layers of the machine learning model. As another example, for a machine learning model implemented as a decision tree, the set of machine learning model parameters can define parameters of a respective splitting function used at each vertex of the decision tree. To generate a model output, the machine learning model can process a model input in accordance with values of the set of machine learning model parameters.

[0103] The system can return the trained model (step 606). The system can use the trained model to process microscopic images of tissue samples to generate data identifying a plurality of cells.

[0104] FIG. 7 shows an example 700 of identifying a region of interest in a cell using a shrinking morphological operation. In this example, the region of interest is a cell membrane. The cell membrane is identified as a region that is within the cell and within a fixed threshold of an outer boundary 702 of the cell. The threshold can be, for example, within 10 pixels of the outer boundary 702 of the cell. An inner boundary 704 of the cell membrane can be found by shrinking the outer boundary 704 by the fixed threshold.

[0105] FIG. 8 shows an example 800 of identifying a region of interest in a cell using an expanding morphological operation. In this example, the region of interest is a cell membrane. The cell membrane is identified as a region that is within the cell and within a fixed threshold of an inner boundary 804 of the cell. The threshold can be, for example, within 10 pixels of the inner boundary 804 of the cell. An outer boundary 802 of the cell membrane can be found by eroding the inner boundary 802 by the fixed threshold.

[0106] FIG. 9 shows an example 900 of a visualization of staining classifications for cells in a microscopic image of a tissue sample. Each cell in the microscopic image has an assigned staining intensity. For example, the membrane 904 of a cell 902 is dark and has a strong staining intensity classification. As another example, the membrane 908 of a cell 906 is light and has a weak staining intensity classification.

[0107] FIG. 10 shows another example 1000 of a visualization of staining classifications for cells in a microscopic image of a tissue sample. Each cell in the microscopic image has an assigned staining intensity. For example, the membrane 1004 of a cell 1002 has a weak staining intensity classification. As another example, the membrane 1008 of a cell 1006 has a strong staining intensity classification.

[0108] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0109] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computerprograms, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine- readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially -gen erated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0110] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.[OHl] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0112] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0113] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0114] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0115] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0116] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messagesor other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0117] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and computeintensive parts of machine learning training or production, i.e., inference, workloads.

[0118] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework.

[0119] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0120] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0121] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0122] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0123] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0124] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: receiving a microscopic image of a tissue sample that has been stained using a cell staining technique; processing the microscopic image of the tissue sample using a machine learning model, in accordance with values of a set of machine learning model parameters, to generate data identifying a plurality of cells shown in the microscopic image of the tissue sample; processing the microscopic image of the tissue sample to generate, for each of the plurality of cells, a classification of a staining intensity of the cell; and generating a staining score characterizing staining intensity in the microscopic image of the tissue sample based on the classifications of the staining intensities of the plurality of cells shown in the microscopic image of the tissue sample.

2. The method of claim 1, wherein the data identifying the plurality of cells shown in the microscopic image of the tissue sample comprises: a respective segmentation of each of the plurality of cells.

3. The method of claim 1, wherein the data identifying the plurality of cells shown in the microscopic image of the tissue sample comprises: a respective segmentation of a nucleus of each of the plurality of cells.

4. The method of any preceding claim, wherein for each of the plurality of cells, generating the classification of the staining intensity of the cell comprises: identifying a region of interest in the cell; generating a respective classification of a staining intensity of each of a plurality of pixels included in the region of interest in the cell; and generating the classification of the staining intensity of the cell based on the classifications of the staining intensities of the plurality of pixels included in the region of interest of the cell.

5. The method of claim 4, wherein for each of the plurality of cells, the region of interest of the cell comprises a cell membrane of the cell.

6. The method of claim 5, wherein for each of the plurality of cells, identifying the region of interest in the cell comprises: obtaining a segmentation of the cell in the microscopic image of the tissue sample; and identifying the cell membrane of the cell by applying a morphological operation to the segmentation of the cell.

7. The method of claim 6, wherein identifying the cell membrane of the cell by applying the morphological operation to the segmentation of the cell comprises: identifying each pixel that is: (i) included in the cell, and (ii) located within a threshold distance from a boundary of the cell, as being included in the cell membrane of the cell.

8. The method of claim 5, wherein for each of the plurality of cells, identifying the region of interest in the cell comprises: obtaining a segmentation of a nucleus of the cell in the microscopic image of the tissue sample; and identifying the cell membrane of the cell by applying a morphological operation to the segmentation of the nucleus of the cell.

9. The method of claim 8, wherein identifying the cell membrane of the cell by applying the morphological operation to the segmentation of the nucleus of the cell comprises: identifying each pixel that is: (i) outside the nucleus of the cell, and (ii) located within a threshold distance from the nucleus of the cell, as being included in the membrane of the cell.

10. The method of claim 4, wherein for each of the plurality of cells, generating the respective classification of the staining intensity of each of the plurality of pixels included in the region of interest of the cell comprises, for each of the plurality of pixels: classifying the staining intensity of the pixel into a predefined set of possible staining intensities based on a comparison of a statistic derived from data defining a color of the pixel to one or more classification thresholds.

11. The method of claim 10, wherein the set of possible staining intensities comprises: a weak staining intensity, a medium staining intensity, and a strong staining intensity.

12. The method of claim 11, wherein for one or more of the plurality of pixels, classifying the staining intensity of the pixel based on the comparison of the statistic derived from data defining the color of the pixel to one or more classification thresholds comprises: classifying the staining intensity of the pixel as a weak staining intensity based on the statistic derived from data defining the color of the pixel being below a first classification threshold.

13. The method of any one of claims 11-12, wherein for one or more of the plurality of pixels, classifying the staining intensity of the pixel based on the comparison of the statistic derived from data defining the color of the pixel to one or more classification thresholds comprises: classifying the staining intensity of the pixel as a medium staining intensity based on the statistic derived from data defining the color of the pixel being between a first classification threshold and a second classification threshold.

14. The method of any one of claims 11-13, wherein for one or more of the plurality of pixels, classifying the staining intensity of the pixel based on the comparison of the statistic derived from data defining the color of the pixel to one or more classification thresholds comprises: classifying the staining intensity of the pixel as being a strong staining intensity based on the statistic derived from data defining the color of the pixel being above a second classification threshold.

15. The method of any one of claims 10-14, wherein the one or more classification thresholds are determined by performing operations comprising: obtaining a set of calibration images, wherein each calibration image is a microscopic image of a calibration tissue sample that has been stained using a cell staining technique; obtaining, for each of the calibration images, a respective target staining score for the calibration image; and iteratively adjusting the one or more classification thresholds to minimize, for each calibration image, a discrepancy between: (i) a predicted staining score generated for the calibration image based on the one or more classification thresholds, and (ii) the target staining score for the calibration image.

16. The method of any one of claims 4-15, wherein generating the classification of the staining intensity of the cell based on the classifications of the staining intensities of the plurality of pixels included in the region of interest of the cell comprises: determining, for each staining intensity in a predefined set of possible staining intensities, a number of pixels included in the region of interest of the cell that are classified as having the staining intensity; and classifying the staining intensity of the cell as a staining intensity associated with a highest number of pixels included in the region of interest of the cell.

17. The method of any preceding claim, wherein generating the staining score characterizing staining intensity in the microscopic image of the tissue sample based on the classifications of the staining intensities of the plurality of cells shown in the microscopic image of the tissue sample comprises: determining, for each staining intensity in a predefined set of possible staining intensities, a fraction of the plurality of cells that are classified as having the staining intensity; and determining the staining score characterizing the staining intensity in the microscopic image of the tissue sample as a linear combination of, for each staining intensity in the set of possible staining intensities, the fraction of the plurality of cells that are classified as having the staining intensity.

18. The method of any preceding claim, wherein the values of the set of machine learning model parameters having are trained values that have been determined by a machine learning training technique.

19. The method of claim 18, wherein determining the trained values of the set of machine learning model parameters by the machine learning training technique comprises: training the machine learning model on a set of training examples, wherein each training example comprises: (i) a training microscopic image of a training tissue sample that has been stained using a cell staining technique, and (ii) target data identifying a plurality of cells shown in the training microscopic image.

20. The method of claim 19, wherein training the machine learning model on the set of training examples comprises, for each of the plurality of training examples: training the machine learning model to reduce a discrepancy between: (i) predicted datagenerated by the machine learning model by processing the training microscopic image of the training example, and (ii) the target data specified by the training example.

21. The method of any preceding claim, wherein the machine learning model comprises a neural network.

22. The method of any preceding claim, further comprising generating a visualization of the classifications of the staining intensities of the plurality of cells.

23. The method of claim 22, further comprising providing data defining the visualization for display on a user device.

24. The method of any preceding claim, wherein the microscopic image of the tissue sample comprises: an immunohistochemistry (H4C) image, or a hematoxylin and eosin stained image, or a fluorescent stained image, Giemsa stained image, or a Wright-Giemsa stained image, or a Nissl stained image.

25. The method of any preceding claim, wherein the microscopic image of the tissue sample comprises an H4C image and wherein the staining score comprises a predicted H-score.

26. The method of any preceding claim, further comprising generating, based on the staining score characterizing staining intensity in the microscopic image of the tissue sample, a recommendation for a treatment to be applied to a subject from whom the tissue sample was obtained.

27. The method of any preceding claim, further comprising generating, based on the staining score characterizing staining intensity in the microscopic image of the tissue sample, a predicted diagnosis of a medical condition in a subject from whom the tissue sample was obtained.

28. The method of any preceding claim, further comprising generating, based on the staining sore characterizing the staining intensity in the microscopic image of the tissue sample, a predicted prognosis for a medical condition in a subject from whom the tissue sample was obtained.

29. The method of any preceding claim, further comprising applying a treatment to a subject from whom the tissue sample was obtained based at least in part on the staining score characterizing staining intensity in the microscopic image of the tissue sample.

30. The method of any preceding claim, wherein the microscopic image of the tissue sample has been generated using a virtual staining technique.

31. The method of any preceding claim, wherein the microscopic image of the tissue sample comprises at least one million pixels.

32. The method of any preceding claim, wherein generating data identifying the plurality of cells shown in the microscopic image of the tissue sample comprises: generating data identifying at least one thousand cells shown in the microscopic image of the tissue sample.

33. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-28 or 30-32.

34. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-28 or 30-32.

Images