Weakly supervised multi-task learning for cell detection and segmentation

Abstract

The present disclosure relates to techniques for segmenting and detecting cells within image data using transfer learning and a multi-task scheduler. In particular, aspects of the present disclosure relate to: accessing a plurality of images of one or more cells; extracting three labels from the plurality of images, wherein the three labels are extracted using a Voronoi transform, a local clustering, and a repel code application; training, by the multi-task scheduler, a convolutional neural network model based on three loss functions corresponding to the three labels; generating, by the convolutional neural network model, a nucleus probability map and a background probability map for each image of the plurality of images based on the training with the three loss functions; and providing the nucleus probability map and the background probability map.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Application No. 62,914,966, filed October 14, 2019, entitled “WEAKLY SUPERVISED MULTI-TASKLEARNING FOR CELL DETECTION AND SEGMENTATION”, the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure relates to biological image analysis, and more particularly to systems and methods for segmenting and detecting cells using weakly supervised transfer learning and multi-task schedulers. Background Technology

[0004] In the analysis of biological samples such as tissue sections, blood, and cell cultures, the samples are typically stained with one or more combinations of staining agents or assays, and then observed or imaged for further analysis. Various processes can be accomplished by observing stained or assayed biological samples, including diagnosing diseases, assessing treatment response, and developing new antiviral drugs. For example, identifying certain objects or structures, such as lymphocytes, cancer cells, and cancer cell nuclei, in biological images (e.g., histopathological images) is often a prerequisite for grading or diagnosing the disease of patients for whom these biological images are obtained. The presence, extent, size, shape, and other morphological appearance of these objects or structures can be important indicators of the presence or severity of a disease. Furthermore, the number or proportion of specific objects or structures (such as cells or cell nuclei) is diagnostically significant for some disease symptoms, further stimulating the need for accurate identification of specific objects or structures.

[0005] In acquiring a biological image, multiple channels of image data can be exported, such as RGB color channels, each observed channel comprising a mixture of multiple signals. Processing of the image data can include color separation, spectral unmixing, color deconvolution, and the like, which are used to determine the concentration of a particular stain from one or more channels of the observed image data. For image data processed by automated methods, displayed on a display, or for assays observed by an observer, a relationship between the color of the tissue and the color of the stain can be determined to determine a model of biomarker distribution in the stained tissue. The local presence and amount of stain can be indicative of the presence and concentration of the biomarker of interest in the tissue. Immunohistochemistry (IHC) slide staining is a technique that can be used to identify specific proteins (e.g., biomarkers) in cells of a tissue section and is widely used to study different types of cells, such as cancer cells and immune cells in biological tissues. For example, in a colorectal cancer (CRC) image of PMS2 IHC nuclear staining, positively tumor nuclei stained brown in different shapes and sizes (e.g., elongated and highly clustered), positively tumor nuclei weakly stained light brown, and negatively tumor nuclei stained blue can be identified and / or quantified to distinguish patients with Lynch Syndrome (LS) and patients with DNA mismatch repair deficiency (dMMR). SUMMARY

[0006] In various embodiments, a computer-implemented method is provided, the method comprising: accessing, by a data processing system, a plurality of images of one or more cells; extracting, by the data processing system, three labels from the plurality of images, wherein the three labels are extracted using a Voronoi transform, local clustering, and applying a Repel code; training, by a multi-task scheduler of the data processing system, a convolutional neural network model based on three loss functions corresponding to the three labels; generating, by the convolutional neural network model, a nucleus probability map and a background probability map for each image of the plurality of images based on the training with the three loss functions; and providing, by the data processing system, the nucleus probability map and the background probability map.

[0007] In various embodiments, a computer-implemented method is provided that includes accessing, by a data processing system, a plurality of images of one or more cells; extracting, by the data processing system, three labels from the plurality of images, wherein the three labels are extracted using a Voronoi transform, a local clustering, and applying a Repel code; training, by a multi-task scheduler of the data processing system, a convolutional neural network model based on three loss functions corresponding to the three labels, wherein the convolutional neural network model includes a plurality of model parameters; generating, by the convolutional neural network model, a nucleus probability map and a background probability map for each image of the plurality of images based on the training with the three loss functions; comparing, by the data processing system, the nucleus probability map and the background probability map to the three labels; updating, by the data processing system, the plurality of model parameters based on the comparison of the nucleus probability map and the background probability map to the three labels to minimize the three loss functions; and providing, by the data processing system, the updated plurality of model parameters for the trained convolutional neural network model.

[0008] In some embodiments, the plurality of images includes a nucleus point label.

[0009] In some embodiments, the Voronoi transform extracts a ridge line between the one or more cells.

[0010] In some embodiments, the local clustering includes applying a k-means clustering algorithm to locally extract background clusters and nucleus clusters based on RGB channel color features concatenated with a distance transform feature that is a distance of each pixel in each cell or polygon created by the Voronoi transform to the nucleus point label.

[0011] In some embodiments, the convolutional neural network model includes a modified U-Net model.

[0012] In some embodiments, the training includes, for each training iteration, selecting, by the multi-task scheduler, one of the three loss functions; and updating, by the data processing system, one or more weights of the convolutional neural network model based on a gradient of the selected loss function.

[0013] In some embodiments, the loss function is selected by the multi-task scheduler based on: if "i%3=0", selecting a loss function associated with the Voronoi label; if "i%3=1", selecting a loss function associated with the Repel label; and if "i%3=2", selecting a loss function associated with the local cluster label, where "i" is an index of a training iteration.

[0014] In some embodiments, generating and providing the kernel probability map and the background probability map includes applying argmax functions and local maxima functions on the kernel probability map and the background probability map, the segmentation binary mask, and the coordinates of the detected cells.

[0015] In various embodiments, a computer-implemented method is provided, the method comprising: obtaining, by a data processing system, a plurality of images of one or more cells; inputting the plurality of images into a convolutional neural network model constructed using a combination of at least three loss functions corresponding to a Voronoi transform, local clustering, and application of Repel codes; generating, by the convolutional neural network model, a kernel probability map and a background probability map for each image of the plurality of images; and providing, by the data processing system, the kernel probability map and the background probability map.

[0016] In some embodiments, the Voronoi transform extracts ridge lines between the one or more cells.

[0017] In some embodiments, the local clustering includes applying a k-means clustering algorithm to locally extract background clusters and kernel clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or polygon created by the Voronoi transform to a kernel point label.

[0018] In some embodiments, the convolutional neural network model includes a modified U-Net model.

[0019] In some embodiments, the convolutional neural network model includes a plurality of parameters identified using a training data set, the training data set including a plurality of medical images having at least three labels extracted using the Voronoi transform, the local clustering, and the application of Repel codes; and the plurality of model parameters are identified using the training data set based on minimizing the at least three loss functions.

[0020] In some embodiments, using the training data set includes, for each training iteration, selecting, by a multi-task scheduler, one of the three loss functions; and updating, by the data processing system, one or more of the plurality of parameters of the convolutional neural network model based on a gradient of the selected loss function.

[0021] In some embodiments, the loss function is selected by the multi-task scheduler based on: if “i%3=0”, selecting the loss function associated with the Voronoi labels; if “i%3=1”, selecting the loss function associated with the Repel labels; and if “i%3=2”, selecting the loss function associated with the local cluster labels, where “i” is an index of a training iteration.

[0022] In some embodiments, generating and providing the core probability map and the background probability map includes applying an argmax function and a local maxima function on the core probability map and the background probability map, the segmentation binary mask, and the coordinates of the detected cells.

[0023] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform part or all of one or more methods disclosed herein.

[0024] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and that includes instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0025] The terms and expressions employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the application claimed. Thus, it should be understood that although the present application has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed can be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this application as defined by the appended claims BRIEF DESCRIPTION OF DRAWINGS

[0026] The present disclosure is described in connection with the appended drawings:

[0027] FIG. 1 An exemplary computing environment for automated cell segmentation and detection is shown in accordance with various embodiments.

[0028] FIG. 2 An overview of the proposed algorithm is shown in accordance with various embodiments.

[0029] FIG. 3 An exemplary U-Net is shown in accordance with various embodiments.

[0030] FIG. 4 Performance of the proposed local clustering algorithm relative to a global clustering algorithm is shown in accordance with various embodiments.

[0031] FIG. 5Segmentation performance according to various embodiments is shown with sample test images (left), overlay segmentation masks based on the algorithm proposed in [1] without CRF (middle), and overlay segmentation masks based on the current algorithm (right): (a) normal tissue region with elongated cells; (b) intragranular region with highly clustered cells; (c) normal tissue region with unstained cells; and (d) tumor region with weakly stained and strongly stained cells. DETAILED DESCRIPTION

[0032] I. SUMMARY

[0033] The present disclosure describes techniques for automated cell segmentation and detection. More specifically, various embodiments of the present disclosure provide systems and methods for segmenting and detecting cells using transfer learning and a multi-task scheduler.

[0034] Single cell segmentation and detection is a fundamental step for performing pathology analysis in biological images. The number and density of cells, morphological information, and staining quality metrics are examples of information that can be obtained from segmented and detected cells in a biological sample (e.g., a tissue slide) for diagnostic purposes and / or staining quality measurement. Compared to traditional image processing methods such as color separation, spectral unmixing, and color deconvolution, recent studies on cell segmentation based on deep learning methods report better performance. Cell detection and instance segmentation face more challenges compared to tasks such as classification and segmentation of objects that are easy to label (e.g., cars, trees, animals, etc.). For example, the contrast of stained cells to background varies greatly; highly clustered cells have touching or overlapping boundaries, which are difficult to segment individually; and most importantly, obtaining ground truth at pixel level for cell instance segmentation is very labor intensive.

[0035] To address these limitations and issues, the techniques for automated single cell segmentation and detection of the present embodiments include using a weakly supervised multi-task learning algorithm for automated, end-to-end single cell segmentation and detection. One illustrative embodiment of the present disclosure involves a method that includes accessing a plurality of images of one or more cells; extracting three labels from the plurality of images, wherein the three labels are extracted using a Voronoi transform, local clustering, and applying a Repel code; training a convolutional neural network model based on three loss functions corresponding to the three labels by a multi-task scheduler; generating a nucleus probability map and a background probability map for each image in the plurality of images by the convolutional neural network model based on the training with the three loss functions; and providing the nucleus probability map and the background probability map. In certain embodiments, generating and providing the nucleus probability map and the background probability map includes applying an argmax function and a local maxima function on the nucleus probability map and the background probability map, a segmentation binary mask, and coordinates of detected cells.

[0036] II. DEFINITIONS

[0037] As used herein, when an action is "based on" something, it means that the action is based at least in part on at least a portion of that thing.

[0038] As used herein, the terms "substantially," "about," and "approximately" are defined as being to a great extent or nearly completely, as understood by a person of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "about," or "approximately" can be replaced with "within [a certain percentage]," where the percentage includes 0.1%, 1%, 5%, and 10%.

[0039] III. TECHNIQUES FOR AUTOMATED CELL SEGMENTATION AND DETECTION

[0040] Image segmentation is a process of separating similar parts of an image that exhibit similarity in different characteristics such as shape, size, color, etc. Segmentation of cells allows visualization of the size and location of cells in a biological sample and can also provide a basis for analyzing staining uptake by cells. For a long time, the gold standard for cell segmentation has been manual segmentation, which is time-consuming and labor-intensive and thus unsuitable for large studies. A lot of research has been done to try to make the process of cell segmentation fully or partially automated. For example, image segmentation techniques such as thresholding, region growing, fuzzy clustering, use of watershed algorithms, etc., have been used for separation of abnormal cells (e.g., cancer cells) from normal cells (e.g., lymphocytes). Nonetheless, the process of segmentation remains challenging due to the diversity in shape, location, and size of cells.

[0041] The multi-task scheduling technique combines information from pixel-level labels obtained using Voronoi transform, color cluster labeling, and Repel codes, which can help improve accurate cell segmentation and detection.

[0042] Described herein is an end-to-end method incorporating a model that uses a convolutional neural network (CNN) to segment and detect cells from images (e.g., images of stained tissue). The developed model is computationally light and is designed to accommodate variability in staining techniques, extreme imbalance between cancer and normal cells, and heterogeneity of input images. The model improves performance of cell segmentation and detection compared to conventional algorithms without the need for increased annotation effort.

[0043] III. A. EXAMPLE COMPUTING ENVIRONMENT

[0044] FIG. 1An exemplary computing environment 100 for cell segmentation and detection using deep convolutional neural networks is shown in accordance with various embodiments. The computing environment 100 can include a deep convolutional neural network (CNN) system 105 to train and execute CNN models. More specifically, the CNN system 105 can include classifier subsystems 110a-110n that can train their respective CNN models. In some embodiments, each CNN model corresponding to the classifier subsystems 110a-110n is individually trained based on one or more images in a set of input image elements 115a-115n (e.g., images from a field of view (FOV) on a slide of a biological sample). In some embodiments, each input image element in the set of input image elements 115a-115n can include one or more digital images depicting cells within a biological sample. Each input image element in the set of input image elements 115a-115n can correspond to a single subject and a day of collecting underlying image data corresponding to the images. The set of input image elements 115a-115n can include one or more training input image elements 115a-115d, validation input image elements 115e-115g, and unlabeled input image elements 115h-115n. It should be appreciated that the input image elements corresponding to the training, validation, and unlabeled groups need not be accessed simultaneously. For example, initial training and validation input image elements can be accessed first and used to train a model, and subsequently, unlabeled input image elements can be accessed or received (e.g., at a single or multiple subsequent times) to test the model.

[0045] In some cases, the CNN models can be trained using weak supervision, and each of the training input image elements 115a-115d and validation input image elements 115e-115g can be associated with a label of one or more “correct” interpretations of the cells (e.g., the center of the nucleus of the cell). The CNN models can be trained using the training input image elements 115a-115d (and validation input image elements 115e-115h to monitor training progress), one or more loss functions, and / or a gradient descent method.

[0046] In some embodiments, the classifier subsystem 110a-110n includes a feature extractor 120, a parameter data store 125, a classifier 130, and a trainer 135 that collectively function to train a CNN model based on training data (e.g., training input image elements 115a-115d) and optimize parameters of the CNN model during weakly supervised training. In some embodiments, the classifier subsystem 110a-110n accesses training data from the training input image elements 115a-115d at an input layer. The feature extractor 120 can pre-process the training data to extract relevant features (e.g., edges) of the training input image elements 115a-115d. In various embodiments, the feature extractor 120 is implemented using a multi-task scheduler to apply at least three transformations and / or encodings on the cell spot labels to extract features or labels of the cells. The at least three transformations and / or encodings can include a Voronoi transformation to extract ridgelines, local clustering of pixel-level labels to extract nuclei, and application of a Repel code as an enhanced center encoding. The classifier 130 can receive the extracted features and transform the features into output metrics of one or more segmentations and detections of one or more cells according to weights associated with a set of hidden layers in a CNN model or models. The trainer 135 can train the feature extractor 120 and / or the classifier 130 by facilitating learning of one or more parameters using training data corresponding to the training input image elements 115a-115d. For example, the trainer 135 can use a backpropagation technique to facilitate learning of weights associated with the set of hidden layers of a CNN model used by the classifier 130. Backpropagation can use, for example, a stochastic gradient descent (SGD) algorithm to accumulate updates to parameters of the hidden layers using one or more of losses from Voronoi features, Repel features, and color cluster features. Learned parameters can include, for example, weights, biases, and / or other hidden layer related parameters, which can be stored in the parameter data store 125.

[0047] A trained CNN model or a collection of trained CNN models (“CNN collection”) can be deployed to process unlabeled input image elements 115h-115n to segment and detect one or more cells. More specifically, the trained version of the feature extractor 120 can generate a feature representation of the unlabeled input image elements, which can then be processed by the trained version of the classifier 130. In some embodiments, image features can be extracted from the unlabeled input image elements 115h-115n based on one or more convolutional blocks, convolutional layers, residual blocks, or pyramidal layers of the CNN model in the classifier subsystem 110a-110n that are optimally used for the dilation of the CNN model. These features can be organized in a feature representation, such as a feature vector of the image. The CNN model can be trained to learn feature types based on subsequent adjustment of parameters in the classification and hidden layers, including fully connected layers of the CNN model. In some embodiments, the image features extracted by the convolutional blocks, convolutional layers, residual blocks, or pyramidal layers include feature maps that are matrices of values representing one or more portions of the image on which one or more image processing operations have been performed (e.g., edge detection, sharpening image resolution). These feature maps can be flattened for processing by the fully connected layers of the CNN model, which output segmentation masks corresponding to predictions related to one or more cells.

[0048] For example, an input image element can be fed to an input layer of a CNN model. The input layer can include nodes corresponding to particular pixels. A first hidden layer can include a set of hidden nodes, each of which is connected to a plurality of input layer nodes. Nodes in subsequent hidden layers can similarly be configured to receive information corresponding to a plurality of pixels or voxels. Thus, the hidden layers can be configured to learn to detect features that span multiple pixels. Each of the one or more hidden layers can include a convolutional block, a convolutional layer, a residual block, or a pyramidal layer. The CNN model can further include one or more fully connected layers (e.g., softmax layers).

[0049] At least a portion of the training input image elements 115a-115d, validation input image elements 115e-115g, and / or unlabeled input image elements 115h-115n can include data collected and received using one or more imaging systems 160, or can have been derived from such data. The imaging systems 160 can include systems configured to collect image data (e.g., FOV images of a slide). The imaging systems 160 can include microscopes. The microscopes can be configured to take images of tissue and / or slides that have been stained on a medium such as a glass slide. In some instances, the cell spot labels associated with the training input image elements 115a-115d and / or the validation input image elements 115e-115g can have been received or can be derived from data received from one or more vendor systems 170, each of which can be associated with, for example, a physician, a nurse, a hospital, a pharmacist, etc. associated with a particular subject. The received data can include, for example, one or more medical records corresponding to the particular subject. The medical records can indicate, for example, a diagnosis or a characteristic description by a professional that indicates whether the subject has a tumor and / or a stage of progression of the subject’s tumor, relative to a time period corresponding to a time at which one or more input image elements associated with the subject were collected or a subsequently defined time period. The received data can further include pixels of locations of centers of cells or nuclei within the one or more input image elements associated with the subject. Thus, the medical records can include one or more cell spot labels or can be used to identify such cell spot labels relative to each training / validation input image element. The medical records can further indicate each of one or more treatments (e.g., drug treatments) that the subject has received and a time period in which the subject received the treatment. In some cases, the images input to the one or more classifier subsystems are received from the vendor systems 170. For example, the vendor systems 170 can receive the images from the imaging systems 160 and can then transmit the images (e.g., along with a subject identifier and one or more labels) to the CNN system 105.

[0050] In some embodiments, data received or collected at one or more of the imaging systems 160 can be aggregated with data received or collected at one or more of the vendor systems 170. For example, the CNN system 105 can identify a corresponding or same identifier for a subject and / or time period in order to associate image data received from the imaging systems 160 with cell spot labeling data received from the vendor systems 170. The CNN system 105 can further use metadata or automated image analysis to process the data to determine which classifier subsystem to feed particular data components to. For example, image data received from the imaging systems 160 can correspond to multiple slides and / or assays. For each image, metadata, automated alignment, and / or image processing can indicate which slide and / or assay the image corresponds to.

[0051] In some embodiments, the computing environment 100 can further include a user device 180 that can be associated with a user that requests and / or coordinates performance of one or more iterations of the CNN system 105 (e.g., where each iteration corresponds to a run of the model and / or a production of output of the model). The user can correspond to a physician, investigator (e.g., associated with a clinical trial), subject, medical professional, etc. Thus, it should be appreciated that in some cases, the vendor systems 170 can include and / or act as the user device 180. Each iteration can be associated with a particular subject (e.g., human) that can (but need not) not be the user. The request for an iteration can include and / or be accompanied by information about the particular subject (e.g., a name or other identifier of the subject, such as a de-identified patient identifier). The request for an iteration can include an identifier of one or more other systems from which data is collected, e.g., input image data corresponding to the subject. In some cases, a communication from the user device 180 includes an identifier for each subject in a particular set of subjects that corresponds to a request to perform an iteration for each subject represented in the set.

[0052] Upon receiving the request, the CNN system 105 can transmit a request for unlabeled input image elements (e.g., including an identifier of the subject) to one or more corresponding imaging systems 160 and / or vendor systems 170. The trained CNN model or set of CNNs can then process the unlabeled input image elements to segment and detect one or more cells. The results for each identified subject can include or be based on cell segmentation and detection from one or more trained CNN models deployed by the classifier subsystems 110a-110n. For example, the cell segmentation and detection can include or be based on outputs generated by the fully connected layers of one or more trained CNNs. In some cases, such outputs can be further processed using, for example, a softmax function. Further, the outputs and / or further processed outputs can then be aggregated using an aggregation technique (e.g., random forest aggregation) to generate one or more subject-specific metrics. The one or more results (e.g., including assay-specific outputs and / or one or more subject-specific outputs and / or processed versions thereof) can be transmitted to and / or provided to the user device 180. In some cases, some or all of the communication between the CNN system 105 and the user device 180 occurs via a network and interface such as a website. It will be appreciated that the CNN system 105 can gate access to results, data, and / or processing resources based on authorization analysis.

[0053] While not explicitly shown, it will be appreciated that the computing environment 100 can further include a developer device associated with a developer. Communications from the developer device can indicate what type of input image elements to use for each CNN model in the CNN system 105, the number of neural networks to use, the configuration of each neural network (including the number of hidden layers and hyperparameters), and how to format data requests and / or which training data to use (e.g., and how to access the training data).

[0054] III. B. MULTITASK SCHEDULER

[0055] FIG. 2 An overview of the prediction algorithm is shown, including pixel-level label extraction based on cell spot labels and a multi-task deep learning approach using a loss scheduler and based on a U-Net model with a ResNet encoder. In some embodiments, the input image 205 is from an image source (e.g., an imaging system 160 or a vendor system 170, such as with respect to FIG. 1An image can be constructed as one or more arrays or matrices of pixel values. A given pixel location can be associated with, for example, a general intensity value and / or an intensity value as it relates to each of one or more gray levels and / or colors (e.g., RGB values). The input image 205 can be labeled with cell point labels 210 that identify one or more pixels at the center of a cell or at the center of a nucleus of a cell.

[0056] In various embodiments, three transformations / encodings are applied on the cell point labels 210: (i) a Voronoi transformation 215, (ii) a local clustering 220, and (iii) a Repel code 225. The Voronoi transformation 215 extracts ridgelines between cells and forms polygons around each cell. These lines help highly clustered cells not to be merged together. The local clustering 220 extracts pixel-level labels of nuclei based on the point labels, the input image 205, and the Voronoi labels. For each extracted Voronoi cell (polygon), a k-means clustering algorithm is used to locally extract background and nucleus clusters based on RGB channel color features concatenated with a distance transform feature that is the distance of each pixel in the Voronoi cell to the nucleus point labels. Using this local k-means, the nucleus pixels are extracted that are located around the point labels and have high local color contrast with the background. The Repel code 225 is used as an enhanced center encoding for cells (the Repel code is described in detail in H. Liang et al., “Enhanced Center Coding for Cell Detection with Convolutional Neural Networks,” arXiv preprint arXiv:1904.08864 (2019), the publication of which is incorporated herein for all purposes). The extracted Repel code 225 can be multiplied by a color cluster mask to make background pixels have zero values.

[0057] The three extracted labels, namely the filtered Repel label, the local color cluster label, and the Voronoi label (see FIG. 2 ), are then compared to the nucleus and background probability maps that are two output channels of the U-Net model 230. The comparison is through three losses L Repel , L Vor , and L Clustto be completed. In some cases, a cross-entropy loss function is used for the binary labels (Voronoi clusters and color clusters), and a mean squared error (MSE) loss function is used for the Repel code labels, as defined in equation (1-3) detailed herein with respect to embodiment 1. Since there are three different losses, the model 230 is trained based on these losses, so these losses need to be combined and the weights of the model need to be updated in each training iteration. A multi-task scheduler 235 is used to combine the losses and update the weights of the model 230. In some cases, one of the three losses (L Repel Vor and L Clust ) is selected in each training iteration, and the weights of the model are updated according to the gradient of that particular loss. If we assume that “i” is the index of the training iteration, if “i%3=0”, the scheduler 235 selects the Voronoi loss, if “i%3=1”, the scheduler 235 selects the Repel loss, and if “i%3=2”, the scheduler 235 selects the color cluster loss, as defined in equation (4) detailed herein with respect to embodiment 1.

[0058] III. C. EXAMPLE U-NET FOR SEGMENTATION AND DETECTION

[0059] Segmentation uses a modified U-Net to extract features from the input image separately, where the encoder part is replaced by the convolutional layers of ResNet50, which is pre-trained on the ImageNet dataset to generate the nucleus and background probability maps. As FIG. 3 ​As shown, the U-Net 300 can include a contracting path 305 and an expansive path 310, which gives it a u-shaped architecture. The contracting path 305 is a CNN network that includes repeated applications of convolutions (e.g., 3x3 convolutions (unpadded convolutions)), each followed by a rectified linear unit (ReLU) and a max-pooling operation (e.g., 2x2 max-pooling with a stride of 2) for down-sampling. At each down-sampling step or pooling operation, the number of feature channels can be doubled. During the contraction, the spatial information of the image data is reduced while the feature information is increased. The expansive path 310 is a CNN network that combines features and spatial information from the contracting path 305 (up-sampling of the feature maps from the contracting path 305) with repeated applications of convolutions (e.g., two 3x3 convolutions) followed by a rectified linear unit (ReLU), concatenation with the corresponding cropped feature maps from the contracting path 305, and a final convolution (e.g., one 1x1 convolution) to generate the nucleus and background probability maps. For localization, high-resolution features from the contracting path 305 are combined with the up-sampled output from the expansive path 310. The U-Net 300 uses the valid part of each convolution without any fully connected layers, i.e., the nucleus and background probability maps contain only pixels for which the full context in the input image is available, and uses skip connections to link the context features learned in the contracting blocks and the localization features learned in the expansive blocks.

[0060] IV. EXAMPLES

[0061] The systems and methods implemented in the various embodiments can be better understood by reference to the following examples.

[0062] IV. A. EXAMPLE 1. - WEAKLY SUPERVISED MULTITASK LEARNING FOR CELL DETECTION AND SEGMENTATION

[0063] Cell detection and segmentation is the foundation of all downstream analysis of digital pathology images. However, obtaining ground truth at the pixel level for single cell segmentation is very labor intensive. To overcome this challenge, an automated, end-to-end deep learning single cell detection and segmentation algorithm was developed, developed as follows: Voronoi labels, Repel labels, and local color cluster labels were extracted based on available point labels from the image data; and a U-Net model was trained using: the Voronoi labels, Repel labels, and local color cluster labels and transfer learning (ResNet pre-trained on ImageNet). The algorithm of this design showed significant improvement in cell detection and segmentation without increasing the annotation effort compared to conventional cell segmentation and detection techniques.

[0064] IV.B. Dataset

[0065] The algorithm of the design was trained, validated, and tested on colorectal cancer and tonsil tissue sections stained with immunohistochemistry (IHC) for PMS2. The dataset included 256 512x512 images with a resolution of 0.5 pm / pixel, covering tumor regions, peritumoral regions, normal tissue regions, intrafollicular, and interfollicular regions in the slides. The dataset had multiple nuclei for detection and segmentation tasks, such as positive cells (dark or light brown staining) and negative cells (blue staining) of different shapes and sizes, with sparse or highly clustered spatial distribution.

[0066] IV. C. PREPROCESSING AND AUGMENTATION

[0067] The dataset was split into training (80%), validation (10%), and test (10%) sets, ensuring that each set had all types of tissue regions (e.g., tumor, peritumoral, normal tissue, etc.). 250x250 pixel patches were extracted from the original images. To increase the size of the training set, data augmentation was performed, including horizontal and vertical flips, random resizing, affine transformations, rotations, and cropping. This resulted in a training set of ~3000 small images. As a final preprocessing step, the training set was normalized by mean subtraction and division by the standard deviation of the RGB channels, respectively. The same normalization was applied to the images in the validation and test sets.

[0068] IV. D. LABEL EXTRACTION

[0069] To overcome the challenges of instance cell segmentation and detection, the designed algorithm was designed in a weakly supervised manner. For example, three transformations / codes were applied to the cell point labels (e.g., point labels at the center of the nuclei of each cell within the FOV): (i) Voronoi transformation, (ii) local clustering, and (iii) Repel code. FIG. 2 An overview of the designed algorithm is shown, including: pixel-level label extraction based on cell point labels and a multi-task deep learning method using a loss scheduler and based on a U-Net model with a ResNet encoder. The Voronoi transformation extracts ridgelines between cells and forms polygons around each cell. These lines help highly clustered cells not to be merged together. Local clustering extracts pixel-level labels for the nuclei based on the point labels, the input image, and the Voronoi labels. For each extracted Voronoi cell (polygon), a k-means clustering algorithm is used to locally extract background and nucleus clusters based on RGB channel color features concatenated with distance transform features, which are the distances from each pixel in the Voronoi cell to the nucleus point labels. Using this local k-means, nucleus pixels are extracted that are located around the point labels and have a high local color contrast with the background. Compared to conventional global color clustering techniques, the local clustering method significantly improves the quality of color cluster labels for weakly stained nuclei.FIG. 4 The performance of the local clustering algorithm is shown compared to the global clustering algorithm used in the conventional segmentation algorithm. As shown, weakly stained cells are well preserved by the local clustering approach, while the global clustering approach fails to detect most of the weakly stained cells.

[0070] Repel encoding is a type of cell enhanced center encoding (see, e.g., H. Liang et al. “Enhanced Center Coding for Cell Detection with Convolutional Neural Networks.” arXiv preprint arXiv: 1904.08864 (2019), which publication is incorporated herein for all purposes) that defines a two-dimensional decay function with its peak at the cell center point label. Compared to the commonly used Gaussian and proximity encodings, the Repel code decays faster for cells with short distances to neighboring cells. Therefore, the Repel code is leveraged in the designed algorithm to facilitate better cell separation in the segmentation task, while better center localization for the detection task. Furthermore, to facilitate better nucleus boundary delineation, the extracted Repel code is multiplied by a local pixel clustering label mask to ensure that background pixels have zero values in the Repel code map, which is referred to as “filtered Repel” in FIG. 2 .

[0071] IV. E. MODEL

[0072] A U-Net model was used, where the encoder part was replaced by the convolutional layers of ResNet50, which was pre-trained on the ImageNet dataset. See, e.g., O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015; K. He, et al., “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016; and O. Russakovsky, et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision 115.3 (2015): 211-252, which are incorporated herein for all purposes.

[0073] IV. E. MULTITASK SCHEDULER

[0074] The three extracted labels, i.e., the filtered Repel label, the local color cluster label, and the Voronoi label (see FIG. 2 ), were compared to the nucleus and background probability maps, which are the two output channels of the U-Net model. This comparison was done during training by three losses: a cross-entropy loss function was used for the binary labels (Voronoi and color cluster), and a mean squared error (MSE) loss function was used for the repel code label, as defined in equation (1-3):

[0075]

[0076]

[0077]

[0078] where o is the model output probability map and t is the corresponding target, i.e. the Voronoi label, the Repel label or the local pixel cluster label. The pixels in the ignore set in (2) are ignored in the Voronoi loss function (as FIG. 1 indicated by the black pixels within the Voronoi sub-region in Fig. 5B), so only the pixels indicated by the red line (used as background) and the green dots (used as foreground) are included.

[0079] Since three different losses are used to train a single model, a strategy is needed to combine them in order to update the model’s weights in each training iteration. For multi-task learning problems, naive summation can not be the best solution as the nature of the various tasks can be very different. To address this problem, a multi-task scheduler is proposed. Specifically, in each training iteration, only one of the three losses is used and the model weights are updated using the following rule: assume “i” is the index of the training iteration, if “i%3=0”, the scheduler selects the Voronoi loss, if “i%3=1”, the scheduler selects the Repel loss, and if “i%3=2”, the scheduler selects the color cluster loss, as follows equation (4):

[0080]

[0081]

[0082] where is the loss selected in the i th th training iteration, is the indicator function that takes the value 1 if i e A, and 0 otherwise. Since the dataset is randomly shuffled before small batches are extracted at each epoch, each data point has a chance to contribute to all three types of loss / task. Compared to the regular naive summation of losses, this multi-task scheduler shows better performance for each individual task, as discussed in the next section. Binary segmentation masks are generated using the argmax function applied to the output probability map, which sets the pixel value to 1 where the probability value of the nucleus is higher than the background, and 0 otherwise. In addition, cells are detected by finding the locations of local maxima in the nucleus output probability map using the max filter with a minimum distance of 2.5 pm (see, e.g., S. van der Walt et al., “scikit-image: Image processing in Python”, PeerJ 2: e453 (2014), which is incorporated herein for all purposes).

[0083] IV. F. SEGMENTATION AND DETECTION RESULTS

[0084] To perform segmentation and detection, the model was trained in PyTorch using a small batch of 8,150 epochs and 60900 total training iterations (see, e.g., A. Paszke et al., “Automatic Differentiation in PyTorch”, NIPS Autodiff Workshop, 2017, which is incorporated herein for all purposes). The Ranger optimizer, as described in L. Wright, “New Deep Learning Optimizer, Ranger: Synergistic combination of RAdam+LookAhead for the best of both” (2019), which is incorporated herein for all purposes, was used to update the parameters and improve the performance of the training. A learning rate of 0.001 was used. For the Repel code, based on equation (4) in H. Liang et al. “Enhanced Center Coding for Cell Detection with Convolutional Neural Networks” arXiv preprint arXiv:1904.08864 (2019), a = 0.05 and r = 70 were used.

[0085] The designed algorithm achieved a pixel-level accuracy of 92.9%, a pixel-level F1 score of 79.1%, an object-level Dice score of 0.784, an object-level Aggregated Jacard Index (AJI) score of 0.599, a detection precision of 94.1%, a detection recall of 92.5%, and a detection concordance correlation coefficient (CCC, a = 0.05 as defined in I. Lawrence and K. Lin. “A concordance correlation coefficient to evaluate reproducibility” Biometrics (1989): 255-268, which is incorporated herein for all purposes) of 0.998. The use of the segmentation performance metrics is defined as follows: H. Qu et al. “Weakly Supervised Deep Nuclei Segmentation using Points Annotation in Histopathology Images” International Conference on Medical Imaging with Deep Learning (IMDL), 2019 [1], which is incorporated herein for all purposes. The detection precision and recall metrics are defined as TP / (TP+FP) and TP / (TP+FN), respectively, where TP, FP, and FN are the number of true positive cells, false positive cells, and false negative cells. Compared to the algorithm proposed in H. Qu et al. “Weakly Supervised Deep Nuclei Segmentation using Points Annotation in Histopathology Images” International Conference on Medical Imaging with Deep Learning (IMDL), 2019, the designed algorithm shows significant improvement in cell segmentation and detection without increasing the annotation effort. In contrast, the conditional random field (CRF) post-processing step, which was used in H. Qu et al. “Weakly Supervised Deep Nuclei Segmentation using Points Annotation in Histopathology Images” International Conference on Medical Imaging with Deep Learning (IMDL), 2019, was excluded.

[0086] The performance of the multi-task scheduler method is compared individually with the multi-task learning method using uncertainty proposed in A. Kendall, Y. Gal, and R. Cipolla. “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018 [2], which is incorporated herein for all purposes. According to the results, the multi-task scheduler method shows better performance compared to the task uncertainty method, where an additional loss layer with learnable weights is defined to combine the three losses. The results of segmentation and detection are summarized in Tables 1 and 2 below.

[0087] FIG. 5 Example segmentation performance of the designed algorithm compared to the algorithms proposed in H. Qu, et al. “Weakly Supervised Deep Nuclei Segmentation using Points Annotation in Histopathology Images” International Conference on Medical Imaging with Deep Learning (IMDL), 2019, including some sample test images and segmentation masks overlaid on the original images. Specifically,

[0088] Table 1. Segmentation performance of the proposed algorithm compared to conventional algorithms.

[0089]

[0090] Table 2. Detection performance of the designed algorithm compared to conventional algorithms.

[0091]

[0092] on Medical Imaging with Deep Learning (IMDL), 2019, including some sample test images and segmentation masks overlaid on the original images. Specifically, FIG. 5 It is shown that: (a) a control region with elongated cells; (b) a control region with negative tumor cells; (c) a tissue region with highly aggregated cells; and (d) a tumor region with weak and well-stained cells. The algorithm performs well in the segmentation of these different types of cells.

[0093] V. OTHER CONSIDERATIONS

[0094] Some embodiments of the present disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-transitory computer- readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine- readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more methods disclosed herein and / or some or all of one or more processes disclosed herein.

[0095] The terminology and expressions employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the application claimed. Thus, it should be understood that although the present application has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed can be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this application as defined by the appended claims.

[0096] The following description provides specific details for a thorough understanding of, and enabling disclosure of, the preferred exemplary embodiments. However, one skilled in the art will understand that the application can be practiced without

[0097] In the following description, numerous specific details are given to provide a thorough understanding of the embodiments. However, it will be apparent that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form, rather than in detail, in order to avoid obscuring the embodiments. In other instances, well-known circuits, processes, algorithms, structures, and techniques have not been shown in detail in order to avoid obscuring the embodiments.

Claims

1. A computer-implemented method comprising: accessing, by a data processing system, a plurality of images of one or more cells; extracting, by the data processing system, three labels from the plurality of images, wherein the three labels are extracted using a Voronoi transform, a local clustering, and a repel code application; training, by a multi-task scheduler of the data processing system, a convolutional neural network model based on three loss functions corresponding to the three labels, wherein the convolutional neural network model comprises a plurality of model parameters; generating, by the convolutional neural network model, a nucleus probability map and a background probability map for each image in the plurality of images based on the training with the three loss functions; comparing, by the data processing system, the nucleus probability map and the background probability map to the three labels; updating, by the data processing system, the plurality of model parameters based on the comparison of the nucleus probability map and the background probability map to the three labels to minimize the three loss functions; and providing, by the data processing system, the updated plurality of model parameters to a trained convolutional neural network model.

2. The computer-implemented method of claim 1, wherein the plurality of images comprises a nucleus point label.

3. The computer-implemented method of claim 1 or 2, wherein the Voronoi transform extracts ridge lines between the one or more cells.

4. The computer-implemented method of claim 2, wherein the local clustering comprises applying a k-means clustering algorithm to locally extract background clusters and nucleus clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or a polygon created by the Voronoi transform to the nucleus point label.

5. The computer-implemented method of any one of claims 1 to 4, wherein the convolutional neural network model comprises a modified U-Net model, wherein the encoder part is replaced by the convolutional layers of a ResNet50, wherein, ResNet50 is pre-trained on the ImageNet dataset.

6. The computer-implemented method of any one of claims 1 to 5, wherein training comprises: for each training iteration, selecting, by the multi-task scheduler, one of the three loss functions; and updating, by the data processing system, one or more weights of the convolutional neural network model based on a gradient of the selected loss function.

7. The computer-implemented method of claim 6, wherein the loss function is selected by the multi-task scheduler based on: if "i%3=0", selecting a loss function associated with a Voronoi label; if "i%3=1", selecting a loss function associated with a repel label; and if "i%3=2", selecting a loss function associated with a local cluster label, where "i" is an index of the training iteration.

8. The computer-implemented method of any one of claims 1 to 7, wherein generating the nucleus probability map and the background probability map comprises applying an argmax function and a local maxima function to the nucleus probability map and the background probability map, a segmentation binary mask, and coordinates of detected cells.

9. A computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform actions including: accessing, by a data processing system, a plurality of images of one or more cells; extracting, by the data processing system, three labels from the plurality of images, wherein the three labels are extracted using a Voronoi transform, a local clustering, and a repel code application; training, by a multitask scheduler of the data processing system, a convolutional neural network model based on three loss functions corresponding to the three labels, wherein the convolutional neural network model includes a plurality of model parameters; generating, by the convolutional neural network model, a nucleus probability map and a background probability map for each image in the plurality of images based on the training with the three loss functions; comparing, by the data processing system, the nucleus probability map and the background probability map to the three labels; updating, by the data processing system, the plurality of model parameters based on the comparison of the nucleus probability map and the background probability map to the three labels to minimize the three loss functions; and providing, by the data processing system, the updated plurality of model parameters to a trained convolutional neural network model.

10. The computer program product of claim 9, wherein the plurality of images includes a nucleus point label.

11. The computer program product of claim 9 or 10, wherein the Voronoi transform extracts ridge lines between the one or more cells.

12. The computer program product of claim 10, wherein the local clustering includes applying a k-means clustering algorithm to locally extract background clusters and nucleus clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or a polygon created by the Voronoi transform to the nucleus point label.

13. The computer program product of any one of claims 9 to 12, wherein: the training includes, for each training iteration, selecting, by the multitask scheduler, one of the three loss functions and updating, by the data processing system, one or more weights of the convolutional neural network model based on a gradient of the selected loss function; and the loss function is selected, by the multitask scheduler, based on: if "i%3=0", selecting a loss function associated with a Voronoi label; if "i%3=l", selecting a loss function associated with a repel label; and if "i%3=2", selecting a loss function associated with a local cluster label, where "i" is an index of the training iteration.

14. The computer program product of any one of claims 9 to 13, wherein generating the nucleus probability map and the background probability map includes applying an argmax function and a local maxima function to the nucleus probability map and the background probability map, a segmentation binary mask, and coordinates of detected cells.

15. A computer-implemented system comprising: one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform acts comprising: accessing, by a data processing system, a plurality of images of one or more cells; extracting, by the data processing system, three labels from the plurality of images, wherein the three labels are extracted using a Voronoi transform, a local clustering, and a repel code application; training, by a multitask scheduler of the data processing system, a convolutional neural network model based on three loss functions corresponding to the three labels, wherein the convolutional neural network model comprises a plurality of model parameters; generating, by the convolutional neural network model, a nucleus probability map and a background probability map for each image in the plurality of images based on the training with the three loss functions; comparing, by the data processing system, the nucleus probability map and the background probability map to the three labels; updating, by the data processing system, the plurality of model parameters based on the comparison of the nucleus probability map and the background probability map to the three labels to minimize the three loss functions; and providing, by the data processing system, the updated plurality of model parameters to a trained convolutional neural network model.

16. The computer-implemented system of claim 15, wherein the plurality of images comprises a nucleus point label.

17. The computer-implemented system of claim 15 or 16, wherein the Voronoi transform extracts a ridge line between the one or more cells.

18. The computer-implemented system of claim 16, wherein the local clustering comprises applying a k-means clustering algorithm to locally extract background clusters and nucleus clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or a polygon created by the Voronoi transform to the nucleus point label. training comprises:

19. The computer-implemented system of any one of claims 15 to 18, wherein: for each training iteration, selecting, by the multitask scheduler, one of the three loss functions; and updating, by the data processing system, one or more weights of the convolutional neural network model based on a gradient of the selected loss function; and the loss function is selected by the multitask scheduler based on: if "i%3=0", selecting a loss function associated with the Voronoi label; if "i%3=l", selecting a loss function associated with the repel label; and if "i%3=2", selecting a loss function associated with the local cluster label, where "i" is an index of the training iteration. ​ 20. The computer-implemented system of any one of claims 15 to 19, wherein generating the kernel probability map and the background probability map comprises applying an argmax function and a local maxima function to the kernel probability map and the background probability map, a segmentation binary mask, and coordinates of detected cells.

21. A computer-implemented method comprising: obtaining, by a data processing system, a plurality of images of one or more cells; inputting the plurality of images into a convolutional neural network model, the convolutional neural network model constructed according to the computer-implemented method of any one of claims 1 to 8; generating, by the convolutional neural network model, for each image of the plurality of images, a kernel probability map and a background probability map; and providing, by the data processing system, the kernel probability map and the background probability map.

22. The computer-implemented method of claim 21, wherein the plurality of images contain a kernel point label.

23. The computer-implemented method of claim 21 or 22, wherein the Voronoi transform extracts ridge lines between the one or more cells.

24. The computer-implemented method of claim 22, wherein the local clustering comprises applying a k-means clustering algorithm to locally extract background clusters and kernel clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or a polygon created by the Voronoi transform to the kernel point label.

25. The computer-implemented method of any one of claims 21 to 24, wherein the convolutional neural network model comprises a modified U-Net model, wherein an encoder portion is replaced with convolutional layers of a ResNet50, wherein, ResNet50 is pre-trained on the ImageNet dataset.

26. The computer-implemented method of any one of claims 21 to 25, wherein: the convolutional neural network model comprises a plurality of model parameters identified using a training dataset, the training dataset comprising a plurality of medical images having at least three labels extracted using the Voronoi transform, the local clustering, and the repel code application; and the plurality of model parameters are identified using the training dataset based on minimizing the at least three loss functions.

27. The computer-implemented method of claim 26, wherein using the training data set comprises: for each training iteration, one of the three loss functions is selected by a multi-task scheduler; and one or more of the plurality of parameters of the convolutional neural network model are updated by the data processing system based on a gradient of the selected loss function.

28. The computer-implemented method of claim 27, wherein the loss function is selected by the multi-task scheduler based on: if "i%3=0", a loss function associated with the Voronoi label is selected; if "i%3=l", a loss function associated with the repel label is selected; and if "i%3=2", a loss function associated with the local cluster label is selected, where "i" is an index of the training iteration.

29. The computer-implemented method of any one of claims 21 to 28, wherein generating and providing the kernel probability map and the background probability map comprises applying an argmax function and a local maxima function to the kernel probability map and the background probability map, a segmentation binary mask, and coordinates of detected cells.

30. A computer program product tangibly embodied in a non-transitory machine- readable storage medium including instructions configured to cause one or more data processors to perform actions comprising: obtaining, by a data processing system, a plurality of images of one or more cells; inputting the plurality of images into a convolutional neural network model, the convolutional neural network model constructed according to the computer-implemented method of any one of claims 1 to 8; generating, by the convolutional neural network model, a kernel probability map and a background probability map for each image of the plurality of images; and providing, by the data processing system, the kernel probability map and the background probability map.

31. The computer program product of claim 30, wherein the plurality of images contain a kernel point label.

32. The computer program product of claim 30 or 31, wherein the Voronoi transform extracts ridge lines between the one or more cells.

33. The computer program product of claim 31, wherein the local clustering comprises applying a k-means clustering algorithm to locally extract background clusters and kernel clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or a polygon created by the Voronoi transform to the kernel point label.

34. The computer program product of any one of claims 30 to 33, wherein the convolutional neural network model comprises a modified U-Net model, wherein the encoder portion is replaced with convolutional layers of a ResNet50, wherein, ResNet50 is pre-trained on the ImageNet dataset.

35. The computer program product of any one of claims 30 to 34, wherein: the convolutional neural network model comprises a plurality of model parameters identified using a training dataset, the training dataset comprising a plurality of medical images having at least three labels extracted using the Voronoi transform, the local clustering, and the repel code application; and the plurality of model parameters are identified based on minimizing the at least three loss functions using the training dataset.

36. The computer program product of claim 35, wherein using the training data set comprises: for each training iteration, one of the three loss functions is selected by a multi-task scheduler; and one or more of the plurality of parameters of the convolutional neural network model are updated by the data processing system based on a gradient of the selected loss function.

37. The computer program product of claim 36, wherein the loss function is selected by the multi-task scheduler based on: if "i%3=0", a loss function associated with the Voronoi label is selected; if "i%3=l", a loss function associated with the repel label is selected; and if "i%3=2", a loss function associated with the local cluster label is selected, where "i" is an index of the training iteration.

38. The computer program product of any one of claims 30 to 37, wherein generating and providing the kernel probability map and the background probability map comprises applying an argmax function and a local maxima function to the kernel probability map and the background probability map, a segmentation binary mask, and coordinates of detected cells.

39. A computer-implemented system comprising: one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform acts comprising: obtaining, by a data processing system, a plurality of images of one or more cells; inputting the plurality of images into a convolutional neural network model, the convolutional neural network model constructed according to the computer-implemented method of any one of claims 1 to 8; generating, by the convolutional neural network model, for each image of the plurality of images, a kernel probability map and a background probability map; and providing, by the data processing system, the kernel probability map and the background probability map.

40. The computer-implemented system of claim 39, wherein the plurality of images contain a kernel point label.

41. The computer-implemented system of claim 39 or 40, wherein the Voronoi transform extracts ridge lines between the one or more cells.

42. The computer-implemented system of claim 40, wherein the local clustering comprises applying a k-means clustering algorithm to locally extract background clusters and kernel clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or a polygon created by the Voronoi transform to the kernel point label. ResNet50 is pre-trained on the ImageNet dataset. the convolutional neural network model comprises a plurality of model parameters identified using a training dataset, the training dataset comprising a plurality of medical images having at least three labels extracted using the Voronoi transform, the local clustering, and the repel code application; 43. The computer-implemented system of any one of claims 39 to 42, wherein the convolutional neural network model comprises a modified U-Net model, wherein the encoder portion is replaced with convolutional layers of a ResNet50, wherein, and 44. The computer-implemented system of any one of claims 39 to 43, wherein: the plurality of model parameters are identified using the training dataset based on minimizing the at least three loss functions. for each training iteration, one of the three loss functions is selected by a multi-task scheduler; and one or more of the plurality of parameters of the convolutional neural network model are updated by the data processing system based on a gradient of the selected loss function.

45. The computer-implemented system of claim 44, wherein using the training dataset comprises:

41. The computer-implemented system of claim 39 or 40, wherein the Voronoi transform extracts ridge lines between the one or more cells.

42. The computer-implemented system of claim 40, wherein the local clustering comprises applying a k-means clustering algorithm to locally extract background clusters and kernel clusters based on RGB channel color features concatenated with distance transform features, the distance transform features being a distance of each pixel in each cell or a polygon created by the Voronoi transform to the kernel point label.

46. The computer-implemented system of claim 45, wherein the loss function is selected by the multi-task scheduler based on: if "i%3=0", selecting the loss function associated with the Voronoi label; if "i%3=l", selecting the loss function associated with the repel label; and if "i%3=2", selecting the loss function associated with the local cluster label, where "i" is the index of the training iteration.

47. The computer-implemented system of any one of claims 39 to 46, wherein generating and providing the core probability map and the background probability map comprises applying an argmax function and a local maxima function to the core probability map and the background probability map, a segmentation binary mask, and coordinates of detected cells.

Images